1.11. Documentation Makefile and Scripts

This section contains details about the documentation’s Makefiles and accompanying scripts. For instructions on how to build the documentation see section Building the GDA Developer and User Guiders at Diamond.

The GDA Developer Guide (this document) and its companion GDA User Guide are built from their sources using the Makfile in their respective top-level directories, i.e.:

  • Developer Guide - <path_to_gda-documentation>/GDA_Developer_Guide

  • User Guide - <path_to_gda-documentation>/GDA_User_Guide

Where <path_to_gda-documentation> is replaced with the path to the location containing clone of gda-documentation repository.

Following sections describe the Makefiles targets available for use, e.g. html that is used to build the documentation, and various Scripts that are used by the Makefile targets.

1.11.1. Makefiles

To build the GDA documentation three Makefiles are used, they are (using path from top-level gda-documentation directory):

  • common/MakefileCommon - Makefile containing targets used to build both GDA Developer Guide and GDA User Guide

  • GDA_Developer_Guide/Makefile - Makefile used to build the GDA Developer Guide (this document)

  • GDA_User_Guide/Makefile - Makefile used to build the GDA User Guide

1.11.1.1. Common Makefile targets

The Makefile targets contained in common/MakefileCommon are available for use via GDA_Developer_Guide/Makefile and GDA_User_Guide/Makefile. This Makefile is a modified version of the one initially generated using the sphinx-quickstart command and therefore contains the default targets provided by Sphinx (see Sphinx Running the build) plus additional GDA documentation specific targets.

The most commonly used targets contained in common/MakefileCommon are described in the following sections.

1.11.1.1.1. help and helpfull

Call these targets to get information about the Makefile targets available.

1.11.1.1.2. clean

This target removes the document’s build directory, created by the html target (see below) in the same directory where make is executed.

This is the Sphinx default make clean behaviour, it has been modified to also remove source/appendix/documents_broken_links.md that is created by the the various broken links targets b_links, b_links_external and b_links_local (see below).

1.11.1.1.3. html

This target builds the HTML pages from the documentation source contained in the respective document’s source directory. On successful completion the document’s build directory will be created/updated and the document can be viewed by opening build/html/contents.html in a browser.

The target is slightly modified from that provided by Sphinx, in that before building the html files it checks whether source/appendix/documents_broken_links.md exists, if it does not the file is created from source/appendix/documents_broken_links.clean. The real documents_broken_links.md (containing details of document’s broken links) is created using the various b_links* targets (see below), however before this is done a clean file explaining how the real file is created is put in its place.

1.11.1.2. GDA Developer Guide Makefile targets

The Makefile contained in gda-documentation/GDA_Developer_Guide is used to build this document. This Makefile includes MakefileCommon and therefore all targets from the common Makefile are available when calling make in this directory.

In addtion to the targets detailed in section Common Makefile targets (see above) the following sections describe the GDA Developer Guide specfic targets provided.

1.11.1.2.1. help

Outputs help message regarding targets contained in this Makefile before calling help in MakefileCommon.

1.11.1.2.2. html

This target extends the MakefileCommon target html (see above). When called it:

  1. Runs the Python script scripts/extractPyDocs.py that extracts doc strings from Python files in GDA source code and creates the Python Code page of this document. For more details see section extractPyDocs.py below.

  2. Copies the .yaml files contained in the scripts/ directory into this document’s build as .txt files so they can be opened directly from most browsers (rather than being prompted to select application to be used). This is the same process used by Sphinx to provide links to documentation source .md files.

  3. Calls the MakefileCommon target html to build the document.

1.11.1.2.3. full

This runs the Python script findDeviceClassesInUse.py using the --full option to update the file containing the list of Devices used across beamlines. This target should only be used for the Jenkins nightly build configuration and is not required locally as this documentation references the file built and hosted on the build server.

1.11.1.2.4. vp_plugins

Build and install the Java GenerateCD class implementing the Visual Paradigm class diagram plug-in used to auto generate class diagrams from the GDA Java source code and used in this document. See GenerateCD Visual Paradigm plug-in.

For full details of how to use this target see How to build and install plug-in.

1.11.1.2.5. vp_plugins_docs

Create the Visual Paradigm GenerateCD class diagram plug-in Javadoc GenerateCD.

For full details of how to use this target see Javadoc.

1.11.1.2.6. vp_plugins_clean

Remove the Visual Paradigm GenerateCD class diagram plug-in.

For full details of how to use this target see Clean.

1.11.1.2.7. create_dia

&&&ajb TODO

1.11.1.3. GDA User Guide Makefile targets

The Makefile contained in gda-documentation/GDA_User_Guide is used to build the GDA User Guide document. This Makefile includes MakefileCommon and therefore all targets from the common Makefile are available when calling make in this directory.

Currently User Guide Makefile contains no User specific targets, i.e. only those described in section Common Makefile targets are used to build the GDA User Guide.

1.11.2. Scripts

A number of scripts are used during the creation of the GDA Developer Guide and GDA User Guide. These are located in the following directories (using path from top-level gda-documentation directory):

  • common/scripts - used during build of both GDA Developer Guide and GDA User Guide

  • GDA_Developer_Guide/scripts - used during build of the GDA Developer Guide (this document)

Routinely the user will not need to call these scripts directly as make is delegated with calling the scripts correctly when required. However for completeness a brief description of the purpose of each script and how it can be used is given below. Note that all scripts are to be run from the same location as make is called, namely gda-documentation/GDA_Developer_Guide for this document and gda-documentation/GDA_User_Guide for the GDA User Guide, which is where make calls them from (and not for example GDA_Developer_Guide/scripts).

Note: currently there are no GDA User Guide specific scripts.

1.11.2.1. Common Scripts

The scripts in common/scripts are used during build of this document and GDA User Guide.

The following sections describe the scripts available.

1.11.2.2. GDA Developer Guide Scripts

The scripts in GDA_Developer_Guide/scripts are used during the build of this document.

The following sections describe the scripts available.

1.11.2.2.1. extractPyDocs.py

Script’s autodoc extractPyDocs.

This script creates the source .md file of this document’s Python Code page, containing GDA Python source code documentation. It parses GDA Python files and extracts docStrings to provide code documentation that can be referenced from within this documentation. The docStrings along with the class/method they are associated with are used to create the file GDA_Developer_Guide/source/python_docs.md.

Currently it extracts docStrings for gda-core.git/uk.ac.gda.core/scripts/ and gda-mx.git/configurations/mx-config/scripts/. More can be added directly into the script by adding to the buildpaths parameter. Additionally it is possible to ignore certain modules within these paths by defining the parameter ‘skipModules’ within the script.

Usage:

python extractPyDocs.py

python extractPyDocs.py help

Prerequisites:

  • workspace_git location: The script relies on knowing the path to the workspace_git directory where it can then navigate the different repos. It assumes a default location for this as simply being two levels above the location that the script is run from (gda-documentation/GDA_Developer_Guide). If the user wishes to define a different location they must create a file in the scripts directory named workspace_root_dir. This file must contain the full path to their workspace_git directory (e.g. /scratch/gda_development/workspace_git). If this file exists then the script will read it and use the path inside as the workspace_git location.

Notes:

  • Once the file has been generated it is possible to create a link to it else where in the document using the Markdown format:

[Link text](python_docs.html#<headingname>)

e.g. [PvManager.py](python_docs.html#pvmanager-py) -> PvManager.py.

Note: when directly referencing a heading, you must reference the heading using the normal markdown format, i.e.:

  • replace all spaces and full-stops with - (dash)

  • change all letters to lower case and remove any special characters (e.g. ').

For example PvManager.py becomes #pvmanager-py.

1.11.2.2.2. extractPyClasses.sh

This bash shell script is a wrapper for the VisParMaker.py python script (see below). It first determines the source path to the Python code, ensures that the output directory is available and then performs a check that the environment is using a correct/compatible version of Python (e.g. Python 2 - see below). Once all of this has been configured it calls the VisParMaker python commands with the defined source path, output and subdirectory.

Usage:

./scripts/extractPyClasses.sh

./scripts/extractPyClasses.sh help

Prerequisites:

  • workspace_git location: This script requires the path to the workspace_git directory to access the git modules. It assumes a default location for this as simply being two levels above the location that the script is run from (gda-documentation/GDA_Developer_Guide). If the user wishes to define a different location they must create a file in the scripts directory named workspace_root_dir. This file must contain the full path to their workspace_git directory (e.g. /scratch/gda_development/workspace_git). If this file exists then the script will read it and use the path inside as the workspace_git location.

Notes

  • The Python file created by VisParMaker.py is output to the GDA_Developer_Guide/vp directory.

  • Currently this script creates ‘pseudo’ classes for the MX python scripts to create class diagrams for the DataCollection', LineScan and GridScan.

  • This script does not need to be run regularly or perhaps even at all by most users. The above Python ‘pseudo’ class files created by VisParMaker.py are committed to the documentation repository in vp/generated_python_classes/. The only reasons to run the script would be if the user wants to create a new class diagram for their Python code so first need to extract the pseudo classes, or if changes are made to the MX python scripts.

1.11.2.2.3. findDeviceClassesInUse.py

Script’s autodoc findDeviceClassesInUse.

This program is used to search through the beamline xml configuration files (using the configuration locations given in daq-platform.git/uk.ac.diamond.daq.server.configuration/beamlineLayouts.cfg) to find all of the classes/Devices that are referenced and then for each class, count how many references are made to it.

An additional option (--full) looks at the Java doc of each device to check whether it has been labelled as a ‘Preferred Method’, i.e. the Device class that should be used going forward (see the Recommended Devices section).

The result is a list of Devices with the number of uses within GDA, whether they are preferred implementations, and finally the beamlines that reference them. Please see the Device section for the output.

Usage:

python findDeviceClassesInUse.py

python findDeviceClassesInUse.py help

python findDeviceClassesInUse.py --full

Prerequisites:

  • workspace_git location: The script relies on knowing the path to the workspace_git directory where it can then navigate the different repos. It assumes a default location for this as simply being two levels above the location that the script is run from (gda-documentation/GDA_Developer_Guide). If the user wishes to define a different location they must create a file in the scripts directory named workspace_root_dir. This file must contain the full path to their workspace_git directory (e.g. /scratch/gda_development/workspace_git). If this file exists then the script will read it and use the path inside as the workspace_git location.

Notes:

  • This script should only be run on the nightly build. Locally this is not required as this documentation references the file generated by the nightly build on the ‘alfred’ web-server.

1.11.2.2.4. updateModelsAndClassDiagrams.sh

Note Dec 2020: Since development of this script the way the documentation uses Visual Paradigm has been reconsidered. Most noteably it was decided not to maintain a Visual Paradigm model of the GDA Java source in version control but instead auto-generate models from latest GDA Java source code, see Auto generation of Class and Sequence diagrams for details. Therefore this script requires updating to follow these developments. At the current time the script’s documentation below is out of date and will require changing once the script has been updated.

The default purpose of this script is to access the GDA Visual Paradigm models and use the Visual Paradigm command line scripts to export the diagrams contained in the models as images in PNG format.

Usage:

./scripts/updateModelsAndClassDiagrams.sh

./scripts/updateModelsAndClassDiagrams.sh help

Prerequisites:

  • Firstly a check is made that Visual Paradigm is installed so the installation directory must be defined at the top of the file as variable vp_path. If it cannot be found at the specified location then the script exits as it cannot continue without Visual Paradigm.

  • Next it checks whether the directory vp (containing the models) is present in the GDA_Developer_Guide directory. The gda-documentation repo ships with a tar file containing this directory (vp.tar.gz), so if vp hasn’t already been extracted from the tar file then it will do so.

  • It next reads through the scripts/definitions.txt file which follows a specific structure to define for each model, the diagrams to extract from it and the output location of the images. An example of the format is given below:

    entry 
model: old_scanning_vp_model.vpp
modelpath: vp
depends: gda-core.git/uk.ac.gda.core/src/gda gda-core.git/uk.ac.gda.api/src 
diagrams: OldScanning_ClassDiagram
output: source/images/use_cases
end

For each entry it will use the Visual Paradigm script ExportDiagramImage.sh to export the named diagrams as PNG files to the output destination. The images will have the same name as the diagram names in the model.

If new models are created they must be added to the scripts/definitions.txt following the above format to have their diagrams be generated automatically by this script.

Options:

  • ./updateModelsAndClassDiagrams.sh html:

With the html option, the script will carry out all the checks detailed above but instead of extracting images from the diagrams, it will use the Visual Paradigm script ProjectPublisher.sh to publish the whole project in html format. This process creates a directory with a collection of files for each element of the model. The purpose of this is so that the class diagrams can be viewed as html pages which allow user interaction with the diagrams. For example, the user can click on a particular class in the class diagram and this then takes them to a page giving all that class’ attributes, operations and description. This means that class diagrams in the documentation provide even more information to the user than a simple image.

For each model the script produces a set of html files and outputs them to GDA_Developer_Guide/vp/html/<model_name>.

1.11.2.2.5. VisParMaker.py

Script’s autodoc VisParMaker.

This Python script is used to parse Python scripts as classes and determine what other scripts they depend on. The output is a single Python file containing a list of classes which can then be used as input for Visual Paradigm to create Python Relation Diagrams. The issue with trying to create Python ‘Class’ diagrams in the standard way in Visual Paradigm is that a lot GDA’s Python scripts do not contain classes and are simply a collection of functions. In order to discern relationships between the Python scripts and enable Visual Paradigm to display them in diagrams, it is first necessary to run the VisParMaker.py script. Given a Python source directory it will define each script as a class of the same name and then parse the script extracting all import statements to determine what other scripts (or ‘pseudo’ classes) it depends on. An example of the resulting extraction printed to a Python output file is shown below.

Some of the Python scripts may also contain classes. In order to capture this information the VisParMaker.py script also searches for classes when parsing each Python script and prints them using the name of each class as an attribute within the new ‘pseudo class’. When importing the Python output files into Visual Paradigm, these will show up as attributes in the class and highlight to the reader that ‘real’ classes are contained in the given script.

Usage:

python VisParMarker.py /path/to/python/src output_filename subdirectory

python VisParMarker.py help

The script can take 3 parameters, the first 2 of which are compulsory:

  1. Path to the Python source code.

  2. The output file location.

  3. The name of any sub-directory with the source directory that should be referenced using subdirectory.scriptname.

E.g. python VisParMarker.py /scratch/user/gda_all/workspace_git/git-mx.git/scripts/ datacollection.py datacollection

Notes

  • There is no need to call this script directly. It is managed by the shell script extractPyClasses.sh which defines the Python source to use and where the outputs should be place.

  • The VisParMarker.py script must be run using the same version of Python as the Python code being parsed. I.e. as all Diamond Python code is currently written in Python 2, this script must be called using Python 2. The Python source code will FAIL to be parsed if the script is run using Python 3.

Example

Below is an extract from the output file produced by running the script in the git-mx.git/scripts/ with subdirectory = datacollection. It highlights 4 different types of output results.

class shutter_control(hutch_utilities_state,SampleChangerLoadApprover,hutch_utilities):
    pass

class synchrotron:
    pass

class data_collection(bart,datacollection.factory):
    pass

class SampleChangerLoadApprover(hutch_utilities):
    SampleChangerLoadApprover = "A Class"
    pass

  1. Here the shutter_control script is labelled as the ‘pseudo’ class and parsed. The import statements within the script indicate that it depends on scripts hutch_utilities_state.py, SampleChangerLoadApprover.py and hutch_utilities.py. This script contains no real classes.

  2. The synchrotron.py script does not contain any imports of modules in the same source path.

  3. The data_collection.pyscript imports bart.py and also imports the factory module which is contained in a subdirectory datacollection within the source path. There may be other factory classes within the source path so it is important to pick out the fact that it is this one and hence the use of the subdirectory argument datacollection.

  4. The SampleChangerLoadApprover.py script imports hutch_utilities.py but this script also contains a class (SampleChangerLoadApprover) which is extracted and printed as an attribute within the ‘pseudo’ class.

1.11.2.2.6. vp_create_models_and_diagrams.py

Script’s autodoc vp_create_models_and_diagrams.

Script used to automatically generate class and sequence diagrams from GDA Java source using Visual Paradigm, see Auto generation of Class and Sequence diagrams section of this document for full details of what this script is used to achieve.

The script’s -h help information can be viewed here. The current contents of the default YMAL configuration files used by the script are defn_models.yaml and defn_meta.yaml.