17.5. Buckminster Build (Legacy)

As of October 2020 the master version of GDA is built using Maven/Tycho. This page is currently kept for legacy releases/general historical info.

17.5.1. The GDA Materialize Process

17.5.1.1. Definition of components involved

17.5.1.1.1. pewma

pewma.py (Python Eclipse Workspace MAnager) is a python script that lives at /dls_sw/dasc/pewma.py and is version controlled at diamond-releng/diamond.releng.buckminster/software/pewma.py. It acts as a wrapper around Buckminster (which has a difficult command line interface itself) and provides lots of features to help manage workspaces. A list of these can be accessed using the command:

pewma --help

It provides aliases for more complicated variables to aid ease of use. E.g.

  • i02-client = uk.ac.gda.beamline.i02.site

  • gdaserver = uk.ac.diamond.daq.server.site

  • dawndiamond = uk.ac.diamond.dawn.site It can also automatically figures out which CQUERY to use. The master is the default version to materialize but a specific version can also be provided as an argument.

17.5.1.1.2. Buckminster

Buckminster is a headless Eclipse for provisioning, building and testing. While it can be considered a general purpose build tool, in reality it is only used for Java and mostly within the use of Eclipse. It is particularly useful for building and sharing distributions of software components which share components across multiple software projects and repositories (much like GDA).The manual can be accessed from a Diamond file location (note that the correct permissions are needed to view it). Equally the archived website can be found here.

It is used within GDA for materializing via the pewma script and building the exported products (i.e. the server and clients).

Cquery

A Buckminster CQUERY (Componenet Query) names a component assembly and Buckminster uses it to locate all the components necessary to complete the materialization process.

The CQUERY:

  • Points to an RMAP (this is key).

  • It allows properties to be set and can define defaults and also pick ups, e.g. ${user.home}/.{dawn,gda}-master.cquery.properties. It can also define how to skip Dawn/GDA test fragment (as appropriate) and how to switch between authenticated and unauthenticated clones.

  • It allows ‘advisor’ notes to be added to specific components, which is used to specify particular git branches when checking out release versions, e.g.   branchTagPath = "dawn-2.10". For this reason it is IMPORTANT to keep CQUERY and RMAP regex synchronised.

An extract from the gda-master.cquery is given below:

<cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="core-master.rmap" properties="file:${user.home}/.gda-master.cquery.properties" shortDesc="CQuery for GDA master">
...

<cq:property key="skip_DAWN_test_fragments" value="true"/>
...

<cq:advisorNode namePattern="^com\.globalphasing\.sdcp\.abstract_beamline$" useTargetPlatform="false" branchTagPath="gda-master">
    <cq:documentation xmlns="http://www.w3.org/1999/xhtml">gphl-abstract-beamline.git repository branch</cq:documentation>
</cq:advisorNode>

RMAP

A Buckminster RMAP (Resource Map) is associated with a CQUERY and lists the repositories in which the defined components can be found. This is how Buckminster loops up where to find the components. An RMAP file essentially acts as a big regex lookup table.

E.g. The location of the beamline configurations is defined in the core-master.rmap, an extract of which is shown below:

<!--
     Git repositories
-->
<rm:propertyElement key="gda.diamond.git.url">
    <bc:constant value="${diamond.dascgerrit.${dascgerrit.authentication.common}.url}/gda/gda-diamond.git"/>
</rm:propertyElement>
...

 <!--#
    Locators                                                        
#-->
<rm:locator pattern="^(all-dls-(configs|clients)|(dls|b07(-1)?|b21|i05(-1)?|i06(-1)?|i08(-1)?|i09(-1|-2)?
|i10|i11|i11-1|i12|i13|i13-1|i14|i15|i15-1|i21|i22|k11|optics|p45|p99)-(config|python|shared)
|uk\.ac\.(diamond\.daq\.(beamline\.(i15|i15-1|i21|k11|p45|p99)|dls\.controls|ispyb\.scan\.info)(\..+)?
|gda\.(beamline\.(b07(-1)?|b21|i05(-1)?|i06(_1)?|i08(-1)?|i09(-1|-2)?|i10|i11|i11-1|i12|i13|i13-1|i14|i15|i15-1|i21|i22)|diamond|dls)
(\..+)?))$" searchPathRef="gda_diamond"/>

They are stored in the diamond-releng repository inside diamond.releng.buckminster/base. There is one for each supported version of GDA/Dawn. They are published to https://alfred.diamond.ac.uk/buckminster/base and used from there by pewma and Buckminster.

CSPEC and CSPECX files

A CSPEC (component specification) lists the attributes of a component such as how to build it and what other components it may depend on. It is used to convert something that Buckminster doesn’t understand unto a component. A CSPECX (CSPEC eXtenstions) are used to add additional information to a component that Buckminster does understand, like a plugin or feature.

CSPEC/CSPECX files are specified in XML and act like an additional manifest. They must be place in the root of a project. They are used by Diamond for the following:

  • To make the ‘plain’ configs directories into components.

  • During the materialization phase they are used to pull in test fragments which a plugin itself does not depend on (i.e. they are not defined in the MANIFEST.MF ). This is because the final product should not contain the test plugins.

  • For site projects they are used to define build actions that can be invoked by Buckminster, e.g. the create.product-linux.gtk.x86 runs an ant script (buckminster.ant) with the create.product target. They are used specifically to set options (e.g. paths) used in the ant build script. The actions are invoked with a property file which Diamond has two of:

    • For ‘normal’ use: buckminster.beamline.properties.

    • For Jenkins use only: buckminster.jenkins.properties.

A simple example of a CSPEC file from the p45-config (buckminster.cspec) is shown below and indicates that the p45-config has a dependency on dls-config.

<?xml version="1.0" encoding="UTF-8"?><cs:cspec xmlns:cs="http://www.eclipse.org/buckminster/CSpec-1.0" name="p45-config" componentType="buckminster" version="1.0.0">
    <cs:dependencies>
        <cs:dependency name="dls-config" componentType="buckminster"/>
    </cs:dependencies>
</cs:cspec>

An example of how a CSPECX file can be used to include a test fragment is shown below for the uk.ac.diamond.daq.mapping project.

<cspecExtension
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0"
	xmlns="http://www.eclipse.org/buckminster/CSpec-1.0">
	<dependencies>
		<dependency name="uk.ac.diamond.daq.mapping.test" componentType="osgi.bundle"/>
	</dependencies>

17.5.1.1.3. P2 repository

P2 is a provisioning platform for Eclipse based applications. A p2 repository (also sometimes referred to as TP) is a location that you can download jars from. They combine the artefacts (bundles), the meta data and can also contain Eclipse features.

When looking for version x.x.x of com.some.library in the repository, it will give you a jar and possible a source jar for that object.

Diamond has its own p2 repository (diamond-orbit) and also mirrors of several Eclipse p2 repositories from www.eclipse.org/downloads.These are published on alfred.diamond.ac.uk, e.g.

17.5.1.1.4. OSGi Bundle

A bundle is a group of Java classes and additional resources with a details manifest file on all its contents as well as additional requirements that may be needed. A META-INF/MANNIFEST.MF is required to be a valid OSGi bundle and it must include the following headers:

Manifest-Version: 1.0                   # Must be 1.0
Bundle-ManifestVersion: 2               # Must be 2 (almost...)
Bundle-Name: MyService bundle           # Not really used for much
Bundle-SymbolicName: com.sample.obj     # This is critical
Bundle-Version: 1.0.0                   # Must be 1.0.0 or higher

In addition, it may also have to define:

Bundle-Activator: com.sample.obj.Activator
Import-Package: org.slf4j;version="[1.7.2,2.0.0)"
Export-Package: com.sample.obj.api;version="1.0.0"

See here for more details on OSGi bundle manifests.

17.5.1.2. Materializing a GDA/Dawn Workspace

The materialization process is initiated with a command such as:

pewma materialize gdaserver

(In this example, it will materialize the gdaserver).

The script carries out the following steps:

  1. gdaserver is a pewma alias, so it will be expanded to uk.ac.diamond.daq.server.site in category gda.

  2. It creates two directories: workspace and workspace_git.

  3. It gets the zipped template workspace from [https://alfred.diamond.ac.uk/buckminster/templates/](https://alfred.diamond.a c.uk/buckminster/templates/). These may be version dependent.

  4. It unzips the template workspace into workspace.

  5. It writes a script file named workspace/pewma-script.txt) which tells Buckminster what to do. This will resolve shortened names, versions, etc.

    import -Dcomponent=uk.ac.diamond.daq.server.site
    https://alfred.diamond.ac.uk/buckminster/base/gda-master.cquery
    
  6. Finally it invokes Buckminster with the script and other options such as the workspace path, e.g.

    buckminster -application org.eclipse.buckminster.cmdline.headless
    --loglevel INFO -data /scratch/all-dls/temp/workspace --scriptfile
    /scratch/all-dls/temp/workspace/pewma-script.txt -vmargs -noverify
    -Xms768m -Xmx1536m -XX:+UseG1GC -XX:MaxGCPauseMillis=1000
    "-Declipse.p2.mirrors=false"
    

The GDA/Dawn workspace is created with objects that come from a lot of different locations. For example, source checkouts come from multiple Git repositories on multiple hosts (e.g. Gerrit, Github, etc). And binary artefacts (jars) come from several different p2 repositories (e.g. Diamond Orbit, Eclipse p2, PyDev p2, etc).

In comparison, a more classic (simple) project may checkout from just one Git repo which contains a build system config (like Maven/Gradle), which is used to fetch the other dependencies that are binary artefacts and then build the project.

The complexity of the Diamond system is handled by Buckminster.

17.5.1.2.1. How does Buckminster work?

  1. Firstly a CQUERY must be specified. This is either done by the user or it could already be specified by pewma in the workspace/pewma-script.txt. From this point (as an example), the CQUERY specified is assumed to be gda-master.cquery.

  2. Buckminster then fetches the CQUERY from: [https://alfred.diamond.ac.uk/buckminster/base/gda-master.cquery](https://alfred .diamond.ac.uk/buckminster/base/gda-master.cquery).

  3. From the CQUERY, it finds out what RMAP to use and fetches it. For example, gda-master.cquery defines the RMAP as core-master.rmap, so Buckminster gets it from: [https://alfred.diamond.ac.uk/buckminster/base/core-master.rmap](https://alfred. diamond.ac.uk/buckminster/base/core-master.rmap).

  4. Using the CQUERY and RMAP, it looks up the components that are asked for (via regex) and this tells it what to fetch. For example, in the core-master.rmap RMAP, uk.ac.gda.core matches with the locator:

 <rm:locator
pattern="^((core|example)-config|diamond\.releng\.tools\.gda|documentation\.gda|org\.apache\.derby\.fragment\
.daq\.persistence|org\.opengda\.lde(\..+)?|uk\.ac\.(diamond\.(daq\.(client\.gui\.(camera|energyfocus.editor|e
xitslit\.configuration|rcpcontroller)|detectors\.addetector(\..+)?|devices\.mobile(\..+)?|experiment(\..+)?|j
ython\.api|mapping(\..+)?|messaging(\..+)?|msgbus(\..+)?|persistence(\..+)?|remotedataset\.feature|scm\.api|s
tage(\.test)?)|scisoft\.gda\.feature)|gda\.(analysis(\.test)?|api(\.test)?|beamline\.synoptics|(client(\.(clo
seactions|commandinfo|experimentdefinition|feature|live\.stream|ncd(\.feature)?|test))?)|common\.client|core(
\.test)?|edxd\.(calibration|common|detector)|epics\.dxp\.client|eventbus(\.api)?|example(\..+)?|logging\.depe
ndencies\.fragment|server\.ncd(\.(epics|test))?|xmleditor\.feature)))$" searchPathRef="gda_core"/>

(notice searchPathRef="gda_core"). So the search path to use is gda_core, which it then goes to look up…

../_images/Buckminster_rmap_contents.png

Above is an extract on the core-master.rmap that defines the searchPath for gda_core. From this Buckminster can tell that it is a Git repository at the uri ${gda.core.git.url} which was set earlier in the RMAP to point to Gerrit via a property:

<rm:propertyElement key="gda.core.git.url">
  <bc:constant value="${diamond.dascgerrit.${dascgerrit.authentication.common}.url}/gda/gda-core.git"/>
</rm:propertyElement>

A more complicated example

The previous case was a fairly simple one compared to some other repositories that are materialized. The MX repository is a good example of this (see here for the gda_mx entry in the RMAP.). The complexity comes from the structure of the repository. It is not a flat directory of components and instead is broken up into separate directories for configs, plugins, features etc. This means that several different directories may need to be checked to locate a specific component. It is this complexity that Buckminster handles.

For example, try locating the p2 org.apache.commons.io:

../_images/Buckminster_mx_rmap1.png

It matches this regex and sees that the searchPath is diamond_orbit_p2 which is defined later in the RMAP as:

../_images/Buckminster_mx_rmap2.png

Once Buckminster has obtained that component (a plugin/feature/config, etc.), it checks what its dependencies are. For plugins it will read the MANIFEST.MF, for features the feature.xml and for configs the buckminster.cspec. It also checks for any cspecx files if present.

It will then do the same process for each dependency as before, look it up in the RMAP, download it or clone it and then import it into the workspace. This is repeated until all reachable dependencies have been obtained. This results in a workspace containing everything asked for and all its dependencies.

17.5.1.3. Product Export

All that has been done so far in the materialization phase is that Buckminster got everything that it was requested for including all the dependencies. This is all that is required to make a development workspace.

However, when GDA/Dawn is run for ‘real’, a workspace is built into a product to ship to users. The development workspace is of no interest to them and it shouldn’t just be lying around. The product should also include other user objects such as a runnable executable (and not an Eclipse launch config). Once it’s packaged it is self-contained, immutable and portable.

17.5.1.3.1. Products / Features / Plugins

Before understanding how to build products, it is first important to understand more about products, features and plugins.

  • Products: These define a set of features or plugins that together make a runnable application. It also points to an ‘application’ defined in a plugin.xml which defines the entry point class for the application. E.g.

    • Server - uk.ac.diamond.daq.server.GDAServerApplication

    • Client - gda.rcp.GDAClientApplication

  • Features: These group together plugins and other features to provide ‘useful’ functionality. They can be used when package imports are used to specify what packages are required but do not define where they are supplied from.

  • Plugins: They are an Eclipse versions of an OSGi bundle. Essentially they are an OSGi bundle with a valid OSGi manifest and declarative services (DS) etc. They can also include Eclipse specific things such as extension points defined by plugin.xml.

GDA uses ‘feature’ based products, e.g.

../_images/ProductBuild_Features.png

Or in the XML:

<product name="Data Acquisition Server"
uid="uk.ac.diamond.daq.server.product" id="uk.ac.diamond.daq.server.product"
application="uk.ac.diamond.daq.server.application" version="9.13.0"
useFeatures="true" includeLaunchers="true">

In GDA a product depends on only one feature (which does not have to be the case in general). Feature dependencies are followed but plugin dependencies are not. The only plugins included are those directly in a feature (see diagram below).

../_images/ProductBuild_Hierarchy.png

17.5.1.3.2. Packaging the product

Once the product -> feature -> plugin tree has been walked (as illustrated by the diagram above), a big list of plugins that make up the product is produced. The actual packaging is handled by an ant script which is called from a Buckminster action, e.g.

<actions>
<public name="create.product-linux.gtk.x86" actor="ant">
<actorProperties>
<property key="buildFile" value="buckminster.ant" />
<property key="targets" value="create.product" />
 Using this
build script
</actorProperties>

The actor="ant" indicates that it is an ant task, the build script (buildFiles) is buckminster.ant and the target (target) is create.product.

Some useful pewma commands to build products are given below.

17.5.1.3.3. Summary

During the materialize phase, all a plugin’s Require-Bundle dependencies will be imported to the workspace. The Import-Package dependencies will not and need to be satisfied by including another plugin.

During the product export phase, plugin dependencies are not resolved at all. Only plugins directly referenced by a feature are included. If the resulting set of plugins is not closed (i.e. all the dependencies are satisfied) then the export will fail.

There is an external web server alfred.diamond.ac.uk that serves the file system under /dls_sw/dasc/static_pages/ this is version controlled by diamond-releng containing the critical build objects that are mostly owned by dlshudson the Jenkins user.

17.5.1.4. Tests

On Jenkins, the test are run using pewma all-tests. This can also be run locally. There are additional commands that can be run:

  • pewma junit-tests: Runs only the junit tests.

  • pewma jyunit-tests: Runs only the jython tests.

  • pewma --include uk.ac.gda.core all-tests: Runs all the tests (i.e. junit and jyunit tests) in uk.ac.gda.core only and accepts global patterns.

These iterate though the workspace and runs the appropriate ant target in each project. The junit-tests and jyunit-tests targets are defined in the project releng.ant.

The results of the tests are put in the directory test-reports inside the project. They are just the JUnit XML output files and so they can be opened in the IDE.

The releng.ant is used to define 2 important things:

  1. Which tests should be run.

  2. The classpath used for the tests.

../_images/releng_ant_example.png

17.5.1.5. FAQ

See the FAQ section for common questions and answers relating to materializing or product builds.

17.5.1.5.1. Useful product build pewma commands

  • pewma sites - returns the list of sites = products buildable from this workspace, e.g.

    /scratch/all-dls/temp $ pewma sites
    2019-05-22 16:17:37 - INFO - "--workspace" defaulted to "/scratch/all-dls/temp/workspace"
    2019-05-22 16:17:38 - INFO - Available sites in /scratch/all-dls/temp/workspace_git: [u'uk.ac.diamond.daq.server.site']
    2019-05-22 16:17:38 - INFO - Run time was 00:00:00 [sites]
    
  • pewma product <site (from above)> - Builds that product, e.g.

    pewma product --recreate-symlink uk.ac.diamond.daq.server.site
    

    The --recreate-symlink option will make the server or client symlink automatically.