17.5. Buckminster Build (Legacy)¶
As of October 2020 the master
version of GDA is built using Maven/Tycho.
This page is currently kept for legacy releases/general historical info.
17.5.1. The GDA Materialize Process¶
17.5.1.1. Definition of components involved¶
17.5.1.1.1. pewma¶
pewma.py (Python Eclipse Workspace MAnager) is a python
script that lives at /dls_sw/dasc/pewma.py
and is version controlled at
diamond-releng/diamond.releng.buckminster/software/pewma.py
. It acts as a
wrapper around Buckminster (which has a difficult command line
interface itself) and provides lots of features to help manage workspaces. A
list of these can be accessed using the command:
pewma --help
It provides aliases for more complicated variables to aid ease of use. E.g.
i02-client = uk.ac.gda.beamline.i02.site
gdaserver = uk.ac.diamond.daq.server.site
dawndiamond = uk.ac.diamond.dawn.site It can also automatically figures out which CQUERY to use. The master is the default version to materialize but a specific version can also be provided as an argument.
17.5.1.1.2. Buckminster¶
Buckminster is a headless Eclipse for provisioning, building and testing. While it can be considered a general purpose build tool, in reality it is only used for Java and mostly within the use of Eclipse. It is particularly useful for building and sharing distributions of software components which share components across multiple software projects and repositories (much like GDA).The manual can be accessed from a Diamond file location (note that the correct permissions are needed to view it). Equally the archived website can be found here.
It is used within GDA for materializing via the pewma script and building the exported products (i.e. the server and clients).
A Buckminster CQUERY (Componenet Query) names a component assembly and Buckminster uses it to locate all the components necessary to complete the materialization process.
The CQUERY:
Points to an RMAP (this is key).
It allows properties to be set and can define defaults and also pick ups, e.g.
${user.home}/.{dawn,gda}-master.cquery.properties
. It can also define how to skip Dawn/GDA test fragment (as appropriate) and how to switch between authenticated and unauthenticated clones.It allows ‘advisor’ notes to be added to specific components, which is used to specify particular git branches when checking out release versions, e.g.
branchTagPath = "dawn-2.10"
. For this reason it is IMPORTANT to keep CQUERY and RMAP regex synchronised.
An extract from the gda-master.cquery
is given below:
<cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="core-master.rmap" properties="file:${user.home}/.gda-master.cquery.properties" shortDesc="CQuery for GDA master">
...
<cq:property key="skip_DAWN_test_fragments" value="true"/>
...
<cq:advisorNode namePattern="^com\.globalphasing\.sdcp\.abstract_beamline$" useTargetPlatform="false" branchTagPath="gda-master">
<cq:documentation xmlns="http://www.w3.org/1999/xhtml">gphl-abstract-beamline.git repository branch</cq:documentation>
</cq:advisorNode>
A Buckminster RMAP (Resource Map) is associated with a CQUERY and lists the repositories in which the defined components can be found. This is how Buckminster loops up where to find the components. An RMAP file essentially acts as a big regex lookup table.
E.g. The location of the beamline configurations is defined in the
core-master.rmap
, an extract of which is shown below:
<!--
Git repositories
-->
<rm:propertyElement key="gda.diamond.git.url">
<bc:constant value="${diamond.dascgerrit.${dascgerrit.authentication.common}.url}/gda/gda-diamond.git"/>
</rm:propertyElement>
...
<!--#
Locators
#-->
<rm:locator pattern="^(all-dls-(configs|clients)|(dls|b07(-1)?|b21|i05(-1)?|i06(-1)?|i08(-1)?|i09(-1|-2)?
|i10|i11|i11-1|i12|i13|i13-1|i14|i15|i15-1|i21|i22|k11|optics|p45|p99)-(config|python|shared)
|uk\.ac\.(diamond\.daq\.(beamline\.(i15|i15-1|i21|k11|p45|p99)|dls\.controls|ispyb\.scan\.info)(\..+)?
|gda\.(beamline\.(b07(-1)?|b21|i05(-1)?|i06(_1)?|i08(-1)?|i09(-1|-2)?|i10|i11|i11-1|i12|i13|i13-1|i14|i15|i15-1|i21|i22)|diamond|dls)
(\..+)?))$" searchPathRef="gda_diamond"/>
They are stored in the diamond-releng
repository inside
diamond.releng.buckminster/base.
There is one for each supported version of GDA/Dawn. They are published to
https://alfred.diamond.ac.uk/buckminster/base
and used from there by pewma and Buckminster.
A CSPEC (component specification) lists the attributes of a component such as how to build it and what other components it may depend on. It is used to convert something that Buckminster doesn’t understand unto a component. A CSPECX (CSPEC eXtenstions) are used to add additional information to a component that Buckminster does understand, like a plugin or feature.
CSPEC/CSPECX files are specified in XML and act like an additional manifest. They must be place in the root of a project. They are used by Diamond for the following:
To make the ‘plain’
configs
directories into components.During the materialization phase they are used to pull in test fragments which a plugin itself does not depend on (i.e. they are not defined in the
MANIFEST.MF
). This is because the final product should not contain the test plugins.For site projects they are used to define build actions that can be invoked by Buckminster, e.g. the
create.product-linux.gtk.x86
runs an ant script (buckminster.ant
) with thecreate.product
target. They are used specifically to set options (e.g. paths) used in the ant build script. The actions are invoked with a property file which Diamond has two of:For ‘normal’ use:
buckminster.beamline.properties
.For Jenkins use only:
buckminster.jenkins.properties
.
A simple example of a CSPEC file from the p45-config (buckminster.cspec
) is
shown below and indicates that the p45-config
has a dependency on
dls-config
.
<?xml version="1.0" encoding="UTF-8"?><cs:cspec xmlns:cs="http://www.eclipse.org/buckminster/CSpec-1.0" name="p45-config" componentType="buckminster" version="1.0.0">
<cs:dependencies>
<cs:dependency name="dls-config" componentType="buckminster"/>
</cs:dependencies>
</cs:cspec>
An example of how a CSPECX file can be used to include a test fragment is shown
below for the uk.ac.diamond.daq.mapping
project.
<cspecExtension
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0"
xmlns="http://www.eclipse.org/buckminster/CSpec-1.0">
<dependencies>
<dependency name="uk.ac.diamond.daq.mapping.test" componentType="osgi.bundle"/>
</dependencies>
17.5.1.1.3. P2 repository¶
P2 is a provisioning platform for Eclipse based applications. A p2 repository
(also sometimes referred to as TP) is a location that you can download jars
from. They combine the artefacts (bundles), the meta data and can also contain
Eclipse features.
When looking for version x.x.x of com.some.library
in the repository, it will
give you a jar and possible a source jar for that object.
Diamond has its own p2 repository (diamond-orbit) and also mirrors of several
Eclipse p2 repositories from
www.eclipse.org/downloads.These are
published on alfred.diamond.ac.uk
, e.g.
17.5.1.1.4. OSGi Bundle¶
A bundle is a group of Java classes and additional resources with a details
manifest file on all its contents as well as additional requirements that may
be needed. A META-INF/MANNIFEST.MF
is required to be a valid OSGi bundle and
it must include the following headers:
Manifest-Version: 1.0 # Must be 1.0
Bundle-ManifestVersion: 2 # Must be 2 (almost...)
Bundle-Name: MyService bundle # Not really used for much
Bundle-SymbolicName: com.sample.obj # This is critical
Bundle-Version: 1.0.0 # Must be 1.0.0 or higher
In addition, it may also have to define:
Bundle-Activator: com.sample.obj.Activator
Import-Package: org.slf4j;version="[1.7.2,2.0.0)"
Export-Package: com.sample.obj.api;version="1.0.0"
See here for more details on OSGi bundle manifests.
17.5.1.2. Materializing a GDA/Dawn Workspace¶
The materialization process is initiated with a command such as:
pewma materialize gdaserver
(In this example, it will materialize the gdaserver).
The script carries out the following steps:
gdaserver
is a pewma alias, so it will be expanded touk.ac.diamond.daq.server.site
in categorygda
.It creates two directories:
workspace
andworkspace_git
.It gets the zipped template workspace from [
https://alfred.diamond.ac.uk/buckminster/templates/
](https://alfred.diamond.a c.uk/buckminster/templates/). These may be version dependent.It unzips the template workspace into
workspace
.It writes a script file named
workspace/pewma-script.txt)
which tells Buckminster what to do. This will resolve shortened names, versions, etc.import -Dcomponent=uk.ac.diamond.daq.server.site https://alfred.diamond.ac.uk/buckminster/base/gda-master.cquery
Finally it invokes Buckminster with the script and other options such as the workspace path, e.g.
buckminster -application org.eclipse.buckminster.cmdline.headless --loglevel INFO -data /scratch/all-dls/temp/workspace --scriptfile /scratch/all-dls/temp/workspace/pewma-script.txt -vmargs -noverify -Xms768m -Xmx1536m -XX:+UseG1GC -XX:MaxGCPauseMillis=1000 "-Declipse.p2.mirrors=false"
The GDA/Dawn workspace is created with objects that come from a lot of different locations. For example, source checkouts come from multiple Git repositories on multiple hosts (e.g. Gerrit, Github, etc). And binary artefacts (jars) come from several different p2 repositories (e.g. Diamond Orbit, Eclipse p2, PyDev p2, etc).
In comparison, a more classic (simple) project may checkout from just one Git repo which contains a build system config (like Maven/Gradle), which is used to fetch the other dependencies that are binary artefacts and then build the project.
The complexity of the Diamond system is handled by Buckminster.
17.5.1.2.1. How does Buckminster work?¶
Firstly a CQUERY must be specified. This is either done by the user or it could already be specified by pewma in the
workspace/pewma-script.txt
. From this point (as an example), the CQUERY specified is assumed to begda-master.cquery
.Buckminster then fetches the CQUERY from: [https://alfred.diamond.ac.uk/buckminster/base/gda-master.cquery](https://alfred .diamond.ac.uk/buckminster/base/gda-master.cquery).
From the CQUERY, it finds out what RMAP to use and fetches it. For example,
gda-master.cquery
defines the RMAP ascore-master.rmap
, so Buckminster gets it from: [https://alfred.diamond.ac.uk/buckminster/base/core-master.rmap](https://alfred. diamond.ac.uk/buckminster/base/core-master.rmap).Using the CQUERY and RMAP, it looks up the components that are asked for (via regex) and this tells it what to fetch. For example, in the
core-master.rmap
RMAP,uk.ac.gda.core
matches with the locator:
<rm:locator pattern="^((core|example)-config|diamond\.releng\.tools\.gda|documentation\.gda|org\.apache\.derby\.fragment\ .daq\.persistence|org\.opengda\.lde(\..+)?|uk\.ac\.(diamond\.(daq\.(client\.gui\.(camera|energyfocus.editor|e xitslit\.configuration|rcpcontroller)|detectors\.addetector(\..+)?|devices\.mobile(\..+)?|experiment(\..+)?|j ython\.api|mapping(\..+)?|messaging(\..+)?|msgbus(\..+)?|persistence(\..+)?|remotedataset\.feature|scm\.api|s tage(\.test)?)|scisoft\.gda\.feature)|gda\.(analysis(\.test)?|api(\.test)?|beamline\.synoptics|(client(\.(clo seactions|commandinfo|experimentdefinition|feature|live\.stream|ncd(\.feature)?|test))?)|common\.client|core( \.test)?|edxd\.(calibration|common|detector)|epics\.dxp\.client|eventbus(\.api)?|example(\..+)?|logging\.depe ndencies\.fragment|server\.ncd(\.(epics|test))?|xmleditor\.feature)))$" searchPathRef="gda_core"/>
(notice
searchPathRef="gda_core"
). So the search path to use isgda_core
, which it then goes to look up…
Above is an extract on the
core-master.rmap
that defines thesearchPath
forgda_core
. From this Buckminster can tell that it is a Git repository at the uri${gda.core.git.url}
which was set earlier in the RMAP to point to Gerrit via a property:
<rm:propertyElement key="gda.core.git.url"> <bc:constant value="${diamond.dascgerrit.${dascgerrit.authentication.common}.url}/gda/gda-core.git"/> </rm:propertyElement>
A more complicated example
The previous case was a fairly simple one compared to some other repositories
that are materialized. The MX repository is a good example of this (see here for the gda_mx
entry in the
RMAP.). The complexity comes from the structure of the repository. It is not a
flat directory of components and instead is broken up into separate directories
for configs, plugins, features etc. This means that several different
directories may need to be checked to locate a specific component. It is this
complexity that Buckminster handles.
For example, try locating the p2 org.apache.commons.io
:
It matches this regex and sees that the searchPath
is diamond_orbit_p2
which is defined later in the RMAP as:
Once Buckminster has obtained that component (a plugin/feature/config, etc.),
it checks what its dependencies are. For plugins it will read the
MANIFEST.MF
, for features the feature.xml
and for configs the
buckminster.cspec
. It also checks for any cspecx
files if present.
It will then do the same process for each dependency as before, look it up in the RMAP, download it or clone it and then import it into the workspace. This is repeated until all reachable dependencies have been obtained. This results in a workspace containing everything asked for and all its dependencies.
17.5.1.3. Product Export¶
All that has been done so far in the materialization phase is that Buckminster got everything that it was requested for including all the dependencies. This is all that is required to make a development workspace.
However, when GDA/Dawn is run for ‘real’, a workspace is built into a product to ship to users. The development workspace is of no interest to them and it shouldn’t just be lying around. The product should also include other user objects such as a runnable executable (and not an Eclipse launch config). Once it’s packaged it is self-contained, immutable and portable.
17.5.1.3.1. Products / Features / Plugins¶
Before understanding how to build products, it is first important to understand more about products, features and plugins.
Products: These define a set of features or plugins that together make a runnable application. It also points to an ‘application’ defined in a
plugin.xml
which defines the entry point class for the application. E.g.Server -
uk.ac.diamond.daq.server.GDAServerApplication
Client -
gda.rcp.GDAClientApplication
Features: These group together plugins and other features to provide ‘useful’ functionality. They can be used when package imports are used to specify what packages are required but do not define where they are supplied from.
Plugins: They are an Eclipse versions of an OSGi bundle. Essentially they are an OSGi bundle with a valid OSGi manifest and declarative services (DS) etc. They can also include Eclipse specific things such as extension points defined by
plugin.xml
.
GDA uses ‘feature’ based products, e.g.
Or in the XML:
<product name="Data Acquisition Server"
uid="uk.ac.diamond.daq.server.product" id="uk.ac.diamond.daq.server.product"
application="uk.ac.diamond.daq.server.application" version="9.13.0"
useFeatures="true" includeLaunchers="true">
In GDA a product depends on only one feature (which does not have to be the case in general). Feature dependencies are followed but plugin dependencies are not. The only plugins included are those directly in a feature (see diagram below).
17.5.1.3.2. Packaging the product¶
Once the product -> feature -> plugin tree has been walked (as illustrated by the diagram above), a big list of plugins that make up the product is produced. The actual packaging is handled by an ant script which is called from a Buckminster action, e.g.
<actions>
<public name="create.product-linux.gtk.x86" actor="ant">
<actorProperties>
<property key="buildFile" value="buckminster.ant" />
<property key="targets" value="create.product" />
Using this
build script
</actorProperties>
The actor="ant"
indicates that it is an ant task, the build script
(buildFiles
) is buckminster.ant
and the target (target
) is
create.product
.
Some useful pewma commands to build products are given below.
17.5.1.3.3. Summary¶
During the materialize phase, all a plugin’s Require-Bundle
dependencies will
be imported to the workspace. The Import-Package
dependencies will not and
need to be satisfied by including another plugin.
During the product export phase, plugin dependencies are not resolved at all. Only plugins directly referenced by a feature are included. If the resulting set of plugins is not closed (i.e. all the dependencies are satisfied) then the export will fail.
There is an external web server alfred.diamond.ac.uk
that serves the file
system under /dls_sw/dasc/static_pages/
this is version controlled by
diamond-releng
containing the critical build objects that are mostly owned by
dlshudson
the Jenkins user.
17.5.1.4. Tests¶
On Jenkins, the test are run using pewma all-tests
. This can also be run
locally. There are additional commands that can be run:
pewma junit-tests
: Runs only the junit tests.pewma jyunit-tests
: Runs only the jython tests.pewma --include uk.ac.gda.core all-tests
: Runs all the tests (i.e. junit and jyunit tests) in uk.ac.gda.core only and accepts global patterns.
These iterate though the workspace and runs the appropriate ant target in each
project. The junit-tests
and jyunit-tests
targets are defined in the
project releng.ant
.
The results of the tests are put in the directory test-reports
inside the
project. They are just the JUnit XML output files and so they can be opened in
the IDE.
The releng.ant
is used to define 2 important things:
Which tests should be run.
The classpath used for the tests.
17.5.1.5. FAQ¶
See the FAQ section for common questions and answers relating to materializing or product builds.
17.5.1.5.1. Useful product build pewma commands¶
pewma sites
- returns the list ofsites = products
buildable from this workspace, e.g./scratch/all-dls/temp $ pewma sites 2019-05-22 16:17:37 - INFO - "--workspace" defaulted to "/scratch/all-dls/temp/workspace" 2019-05-22 16:17:38 - INFO - Available sites in /scratch/all-dls/temp/workspace_git: [u'uk.ac.diamond.daq.server.site'] 2019-05-22 16:17:38 - INFO - Run time was 00:00:00 [sites]
pewma product <site (from above)>
- Builds that product, e.g.pewma product --recreate-symlink uk.ac.diamond.daq.server.site
The
--recreate-symlink
option will make theserver
orclient
symlink automatically.