17.3. Building GDA

Maven is used to build GDA on the command line. This includes building GDA products. It is also used by the CI to build and run tests.

On DLS workstations it can be made available from the module system with:

module load maven

GDA is comprised of Eclipse PDE components. The PDE ecosystem varies substantially from plain Maven built Java projects. In order to integrate the two ecosystems Eclipse Tycho is used. Tycho is a set of Maven plugins to facilitate building PDE projects in Maven.

17.3.1. Structure of the build

Each PDE component, e.g. plug-ins, features and products, is mapped to a Maven project (which can also be referred to as an artifact or a module). The overall build is an aggregation of these projects. In Maven this is known as a multiple module build or a Maven reactor.

17.3.1.1. Tycho Pomless

The central structure in Maven is the POM. This defines the identifiers for a project as well as providing properties and build (Maven plugin) configuration. Project dependencies are also specified in the POM. The POM is specified in XML and is located in the project root (pom.xml).

Three key attributes of a Maven project are the groupId, artifactId, and version.

PDE employs its own mechanism for determining key project identifiers and dependencies based on the OSGi specification - manifests. Manifests define plug-in ID, version and dependencies. For more information on OSGi components see Products/Features/Plugins.

In order to prevent some of this metadata duplication, and to avoid maintaining a large number of POMs, Tycho contains a component called Tycho Pomless which maps PDE components to Maven projects by generating an ephemeral POM at build time. This allows the GDA build to contain a minimum number of pom.xml files.

External dependencies are specified in a PDE target platform rather than as direct dependencies in the POMs. This target platform is incorporated into the Tycho build ensuring a consistent environment between the IDE and Maven.

17.3.1.2. Project hierarchy

In addition to project aggregation, project inheritance also plays an important role in the GDA multi module build. For some further reading see Project Inheritance vs Project Aggregation.

The GDA build consists of multiple source code repositories, inside these the directory structure is not always consistent. Each repository contains at least one POM aggregating its containing projects. The repositories are aggregated in total in the daq-aggregator repository. This POM daq-aggregator.git/pom.xml is the entry point to the build.

Note that this is not the highest parent in terms of Maven inheritance, the parent project of daq-aggregator.git/pom.xml is daq-aggregator.git/releng/pom.xml.

17.3.1.3. Maven profiles

The GDA code base is built as a single unit - an aggregation of all projects. This simplifies the build in some ways and provides benefits such as ensuring that all Java code will compile when making a change to any GDA repository. This, however, is with the exception of the product projects and their corresponding features. These projects don’t contain any compilable code but are rather recipes to produce products. Assembling products takes a long time therefore these projects are not part of the build by default but are built by enabling specific Maven profiles.

In addition to these “product” profiles there are some additional profiles defined in the build to be aware of.

17.3.1.3.1. gda & dawn

Since the aggregated build is comprised of source repositories for both GDA and Dawn these profiles are used to determine which to build. If no profiles are specified at all, the gda profile will be enabled by default. These profiles largely just determine which of the repositories to traverse into to find child Maven modules.

17.3.1.3.2. server

This is the profile to include the GDA server product and feature projects in the build. The POM for the server has been augmented to additionally extract the product from the usual build output location target/ and place it in a timestamped directory in the deployment

For example:

/dls_sw/p45/software/gda/
├── servers
│   └── server_20240305-1540
└── workspace_git
    ├── ...
    ├── ...

17.3.1.3.3. iXX

Each beamline has its own client product definition. The iXX profiles (e.g. p45, i03, b16) enable the beamline client product to be included in the build.

As is the case with the server, the product will also be deployed to the standard location:

/dls_sw/p45/software/gda/
├── clients
│   └── client_20240305-1540
└── workspace_git
    ├── ...
    └──...

17.3.2. Performing a build

Maven plugins contribute goals, some of which are attached to specific phases in the Maven build lifecycle.

For example the tycho-compiler-plugin’s compile goal is attached to the compile phase by default and would be executed when running mvn compile.

Tycho has many other goals required which are attached to various phases. The result is that generally Maven is always required to be invoked up to the verify phase. This is the highest phase in the default lifecycle apart from install and deploy which we do not use as part of the GDA build/deploy procedure.

Additionally is is generally recommended (at least historically) to run clean before building when using tycho, i.e. mvn clean verify

17.3.2.1. Typical command to build products for a beamline

# Switch to daq-aggregator.git
cd <PATH-TO-GDA-DEPLOYMENT>/workspace_git/daq-aggregator.git

# Run Maven
mvn -DskipTests -Pgda,server,p45,recreate-symlink clean verify

This will invoke Maven to compile all the projects, materialize the products for the GDA server and the p45 beamline client, deploy them and create the symlinks as referred to above. The system property to skip the tests will almost always be used for beamline deployments as the tests are usually only run with Maven in the CI.

17.3.2.1.1. Optimised Maven command

For beamlines GDA is deployed to the /dls_sw filesystem. This filesystem can suffer from performance limitations. A special property has been added to the build configuration to override the build output directory which allows a local disk to be used as the build output (the default build output is target/, inside the project source). The final product artifact will still be deployed to the expected location.

In addition Maven supports parallel execution which can be enabled with the -T switch.

Combing these two points leads to:

mvn -DskipTests -Dalt.build.dir='/scratch/30day_tmp/build' \
-Pgda,server,p45,recreate-symlink -T1C clean verify

This will result in significantly faster build times compared to the command above.

The location /scratch/30day_tmp is present on all DLS workstations.

The setting -T1C will instruct Maven to use one thread per CPU core. This may be modified if desired.

17.3.3. Appendix

17.3.3.1. Managing local m2 repository

The default location for a user’s local maven repository is ~/.m2. For DLS workstations this is subject to the staff home quota of 8GB. There is a Maven property to override this location however it is probably simpler to replace the directory with a symlink to an alternative. Two examples of this are shown below. Before doing this, delete the original m2 directory if it exists (~/.m2).

# Link to /dls/science/users/<fedid> - users have a 50GB quota here
# This is also NFS so will be available on any Diamond machine.
# This is consistent with the dev-inst script meaning the .p2, .m2 and .eclipse
# folders would all be under the same parent
mkdir /dls/science/users/<fedid>/m2 # or call it anything you like
ln -s /dls/science/users/<fedid>/m2 ~/.m2

# Link to scratch - this will be a bit faster but limited to the current machine
mkdir -pv /scratch/<fedid>/m2 # or call it anything you like
ln -s /scratch/<fedid>/m2 ~/.m2