Introduction

The Amygdala GDA launcher is a daemon running on the control machine enabling clients to start, stop and query the status of the GDA server.

This was previously managed by a series of scripts copied and modified between beamlines and relied on the default command of an SSH connection. This launcher offers several benfits.

• Clear command line interface

  The arguments and options available are easy to find and are structured in a more maintainable way. They are documented and autocompletion is available.

• Re-uses configuration

  Where the previous script based solution offered a 'fire and forget' approach to starting the server where every time it ran it had to re-establish the configuration from scratch (twice due to running the same scripts on the server and client machines), the daemon is a long running process that loads its configuration once at start up and re-uses it for subsequent launches of GDA.

• Tracks the running GDA server

  It also keeps track of the server as it is running so that shutting down the server can be done cleanly without scripts having to grep through the output of ps.

  Tracking the running server and using that information when starting the client ensures that the correct client is always run and provides instant feedback when a client is started when there is no server running.

• Well defined API

  Users connect to the launcher via a CLI client (gda) or a GUI equivalent, but as the daemon offers a well defined API, there is nothing stopping alternative clients being used if required (potentially within GDA itself).

• Stand-alone compiled executable

  The code for the launcher is in its own repository, separate from the GDA application. It is compiled to a single executable that can be shared between all instances instead of a series of scripts duplicated for each server. It is written in rust offering a more readable codebase than bash and meaning minimal resources are required for the daemon (roughly 6MB of memory, negligible CPU use).

• Separate Configuration

  Where the scripts had several places where configuration values were set or derived, spread over several files, amygdala uses a separate TOML file providing a single point of reference for all configuration used by the process.

• No SSH keys

  As the client makes its own connection to the daemon (via the gRPC protocol), there is no need to manage the SSH configuration of users or share keys to access the gda2 user. If access needs to be restricted it can be limited to only the beamline network by setting a subnet mask.

Components

amygdala

This is the server daemon that accepts start/stop/status requests and controls the GDA server

gda

This is the client application used to communicate with amygdala. It is intended as a replacement to the existing gda scripts although while it offers a similar interface, there are differences (that hopefully make it more intuitive to use). See the migration section for more information about the changes.

Why Amygdala?

Mainly because it's one of the few words containing GDA and its overlap with 'GDALAuncher' but also its role as a central component of the brain handling decision making and recall of previous experiences is tangentially relevant (if you can get over the stress and anxiety it causes).

Installation

Installation

Both the amygdala daemon and the gda client are single, stand-alone binaries that require no installation beyond ensuring the executable is on your $PATH.

Using the module system, the launcher is available via

$ module load gda_launcher

This will make the most recent stable release available. If you would like to test a pre-release version you can use

$ module load gda_launcher/nightly

or see a list of available versions with

$ module avail gda_launcher

You can check the path is set up correctly and you have the version you expect with --version.

$ gda --version
gda - 0.2.2
Built: Fri, 14 Jun 2024 16:27:39 +0000
Commit: db1054466befd101e67687f3e993bffed41a9312
$ amygdala --version
amygdala - 0.2.0
Built Fri, 24 May 2024 16:20:13 +0000
From acdf800d085bcc3167e7b8b0fb5d21e2863e186b

Downloading binaries

The built binaries for each version will be made available by CI in the repo and can be downloaded from the release page here.

Building from source

Both components can be built by cloning the source from here and running

cargo build --release --all

You will need the rust/cargo toolchain to be available - see rustup.rs. See development section for more details.

Basic Use

Using the gda client command

Assuming the daemon is running on the control machine for a beamline and the client is being run on one of the beamline workstations, no configuration or arguments should be needed.

$ # (re)start the server
$ gda server
$ # Stop the server
$ gda server stop
$ # Start the client
$ gda client

The interface has remained largely unchanged from the previous scripts except where it made sense to make things more consistent or better follow conventions. For the equivalent to common commands that have changed, see the migration page.

If $BEAMLINE is defined, control machine will be assumed to be http://$BEAMLINE-control:50051. If $BEAMLINE is not defined and no host is specified on the command line, localhost will be used.

Alternative servers

If you are not running on a beamline workstation or you would like to interact with a different beamline, you can specify the host with the -H or --host option.

$ # start the server on ixx-control
$ gda --host ixx-control server

Accessing help

The full CLI docs are available via the help command. Passing --help to any subcommand will give more detailed help on that specific command.

There is also more documentation available in the cli reference section and its sub-pages. If anything is unclear or does not behave as expected, please open an issue here.

Running the amygdala daemon

The amygdala daemon takes its configuration from two sources. The fields that determine how the process itself runs, are passed as command line arguments - for full details see the amygdala CLI reference. The fields that determine how the GDA processes it launches are started and managed, are loaded from a configuration file. This file is detailed in the config file reference section and can either be set via the -c/--config CLI option, or via the $AMYGDALA_CONFIG environment variable.

For beamline controls machines, the process should be running automatically via systemd. To restart it or to check its status you can use systemctl as a user with sufficient privileges on the control machine (with sudo if required).

$ # Check the status - this should include the path to the binary that is
$ # running if needed
$ systemctl status gda_launcher.service
● gda_launcher.service - "Amygdala - GDA server launcher for ixx"
...
$ # Restart the daemon (see also `gda manage restart` command)
$ sudo systemctl restart gda_launcher.service
...
$ # Stop the daemon (eg to try running the nightly version for testing)
$ sudo systemctl stop gda_launcher.service
...
$ # To start the nightly version
$ sudo systemctl start gda_launcher_nightly.service
...

Running locally

To run the amygdala daemon on a local workstation for development, you can either set the $AMYGDALA_CONFIG environment variable to a config file, or pass the path as an option on the command line:

$ export AMYGDALA_CONFIG=/path/to/config.toml
$ amygdala serve
<...> Starting Amygdala server version="0.3.0" ...
...

The process will then run in the foreground and can be interrupted using Ctrl-C. If a GDA server is running, it will be shutdown gracefully before the process exits. If it has crashed for some reason and amygdala is waiting for it, pressing Ctrl-C again while it is shutting down will send a SIGKILL to the GDA server and terminate immediately.

High level logging related to the service itself is written to the terminal, while more detailed debug logs are available via the -v/--verbose flag. This can be repeated to increase the logging level to DEBUG and then TRACE. Logs can also be written to a date-stamped file in the directory specified by log_directory. The directory being used should be printed to the terminal at start up.

To develop the amygdala codebase itself, see the developer section.

Common Commands

Most of the common commands have intentionally not changed to minimise disruption

Restart the GDA server

$ gda server restart
$ # As restart is the most common server command, it is the default and can be omitted
$ gda server

Start the client

$ gda client

Start the logpanel

$ gda logpanel

Offline Use

Running Amygdala locally to run GDA on a dev machine or in a lab is very similar to running on a beamline control machine but requires some configuration and directories to be created/available.

Initial Setup

Create Required Directories

Deployment root

This is the directory where the GDA deployments are and for dev machines likely already exists. For a dev machine this could be something like /scratch/gda_workspaces/. It is the equivalent of /dls_sw/ixx/software/gda_versions/ on live systems.

Log directory

This is the directory where the stdout/stderr logs from the gda processes will be written as well as well as the logs from amygdala itself.

This should have subdirectories gda_launcher, gda-client-output and gda-server-output eg,

/scratch/local_gda/logs
├── gda-client-output
├── gda-servers-output
└── gda_launcher

Runtime root directories

These are the parent directories of the eclipse workspaces and configs. These both default to /tmp which is fine if you don't need workspaces to persist but can be set to a local directory if required

Write Config File

The config file for amygdala lists all the directories created above

deployment_root = "/scratch/gda_workspaces"
default_deployment = "master"
log_directory = "/scratch/local_gda/logs"
server_runtime_root = "/scratch/local_gda/server"
client_runtime_root = "/scratch/local_gda/client"

The default_deployment field should be the name of the deployment directory within the deployment_root that should be used by default.

Start Amygdala

On Diamond machines, amygdala is available via module load amygdala or module load amygdala/nightly.

$ AMYGDALA_CONFIG=/path/to/config/file BEAMLINE=ixx amygdala

The BEAMLINE variable being set determines the name of the client executable to run within the built product, eg gda-ixx. It also ensures that the server runs with the environment variable set. If this is not going to change, it can also be specified in the config file by using the full path in the layout section of the config.

[layout]
client = "client/gda-ixx"

Starting GDA

Once amygdala is running, it should behave the same as a live version on beamlines. The same gda command can be used as normal.

Determining which amygdala to use

By default (if $BEAMLINE is not set), gda will use amygdala running on localhost. A host can be specified if required using the -H/--host option to the gda command (prior to any subcommand, eg gda -H localhost server status).

Shutting Down Amygdala

When amygdala is no longer required it can be shutdown using Ctrl-C if running in the foreground. Any running GDA server will be shut down before amygdala exits.

If amygdala is running in the background it can be shutdown using the linux kill or pkill commands. SIGINT or SIGSTOP should be used if the GDA server should be shutdown as no clean up work can be done after receiving SIGKILL.

Setting up a new beamline

Because every beamline is sharing the same executable and there are differences in layout between beamlines, there is some amount of configuration required for the daemon to run.

Steps for all beamlines

Create a config file

All configuration should be in a toml file. By default for the beamline control machines, this should be available at /dls_sw/ixx/etc/gda_launcher.toml. It should have dls_dasc as the group and have 464 permissions (world readable, writeable only by dls_dasc).

Add a symlink in the gda_versions directory

This is optional but recommended. The link should usually be named gda and point to the deployment directory next to it that should be the default version started by gda server. See Deployment Root below.

Ensure the GDA supports the new configuration loading

The launcher daemon and client depend on the new (9.33+) configuration loading services in GDA. The default deployment configured should be using the new configuration.

Check PATH variable in beamline profile

The $PATH variable used by the Diamond Launcher needs to include the gda_launcher directory. If a beamline is migrating from the previous scripts, this should be added so that it is later in the list than the current /dls_sw/ixx/software/gda/config/bin directory. See the launcher transition section for more details for working with both approaches to launching GDA.

Check the commands run by the Diamond Launcher

These are configured in the applications.json file in /dls_sw/ixx/etc and any files in the /dls_sw/ixx/etc/dls-launcher-applications directory. These are often symlinks to GDA config so maintaining two versions should be possible. See the Diamond Launcher section of the transition page for more.

Testing Beamline Setup

The new configuration file and beamline setup can be tested without changing the existing pre-9.33 deployments of GDA. The amygdala launcher should be running on all beamlines and can be left running while previous versions of GDA are being used. If it is not running, see the getting started page. For testing see Initial Testing section for details.

For supporting both pre and post 9.33 versions of GDA on the same beamline, see Managing the Transition.

Minimal Configuration File

The minimal configuration required for most beamlines would be

log_directory = "/dls_sw/ixx/logs"
server_runtime_root = "gda_server"
client_runtime_root = "/scratch"

deployment_root = "/dls_sw/ixx/software/gda_versions"
default_deployment = "gda"

[env]
"GDA_MODE" = "live"
"GDA_VAR" = "/dls_sw/ixx/software/gda_var"

The values for log and var directories should be adjusted to match the layout of the beamline.

For full details of what can be configured, see the server configuration section.

Deployment Root

The way that the default deployment is chosen is changing from the previous system. The motivation for this is to allow beamline staff to launch alternative deployments meaning support engineers can deploy a test version for a new feature and let the beamline test it in their own time; they do not need to be available to switch symlinks around (which have traditionally been readonly to beamline staff).

The new launcher has a 'deployment root' configured (this would usually be /dls_sw/ixx/software/gda_versions/) and a default deployment (name of the deployment directory within the root). The default is intended to be a gda symlink to the current release in the same way that the /dls_sw/ixx/software/gda symlink is for the previous scripts. When a new release is made, the symlink is updated so that by default the new version is used. The existing gda symlink can remain and point to the new one for convenience or to keep existing scripts working if required.

The file hierarchy then becomes something like this

/dls_sw/ixx/software
├── gda -> gda_versions/gda
├── gda_versions
│  ├── gda -> gda-9.33
│  ├── master
│  │  ├── client -> clients/client_20240119-1419
│  │  ├── clients
│  │  ├── config -> workspace_git/gda-diamond.git/configurations/i22-config
│  │  ├── server -> servers/server_20240119-1419
│  │  ├── servers
│  │  └── workspace_git
│  └── gda-9.33
│     ├── client -> clients/client_20240215-1232
│     ├── clients
│     ├── config -> workspace_git/gda-diamond.git/configurations/i22-config
│     ├── server -> servers/server_20240215-1232
│     ├── servers
│     └── workspace_git

If the beamline staff want to run an alternative version, they can specify the deployment via the -d/--deployment options to the gda server start command. This will look up the chosen version in the deployment root and use that instead.

For the above file tree and the default_deployment set to gda:

$ # Launch the default server (gda-9.33)
$ gda server
$ # Launch the master deployment for testing
$ gda server start --deployment master

The client does not have the deployment option as it will always use the same deployment as the currently running server to ensure the two can't be out of sync with each other.

Initial testing with launcher

Making sure that the new launcher works with your beamline's configuration can be done without affecting the beamline. For more thorough testing including the 'infrastructure' changes (beamline environment, DLS launcher etc) and the considerations needed to ensure pre-9.33 installations can still be used, see the Managing the Transistion page.

Setting up the beamline

For full details of setting up the beamline see Setting up a new beamline. For testing the new config and launcher combination, a smaller set of minimal changes are required.

1. Write the config file to somewhere accessible to gda2.
2. Make a new deployment of GDA with the updated config (in gda_versions)
3. Check that the default_deployment config entry is the name of the new deployment and the directory is gda_versions.

Running laucher daemon on beamline

While the launcher daemon will eventually be managed by systemd, it is not currently running on control machines so for testing the new setup it needs to be run manually.

1. Log into control machine

   $ ssh ixx-control
   

2. Log in as gda2

   $ sudo su - gda2
   

3. Load new launcher from module system

   $ module load gda_launcher
   

   For now while the launcher is still being tested, this is loading a pre-release/nightly version. When a final release is made and the beamlines start using it live, this will change to default to a stable version with a nightly option for testing.

4. Load current java version from launcher

   $ module load java/gdamaster
   

   This step should only be required until the JRE is bundled in with the product

5. Set the config file

   $ export AMYGDALA_CONFIG=/dls_sw/ixx/etc/gda_launcher.toml
   

   This could be to a local file instead but as nothing else will be reading the live file it could be in the 'real' location for testing.

6. Enable debug logging (optional but useful for testing)

   $ export RUST_LOG=amygdala=debug
   

   There is also trace logging that can be enabled but this is noisier and doesn't add much if you're not developing the daemon itself. If there is information missing from the logs that would be useful, add a comment in slack.

7. Start the daemon

   $ amygdala serve
   

This will run the launcher process in the foreground with logs written to stdout. The initial lines should include the version and build of the launcher being used. It'd be useful to include it if there are any issues as the launcher is still being developed.

The daemon should be left running while running tests. It can be shutdown via Ctrl-C if it needs to be restarted (eg to reload config). It will cleanly shutdown any GDA server that is running first.

Testing the client commands

From a beamline workstation

1. Close any open clients

2. Shutdown any previous servers

   The new launcher won't interfere with any old servers but it will give an error when you try and start the server saying there is already a server running. It will not shutdown servers that it didn't start.

3. Load the new launcher

   $ module load gda_launcher
   

4. Check it is connecting

   $ gda server status
   

   You should get a message that the server is not running

5. Test any gda commands your beamline is likely to run. A list of commands/subcommands is available from the cli via either the help command

   $ gda help server
   

   or by passing --help to any command

   $ gda server start --help
   

   While the CLI help will include all available commands, the documentation is not yet very expansive. Most commands should be self descriptive, any that aren't could probably be named better - feedback is welcome.

Returning the Beamline to pre-9.33 deployment

Nothing above should affect the existing beamline configuration so shutting down the launcher and logging out should be enough to get the beamline back to its initial state.

A new terminal will be needed when restarting the old servers to ensure the new command isn't on the $PATH (or use module unload and hope?).

Next Steps

See Managing the Transition for changes that need to be made that may need to be reverted after testing.

Managing the transition

For the period while we have supported versions of GDA on both sides of the configuration changes (until 9.35 is released), we need to able to easily support switching between versions.

Most of the suggestions below rely on the suggested layout in the deployment root section - specifically keeping the current gda symlink and pointing it to the new gda_versions/gda symlink when running with the 9.33+ deployments.

Diamond Launcher

The diamond launcher applications are configured in three places

• Core applications (beamline synoptics etc) are configured centrally with the launcher. None of these relate to GDA and can be ignored.
• User home directories. These will need to be changed by individual users but will mainly follow the same steps as below.
• Beamline specific applications only included on beamline machines. This is where GDA is usually configured and needs updating

The applications are defined in json format in /dls_sw/ixx/etc/diamond_launcher_applications. Any json file in that directory will be included and all entries merged into the entries on the launcher.

How much needs to change depends on what beamlines are doing in their launchers.

The simple cases

Any launcher entry that consists only of gda servers, gda client or gda logpanel (after merging the command and args sections of the config), can be left as they are. However this does not apply if there are additional arguments e.g. gda --start servers, if so see below.

Assuming the path is modified (see below), to include the /dls_sw/ixx/software/gda/config/bin directory and the new launcher in that order, if your currently active deployment has a gda script (and is therefore pre 9.33), it will be called. If there is no gda script, the new launcher will be used. For the basic cases where the commands are unchanged, the application entries will work with either.

This also applies if the DLS launcher uses gdaserver or gdaclient scripts (or any other non-gda script). These will be in the config so will always be correct for the deployment they are in. Any of these non-standard scripts will need updating as part of the config change.

Any beamlines using the gda_conf.py script for selecting spring profiles should also be ok as the script will be updated for the new release and whichever deployment is active will contain the correct version.

The less simple cases

If the application entries use versions of the gda command that have changed, it is more difficult to keep them correct when changing between pre and post 9.33 deployments. The current suggestions are below but any alternatives are welcome.

Wrap the commands in a new script in config/bin

Replace a command that runs (eg) gda --restart servers --springprofiles='one,two' with a gda_one_two script in the beamline config that calls the same command in pre-9.33 deployments and the new gda server restart -P -p one -p two command in newer configs. For example there would be a new gda_one_two script in config/bin/ for all supported GDA deployments, including GDA 9.33, and the launcher entry would be updated to call this.

Switch the application.json file

Remove all GDA commands from the json file in /dls_sw/ixx/etc/diamond_launcher_applications and add a new symlink version to a json file in the config. This config version would be up to date with the currently active deployment and the DLS launcher (is supposed to be) automatically refreshed every so often to keep launchers up to date. This needs testing to ensure the update frequency is regular enough for beamlines not to get left with out of sync launchers.

Note that some beamlines already have the launcher files configured this way. This can be checked with:

ls -l /dls_sw/iXX/etc/dls-launcher-applications.json
ls -l /dls_sw/iXX/etc/dls-launcher-applications/

and examining from the output whether the json files are symlinks or files.

Add multiple entries

Add new entries to the launcher that work with the 9.33+ versions and have the option to launch either.

This is may be the simplest of the non-simple options but could easily result in the wrong server being started by mistake.

Removing old launcher XML files

This isn't related to the new launcher changes however is mentioned here for completeness. Several years ago the launcher configuration was migrated to use json instead of xml. When this was carried out, the existing xml files were automatically converted but the xml files were never removed. These xml files can now be deleted to avoid potential confusion in the future.

Example launcher entries for common use cases

These examples are currently hardcoding the path to a particular version of the GDA launcher. If the PATH is setup correctly (see below) this should not be required.

Starting the server

The easiest approach is to start it via the GUI:

{
    "type": "button",
    "text": "Start GDA Server",
    "args": " ",
    "command": "/dls_sw/apps/gda_launcher/stable/gda-gui",
    "icon": "sgda.png"
}

Otherwise, at the moment at lest, the command must open a terminal so that the progress blocking start command can be monitored:

{
    "type": "button",
    "text": "Start GDA Server",
    "args": "-- bash -c '/dls_sw/apps/gda_launcher/stable/gda server restart; exec bash'",
    "command": "gnome-terminal",
    "icon": "sgda.png"
}

Starting the client

The launcher requires that processes are run in the foreground.

{
    "type": "button",
    "text": "Start GDA Client",
    "args": "client --foreground",
    "command": "/dls_sw/apps/gda_launcher/stable/gda",
    "icon": "cgda.png"
}

Starting the logpanel

Using gda logpanel works from the terminal but not from the Diamond Launcher as it requires the process to be launched in the foreground. The logpanel sub script can be invoked instead:

{
    "type": "button",
    "text": "Start GDA Log Panel",
    "args": " ",
    "command": "/dls_sw/apps/gda_launcher/stable/gda-logpanel",
    "icon": "lgda.png"
}

PATH variable

The new launcher command was named gda to minimise the disruption converting from the previous scripts to the new version. This does have the temporary downside during the time where both versions are required of increasing the likelihood of the wrong command being used.

The current suggestion (again, alternatives welcome) is to add the new launcher directory to the $PATH variable after the current config/bin directory. This means that if the old gda script exists (because a pre 9.33 deployment is active) it will be used in preference to the new one but if it doesn't the new launcher will already be on the path as a fallback.

The $PATH is set up in /dls_sw/ixx/etc/ixx_profile.sh, either directly or via other sourced files. As this script is sourced once when the user logs in and subsequent changes are ignored, it needs to be able to handle both configuration versions.

Adding module load gda_launcher before prepending the config/bin directory should have the right effect but this is something to be tested on each beamline as many different approaches are used regarding sourcing external files.

The Amygdala Daemon

The daemon running on the server can be left running while the active deployment is reverted to a pre-9.33 deployment. It won't work to launch GDA but won't interfere with the existing bash scripts.

Trying to launch a 9.33 deployment while a pre-9.33 server is running will result in an error along the lines of 'Another instance of the server is already running', whereas trying to run a pre-9.33 server while the 9.33 server is running will probably work correctly but may cause an error or warning from the amygdala daemon the next time it is used as its instance of the server will have been shut down by an external process. It should recover from this error and return to normal operation on subsequent launches.

The server status and client-details commands will not be correct when a pre-9.33 server is running.

Migration from previous scripts

While the interface of the previous gda command has been kept where possible for the main uses, there are differences to make it more consistent and predictable.

While these are the currently implemented commands, they are not fixed indefinitely and if there is a valid reason why alternative syntax would be more useful, it may be possible to accommodate it.

Starting and Stopping the server

Most of the time the servers are not stopped manually. Instead, gda servers is used to restart the server, shutting down the server if it is already running. This behaviour is unchanged.

If a user wishes to stop the server but not restart it, or start it only if there is not a server already running, the command has changed

Motivation

By convention command options reflect the command they follow, given the --start, --stop, --restart options do not apply to the other subcommands, it did not make sense for them to be at the top level of the gda command.

The new gda command also adds additional server commands where it did not make sense for start or stop to be options (eg status).

Setting Spring profiles

Spring profiles are used to enable or disable specific aspects of the server or client. There can be a configurable set of profiles enabled by default. With the new launcher, it has been easier to set and clear these profiles.

Motivation

Being able to add to the default profiles without overriding them was a big motivation for this change. The previous command reserved -p for GDA profiles, a feature that has not been used for several versions. Repurposing it for spring profiles makes it much more convenient, while allowing repeated use to specify multiple better follows conventions set by other commands. There is a more verbose --profile option that behaves the same way if it's being used in scripts where being more descriptive is beneficial.

Setting client workspace directory

The previous command had a --workspace option to specify an alternative location for the client workspace directory. This allowed multiple clients to be started on the same machine without clashing. This has been replaced with a --tag option that creates the workspace in the same location but appends a user provided tag to the directory name. This still allows multiple clients to be run and makes it clear what each workspace was used for but prevents the user having to provide a full path to a suitable workspace location.

Motivation

This makes it simpler to use an alternative workspace while keeping all workspaces in a single place for consistency making it easier to go back to a previous workspace without having to know to specific directory.

Setting custom system properties

The previous scripts used GDA_CLIENT_VMARGS and GDA_SERVER_VMARGS variables and passed them as they were to the relevant command. These were generally set in the custom bash scripts for a beamline.

The new launcher has two approaches for custom system properties. For those that should be set every time, there is the system section of the launcher configuration (see configuration section for details), and there is a -D option (to mirror the same option when calling java).

Previous commands

$ export GDA_SERVER_VMARGS="${GDA_SERVER_VMARGS} -Dnew.property=value"
$ gda server

New command

$ gda server start -Dnew.property=value

Motivation

It makes it much easier to set system properties on the fly.

Debugging the server or client

The --debug flag to the server is still present but is now also required to be after the component to start, ie gda server --debug and gda client --debug. The same is true for the equivalent --debug-wait flag.

There is now also a --debug-port option that allows the debug port to specified instead of defaulting to 8000. This option requires either --debug or --debug-wait to be present.

Motivation

Similar to the start and stop commands, it is more predictable for the flags and options to appear after the command they relate to.

gdaclient and gdaservers commands

These changes refer only the gda command provided on all beamlines. If beamlines use gdaclient or gdaserver (or similar) scripts, these are not changed although if they rely on the gda script they may need to be modified if they use any of the features listed above.

Extending the client

While they were often difficult to work with, the previous scripts did allow for arbitrary commands to be run and for the gda command to be modified as required on a per beamline basis.

While some of this ability is lost by making the command into a built executable, it retains some customisation via extension commands. In a similar way to other tools (such as git and cargo), if gda is called with a subcommand xyz that it doesn't recognise, it looks in the user's $PATH for a command named gda-xyz and passes all arguments to that command instead. This allows gda to provide a unified interface to any number of beamline specific scripts.

$ gda unknown one two -f three
$ # is equivalent to
$ gda-unknown one two -f three

When gda is run with the --help flag, it will look at the user's $PATH and list any matching commands that it finds and list them along with the built in subcommands (along with their paths and instructions on how to add new commands).

$ gda --help
...
Extending gda
    Additional subcommands can be provided by naming external commands
    gda-* and ensuring they are executable and on your $PATH

    The currently available commands are:
    usercommand: /home/user/.local/bin/gda-usercommand

To ensure that gda can find your extension command, make sure that it is executable and that it is in one of the directories listed in your $PATH variable.

Amygdala Server Configuration

The configuration for the Amygdala launcher is split into two sections.

The Launcher Configuration

This section configures the behaviour of the launcher itself and how it runs, for instance the port that clients should connect to and the logging levels and destinations.

The majority of these options are read from the CLI arguments given when starting the launcher and are described in the CLI reference section.

The GDA Client/Server Configuration

This section configures the way GDA processes are started and managed by GDA, for instance, this includes the location of the GDA deployments and the default deployment to run when one is not specified as well as hooks to run before and after starting/stopping the server.

This configuration is read from a TOML file described in the Amygdala Config Section

Amygdala CLI

The amygdala command currently only has a single sub-command (other than help for displaying usage instructions and completions for shell support), but is intended to prevent breaking changes being required when additional commands are added (for instance if an interactive config file generator is added).

$ amygdala --help
Usage: amygdala <COMMAND>

Commands:
  serve        Start the launcher process, listening for commands via gRPC
  completions  Generate shell completions for amygdala
  help         Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

Serve

The serve subcommand starts the launcher in the foreground listening for incoming requests from the gda client.

It is possible to run amygdala with no further options or arguments if the defaults are acceptable, but the following configuration is available.

$ amygdala serve --help
Start the launcher process, listening for commands via gRPC

Usage: amygdala serve [OPTIONS]

Options:
  -c, --config <CONFIG>
          Configuration file for the GDA launcher
          
          If not present the config file will be read from the first defined of
          the following:
          * $AMYGDALA_CONFIG
          * $XDG_CONFIG_HOME/amygdala/config.toml
          * $HOME/.config/amygdala/config.toml

  -p, --port <PORT>
          Port opened for gRPC requests
          
          [default: 50051]

  -h, --help
          Print help (see a summary with '-h')

Terminal Logging:
  -v, --verbose...
          Increase the level of logs written to stderr

  -q, --quiet
          Disable all output to stderr/stdout

File Logging:
  -l, --log-directory <LOG_DIRECTORY>
          Parent directory of log files written by the amygdala process (not
          those written by the GDA processes it launches)
          
          If not present, logging will not be written to file.

  -L, --level <LEVEL>
          Level threshold of logs that should be sent to file

Graylog:
      --graylog <FILE>
          Configuration file containing graylog host and port if not passed
          individually

      --graylog-host <GRAYLOG_HOST>
          Hostname of graylog server

      --graylog-port <GRAYLOG_PORT>
          Port of graylog server

General options

--config/-c

$ amygdala serve --config path/to/config.toml

The file used to configure how the GDA processes are started and managed can be set via the --config/-c option. If this is not given, the file is read from the first of the following to be defined.

• $AMYGDALA_CONFIG environment variable.
• $XDG_CONFIG_HOME/amygdala/config.toml
• $HOME/.config/amygdala/config.toml

Note that if one of these is defined (the environment variable is set), the subsequent files are not checked, even if the file is not found.

Port

$ amygdala serve --port 9876

The port which should be opened for the gRPC end points and to which the gda client should connect.

This defaults to the same 50051 that the client defaults to so both should be changed if either of them are.

Logging

By default, minimal logging (info level and above) is written to stdout, no log files are written and no messages are sent to graylog.

Terminal Logging

The level of logging that is written to stdout can be adjusted using -v/--verbose flags. These can be repeated to increase logging to debug and then to trace. Repeating further has no effect.

If no logs are required, -q/--quiet can be used to remove all terminal output.

File Logging

-l/--log-directory

If a log directory is given, the logs will be written to file named gda_launcher.yyyy-mm-dd.log in that directory. A new date-stamped file is created each day and will be kept for 2 weeks.

$ amygdala serve --log-directory /path/to/logs

-L/--level

The logs written to file can be filtered by level. This should be one of the keywords ERROR, WARN, INFO, DEBUG or TRACE (case insensitive)

If a log file level is set, a log directory is also required (as the level does not make sense is no logs are being written).

By default, logs at debug level and above are written to file (if a log directory is set).

$ amygdala serve -L TRACE # --log-directory also required if setting level

Graylog

Log events can also be sent to a central Graylog instance. For this a host name and port are required. The graylog server should be configured to accept TCP connections (instead of UDP). The host name should not include any schema or protocol information, eg graylog.example.com instead of tcp://graylog.example.com or similar.

There are two ways of passing graylog configuration.

External File

If the host and port are written in TOML format to a file, the path to that file can be passed in. This is useful if multiple instances of amygdala should share a common configuration to allow it to be updated in a single place if required.

This file can be passed in using the --graylog option.

$ amygdala serve --graylog /path/to/graylog.toml

The file should have the format

host = "graylog.example.com"
port = 12345

Individual Fields

The host and port can also be passed directly. In this case both are required.

$ amygdala serve --graylog-host graylog.example.com --graylog-port 12345

It is not possible to use these field to override a value from an external graylog config file.

Amygdala Configuration File

The majority of the configuration is optional and can be omitted. For a list of the required fields, see the migration guide.

TL;DR

• Required Server Fields
  • deployment_root
  • default_deployment
  • server_runtime_root
• Required Client Fields
  • client_runtime_root

Lazy Loading

The configuration file is read once when it is first required. All subsequent uses will use the cached version of the config and changes to the file will require a restart of the launcher. If the process is being managed by systemd, it is possible to request a restart from the client.

If it is not possible to read the config file for any reason, the request will fail and the client will be sent details of the error. In this case, subsequent requests will attempt to read the config again and changes will be reflected if they haven't already been loaded.

Full Config Example

The majority of these fields are optional

subnet = "12.34.56.0/23"

deployment_root = "/path/to/deployment_root"
default_deployment = "version_1"
shutdown_timeout = 13
status_port = 23456
startup_file_directory = "/tmp/startup_files"
log_directory = "/tmp/amygdala_logs"
client_runtime_root = "/tmp/gda_client"
server_runtime_root = "/tmp/gda_server"

[layout]
client = "client/client-bin"
server = "server/server-bin"
config = "config"

[env]
COMMON_FOO = "BAR"
[system]
"common.foo" = "bar"

[server.env]
SERVER_FOO = "BAR"
[server.system]
"server.foo" = "bar"

[client.env]
CLIENT_FOO = "BAR"
[client.system]
"client.foo" = "bar"

[[hooks.pre_start]]
command = "pre_start_command"
args = ["one", "two"]
env = {"VARIABLE": "VALUE"}

[[hooks.post_start]]
command = "post_start"

Launcher specific configuration

These items control the way the launcher daemon is run and doesn't affect the server or clients it starts.

All launcher fields are optional

Optional Fields

Port

amygdala serve --port 8765

port = 9876

Subnet

subnet = "12.34.56.00/21"

This restricts the hosts from which the client can connect. It is intended to restrict clients to the beamline network so that the server for a beamline cannot be started unless you are on the beamline. This is intended mainly to prevent the wrong server being started more than as a security feature as it is easy to circumvent by SSHing to a beamline workstation.

There is no default subnet and clients will be able to connect from any machine that can reach the control machine. No default

Server configuration

Required Fields

Deployment Root

deployment_root = "/dls_sw/ixx/software/gda_version/"

The parent directory of all the GDA deployments for a beamline. For Diamond, this is will almost always be the gda_versions directory.

Default Deployment

default_deployment = "gda"

The directory within the deployment root that should be used by default if an alternative deployment is not specified. This is currently required but may become optional in future if gda becomes the de facto default in practice.

Optional Fields

Shutdown Timeout

shutdown_timeout = 10

The time in seconds when shutting down a server between sending SIGINT interrupt and assuming it has crashed and sending a SIGKILL. Default: 10s

Status Port

status_port = 19999

This is the port used to determine when the server has reached a 'ready' state. It is also used to determine if a server is running that was started by an external process (eg the previous script launcher or the IDE if developing locally). Default: 19999

Startup File Directory

startup_file_directory = "/tmp/"

The directory where GDA should write any errors encountered during start up. This is used to relay the top level error to the client without having to read through the main logs. The individual files only exist for the startup process and are immediately removed on either failure or success. Default: $XDG_RUNTIME_DIR (or /tmp if undefined)

Log Directory

log_directory = "/dls_sw/ixx/logs"

The parent directory of the gda-servers-output and gda-client-output. It has no effect on the directory where GDA writes its main log files. Note that the output subdirectories are not created and must already exist. Default: /tmp/

Server Runtime Root

server_runtime_root = "gda_launcher"

The directory where the server workspace and config directories should be created. In the previous scripts this would be in gda2's home directory. A relative path here will be relative to the working directory when the daemon is started. Default: /tmp/

Layout

[layout]
client = "client/gda-ixx"
config = "ixx-config"
server = "server/gda-server"

This describes the layout of each deployment. It is optional and individual fields can be overridden if needed (it's not all or nothing). For most beamlines, the defaults will be ok. Relative paths are resolved against the deployment being used. Absolute paths are used as they are but aren't recommended as changing deployments will have no effect.

• Default server: server/gda-server
• Default client: client/gda-$BEAMLINE (or gda-example if $BEAMLINE is not set).
• Default config: config

System Properties and Environment Variables

[server.system]
"gda.system.property" = "value"
"other.system.property" = "other value"
[server.env]
"SERVER_ENV" = "value"

Any system properties that should be passed to the server or environment variables that should be set for the server process. This is where defaults specific to this installation should go (as opposed to the config.toml file in the beamline configuration). No defaults

Hooks

It is possible to specify external programs that should be run on certain events in the life cycle of the GDA server. The four currently available extension points are before and after the GDA server starts and shuts down. There can be multiple hooks at each point in which case they will run sequentially in the order they are defined in the configuration.

For each hook, a path to the command is required. This can either be an absolute path to a command or a command name on the $PATH of the user running the daemon. There can also, optionally, be a list of arguments to pass to the command as well as any environment variables that should be set.

The four hook names are

• pre_start - If these exit with a non-zero exit code, the launch is aborted.
• post_start - These are not run if the server fails to start. If these fail to run after an otherwise successful launch, the error is reported to the user but the server is left running.
• pre_shutdown - This includes before shutting down the server when the user has requested a restart. If these fail, the shutdown is aborted (and therefore also the subsequent start for a restart).
• post_shutdown - These don't currently block the server starting if they fail during a restart to maintain consistency with the case where there is nothing that can be done if a post_shutdown hook fails when calling stop.

Each hook should be defined in the same way with the appropriate hook name used.

[[hooks.pre_start]]
command = "/path/to/command"
args = ["one", "two"]
env = { "VAR_ONE": "VALUE_ONE", "VAR_TWO": "VALUE_TWO" }

Hooks can be specified in any order but should all be together at the end of the config file due to TOML restrictions. No defaults

Client Configuration

Required Fields

Client Runtime Root

client_runtime_root = "/tmp/"

The parent directory of the client workspace and config directories. The actual directories used will be in a tree below this directory based on the user name of the user starting the client and the deployment and build of the client being run.

Optional Fields

System Properties and Environment Variables

[client.system]
"gda.system.property" = "value"
"other.system.property" = "other value"
[client.env]
"CLIENT_ENV" = "value"

The client equivalent of the server fields. No defaults

Common Configuration

Optional Fields

System Properties and Environment Variables

[system]
"common.system.property" = "common value"
[env]
GDA_VAR = "/path/to/var/directory"
GDA_MODE = "live"

Any system properties or environment variables that are common to both the server and the client. This is probably the best place for GDA_VAR and GDA_MODE to be defined. No defaults

GDA CLI Client

The gda client command allows users to manage GDA servers and clients, handling communication to start servers on remote machines.

The client is divided into serveral main subcommands. These can be listed using gda --help

$ gda --help
GDA launcher

Usage: gda [OPTIONS] <COMMAND>

Commands:
  server          Start, stop or check the status of the GDA server
  client          Start the client
  client-details  List the configuration that would be used to start the client
  launcher        Interact with the launcher process itself
  completions     Generate shell autocompletion - see help for details
  doctor          Print full diagnostics info for debugging (Unstable)
  help            Print this message or the help of the given subcommand(s)

Options:
  -H, --host <HOST>
          Host running the server process

  -p, --port <PORT>
          Port of the launcher service

  -G, --no-gui
          Prevent GUI pop-up being shown even if display is present
          
          This can also be made default locally by setting the `AMYGDALA_NO_GUI` environment
          variable (to anything - the value is ignored).

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Extending gda command
    Additional subcommands can be provided by naming external commands
    gda-* and ensuring they are executable and on your $PATH

Common Options

These options affect how the client runs and should be passed before any subcommand.

-H/--host

$ gda --host ixx-control

Set the address of the machine where amygdala is running. This defaults to $BEAMLINE-control is $BEAMLINE is set and localhost otherwise.

(NB, -H is used instead of -h to avoid clash with help)

-p/--port

$ gda --port 12345

Set the port on which the amygdala server is running. This defaults to 50051 if not set.

-G/--no-gui

$ gda --no-gui

Force the gda command to use the terminal for feedback instead of using the built-in GUIs. By default, the GUI is used if $DISPLAY is set and the terminal UI is used as a fallback if not, for instance if run remotely via SSH.

As the terminal UI does not show a confirmation prompt, using this option in a situation where there is no terminal available (via launcher menu or desktop entry etc) will result in the command running as normal but with no feedback given to the user.

If the GUI is never required for a user, setting $AMYGDALA_NO_GUI (to anything, the value is ignored) will be equivalent to passing --no-gui to every command.

-h/--help

$ gda --help

Show the help text (as shown above).

Subcommmands

• Server Interact with the GDA server managed by the amygdala daemon.

• Client Start the GDA client on the local machine

• Client Details Show the configuration that would be used to launch the client

• Launcher Manage the amygdala process itself

• Completions Generate shell completion functions to help interactions with gda command.

Help

Help is available at all levels of the gda command. Passing --help to any other command should generally provide help for that command.

If any help provided by the built-in help text is unclear or incorrect, this should be considered a bug. Please consider opening an issue in the bug tracker.

Server Subcommand

The server command deals with all actions interacting with the GDA server running on the machine where amygdala is running. It is further divided into subcommands.

Help is available via either gda help server or gda server --help.

$ gda server --help
Start, stop or check the status of the GDA server

Usage: gda server [COMMAND]

Commands:
  start    Start the GDA server - will not shutdown an existing server
  restart  Start the GDA server shutting down an existing server if running
  stop     Shutdown the current server if one is running
  status   Check the status of the server if one is running
  help     Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

start

Start the server on the machine running the amygdala daemon (set via host above). The options passed here are combined with any defaults and passed to the server process.

$ gda server start --help
Start the GDA server - will not shutdown an existing server

Usage: gda server start [OPTIONS]

Options:
  -h, --help  Print help

GDA Options:
  -c, --config <CONFIG>
          Set the path to the config directory
  -m, --mode <MODE>
          Set the `gda.mode` property
  -p, --profile <PROFILE>
          Add a spring profile to enable when starting the server
  -P, --no-default-profiles
          Prevent default profiles being used - use only the given profiles
  -x, --xml <path/to/file.xml>
          Add a spring beans XML file to be included when starting the server
      --no-default-xml
          Do not include any default spring XML - use only those given here
  -k, --properties <path/to/file.properties>
          Add a properties file to be loaded by the started server
      --no-default-properties
          Do not load properties from any of the default files
  -l, --logging <path/to/logging.xml>
          Add a logging configuration file to be loaded by the server at start-up
      --no-default-logging
          Do not include logging configuration from any of the default locations
  -D <property=value>
          Pass arbitrary JVM args to the process
  -d, --deployment <DEPLOYMENT>
          The name of the deployment to use if different from default

Debugging:
      --debug                    Enable debugging but don't wait for the debugger to attach
      --debug-wait               Enable debugging and wait for debugger to attach before starting
      --debug-port <DEBUG_PORT>  The port to open to enable debugging

stop

Shutdown the GDA server on the remote machine if one is running.

$ gda server stop --help
Shutdown the current server if one is running

Usage: gda server stop [OPTIONS]

Options:
  -t, --shutdown-timeout <TIMEOUT>  Time to wait for server to shutdown before terminating process
  -h, --help                        Print help

If no server is running, this command has no effect.

restart

Equivalent to calling stop followed by start. The options available are the combination of both previous commands.

$ gda server restart --help
Start the GDA server shutting down an existing server if running

Usage: gda server restart [OPTIONS]

Options:
  -h, --help  Print help

GDA Options:
  -c, --config <CONFIG>
          Set the path to the config directory
  -m, --mode <MODE>
          Set the `gda.mode` property
  -p, --profile <PROFILE>
          Add a spring profile to enable when starting the server
  -P, --no-default-profiles
          Prevent default profiles being used - use only the given profiles
  -x, --xml <path/to/file.xml>
          Add a spring beans XML file to be included when starting the server
      --no-default-xml
          Do not include any default spring XML - use only those given here
  -k, --properties <path/to/file.properties>
          Add a properties file to be loaded by the started server
      --no-default-properties
          Do not load properties from any of the default files
  -l, --logging <path/to/logging.xml>
          Add a logging configuration file to be loaded by the server at start-up
      --no-default-logging
          Do not include logging configuration from any of the default locations
  -D <property=value>
          Pass arbitrary JVM args to the process
  -d, --deployment <DEPLOYMENT>
          The name of the deployment to use if different from default

Debugging:
      --debug                    Enable debugging but don't wait for the debugger to attach
      --debug-wait               Enable debugging and wait for debugger to attach before starting
      --debug-port <DEBUG_PORT>  The port to open to enable debugging

Shutdown Options:
  -t, --shutdown-timeout <TIMEOUT>  Time to wait for server to shutdown before terminating process

status

Return the status of the server managed by amygdala.

This command takes no options although --help is available for a description of what it does.

$ gda server status --help
Check the status of the server if one is running

Usage: gda server status

Options:
  -h, --help  Print help

The output of this command is intended to be human-readable and should not be relied on for scripting. See extending the client for details of how to access status programmatically.

The returned status is either that the server is not running, or the following fields:

• Server Executable The full, absolute path to the server executable that is running. This can be useful to determine if the server has been restarted since a rebuild, or links being updated.
• Start Time The exact time that the current server process was launched.
• Uptime Approximate running time of the server. The accuracy is not intended to be more than a rough guide (the start time can be used for that if required), and will round to nearest sensible unit eg 3 minutes or 7 days 12 hours.
• Options The options provided by the user at the command line when the server was started. These options do not include any defaults that may be included via either the amygdala config or the beamline config.

Client Subcommand

The client subcommand starts the GDA RCP client on the user's local machine. It uses information from the amygdala server it connects to to determine which executable should be run as well as the startup information such as runtime and logging directories.

$ gda client --help
Start the client

Usage: gda client [OPTIONS]

Options:
  -h, --help
          Print help (see a summary with '-h')

Launcher Options:
      --console
          Start the client in the foreground
          
          [aliases: cs]

  -f, --foreground
          Start the client in the foreground with stdout/stderr redirected to file
          
          [aliases: fg]

      --background
          Start the clieng in the background with stdout/stderr redirected to file and process
          detached from terminal session
          
          [aliases: bg]

      --reset
          Delete user config/workspace directories - can help fix corrupt sessions

      --keep
          Rename previous config/workspace directories rather than deleting them when running
          `--reset`

      --force
          Start client even if the server is not running
          
          Can be useful if the server is started manually or via the IDE instead of using the
          launcher

      --tag <TAG>
          Tag the client config/workspace directories to allow multiple clients to be run by the
          same user on the same machine - must be alphanumeric (or '_')

Client Options:
      --perspective <PERSPECTIVE>
          ID of the perspective to open when the client is started

      --plugin-customisation <CUSTOMISATION>
          Path to a plugin customisation file to set client preferences

  -c, --config <CONFIG>
          Set the path to the config directory

  -m, --mode <MODE>
          Set the `gda.mode` property

  -p, --profile <PROFILE>
          Add a spring profile to enable when starting the server

  -P, --no-default-profiles
          Prevent default profiles being used - use only the given profiles

  -x, --xml <path/to/file.xml>
          Add a spring beans XML file to be included when starting the server

      --no-default-xml
          Do not include any default spring XML - use only those given here

  -k, --properties <path/to/file.properties>
          Add a properties file to be loaded by the started server

      --no-default-properties
          Do not load properties from any of the default files

  -l, --logging <path/to/logging.xml>
          Add a logging configuration file to be loaded by the server at start-up

      --no-default-logging
          Do not include logging configuration from any of the default locations

  -D <property=value>
          Pass arbitrary JVM args to the process

Debugging:
      --debug
          Enable debugging but don't wait for the debugger to attach

      --debug-wait
          Enable debugging and wait for debugger to attach before starting

      --debug-port <DEBUG_PORT>
          The port to open to enable debugging

Many of the options are equivalent to those available for starting the server. These are passed to the application. In addition, there are also options to configure how the launcher handles the setup for the process.

Client Launcher Options

Foreground/Background/Console

These are mutually exclusive and control how the process is handled.

• --foreground. This is the default and causes the client to run in the foreground in the terminal where it is launched. The stdout/stderr of the process are written to files in the configured logging directory. These files are displayed to the user for reference.

  If the terminal is closed, the client will be killed.

• --console. This is similar to --foreground in that the process runs in the foreground and is tied to the life of the terminal sessions. The difference is that stdout/stderr are printed to the terminal instead of being redirected to file. This is most useful for debugging. If used from a situation where there is no terminal available (eg a DLS launcher), there will be no logs available.

• --background. This disconnects the client process from the terminal where it is launched. Logs are written to file and the client is started in a new process group so that closing the terminal sessions does not affect it. This is roughly equivalent to running $ gda client & disown without passing --background.

Runtime directory tagging

$ gda client --tag <local_name>

The GDA client uses a workspace and config directory to store user data at runtime. These directories are locked by the client and cannot be used by two instances of the application at once and as the directories are chosen automatically by a combination of username, GDA deployment and client build, this prevents a user having two clients open. To support this where it is required, the gda command supports client tagging. This appends a user supplied string to the auto-generated runtime directory names allowing them to open multiple clients. As the runtime directories are also used to store user preferences and client state (perspectives/layouts etc), this can also be used to support having multiple configurations of GDA that can be launched as required, even if only one is ever going to be used at any one time.

Resetting runtime directories

To clear any user configuration and return the client to default settings, it can be useful to use new runtime config and workspace directories. This can be done via the --reset option which will delete the directories if they exist before reusing the same path for the new client.

$ gda client --reset

If the previous state/configuration should not be deleted (eg to allow later debugging of an issue), the --keep option can be passed (as well as --reset) to move the existing directories to a timestamped backup instead of deleting them.

$ gda client --reset --keep

Note that the time used for the timestamp is the time the directories are replaced, not the time they were originally created.

Starting the client without the server

By default, the client will not start if the server is not running. This allows it to fail fast in situations where the client is likely to encounter errors later in the startup process and with a concise error message. However, there may be occasions where amygdala is not aware of the server running, for instance if the server has been started from an IDE. In these cases, the --force option can be used to skip the check. As amygdala is not aware of which server is running, it is not possible to ensure that the correct client is started so additional errors may be encountered for instance if the client and server versions don't match. The client from the default deployment will be used in this case.

Client Detail Subcommand

$ gda client-details

This command is intended for debugging. It uses the currently running server if there is one, as well as the amygdala configuration to list the paths that would be used to start the client.

The output of this command is intended to be human readable and should not be relied on for scripting. It may change without notice.

The details provided include

• running - whether or not the GDA server is running. This will may return a false negative if a server has been started via an alternative method that amygdala is not aware of, for instance via an IDE.
• executable - the client executable that will be started. This is the client in the deployment of the server if one is currently running.
• config - the absolute path of the GDA configuration that will be passed to the client when it is started. It will be overridden if the user provides an alternative config via the --config option.
• logs - the directory where the stdout/stderr logs will be written if the client is started without the --console option. The exact paths are reported to the user when the client starts.
• runtime - the parent directory of the config and workspace directory trees. The actual directories used will be dependent on the exact client build when the client is started.
• deployment - the root of the deployment of the server if one is running, and the default deployment otherwise.

$ gda client-details --help
List the configuration that would be used to start the client

Usage: gda client-details

Options:
  -h, --help  Print help

Doctor subcommand

The 'doctor' subcommand provides an easy way to gather many relevant details required to help diagnose issues with gda and amygdala.

The output should not contain any sensitive information but as there is no automated submission, all details can be checked before they are added to tickets or similar.

The details gathered will be similar to those below:

Local Machine:
    OS: Red Hat Enterprise Linux 8.9 (Ootpa)
    Arch: x86_64
    Desktop: Unknown: Unknown
    Hostname: ws419.diamond.ac.uk
    User: qan22331
    SSH from: 172.23.122.70 (i22-ws001.diamond.ac.uk)
          to: 172.23.240.178 (ws419.diamond.ac.uk)
    $DISPLAY not set
    $BEAMLINE not set
    $JDK_OPTIONS not set

Client (gda):
    Version: 0.3.0-dev
    Built: Thu, 30 May 2024 08:49:55 +0000
    Commit: 43e5c34e4a0d6611ba6c880fa6490892134dbb69 (+unstaged changes)
    Executable: "/scratch/dev/amygdala/target/debug/gda"
    arg[0]: "target/debug/gda"
    Available extensions:
        gui: "/dls_sw/apps/gda/latest/gda-gui"
        logpanel: "/dls_sw/apps/gda/latest/gda-logpanel"

Configuration:
    Host: localhost
    Port: 50051

Amygdala:
    Version: 0.3.0-dev
    Commit: 489931482d6463716fafd20d9a8d2295996af086
    Executable: "/scratch/dev/amygdala/target/debug/amygdala"
    Start time: 2024-05-29 16:28:25 UTC
    Current Server:
        Server running: false
        Defaults:
            Deployment: "/scratch/sample_deployments/gda_versions/master"
            Config: "/scratch/sample_deployments/gda_versions/master/workspace_git/gda-diamond.git/configurations/i22-config"
            Client Runtime: "/scratch/sample_deployments/client_root"

More fields may be added in future versions and the layout may change. It is intended to be human readable and should not be used for scripting.

When opening an issue, please include the output of gda doctor.

$ gda doctor --help
Print full diagnostics info for debugging (Unstable)

Usage: gda doctor

Options:
  -h, --help  Print help

Launcher Subcommand

$ gda launcher

The launcher subcommand provides control of the amygdala process itself rather than any of the processes it manages. It is further split into several subcommands.

Help is available via either gda launcher --help or gda help launcher.

$ gda launcher --help
Interact with the launcher process itself

Usage: gda launcher <COMMAND>

Commands:
  interrupt  Force the launcher to cancel any commands currently in progress
  restart    Request that the launcher be restarted
  version    Request version information of the running launcher
  help       Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

interrupt

As the amygdala daemon is managing a single instance of the server process, it does not make sense to handle multiple server requests at once. For this reason if one request is in process, any other requests will return an error saying the server is busy.

There are situations where this can cause issues, for instance if a server does not start correctly but appears to hang instead of exiting with an error, the start request will not return. If the user that requested it does not cancel the request, all other users will locked out of the server.

The interrupt command offers other clients the chance to cancel pending requests from other clients. Any process being started when the interrupt is called will be interrupted and the client that requested it will receive an error saying the server was interrupted.

restart

If the amygdala config is updated, the process needs to be restarted for it to be re-read. To make this easier for remote users, the restart command can be used to shutdown the existing process and request that it be restarted.

version

Display version information of the amygdala process. The output is intended to be human readable and should not be used for scripts. It may change without notice.

• executable - the absolute path the executable that is running. This can be useful if the daemon was started via a symlink that has since been updated.
• version - the version string of the amygdala daemon
• commit - the exact git commit from which the executable was built. Useful for pre-release versions where multiple different executables may be listed as having the same version.
• started - the exact time that the daemon was started
• uptime - approximate running time of the daemon. The accuracy is not intended to be more than a rough guide and will be rounded to the nearest appropriate unit eg 3 minutes, 21 hours or 4 days. The start time can be used to get an exact uptime.

Completions Subcommand

The completions command provides scripts to generate autocompletion functions for bash and zsh shells. How they are used depends on the shell in use. To determine which shell is being used, you can run echo $0 in a terminal.

Running either gda completions zsh or gda completions bash will generate a script used by the shell to provide autocompletion suggestions when running gda commands.

Setting up completions

The way to enable completions depends on the shell being used.

Bash

The command can be run directly so the generated script can be reviewed but to setup completions the script should be sourced in the current shell.

$ source <(gda completions bash)

This can be automated by including the line in a script that is sourced at start-up, for instance ~/.bashrc. This will require gda to already be on the path so it may be useful wrap the call so it is only run if gda is available.

if [[ $(command -v gda >/dev/null) ]]; then
    source <(gda completions bash)
fi

Zsh

The generated script can be reviewed by running the completions command directly, but to setup completions the script should be written to a file called _gda in a directory on the shell's fpath. A possible setup is below although it assumes no existing configuration and should be modified to match user's existing configuration.

$ mkdir -p ~/.zsh/completions
$ gda completions zsh > ~/.zsh/completions/_gda
$ echo "fpath=(~/.zsh/completions $fpath)" >> ~/.zshrc
$ echo "autoload -Uz compinit && compinit" >> ~/.zshrc

For a full explanation of the completion setup see the ZSH docs

Help

$ gda completions --help
Generate autocompletion scripts for use with the gda CLI

For Bash, enable the completions by running

    $ source <(gda completions bash)

This can be automated by adding to a bash startup script somewhere (.bashrc etc)

For zsh, the completions can be enabled by writing the script to a file in one
of the directories in your `$fpath` variable. This can be set up in your .zshrc
file via something like `fpath+=~/.zsh/completions`

    $ gda completions zsh > ~/.zsh/completions/_gda

`compinit` will then need to be run although this is often already run during
startup.

Usage: gda completions <SHELL>

Arguments:
  <SHELL>
          Shell to generate completions for
          
          [possible values: bash, zsh]

Options:
  -h, --help
          Print help (see a summary with '-h')

Developer Reference

The main requirement is to have the rust toolchain installed including cargo and rustfmt. This can be installed and configured via rustup and following the guide for your OS.

The main project runs on recent (1.75+) stable versions of rust but the code-style checks require a nightly version to also be available (see below).

Building

Once the build toolchain is available, all components can be built using cargo.

$ git clone git@gitlab.diamond.ac.uk:daq/amygdala.git
Cloning into 'amygdala'...
remote: Enumerating objects: 2071, done.
remote: Counting objects: 100% (2071/2071), done.
remote: Compressing objects: 100% (949/949), done.
remote: Total 2071 (delta 1190), reused 1800 (delta 1049), pack-reused 0
Receiving objects: 100% (2071/2071), 585.96 KiB | 25.48 MiB/s, done.
Resolving deltas: 100% (1190/1190), done.
$ cd amygdala
$ cargo build --workspace
// ... Compilation output removed
   Compiling amygdala v0.2.0-dev (/path/to/amygdala/amygdala)
   Compiling gda v0.2.0-dev (/path/to/amygdala/gda)
   Compiling amygdala-api v0.1.2 (/path/to/amygdala/amygdala-api)
    Finished dev [unoptimized + debuginfo] target(s) in 1m 38s
$

This will build both gda and amygdala into the ./target/debug/ directory.

Code style

The project should conform to the rustfmt code-style including the settings provided in rustfmt.toml in the project root (these will automatically be used when running cargo fmt).

As these options are still unstable in rustfmt it requires the nightly version of rust so should be run as cargo +nightly fmt.

Lints and Warnings

Wherever possible, code should address any warnings or suggestions from the compiler or the default settings of cargo clippy.

Use of unwrap

The only additional, non-default restriction enforced is the inclusion of the unwrap_used = "deny" lint setting.

This lint prevents unwrap() being used on any Option or Result in the codebase. Panics cause a very negative experience for users and, with the default error message, it is not immediately obvious which pre-condition was not met.

If it is not possible to handle the error state cleanly or there are reasons why the unwrap can never fail, expect() should be used instead with the justification for why it is safe to do so as the message. While it doesn't prevent the hard-crash of the application, it does at least make it obvious which assumption was incorrect.

[workspace.lints.clippy]
unwrap_used = "deny"

[lints]
workspace = true

Git workflow

All additions to any of the projects should be made through PRs in gitlab. Commits should be reduced to logical steps with commits being rebased and reworded to provide useful context when viewed via git log without having to reference gitlab issues or PRs. The PR message can be more conversational and don't always make good commit messages.

If commits are squashed by gitlab when a PR is merged, care should be taken to ensure the generated message is still descriptive and doesn't just contain the PR title.

PRs in gitlab are required to be fast-forward only and merging a PR will not preserve the feature branch. If a PR consists of multiple commits where intermediate stages do not leave the repo in a working state. The PR should be for the merge commit to keep the history in a form that keeps the main branch in a buildable state. If a PR includes multiple commits including things like 'fix typos' and 'sort clippy lints', they should be squashed, either when submitting the PR or manually by the author.

Unfortunately, gitlab does not offer settings to enforce these preferences so care should be taken to keep the history coherent.

Commit messages

All commit messages should be useful and provide context without requiring access to gitlab (issues can be be linked in the text if required but should not be the entire content). A good style guide for commit messages is here. The main points:

• Short single line summary (< 50 characters if possible)
• Capitalise summary and use imperitive tense
• Split summary and body with a blank line
• Wrap all lines at 72 characters
• Explain what and why rather than how

Changelog

All changes that affect either users or future developers should be included in the changelog of the relevant project. The entries should be added to the unreleased section to one of the subheadings listed in the Keep A Changelog guide:

• Added for new features.
• Changed for changes in existing functionality.
• Deprecated for soon-to-be removed features.
• Removed for now removed features.
• Fixed for any bug fixes.
• Security in case of vulnerabilities.

Changelog entries should be targeted more at users than commit messages are and copying the commit message verbatim into the changelog is rarely useful.

Review Checklist

Feel free to review any PR, not only those to which you have been added as a reviewer. While some minor checks do not require thorough checks (typos, reformatting etc), a good starting point for reviews would be

• Is the change needed? This should probably have been covered in an issue before hand but it's always good to check.
• Is this the best approach? Look at the higher level design first. Is the change being made to the right level of abstraction in the right area of the codebase?
• Does it work? Ideally checkout the change (glab CLI can be useful) and test it locally rather than relying on the gitlab diff view.
• Code level review Are there better ways of doing it? Could it be more optimised (only if not at the cost of clarity and functionality)? Are there more idiomatic approaches?
• Are the docs still up to date?
• Is there an entry in the changelog? If the change is going to be noticeable by someone using the application, it should be mentioned. If it is a very minor change and there is already a comment for a similar change, this may not be required, eg changes to help messages could all be covered by a single, "Improved CLI help text", entry.
• Is it tested? There are very few tests in the project so far but it would be good if new changes could be tested, especially if they're fixing bugs that could be unintentionally reintroduced in future.

Layout of project

The project consists of several components in a cargo workspace.

• The three main crates, gda, amygdala and amygdala-api. These are all at the root level of the project, each in their own directory. They are referenced as workspace members in the root Cargo.toml file.
• ci.py and the releng python module used to support building, testing and deployment via CI. Can also be used locally - see releng section
• The book. This documentation built using mdbook
• catalog-info.yaml - project information for the DLS dev-portal

Crate overview

GDA

The gda client command used to communicate with amygdala and control GDA.

Amygdala

The amygdala daemon that manages the running instance of the GDA server.

Amygdala API

The amygdala-api crate contains the message and service definitions used for communication between the other two applications. These definitions could also be used in future to support new applications, potentially in different languages.

CI scripts

The main ci.py script at the root of the project is used for CI to lint, test, build and deploy the project. It can also be used locally.

It depends of some types and utilities defined in the releng package.

The Book

The high level documentation (this site) is written as an mdbook. The source is in book/src and it can be built using mdbook build book from the root of the repo. This builds in into book/book. It can also be served by a development server offering live updates and reloading etc via mdbook serve book. See mdbook docs for more configuration options.

Generated content

The book contains the output of gda help commands. To keep these up to date, they are generated when the book is compiled. The commands that are run and the files that are written are listed in book/help_commands.csv and built by the mdbook-gen-help script. This is run automatically by mdbook as a preprocessor (configured in book/book.toml) and requires the path to the built applications via the BUILD_ARTIFACTS environment variable. If using the ci script to build everything (./ci.py build), this should be artifacts.

BUILD_ARTIFACTS=artifacts mdbook serve book

Remaining files

The remaining files are informational

• README.md - introduction to the project and the main homepage when viewing the repo on gitlab
• DEPLOYMENT.md - rough guide to running the amygdala process
• DEVELOPMENT.md - guidelines to follow for developers - git workflow, review checklist etc. This is included in this book as Developer Reference
• DLS_DEPLOYMENT.md - DLS specific description of the release procedure and version handling

Amygdala crate

This is a binary crate so main.rs and the main() method are the main entry points.

Process Overview

The central type in amygdala is the Launcher and its implementation of the Launch trait (from the amygdala-api crate). This is wrapped in tonic infrastructure to make it available via gRPC.

The Launcher holds an instance of a ProcessManager that manages the actual GDA server while it handles incoming requests, delegating them to the process manager and converting the response as required. As multiple requests can be received concurrently, the process manager is held in an Arc<Mutex< >> so that the ProcessManager type doesn't have to be thread safe.

The Launcher also holds the configuration loaded from the configuration file. This configuration is a wrapper around a lazy loading container (an Arc around a OnceLock) allowing the configuration to be cloneable and to be created before the configuration file it is based on is available.

When requests are received from clients, the request is combined with the configuration (which is loaded if needed) to create the specific configuration required for that request. This new derived configuration is passed into the ProcessManager where, if it was starting a new server, it is held for the lifetime of that process allowing relevant information to be queried for the current instance of the server.

Configuration

There are two kinds of configuration.

Provided Configuration

The first, the 'provided' configuration, is that read from the config file. It is constant for the lifetime of the amygdala process.

Derived Configuration

The second, the 'derived' configuration, is the specific configuration required to start the server. It includes options provided in the start request as well as timestamped log files and build specific runtime directories. This configuration is created when the GDA server is started and dropped when it is shutdown. This configuration is the source of client details as they depend of the specific server that is running.

Startup Files

If the GDA server fails to start, it writes the fatal error message to a startup file. This provides an easy mechanism to determine the error that killed it instead of reading through the logs and guessing. The file that is written is set via the OBJECT_SERVER_STARTUP_FILE environment variable. Amygdala creates this file beforehand as a named pipe/fifo so that it can be instantly notified when it is written to instead of polling the filesystem. This file is deleted when it is dropped so that the file is not left behind in the case where the server did not write to it.

Cancelling Requests

There are two kinds of request. The simpler request/response model used for version and manage where the client makes a request and the server sends a reponse (as in 'normal' http requests), and the more complex streaming request where the client makes a request and then the server sends a possibly variable number of response messages asynchronously. These streaming requests are used for start and stop requests where the process is long running and there are updates at various points through the process.

When a client connects to make a streaming request, eg start, the processManager mutex is locked and the start task is spawned in the background. To allow the client to be sent messages, a channel is created to allow the updates to be sent. If the client, receiver side of the channel is closed (for instance if the client disconnects), the spawned process aborts and cancels its current task.

Interrupting the service

If a task is in process, the server is blocked from accepting other requests (as the process manager is locked in a mutex). While this is intentional and not an issue in normal use, if a request takes a long time (for instance the server hangs on startup), and the client is left waiting, no other commands would normally be able to access the server. If the original client is run by a user who is no longer able to cancel the request, there needs to be a way for other users to interrupt the blocked task.

For this reason, as well as the task/client channel described above, an interrupt channel is passed in. This is exposed via the manage endpoint and allows other clients to trigger an interrupt of a running task.

This same process is used when the amygdala process itself is interrupted to allow it to exit promptly.

GDA Server Status Port

The GDA server can take a while to start up and errors can occur at any time in this process. To indicate to external processes that the server has started and is in a stable state, it opens a status port when the initialisation process is complete. Amygdala uses this port to monitor the startup and block until it is complete. This lets clients assume that, as long as an error is not returned, when the requests completes, the server is available. This status port can also be queried for basic server status information such as version, although amygdala does not make use of this yet.

The status port is also used at shutdown as a check to ensure that the server has shutdown correctly, as the process amygdala holds is the eclipse launcher rather than the server itself.

Interactions with Externally Started Servers

The status port described above is used when starting the server to ensure that the previous server is not still running. This means that as amygdala cannot be aware of servers that it did not start, it is possible to get into a state where amygdala reports that it can't start a server as there is already one running and also that it can't stop the server as there is no server running.

For now, the fix for this situation is to shutdown the server in the same way that is was started which will allow amygdala to return to normal operation.

Restarting Amygdala

The manage endpoint has the option to restart the amygdala process. This is not a true restart and relies on external infrastructure to start a new process.

It shuts down any running servers, then exits with a sentinel exit code intended to be used with systemd RestartForceExitStatus to trigger a restart. To help prevent the service shutting down when not in a situation where it will be restarted, it checks for the presence of a AMYGDALA_SYSTEM environment variable. If this is not present the manage endpoint will return a precondition_failed error and not shutdown.

Abnormal Shutdown

Normally, when amygdala is shutdown it will shutdown any servers it has started first. If it is sent a SIGKILL signal it is not able to do any cleanup work and can leave a started server running in the background. In this situation, manual intervention is required to shutdown the server so for this reason, it is recommended that amygdala be shutdown via an interrupt or SIGSTOP rather than SIGKILL.

GDA crate

This a binary crate so main.rs and the main() method are the main entry points.

The main module mainly handles reading the CLI input and delegating execution to the various dedicated modules.

Modules

cli

The CLI parsing and type coercion is handled by clap and is contained within the cli module. The common types are in the root of the module with the larger subcommands broken into their own sub-modules.

connection

The connection module wraps up all communication with the amygdala process abstracting all gRPC details away from the rest of the application.

It provides a Connection type with a method for each service endpoint as well as standalone functions that handle setting up async runtimes and creating the initial connection.

client

The client module handles all aspects of starting the GDA client locally. It provides a ClientCommandOutline type that can be built using the client details returned by the server, this can then be passed user configuration from the CLI to build a runnable command.

It also handles the spawning of the new process taking into account user preferences regarding log redirection and foreground/background options.

server

The server module is split into two implementations, one for the gui components and the other for running the same commands from the terminal. The top level entry point method (server_command) of each is re-exported from the server module. All other implementation details are hidden from the rest of the crate.

gui_utils

This is a bit of a dumping ground for reusable UI components. It defines the common base style for GUIs so there is some consistency for each use. This also defines an error dialog used when starting the client.

build_info

This module exposes information gathered by the built crate at build time so that commit info and build time can be made available to help debugging.

API crate

The API crate contains the message and service definitions used for communication between the server and client. The definitions are in the language agnostic protobuf format.

The services are implemented as gRPC services.

Protobuf definitions

The protobuf definitions (in the proto subdirectory) are split into messages and services. The former defines all the types that can be created and sent between processes and the latter defines the services and what types they expect and return.

Much of the code exposed by the amygdala-api crate is auto-generated from the protobuf definitions by the prost and tonic crates. The remaining code that augments the generated types is in two modules corresponding to the two protobuf files. Most of this code is providing utility methods to ease use.

There are two features (server and client) configured so that the server and client types are only included where they are required. The message types are always available.

Due to the design of protobuf, fields can be added in future changes without breaking existing code as long as the field numbers of existing fields are not changed.

Build script

The build script in the amygdala-api crate uses prost and tonic to generate the rust implementations of the message and service types, adding attributes required to enable the features and custom debug implementations mentioned above.

Protoc binary

Prost relies on an external binary, protoc to generate code. To remove the requirement for developers (and CI) to have this in their environments as well as to ensure a consistent version is used, this is provided by the protobuf_src crate which builds and caches a version locally. This adds a few seconds to the build process the first time it is run but prevents dependency and environment issues sourcing from relying on external applications.

CI Scripts

The project has pipelines set up to run at regular points in the development cycle. These are all run via the ci.py script.

$ ci.py --help
usage: ci [-h] [--deploy-dir DEPLOY_DIR]
          {check,build,deploy,versions,docs,clean-nightly} ...

Amygdala CI runner

positional arguments:
  {check,build,deploy,versions,docs,clean-nightly}
    check               Lint, format, test etc
    build               Build and save artifacts
    deploy              Deploy built products
    versions            List versions of tools being used
    docs                Build and publish the documentation book
    clean-nightly       Remove old nightly releases from deployment directory

options:
  -h, --help            show this help message and exit
  --deploy-dir DEPLOY_DIR
                        Directory where releases should be deployed

CI tasks

Check

Check that the code passes all required checks

$ ci.py check --help
usage: ci check [-h]

options:
  -h, --help  show this help message and exit

This checks that:

• the versions make sense

  A release version of either gda or amygdala cannot depend on a pre-release version of amygdala-api. In practice, amygdala-api should be kept on release versions as each change should be usable in its own right and not tied into application changes.

• the versions don't conflict with previously released versions.

  A version cannot match a previously released version, that is a new commit to the project cannot be marked as a release if that version has already been deployed to the filesystem.

  If a crate is marked as a pre-release version the version it is previewing (eg 1.2.0 for 1.2.0-dev) must not already have been released.

  Additionally, the previous version must have been release. Eg, if a crate is versioned as 1.2.3, version 1.2.2 must have already been published. This prevents versions being skipped. If a major or minor version is bumped, the next number in the version must be 0, eg, going from 1.2.3 to 1.3.0 is fine whereas 1.2.3 to 1.3.3 is not.

• the code is well formatted

  This runs cargo +nightly fmt --check and enforces the code style standards in all rust code. Running this locally (without the --check flag) will fix any issues.

• the code compiles and the tests pass

  The workspace is compiled (with the dev profile for speed) and the tests are run. As part of this clippy is run to ensure any denied lints are caught. Warnings will be logged but do not prevent the build continuing. These should be fixed wherever possible.

Build

Build the workspace in release mode and store the build products as artifacts.

$ ci.py build --help
usage: ci build [-h]

options:
  -h, --help  show this help message and exit

This will also include any resources listed in the [package.includes] section of each crates Cargo.toml. This enables scripts to be deployed automatically without having to be handled explicitly anywhere.

artifacts
├── amygdala
│  └── amygdala
└── gda
   ├── gda
   ├── gda-gui
   └── gda-logpanel

Deploy

The deploy subcommands deploys the build applications to various locations

$ ci.py deploy --help
usage: ci deploy [-h] {apps,registry} ...

positional arguments:
  {apps,registry}
    apps           Deploy to apps directory
    registry       Deploy to CI registry

options:
  -h, --help       show this help message and exit

Apps

Applications at Diamond are made available via the module system. This requires the executables to be copied to a specific location (set via $DEPLOY_DIR or --deploy-dir to the main ci command) and a module file describing how the application be made available to be written to a directory included on users' $MODULEPATH (set via $MODULE_DIR or --module-dir).

Registry

Release versions of the applications (i.e. not the nightly builds) are published to the gitlab 'generic package registry' for the project. These builds are kept indefinitely and can be linked from releases.

Documentation

The docs subcommand builds and deploys the documentation (this book) to alfred.

$ ci.py docs --help
usage: ci docs [-h] {build,publish} ...

positional arguments:
  {build,publish}
    build          Build the documentation site
    publish        Publish the documentation

options:
  -h, --help       show this help message and exit

The documentation site is only a single version so will only reflect the latest version of the repository. For this reason docs should be make note of features that require specific versions of the applications and not immediately remove docs when features are removed (instead marking them as deprecated/unsupported).

Clean Nightly

As nightly builds have a fairly short useful lifetime, old versions should be deleted regularly. The exact number to keep is not fixed as if many updates are made in a short time frame, it can be useful to retain more past versions to help compare new features.

The clean-nightly command deletes nightly builds from the app directory. It has three options to control which builds are deleted. A minimum and maximum count. The newest builds up the minimum count will always be kept however old they are. Any builds over the maximum count will always be deleted however new they are. Between the two limits, anything older than the maximum age will be deleted.

$ ci.py clean-nightly --help
usage: ci clean-nightly [-h] [--min-builds MIN_BUILDS]
                        [--max-builds MAX_BUILDS] [--max-age MAX_AGE]

options:
  -h, --help            show this help message and exit
  --min-builds MIN_BUILDS, -n MIN_BUILDS
                        Minimum number of builds to keep - these will be kept
                        even if older than threshold
  --max-builds MAX_BUILDS, -x MAX_BUILDS
                        Maximum number of builds to keep - beyond this, builds
                        will be deleted even if newer than threshold
  --max-age MAX_AGE, -a MAX_AGE
                        Age (in days) of oldest builds to keep

Versions

This subcommand does nothing other than list the versions of all the applications in the environment being used to help debugging any discrepancies between jobs run in CI and across developer environments.

$ ci.py versions --help
usage: ci versions [-h]

options:
  -h, --help  show this help message and exit

Pipelines and when they run

Merge Requests

For every merge request to the gitlab repo (and every commit to an existing MR) the check ci task is run. This must complete successfully for the MR to be mergable.

Every Commit to main

For every commit to main (i.e. every MR that gets merged), the check, build, deploy and docs commands are run, updating the nightly module version, the latest module version (for release versions) and the documentation site with the latest versions.

On Request

As the check stage of the CI does not store any artifacts, it is not possible to download builds of merge requests for testing. To make this feasible without having to build locally, it is possible to trigger a build for a merge request.

From the gitlab repo page, go to Build > Pipelines > Run Pipeline, then select the branch backing the merge request from the branch drop down and select Run Pipeline. This will just run the build stage and keep the built artifacts for 7 days. These can be downloaded via Build > Artifacts and selecting the relevant file from the browse menu next to the pipeline once it completes.

Schedule

Scheduled CI runs only run the clean-nightly job to remove any old nightly builds. This is currently configured to run weekly.

Gitlab specific features

Section headers/footers

The functions in the ci script and import utilities are wrapped in section headers. These are managed by the section decorator and allow the individual stages in the pipelines to be collapsed in the pipeline log viewer in gitlab. Due to the way they are designed, the scripts still work when run from the terminal but may produce surprising results when redirected to file, eg running ./ci.py versions gives

$ ci.py versions
section_start:1761823901:run_versions-0[collapsed=true]
Version Report
$ /usr/bin/python3 --version
Python 3.12.11
$ cargo --version
cargo 1.90.0 (840b83a10 2025-07-30)
$ rustc --version
rustc 1.90.0 (1159e78c4 2025-09-14)
$ mdbook --version
mdbook v0.4.52
section_end:1761823901:run_versions-0


Gitlab package repository

As part of the deploy stage, if the version of either application is a release version, it is published to the gitlab package repository for the project. This allows the built products to be linked to releases and made downloadable in a way that will not expire.

This uses several environment variables defined by gitlab

• CI_API_V4_API The root api address for the gitlab instance
• CI_JOB_TOKEN An authentication token that allows the job to publish to the repository.
• CI_PROJECT_ID A unique ID for the project within gitlab used to identify the package repository.

For details on publishing to the package repository, see gitlab documentation. The generic package repository is used as gitlab currently doesn't support rust crates.

GDA - Launcher CLI

Unreleased

Released YYYY-MM-DD

0.2.3

Released 2025-02-28

Improve error messages and support for server side config changes.

Changed

• Change the client runtime directories to be more consistent. Use {root}/{user}/{version} for both workspace and config, with the config directories then having an additional level for each client build. See #76
• Errors when starting client now include more details for what went wrong. See #81 and #83
• Removed 'xyz is not a recognised subcommand...' message when running extension commands from gda.

Added

• Check for existence of AMYGDALA_NO_GUI environment variable to disable GUI popups for all commands without requiring -G/--no-gui flag.
• Added support for new error states from amygdala related to the configuration not being available (after the change to lazy loading).

Fixed

• Support running gda-gui script directly again. Now uses default host and port so that anything that relied on the previous binary being callable directly can continue to do so instead of requiring it to be called via the gda extension system.
• Use canonical client build directory when determining the config directory. This prevents the same config directory being used for every client, including across versions of GDA. See #75.
• Client stderr output now gets written to the correct file (instead of stdout).

0.2.2

Released 2024-06-14

Mainly a bug fix release to address the JVM arg handling problems.

Changed

• Tags used to create alternative client runtime directories are now restricted to alphanumeric + '_' characters. This is to prevent tags being used to create directories outside the configured parent directory. See #58.

Added

• doctor subcommand to help diagnose any issues with launcher. Intended to provide easy way for users to gather relevant information about versions and the environment.

Fixed

• System properties passed by -D on the command line had the -D prefixed twice before being passed to the server/client commands. They are now passed correctly. See #68.
• Launcher (previously manage) commands no longer fail silently when server is not running. See #70.

0.2.1

Released 2024-05-29

Bug fix patch release to deal with newer egui version not running well on RHEL7 machines.

Fixed

• egui and eframe versions rolled back to support RHEL7 again

0.2.0

Released 2024-05-24

This is a major rewrite of the gda application. While the main interface has remained the same, there may be subtle changes in behaviour in addition to those listed below.

Changed

• gda-gui has been merged into main gda application.
  • gda server commands now show a gui if $DISPLAY is available.
  • gda client will now show an error dialog if it can't be started. If the process starts but then fails (eg for config errors), no error is shown as the application will show its own errors.
  • The previous behaviour is maintained behind a --no-gui/-G option.
• manage subcommand has been renamed to launcher to better reflect what it does - manage the launcher process. manage has been kept as an alias for now.
• Client now defaults to starting in the foreground with the output written to logs (to match the previous script behaviour). There are also --console/--cs and --background/--bg options to write output to the terminal or start in the background respectively.

Added

• External commands now have AMYGDALA_STATUS set in their environment with a value of either RUNNING or NOT_RUNNING.
• Errors showing the GUI components are now sent to stdout instead of being dropped.
• Client subcommand now has --console and --background options in addition to --foreground.

Removed

• Exit code is now reduced to 0 or 1. If finer grained errors are required for scripts, they can be re-added in future.

Fixed

• The parent directory of runtime (config + data) directories is now created with the correct permissions so clients can be started by other users.

Deprecated

• The previous gda-gui application is now redundant as its features are now included in the main gda application. It is replaced by a shell script that redirects calls through to gda server restart.
• The manage subcommand has been renamed to launcher and will be removed in a later version.
• The -f short flag for --foreground will be removed in future as it is the default.

0.1.0

Released 2024-05-09

• Initial version of gda cli

Amygdala

Unreleased

Released YYYY-MM-DD

0.3.0

Released 2025-02-28

Breaking Changes

• amygdala should now be started as amygdala serve instead of amygdala.

Added

• Extended CLI. Includes options to set port and logging etc.
• Graylog support. The logging from the launcher daemon can now be sent to graylog. See the server cli section of the book for details.

Changed

• Main configuration moved from the ProcessManager to the Launcher. The process specific configuration used to launch a server is still included with the server process but the general config read from the file is now held outside the process manager.
• Configuration file is loaded lazily when it is first required instead of at startup.
• Check for external servers when shutting down server.
• Add specific error variants for when external servers are running. See #83
• Improve error messages when server fails to start. See !96

0.2.0

Released 2024-05-24

Changed

• The crate is now a 'binary only' crate. The lib.rs module has been removed and the modules are now imported directly in main.rs.

Security

• The deployment passed when starting the server is now checked to ensure server command is within the configured deployment root. Prevents absolute paths allowing arbitrary commands to be run as the user running the amygdala daemon.

0.1.0

Released 2024-05-09

• Initial release of amygdala - GDA launcher daemon

Amygdala API

0.1.5

Released 2024-09-11

Changed

• Add From<Option<Duration>> implementation for StopRequest

0.1.4

Released 2024-06-14

Changed

• Include -D flag in args returned by GdaOptions::vmargs. This prevents the server and client both having to handle building the args.

0.1.3

Released 2024-05-29

Added

• Utility methods to Version (executable, start_time)

0.1.2

Released 2024-05-14

Added

• Utility methods for StatusResponse (uptime_string, start_time, executable)
• is_restart method for StartRequest

0.1.1

Released 2024-05-14

Added

• From<Duration> for StopRequest

0.1.0

Released 2024-05-09

• Initial version of API used for gda and amygdala