20. Key Patterns

20.1. OSGi Classloading

A ClassLoader in Java is simply another Java class which is used to load class files. It takes a full qualified class name (FQCN) as a string and returns the Class object. However the stand Java ClassLoader does come with issues, for example if you wanted to use two versions of a library but both jars contain classes with the same fully qualified name, you cannot have both loaded at the same time.

OSGi (Open Service Gateway Initiative) is a Java framework which allows the development and deployment of modular software. In this framework, the ClassLoader is segmented into bundles (or plugins for Eclipse). Bundles are comprised of Java classes and other resources. They are the only entities for deploying Java-based applications in OSGi and provide functions to the end user. The main benefits of OSGi are that it reduces complexity, it is dynamic so you can swap bundles at runtime and also provides additional services (e.g. logging).

Manifest

A bundle carries descriptive information about itself in the manifest file located at <plugin.name>/META-INF/MANIFEST.MF.

Below is an extract from the uk.ac.gda.core MANIFEST.MF:

uk.ac.gda.core/META-INF
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: uk.ac.gda.core plugin
Bundle-SymbolicName: uk.ac.gda.core;singleton:=true
Bundle-Version: 9.12.0.qualifier
Require-Bundle: org.aopalliance,
 javax.servlet;bundle-version="2.5.0",
 uk.ac.gda.common,
Eclipse-RegisterBuddy: uk.ac.diamond.scisoft.analysis,
 uk.ac.gda.api,
 uk.ac.diamond.org.springframework
Eclipse-BuddyPolicy: registered
Export-Package: gda.beamline.beam,
 gda.beamline.beamline,
 gda.beamline.video,
 gda.commandqueue,
 gda.data
Bundle-ClassPath: .,
 scripts/
Bundle-Vendor: Diamond Light Source
Bundle-RequiredExecutionEnvironment: JavaSE-1.8
Import-Package: javax.jms,
 org.apache.commons.collections.map,
 org.apache.commons.logging
Bundle-ActivationPolicy: lazy
Service-Component: OSGI-INF/*.xml
Bundle-Activator: uk.ac.gda.core.GDACoreActivator
Automatic-Module-Name: uk.ac.gda.core

See the specification for more details on each of the parameters contained in the file.

The class space is all the classes reachable from a given bundles class loader and so contains classes from:

  • The parent class loader - java.* packages from the boot class path.

  • Imported Packages (Import-Package, e.g. javax.jms) - defines the constraints on the imports of shared packages. Many packages can be imported. Packages must be resolved.

  • Required bundles (Require-Bundle, e.g. org.aopalliance) - contains a list of required bundle symbolic names. It will take all exported packages from a required bundle at wire these to this bundle.

  • The bundle’s classpath (Bundle-ClassPath, ".", scripts/) - defines a list of directories (or JAR file path names) that are inside the bundle that contain classes. The “.” specifies the root directory of the bundle.

  • Attached fragments (Fragment-Host) - links the fragment to its potential hosts. E.g.

    uk.ac.diamond.daq.stage.test/META-INF/MANIFEST.MF:Fragment-Host: uk.ac.diamond.daq.stage;bundle-version="1.0.0"
    

A class must be consistent so never contain two classes with the same fully qualified name.

The OSGi classloading flowchart is shown below (taken from the OSGi Core Release document.)

scale60

20.1.1. Eclipse buddying

Eclipse buddying is Eclipse specific and so not directly implemented by OSGi. It is only implemented by Equinox (an implementation of the Eclipse OSGi framework). In terms of ordering, it is implemented at the end of the flowchart above, so if no dynamic import package is specified (dynamic import? -> no), then try budding. Note that this does imply that a bundle can only use with dynamic import package or buddying and not both.

Buddying is when bundles help each other out like ‘buddies’ and allows for the dynamic discovery and loading of classes. A bundle declares that it needs help from other bundles to load classes. When the bundle class loader fails to find the desired class through the normal OSGi model (above), its buddy policy is invoked. This discovers a set of buddies and consults each one until either the class is found or there are no more buddies to consult. This can all be set-up in the bundle manifest. For example in the above manifest for uk.ac.gda.core the following lines define the name of the bundle and set the Buddy policy to registered so all dependent bundles that explicitly register themselves as buddies to that bundle are consulted (more details).

Bundle-SymbolicName: uk.ac.gda.core
Eclipse-BuddyPolicy: registered

Then any bundle which can help manifest will have the lines:

Eclipse-RegisterBuddy: uk.ac.gda.core
Eclipse-BuddyPolicy: registered

For example, the uk.ac.gda.api bundle MANIFEST.MF contains these line. There can be multiple bundles defined under the RegisteredBuddy. For example, the manifest from uk.ac.gda.core above states that it is also a registered buddy for uk.ac.diamond.scisoft.analysis, uk.ac.gda.api and uk.ac.diamond.org.springframework.

20.1.2. A note on Dynamic Import Package

The last stage in the flowchart is to check is DynamicImport-Package is called. Dynamic imports are matched to export definitions to form package wirings during class loading. It works in a similar way to Import-Package except it is not required for the bundle to resolve the module. They are used as a last resort if no other definition can be found.

Currently they are not used in GDA due to the issue of split packages.

20.1.3. Module-finding Libraries

Libraries like Jackson may dynamically search for and load modules with extra functionality, this may require additional configuration using GDA’s custom ClassLoaders.

For example, if adding some Jackson modules:

<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
	<artifactId>jackson-datatype-jdk8</artifactId>
	<version>2.12.1</version>
	<type>jar</type>
</dependency>
<dependency>
	<groupId>com.fasterxml.jackson.datatype</groupId>
	<artifactId>jackson-datatype-jsr310</artifactId>
	<version>2.12.1</version>
	<type>jar</type>
</dependency>

Some additional setup may be needed for an ObjectMapper:

var mapper = new ObjectMapper();
var classLoader = GDAClassLoaderService
		.getClassLoaderService()
		.getClassLoaderForLibraryWithGlobalResourceLoading(
				ObjectMapper.class,
				Collections.emptySet());
mapper.registerModules(ObjectMapper.findModules(classLoader));

For OSGI applications the dependencies will need to be included in a feature. For unit testing purposes they will need to be included in the manifest of the test plugin (or one of its dependencies)

20.2. Queueing Scans

20.2.1. GDA9 (New Scanning) Queue System

A description and walk through of new style scanning can be found in the use case section.

20.2.1.1. Key Classes

There are three main classes in the GDA9 queue system. These are defined by the four interfaces: IConsumer, IPublisher and ISubscriber. The first of these relates processing a queue using the IConsumer which consumes beans from the queue. This interface implements the interface IQueueConnection. IPublisher and ISubscriber define a publish/subscribe events system. These two interfaces extends ITopicConnection.

The classes that implement each of these components are ConsumerImpl, PublisherImpl and SubscriberImpl respectively. ConsumerImpl directly extends AbstractConnection while PublisherImpl and SubsciberImpl extends AbstractTopicConnection which then extends AbstractConnection.

The interface IEventService defines factory methods for all of these classes. The event service is an OSGi component that can injected, it is mostly accessed through a service holder. The implementing class is EventServiceImpl.

There can be multiple instances of each of these classes in the system, possibly configured for different queues and/or topics. Additionally, they may make use of each other. For example, ConsumerImpl uses an ISubscriber to subscribe to the command topic.

20.2.1.2. ConsumerImpl

The central object on the server is the ConsumerImpl. This is essentially the queue as it contains the queue as a persistent object implemented using MVStore (a persistent, log structures key-value store). The queue is also backed up. There can be only one consumer containing items in the submission queue. This consumer is created by the ScanServlet* which is configured in Spring. ScanServlet extends AbstractConsumerServlet which defines most of its behaviour. The only method that ScanServlet needs to define itself is the method to create a new process to run the consumed bean. This method is defined in the interface IConsumerServlet. A method reference to this method is passed to the consumers setRunner() method in AbstractConsumerServlet. In this instance the ScanServlet creates the process ScanProcess which is used to process the bean and perform the scan.

Aside on Servlets

* Note that the ScanServlet is not a servlet by definition. It is just a configurable/object that creates and starts a consumer. Potentially it could be removed if the consumer was instead declared directly in Spring.

Equally note that there are other ‘Servlets’ that use queues including:

  • PositionServlet - a servlet to handle a PositionRequest to move to an IPosition.

  • AcquireServlet - a servlet to acquire data from a particular detector.

  • DeviceServlet - a servlet to get the available devices from the IDeviceService.

These all extend AbstractResponderServlet which itself implements IResponderServlet (a Request/Response servlet) used for processing a queue. It creates a response for each request processed for the given topic.

20.2.1.3. Queues and Topics

The core of the GDA9 queueing/events system uses ActiveMQ queues and topics (see here for the difference between the 2 types).

Name

Type

Value

Description

Submission Queue

Queue

org.eclipse.scanning.submission.queue

This is the queue that is contained in the IConsumer.

Status Topic

Topic

org.eclipse.scanning.status.topic

Used to publish or listen for the bean’s status during running and processing.

Command Topic

Topic

org.eclipse.scanning.command.topic

The consumer listens for ConsumerCommandBeans with its consumer ID and acts on the commands accordingly (see description below for controlling the consumer).

Acknowledgement Topic

Topic

org.eclipse.scanning.ack.topic

Used for acknowledgement from the consumer to commands sent on the command topic, e.g. client side submits a command and receives an acknowledgement on this topic.

20.2.1.4. Server side usage

Creating the consumer

  • The API for a consumer is defined by the interface IConsumer and implemented by the class ConsumerImpl which extends AbstractConnection. The consumer ‘consumes’ the items from the contained queue and runs them. Consumers generally run on the server, and there should be only one consumer for each submission queue. Most consumers belong to servlets. The consumer for the default (scanning) submission queue is created by the ScanServlet which is defined in Spring. ScanServlet extends AbstractConsumerServlet which implements the interface IConsumerServlet.

  • The AbstractConsumerServlet.connect() method is called. This calls IEventService.createConsumer() to create a consumer with the URI broker, queues and topics. The implementation of this is EventServiceImpl which instantiates (and then returns) a new ConsumerImpl.

  • The ConsumerImpl contains the queues internally and within the constructor it creates 2 SynchronizedModifiableIdQueue queues: one for the submission queue and another for the set of running and completed jobs (e.g. status queue). The SynchronizedModifiableIdQueue implement IPersistentModifiableIdQueue. The last line of the constructor calls connect(), which creates a publisher (PublisherImpl -> IPublisher) to the status topic and two subscribers (SubscriberImpl -> ISubscriber); one to the status topic and the other to the command topic.

  • After creating the consumer, it is configured in the AbstractConsumerServlet.connect() method including setting the runner which defines an IProcessCreator to create a process from a bean, e.g. a ScanProcess for a ScanBean.

Starting the consumer

  • Following configuration of the consumer, AbstractConsumerServlet.connect() calls start() on it. This method creates a new thread which simply calls the run() method within a try/catch block that logs any thrown exceptions.

  • The ConsumerImpl.run() method calls init() to initialise the consumer and starts the main consumer loop.

  • [LEGACY: Finally, AbstractConsumerServlet.connect() creates a JMSQueueReader (from the IEventService method) and calls its start() method which runs in a separate thread. This calls the JMSQueueReader.run() and then initialize() to create a consumer for the JMS queue (a MessageConsumer). NOTE: the JMS queue is no longer used so this last point is essentially a legacy implementation that remains as some users still submit straight to the JMS queue from Python scripts.]

Running the consumer

  • After the call to init() in the ConsumerImpl.run() method, the main loop begins. The main loop runs as long as isActive() returns true. For each loop of the event loop, consume() is called. Within the loop checkPaused() is called first to check if the consumer should pause, then as long as isActive() still returns true we call executeBean().

  • The executeBean() method calls the runner’s createProcess() method to create the IConsumerProcess to run for the bean. For scanning this will call ScanServlet.createProcess() which creates and returns a new ScanProcess.

  • The start() method is called on the process which calls IConsumerProcess.execute() -> ScanProcess.execute() which begins to process the bean and run the scan.

Submitting to the consumer

Controlling the consumer

For example:

  • PAUSE:

If QueueCommandBean = PAUSE_QUEUE, the pause() method is called. This sets the variable awaitPause to true. The IPublisher is used to publish this. During the next iteration of consume(), the checkPaused() method will block and cause the consumer to wait until it has been lifted.

  • RESUME:

If QueueCommandBean = RESUME_QUEUE, the resume() method is called. This sets the variable awaitPause to false and publishes this notification using the IPublisher. Now the checkPaused() method finds awaitPause = False and is freed so resumes the running of the consumer.

_images/NewScanningQueues_ClassDiagram.png

20.2.1.5. Client side

The client uses the ConsumerProxy to submit to the queue. It implements the same IConsumer interface as the server (detailed above) and sends commands on the command topic and receives replies on a command acknowledgement topic. The commands are represented by a QueueCommandBean which has an enum QueueCommandBean.Command to identify the command. This includes SUBMIT_JOB to submit a job. The bean to submit is also contained in the QueueCommandBean, in the jobBean field.

_images/NewScanningQueuesClient_ClassDiagram.png

20.2.2. Old Scanning Queue System

Old scanning uses a different system for queuing scans. This is predominately used in MX scanning (e.g. data collections, line scans, grid scans etc). It allows users to submit a scan and then queue up others to run.

The key class that manages the queue on the server side is FindableProcessorQueue. It implement IFindableQueueProcessor, which further extends Processor and Queue. It can be defined in Spring as the processor of a given queue. Most frequently used is the CommandQueue which implements Queue and handles Command objects. Command extends IObservable. The Commands are subsequently stored in CommandQueue and processed by the FindableProcessorQueue.

A Spring object definition for the CommandQueue is given below (taken from MX server config).

<bean id="commandQueue" class = "gda.commandqueue.CommandQueue">
	</bean>

	<bean id="commandQueueProcessor" class = 
"gda.commandqueue.FindableProcessorQueue">
		<property name="queue" ref="commandQueue"/>
		<property name="startImmediately" value="true"/>
	</bean>

	<gda:rmi-export
		serviceName="gda/commandQueueProcessor"
		service="commandQueueProcessor"
		serviceInterface="gda.commandqueue.IFindableQueueProcessor" />

20.2.2.1. Server side

Set-up the queue processor

  • The method setQueue() is called during the GDA start-up on FindableProcessorQueue. The queue is defined in Spring as the CommandQueue (as above) and this is set as the queue that FindableProcessorQueue will process.

  • Spring can also define whether the queue should start straight away by setting startImmediately to true. The start() method within FindableProcessorQueue is called and a new Thread is started. This thread is managed by FindableProcessorQueue through the method sendCommandToManagerThread().

Running the queue processor

  • The run() method is called within FindableProcessorQueue. This will enter a loop to wait for a change to the queue or a change of the running status. It then calls the method removeQueueHead() which calls the CommandQueue removeHead() method. This takes the top command off of the queue and returns it to FindableProcessorQueue as a Command. The Command returned is a JythonCommand that contains a script file to run along with other attributes such as a description and status. E.g.

    JythonCommand [scriptFile=/tmp/JythonCommandRunnerCommand_7389848664369882100.py, description=Data collection. Prefix= test2_0_:NOT_STARTED]
    
  • Next FindableProcessorQueue calls run() on the Command. This calls the run() method from JythonScriptFileRunnerCommand which gets the ICommandRunner (the JythonServerFacade) and calls the runScript() from it with the script defined in the original command as the argument.

  • The JythonServerFacade (which implements ICommandRunner) then uses the JythonServer to run the script which will run the data collection/ grid scan etc.

  • FindableProcessorQueue will monitor the progress of the progress and notify any observers.

  • FindableProcessorQueue continues to remove commands from the CommandQueue via the removeQueueHead() method until the CommandQueue returns null indicating that there are no more commands in the queue.

Class diagram

_images/OldScanningQueues_ClassDiagram.png

Managing the queue

FindableProcessorQueue also has control over the queue and tasks. This allows the user to interact with the scan/queue, for example, abort it, pause it, resume it etc. The calls made to the processor FindableProcessorQueue are initiated from the client side (see client side for more details).

  • Pause:

The PauseCommandQueueHandler on the client side calls the pause() method in FindableProcessorQueue which, via the sendCommandToManagerThread() method, will pause the task by calling pause() on the Command (i.e. the JythonScriptFileRunnerCommand) which then gets the script and scan controllers to pauses the scan/script. This calls pausesCurrentScan() and pauseCurrentScript() in JythonServerFacade which tells the JythonServer to pause. The FindableProcessorQueue will also pause the queue.

  • Stop:

The StopCommandQueueHandler on the client side calls the stop() method in FindableProcessorQueue which, via the sendCommandToManagerThread() method, will stop the current task by sending a call to abort() on the Command (i.e. the JythonScriptFileRunnerCommand) which then calls abortCommands() on an ICommandAborter implemented by the JythonServerFacade which tells the JythonServer to abortCommands(). The FindableProcessorQueue will also stop the queue.

  • Start:

The StartCommandQueueHandler on the client side calls the start() method in FindableProcessorQueue which, via the sendCommandToManagerThread() method, will send a call to resume() on the Command (i.e. the JythonScriptFileRunnerCommand) if it had been in a paused state. The JythonScriptFileRunnerCommand calls resumeCurrentScan() and resumeCurrentScript on the ICurrentScanController and the IScriptController implemented by the JythonServerFacade which tells the JythonServer to resumeCurrentScan() and resumeCurrentScript().

  • Skip the scan:

The SkipCommandQueueHandler on the client side calls the skip() method in FindableProcessorQueue which calls abort() on the current Command being run. The does not disrupt the thread running so the next Command will be taken from the CommandQueue.

  • Stop after current scan:

The StopAfterCurrentCommandQueueHandler on the client side calls the stopAfterCurrent() method in FindableProcessorQueue which creates a new StopCommand which is added to the CommandQueue and then moved to the top of the queue. This command will stop the processor.

20.2.2.2. Client side

The client side uses CommandQueueViewFactory to display the contents and status of the queue. A client side manager must get the queue from the CommandQueueViewFactory and call addToTail() on the queue (CommandQueue method) providing a CommandProvider as an argument. It uses the JythonCommandCommandProvider (which implements CommandProvider) to create a command to run with a description and the other elements needed. The CommandQueue.addToTail(CommandProvider) method then calls getCommand() from the JythonCommandCommandProvider to add the Command to the CommandQueue. These commands will subsequently be picked off the queue by the server as described above.

An example of this implementation can be found in the MX DataCollectionScanManager.

Interactions

The client side has a number of AbstractHandlers to manage user interactions:

The execute() method is triggered when the ‘pause queue’ button is pressed. It gets the processor from the CommandQueueViewFactory (FindableProcessorQueue) and calls the pause() method.

The execute() method is triggered when the ‘stop queue’ button is pressed. It gets the processor from the CommandQueueViewFactory (FindableProcessorQueue) and calls the stop() method.

The execute() method is triggered when the ‘skip scan’ button is pressed. It gets the processor from the CommandQueueViewFactory (FindableProcessorQueue) and calls the skip() method.

The execute() method is triggered when the ‘Resume scan’ button is pressed. It gets the processor from the CommandQueueViewFactory (FindableProcessorQueue) and calls the start() method.

The execute() method is triggered when the ‘Stop after current scan’ button is pressed. It gets the processor from the CommandQueueViewFactory (FindableProcessorQueue) and calls the stopAfterCurrent() method.

20.3. Asynchronous Threads - The Async Class

If tasks need to be run in asynchronously within a process then GDA developers are strongly encouraged to use the Async.java class. This is available from the gda-core.git repo and located in the uk.ac.diamond.daq.concurrent package within.

The Async class runs tasks in their own thread. Existing idle threads will be reused if possible to minimise the overhead of creating new threads. It manages an internal thread pool to do this. It uses an ExecutorService to manage asynchronous tasks which returns a ListenableFuture so that callbacks can be added. It can also run and manage repeating tasks using ScheduleExectuorServices.

More specifically the Async class provides the ability to:

  • execute() - run a Runnable in its own thread.

  • submit() - submit a task to be run in its own thread and return a Future giving access to the completion or error state of the task. This means that task can be cancelled if necessary. The Future also gives access to the return value of the Callable target. See below for more details or alternatively an explanation of the use of Futures is available here.

  • submitAll() - submit a list of callables to be executed concurrently.

  • executeAll() - submit a collection of Runnables to be executed concurrently.

  • scheduleAtFixedRate() - schedule a task to be run repeatedly at a fixed rate after an initial delay.

  • schedule() - schedule a task to run after a given delay.

  • scheduleWithFixedDelay() - schedule a task to be run repeatedly with a fixed delay between executions.

There are variants of each of the base methods given above which allow threads to be renamed or allow varying periods and/or times between executions.

Aside: A Future is a future result of an asynchronous computation. A result will eventually appear in the Future after the processing is complete. A FutureTask is a cancelable asynchronous computation and upon running it will execute the given callable. The use of the ExecutorService means that a reference to the running thread is maintained and so it can be cancelled and it can wait for it to finish before getting the result. Submitting a Callable to the pool will return the value of the call method when Future.get() is implemented.

20.3.1. Usage

Use the Async class to manage your asynchronous threads/tasks as such:

Async.execute(() -> {
	 // do something
	});

Users are strongly discouraged from creating their own threads. i.e. DO NOT USE

new Thread(new Runnable()) {
	public void run() {
		// do something
	}
}

In the above example there is no callbacks available so the thread is simply started and then lost.

20.3.2. Examples

EXAMPLE 1: Readout of detectors

The following extract is taken from the ConcurrentScanChild() class (available in gda-core) which contains the readoutDetectorsAndPublish() method. This method creates a FutureTask to readout the detectors using parallel threads (asynchronously) into a ScanDataPoint (during scanning). The subtleties of running parallel threads so that each detector is readout in its own thread are skipped here but see the actual for details on this if interested.

  • Create the FutureTask and override the call() method with the tasks that should be carried out in the thread. The call method will run when the task is submitted.

    detectorReadoutTask = new FutureTask<Void>(new Callable<Void>() {
    
        List<Future<Object>> readoutTasks;
    
       /**
         * Readout each detector in a thread, add the resulting data to the 
         * ScanDataPoint and publish.
        */
        @Override
        public Void call() throws Exception {
    
        	try {
              // do something
    
    
      	} catch (Exception e) {
      		// catch it
      		}
      		throw e;
      	}
      	return null;
        }
    });
    
  • Then execute this using Async:

    Async.execute(detectorReadoutTask);
    

EXAMPLE 2: Update the state of a motor (using the ScheduleExectorService)

This is an example of how to schedule a command to run periodically so that the status of a motor can be reported on throughout a scan.

  • Create the target task to run. This task will log the motor positions and status.

    private void runUpdates() {
    	logger.trace("Updating motor {} - position: {}, target: {}, status: {}",
      				getName(), position, targetPosition, getStatus());
    
      updatePosition();
    }
    
  • Run at a fixed rate using the Async scheduleAtFixedRate() method:

    updateProcess = Async.scheduleAtFixedRate(
      				this::runUpdates,
      				200,
      				500,
      				MILLISECONDS,
      				"%s - Motor update"
      				getName());
    

Here the arguments are: the task, delay to starting, interval, unit of time for delay and interval, name format for the thread, argument object to format the name with (i.e. the name of the motor). This task will return a value which can be used to determine whether the motor has reached its final position or if there has been an issue.

EXAMPLE 3: Run sequential continuous moves

This example create a Runnable to move a mechanism sequentially given a list of moves.

  • Create the Runnable which defines what actually needs to happen by overriding the run() method.

    private class MoveRunner implements Runnable {
    
      private final List<Move> moves;
    
      public MoveRunner(final List<Move> moves) {
      	this.moves = moves;
      }
    
      @Override
      public void run() {
      	performingSequentialMoves = true;
      	try {
      		 do something to move
      	} catch (DeviceException e) {
      		// catch response
      	} 
      }
    }
    
  • Execute the Runnable giving the list of moves as an argument to the constructor.

    Async.execute(new MoveRunner(sequentialMoves));
    

20.4. Scan Messaging Service

The MessagingService provided an interface that allows simple objects to be sent as messages for any non-GDA application listening to it. A ScanMessage can be constructed during a scan which incorporates information about the scan and current progress. It can be used to update listeners on when the scan starts and finishes.

If scan progress information is required by some piece of code or application it is strongly recommended that this Messaging Service is used as a means of listening and obtaining the information as opposed to trying to query or interact with the actual scan process.

20.5. Finding Services

Much of the functionality of GDA is accessible via OSGi services. This is especially true for new scanning, new nexus writing and the parts of DAWN that are used in GDA. Typically a service will be defined by an interface and implemented - often in a different plug-in by an implementation class, e.g. IRunnableDeviceService vs RunnableDeviceServiceImpl. Services can be accessed via the ServiceProvider by calling the method getService with the required service class object, as below:

final IEventService eventService = ServiceProvider.getService(IEventService.class)

The getService method will throw an IllegalArgumentException if no service is registered for the given service class object.

When running GDA, this class looks up the requested service in the OSGi context. When writing JUnit tests you may need to set up services in the ServiceProvider by calling the setService method. Any test that uses ServiceProvider should always call ServiceProvider.reset() in its teardown method (e.g. typically the method annotated with @AfterAll in JUnit 5). This will clear the ServiceProvider for the next test. If this is not done this may cause a build failure in Jenkins, as registering a service with an interface for which the ServiceProvider has a service will cause an exception to be thrown. This exception will include the stack trace of the call that initially registered the service, to make it easier to find.

In Jython scripts the service provider can be accessed by first importing it as below:

from uk.ac.diamond.osgi.services import ServiceProvider
from org.eclipse.scanning.api.device import IRunnableDeviceService
runDeviceService = ServiceProvider.getService(IRunnableDeviceService)

20.6. Scripting Framework

In general it would be good practice to have a more clear cut separation between the scripts created and used by beamline scientists to run their experiments and GDA. While each beamline has its own unique requirements, there are many tasks that are common to most and so some sort of UI framework to accommodate these task could be developed with extra scripting ability available on top, e.g. a gateway script where specifics are added by the beamline staff.

One such example of a task that this could be applied to is the changing of the beam energy. The beam energy delivered to the beamline is fixed but can be altered using mirrors on the actual beamline. In the current environment scientists have Python scripts that will move the mirrors or other Scannables to the correct place to achieve the desired energy.

20.6.1. Tomography

A tomography experiment will carry out a series of scans which are then sent off for processing to build up a 3D image. Before the experiment a set of dark and flat fields are taken (the former involving no beam and the latter with the beam but no sample in position). These may also be taken midway through and/or at the end of the experiment.

All tomography experiments will take N x number of darks and M x number of flats, then take actual data and then N2 x number of darks and M2 number of flats (where N2 and M2 could be 0). All experiments will also involve the rotation of the sample. What will differ across beamlines is the actual number of positions and the angles to rotate at. Equally the format of the scan may vary as they may want to can multiple times, restart from the beginning or continue round.

Currently each beamline will create and use their own Python scripts to describe the scans that they want to do. While certain parts of the scan will be unique to each beamline/experiment, there are many parts which are generic for all (as detailed above). A basic scan can be performed from the Tomography UI screen where users can input specific parameters and click submit to run the scan. Behind the scenes this calls a selection of Python scripts that carry out the scan.

However there should be a more clear cut separation between the scientists working on the beamline and GDA. A framework of scripts and plugins could be created to perform the core tasks common to all Tomography experiments, which beamline users have access to and can wrap then into their own Python scripts should they require additional functionality.

20.7. Scannables and Monitors

The GDA Monitor provides feedback on an element of the scan. For example, the temperature of a device might have to be monitored during the scan. They observe a value from a device but have no control over the underlying device. The Monitor class extends the Scannable class (although note that a Monitor actually has less functionality than a Scannable). This is because the Scannable class already contains the functionality to getPosition() for the device in question. In the above example, the temperature of a device is not what is scanned meaning it is not a Scannable, but this variable may well change during the scan and as a result of it, hence the need for the Monitor class.

A Monitor can be added to scan and configured to monitor items per scan (i.e. read back a value once for each scan) or per point (i.e. for each scan point/step). In New scanning this is distinguished be the terms ‘per point’ and ‘per scan’, whereas in Old scanning the monitor is added at the end of the scan description (for per point) or via the scan metadata (for per scan).

Other such examples include using EpicsMonitors (extends MonitorBase which implements Monitor) to monitor the ring energy, ring current, topup etc. These can also be grouped together as a ScannableGroup and defined as such in Spring.

Monitors and Scannables can be used as simple in and out processors (Scannable), where information can be fed in and read out. Because of this they are often used on beamlines as a ‘mutator’.

An example in gda-core is the ScaledScannable (gda.devices.scannable) which extends ScannableBase (which itself implements Scannable). In this Scannable the position to move the device to is scaled by some factor, e.g.

@Override
public void asynchronousMoveTo(Object externalPosition) throws DeviceException {
	final Double[] convertedPos = ScannableUtils.objectToArray(externalPosition);
	final double actualPos = convertedPos[0] / scalingFactor;
	scannable.asynchronousMoveTo(actualPos);
}

and equally the value returned from the getPosition() method is also mutated in a similar way:

@Override
public Object getPosition() throws DeviceException {
	final double actualPos = ScannableUtils.getCurrentPositionArray(scannable)[0];
	final double convertedPos = actualPos * scalingFactor;
	return convertedPos;
}

20.7.1. Diffraction Scannables

Another commonly used example of using Scannables as mutators is in diffraction experiments where scientist want to work in reciprocal space (h,k,l) instead of Cartesian x, y, z which are the coordinates that describe the devices movements. In this case a ScannableGroup can be defined (which is itself a Scannable). This provides composite motion so that a move in x, y and z can be defined by a single projection. Specific maths can then be inserted into the defined ScannableGroup so that scientist can specify the movement they want in the hkl reciprocal space which is then converted to a movement in xyz which is sent to the devices.

Diffcalc is used to perform the calculations for controlling diffractometers within reciprocal lattice space. It is a Python/Jython based calculator. For more details see here.

20.7.2. Adding metadata and entries to a NeXus file

Once other example of using Scannable a mutators has been demonstrated by the i15-1 beamline who have created Scannables to add metadata information to their scans and unique entries to the NeXus file that is consequently created. The section on NeXus File Writing covers this implementation.

20.8. Devices

20.8.1. Usage within Beamlines

A list of all devices specified within beamline configurations is available here. The list is ordered by usage (i.e. how many times they are specified). The beamlines where each are in use is also displayed. Devices that have been labelled as the preferred implementation are also highlighted (see below).

This list can be used to identify both the most commonly used Devices (hence recommended) and those which have very few usages which perhaps could be removed or merged with another. Devices labelled with ‘Preferred Method’ have been identified as the recommended Device to use for a given implementation by a specific comment that has been added to their javadoc (see below).

Low usage

There are essentially two reasons that the Device might have a very small use count.

  1. They provide a very specific task that is really only a feature of one beamline and not used by others. If this is the case then obviously this Device and its count is warranted.

  2. The Device was created for a task that may be similar to that carried out by another Device but differs in a few small areas that provide specific functionality for that beamline. In this case it may be worth seeing how the two could be combined to cover all required functionality into one Device.

  3. The Device was created for a task that may be similar to that carried out by a pre-exisitng Device and the developer did not realise that it existed. In this case it is essentially a duplicate and if it has no specific reason for use over another existing Device then it should be removed.

Further investigation in these Devices is required.

Recommended Devices

In many cases there are Devices that should be used to carry out certain tasks

  • i.e. ‘preferred’ methods of implementation. These will often be the Devices with the highest counts (although not always if a new implementation has yet to be fully adopted). One example is the RMIAutomatedExporter used to export objects over RMI and the recommended approach as it does not require having to list all of the objects as beans.

If a Device is identified as being the recommended/preferred implementation then a note should be made in the JavaDoc to reflect this. The standard template for the ‘preferred’ method note should take the form:

 * <b><font color="green"> This is the PREFERRED method for <job this performs>.</font></b>

A link to a section in the GDA Developers Guide can also be added as such:

 * <b><font color="green"> This is the PREFERRED method for <job this performs>. For more details on usage see
 * <a href="https://alfred.diamond.ac.uk/documentation/manuals/GDA_Developer_Guide/master/<page>.html#<section>"> here.
 * </a></font></b>

IMPORTANT: In order for the Device to be labelled in the Device list generate above as the ‘Preferred Method’, it MUST contain the following string in the javadoc:

This is the PREFERRED method

For example take the RMIAutomatedExporter. Javadoc source code:

<b><font color="green"> This is the PREFERRED method for exporting objects over 
RMI. For more details on usage see
<a href="https://alfred.diamond.ac.uk/documentation/manuals/GDA_Developer_Guide/master/remoting.html#automated-rmi-exporting"> here.</a></font></b>

Output:

This is the PREFERRED method for exporting objects over RMI. For more details on usage see here.

20.8.2. EnumPositioners

Below is a list of existing EnumPositioners currently in use on beamlines (as of August 2019). The list is in order of usage. For the exact usage value see the Device list attached to the section above.

EnumPositioner

Beamlines

–> High usage (81 - 500+)

gda.device.enumpositioner.DummyEnumPositioner

b07-1, b16, b18, b21, i02-1, i02-2, i04-1, i05, i05-1, i06, i06-1, i07, i08, i08-1, i09, i09-1, i09-2, i10, i11, i11-1, i12, i13, i13-1, i14, i18, i19-1, i19-2, i19-shared, i20, i20-1, i21, i22, i23, i24, k11, lab44, mx, optics

gda.device.enumpositioner.EpicsPneumaticCallback

b07-1, b18, b21, i02-2, i04-1, i05-1, i06, i06-1, i07, i08, i09, i09-1, i09-2, i10, i11, i11-1, i12, i13, i13-1, i14, i15, i19-1, i19-2, i19-shared, i20, i21, i22, i23, i24, mx

gda.device.enumpositioner.EpicsPositionerCallback

b07-1, b18, i06-1, i07, i08, i09, i09-1, i09-2, i11, i11-1, i13, i18, i20, i20-1, i21, i23, i24

gda.device.enumpositioner.EpicsPositioner

i02-2, i04-1, i05, i05-1, i06-1, i07, i08-1, i10, i12, i13, i13-1, i14, i19-2, i20-1, i23, i24, k11, optics

gda.device.enumpositioner.DummyValve

b18, b21, i02-1, i06, i09, i09-1, i09-2, i11, i11-1, i13, i13-1, i14, i18, i19-2, i20, i21, i22, i23, i24, lab44, mx

–> Medium usage (11 - 80)

gda.device.enumpositioner.EpicsSimplePositioner

b18, i18, i20, i20-1, i23, i24, k11, mx

gda.device.enumpositioner.EpicsSimpleBinary

b07-1, b18, i02-1, i04-1, i05-1, i09, i09-1, i09-2, i19-1, i19-2, i23, i24, mx

gda.device.enumpositioner.DummyNamedEnumPositioner

i09, i09-1, i09-2, i11, i21

gda.device.enumpositioner.EpicsSimpleMbbinary

i02-1, i06, i10, i11, i11-1, i12, i13, i13-1, i21, i22, mx

gda.spring.EpicsEnumPositionerFactoryBean

b21, b24, i05, i05-1, i22

gda.device.enumpositioner.EpicsCurrAmpQuadController

i06, i07, i10, i11, i11-1, i24

–> Low usage (1 - 10)

uk.ac.gda.dls.client.views.EnumPositionerCompositeFactory

i06-1

gda.device.enumpositioner.EpicsLimitBasedPositioner

i02-2

gda.device.enumpositioner.EpicsAirBearingControl

i20-1, i21

gda.device.enumpositioner.EpicsSimplePneumatic

i06-1

gda.device.enumpositioner.EpicsEditablePositionerCallback

i20, i20-1

gda.device.enumpositioner.IntegerMapperBasedEnumPositioner

i13, i13-1

gda.device.enumpositioner.SelectorControlledEnumValueMapper

i13, i13-1

gda.device.enumpositioner.EpicsValveCallback

i05, i20, i20-1

gda.device.enumpositioner.NameMappedEpicsPneumaticCallback

i20

gda.device.enumpositioner.EpicsValve

i18, i20, i20-1

gda.device.enumpositioner.DummyPersistentNamedEnumPositioner

p45