20. Key Patterns¶
20.1. OSGi Classloading¶
A ClassLoader in Java is simply another Java class which is used to load class files. It takes a full qualified class name (FQCN) as a string and returns the Class object. However the stand Java ClassLoader does come with issues, for example if you wanted to use two versions of a library but both jars contain classes with the same fully qualified name, you cannot have both loaded at the same time.
OSGi (Open Service Gateway Initiative) is a Java framework which allows the development and deployment of modular software. In this framework, the ClassLoader is segmented into bundles (or plugins for Eclipse). Bundles are comprised of Java classes and other resources. They are the only entities for deploying Java-based applications in OSGi and provide functions to the end user. The main benefits of OSGi are that it reduces complexity, it is dynamic so you can swap bundles at runtime and also provides additional services (e.g. logging).
A bundle carries descriptive information about itself in the manifest file
located at <plugin.name>/META-INF/MANIFEST.MF
.
Below is an extract from the uk.ac.gda.core MANIFEST.MF
:
uk.ac.gda.core/META-INF
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: uk.ac.gda.core plugin
Bundle-SymbolicName: uk.ac.gda.core;singleton:=true
Bundle-Version: 9.12.0.qualifier
Require-Bundle: org.aopalliance,
javax.servlet;bundle-version="2.5.0",
uk.ac.gda.common,
Eclipse-RegisterBuddy: uk.ac.diamond.scisoft.analysis,
uk.ac.gda.api,
uk.ac.diamond.org.springframework
Eclipse-BuddyPolicy: registered
Export-Package: gda.beamline.beam,
gda.beamline.beamline,
gda.beamline.video,
gda.commandqueue,
gda.data
Bundle-ClassPath: .,
scripts/
Bundle-Vendor: Diamond Light Source
Bundle-RequiredExecutionEnvironment: JavaSE-1.8
Import-Package: javax.jms,
org.apache.commons.collections.map,
org.apache.commons.logging
Bundle-ActivationPolicy: lazy
Service-Component: OSGI-INF/*.xml
Bundle-Activator: uk.ac.gda.core.GDACoreActivator
Automatic-Module-Name: uk.ac.gda.core
See the specification for more details on each of the parameters contained in the file.
The class space is all the classes reachable from a given bundles class loader and so contains classes from:
The parent class loader - java.* packages from the boot class path.
Imported Packages (
Import-Package
, e.g.javax.jms
) - defines the constraints on the imports of shared packages. Many packages can be imported. Packages must be resolved.Required bundles (
Require-Bundle
, e.g.org.aopalliance
) - contains a list of required bundle symbolic names. It will take all exported packages from a required bundle at wire these to this bundle.The bundle’s classpath (
Bundle-ClassPath
,".", scripts/
) - defines a list of directories (or JAR file path names) that are inside the bundle that contain classes. The “.” specifies the root directory of the bundle.Attached fragments (
Fragment-Host
) - links the fragment to its potential hosts. E.g.uk.ac.diamond.daq.stage.test/META-INF/MANIFEST.MF:Fragment-Host: uk.ac.diamond.daq.stage;bundle-version="1.0.0"
A class must be consistent so never contain two classes with the same fully qualified name.
The OSGi classloading flowchart is shown below (taken from the OSGi Core Release document.)
20.1.1. Eclipse buddying¶
Eclipse buddying is Eclipse specific and so not directly implemented by OSGi. It is only implemented by Equinox (an implementation of the Eclipse OSGi framework). In terms of ordering, it is implemented at the end of the flowchart above, so if no dynamic import package is specified (dynamic import? -> no), then try budding. Note that this does imply that a bundle can only use with dynamic import package or buddying and not both.
Buddying is when bundles help each other out like ‘buddies’ and allows for the dynamic discovery and loading of classes. A bundle declares that it needs help from other bundles to load classes. When the bundle class loader fails to find the desired class through the normal OSGi model (above), its buddy policy is invoked. This discovers a set of buddies and consults each one until either the class is found or there are no more buddies to consult. This can all be set-up in the bundle manifest. For example in the above manifest for uk.ac.gda.core the following lines define the name of the bundle and set the Buddy policy to registered so all dependent bundles that explicitly register themselves as buddies to that bundle are consulted (more details).
Bundle-SymbolicName: uk.ac.gda.core
Eclipse-BuddyPolicy: registered
Then any bundle which can help manifest will have the lines:
Eclipse-RegisterBuddy: uk.ac.gda.core
Eclipse-BuddyPolicy: registered
For example, the uk.ac.gda.api bundle MANIFEST.MF contains these line. There
can be multiple bundles defined under the RegisteredBuddy
. For example, the
manifest from uk.ac.gda.core above states that it is also a
registered buddy for uk.ac.diamond.scisoft.analysis, uk.ac.gda.api and
uk.ac.diamond.org.springframework.
20.1.2. A note on Dynamic Import Package¶
The last stage in the flowchart is to check is
DynamicImport-Package
is called. Dynamic imports are matched to export
definitions to form package wirings during class loading. It works in a similar
way to Import-Package
except it is not required for the bundle to resolve the
module. They are used as a last resort if no other definition can be found.
Currently they are not used in GDA due to the issue of split packages.
20.1.3. Module-finding Libraries¶
Libraries like Jackson may dynamically search for and load modules with extra functionality, this may require additional configuration using GDA’s custom ClassLoaders.
For example, if adding some Jackson modules:
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jdk8</artifactId>
<version>2.12.1</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
<version>2.12.1</version>
<type>jar</type>
</dependency>
Some additional setup may be needed for an ObjectMapper
:
var mapper = new ObjectMapper();
var classLoader = GDAClassLoaderService
.getClassLoaderService()
.getClassLoaderForLibraryWithGlobalResourceLoading(
ObjectMapper.class,
Collections.emptySet());
mapper.registerModules(ObjectMapper.findModules(classLoader));
For OSGI applications the dependencies will need to be included in a feature. For unit testing purposes they will need to be included in the manifest of the test plugin (or one of its dependencies)
20.2. Queueing Scans¶
20.2.1. GDA9 (New Scanning) Queue System¶
A description and walk through of new style scanning can be found in the use case section.
20.2.1.1. Key Classes¶
There are three main classes in the GDA9 queue system. These are defined by the
four interfaces: IConsumer, IPublisher and
ISubscriber. The first of these relates processing a queue
using the IConsumer
which consumes beans from the queue. This interface
implements the interface IQueueConnection. IPublisher
and ISubscriber
define a publish/subscribe events system. These two
interfaces extends ITopicConnection.
The classes that implement each of these components are
ConsumerImpl, PublisherImpl and
SubscriberImpl respectively. ConsumerImpl
directly extends
AbstractConnection while PublisherImpl
and
SubsciberImpl
extends AbstractTopicConnection
which then extends AbstractConnection
.
The interface IEventService defines factory methods for all of these classes. The event service is an OSGi component that can injected, it is mostly accessed through a service holder. The implementing class is EventServiceImpl.
There can be multiple instances of each of these classes in the system,
possibly configured for different queues and/or topics. Additionally, they may
make use of each other. For example, ConsumerImpl
uses an ISubscriber
to
subscribe to the command topic.
20.2.1.2. ConsumerImpl
¶
The central object on the server is the ConsumerImpl. This
is essentially the queue as it contains the queue as a persistent object
implemented using MVStore (a persistent, log structures key-value store). The
queue is also backed up. There can be only one consumer containing items in the
submission queue. This consumer is created by the ScanServlet*
which is configured in Spring. ScanServlet
extends
AbstractConsumerServlet which defines most of its
behaviour. The only method that ScanServlet
needs to define itself is the
method to create a new process to run the consumed bean. This method is defined
in the interface IConsumerServlet. A method reference to
this method is passed to the consumers setRunner()
method in
AbstractConsumerServlet
. In this instance the ScanServlet
creates the
process ScanProcess which is used to process the bean and
perform the scan.
Aside on Servlets
* Note that the ScanServlet
is not a servlet by definition. It is just a
configurable/object that creates and starts a consumer. Potentially it could be
removed if the consumer was instead declared directly in Spring.
Equally note that there are other ‘Servlets’ that use queues including:
PositionServlet
- a servlet to handle aPositionRequest
to move to anIPosition
.AcquireServlet
- a servlet to acquire data from a particular detector.DeviceServlet
- a servlet to get the available devices from theIDeviceService
.
These all extend AbstractResponderServlet
which itself implements
IResponderServlet
(a Request/Response servlet) used for processing a queue.
It creates a response for each request processed for the given topic.
20.2.1.3. Queues and Topics¶
The core of the GDA9 queueing/events system uses ActiveMQ queues and topics (see here for the difference between the 2 types).
Name |
Type |
Value |
Description |
---|---|---|---|
Submission Queue |
Queue |
|
This is the queue that is contained in the |
Status Topic |
Topic |
|
Used to publish or listen for the bean’s status during running and processing. |
Command Topic |
Topic |
|
The consumer listens for |
Acknowledgement Topic |
Topic |
|
Used for acknowledgement from the consumer to commands sent on the command topic, e.g. client side submits a command and receives an acknowledgement on this topic. |
20.2.1.4. Server side usage¶
Creating the consumer
The API for a consumer is defined by the interface IConsumer and implemented by the class ConsumerImpl which extends AbstractConnection. The consumer ‘consumes’ the items from the contained queue and runs them. Consumers generally run on the server, and there should be only one consumer for each submission queue. Most consumers belong to servlets. The consumer for the default (scanning) submission queue is created by the
ScanServlet
which is defined in Spring. ScanServlet extends AbstractConsumerServlet which implements the interface IConsumerServlet.The
AbstractConsumerServlet.connect()
method is called. This callsIEventService.createConsumer()
to create a consumer with the URI broker, queues and topics. The implementation of this is EventServiceImpl which instantiates (and then returns) a newConsumerImpl
.The
ConsumerImpl
contains the queues internally and within the constructor it creates 2 SynchronizedModifiableIdQueue queues: one for the submission queue and another for the set of running and completed jobs (e.g. status queue). TheSynchronizedModifiableIdQueue
implement IPersistentModifiableIdQueue. The last line of the constructor callsconnect()
, which creates a publisher (PublisherImpl -> IPublisher) to the status topic and two subscribers (SubscriberImpl -> ISubscriber); one to the status topic and the other to the command topic.After creating the consumer, it is configured in the
AbstractConsumerServlet.connect()
method including setting therunner
which defines an IProcessCreator to create a process from a bean, e.g. a ScanProcess for a ScanBean.
Starting the consumer
Following configuration of the consumer,
AbstractConsumerServlet.connect()
callsstart()
on it. This method creates a new thread which simply calls therun()
method within a try/catch block that logs any thrown exceptions.The
ConsumerImpl.run()
method callsinit()
to initialise the consumer and starts the main consumer loop.[LEGACY: Finally,
AbstractConsumerServlet.connect()
creates a JMSQueueReader (from the IEventService method) and calls itsstart()
method which runs in a separate thread. This calls theJMSQueueReader.run()
and theninitialize()
to create a consumer for the JMS queue (a MessageConsumer). NOTE: the JMS queue is no longer used so this last point is essentially a legacy implementation that remains as some users still submit straight to the JMS queue from Python scripts.]
Running the consumer
After the call to
init()
in theConsumerImpl.run()
method, the main loop begins. The main loop runs as long asisActive()
returns true. For each loop of the event loop,consume()
is called. Within the loopcheckPaused()
is called first to check if the consumer should pause, then as long asisActive()
still returns true we callexecuteBean()
.The
executeBean()
method calls the runner’screateProcess()
method to create the IConsumerProcess to run for the bean. For scanning this will callScanServlet.createProcess()
which creates and returns a newScanProcess
.The
start()
method is called on the process which callsIConsumerProcess.execute()
->ScanProcess.execute()
which begins to process the bean and run the scan.
Submitting to the consumer
The
ConsumerImpl.submit()
method calls theadd()
method on the submission queue (SynchronizedModifiableIdQueue) to add the bean to the queue.
Controlling the consumer
When the consumer starts up, it creates a subscriber to the command topic. A listener, an instance of the inner class CommandListener, is added to the subscriber. The
CommandListener
implements the class IBeanListener. This has a methodbeanChangePerformed()
which is called by the subscriber when an appropriate bean is sent to the command topic.If the command is for this consumer it calls
processQueueCommand()
which contains aswitch
case for the different commands (represented by the enum QueueCommandBean.Command) received in the QueueCommandBean. E.g. PAUSE_QUEUE, RESUME_QUEUE, SUBMIT_JOB, GET_RUNNING_AND_COMPLETED, GET_INFO, (see class’s Java docs for all available commands). The consumer carries out actions according to the command.
For example:
PAUSE:
If
QueueCommandBean = PAUSE_QUEUE
, thepause()
method is called. This sets the variableawaitPause
to true. TheIPublisher
is used to publish this. During the next iteration ofconsume()
, thecheckPaused()
method will block and cause the consumer to wait until it has been lifted.
RESUME:
If
QueueCommandBean = RESUME_QUEUE
, theresume()
method is called. This sets the variableawaitPause
to false and publishes this notification using theIPublisher
. Now thecheckPaused()
method findsawaitPause = False
and is freed so resumes the running of the consumer.
20.2.1.5. Client side¶
The client uses the ConsumerProxy to submit to the queue. It
implements the same IConsumer
interface as the server (detailed above) and
sends commands on the command topic and receives replies on a command
acknowledgement topic. The commands are represented by a
QueueCommandBean which has an enum
QueueCommandBean.Command to identify the command.
This includes SUBMIT_JOB to submit a job. The bean to submit is
also contained in the QueueCommandBean
, in the jobBean
field.
20.2.2. Old Scanning Queue System¶
Old scanning uses a different system for queuing scans. This is predominately used in MX scanning (e.g. data collections, line scans, grid scans etc). It allows users to submit a scan and then queue up others to run.
The key class that manages the queue on the server side is
FindableProcessorQueue. It implement
IFindableQueueProcessor, which further extends
Processor and Queue. It can be defined in Spring as the
processor of a given queue. Most frequently used is the
CommandQueue which implements Queue
and handles
Command objects. Command
extends IObservable. The
Command
s are subsequently stored in CommandQueue
and processed by the
FindableProcessorQueue
.
A Spring object definition for the CommandQueue
is given below (taken from MX
server config).
<bean id="commandQueue" class = "gda.commandqueue.CommandQueue">
</bean>
<bean id="commandQueueProcessor" class =
"gda.commandqueue.FindableProcessorQueue">
<property name="queue" ref="commandQueue"/>
<property name="startImmediately" value="true"/>
</bean>
<gda:rmi-export
serviceName="gda/commandQueueProcessor"
service="commandQueueProcessor"
serviceInterface="gda.commandqueue.IFindableQueueProcessor" />
20.2.2.1. Server side¶
Set-up the queue processor
The method
setQueue()
is called during the GDA start-up on FindableProcessorQueue. The queue is defined in Spring as the CommandQueue (as above) and this is set as the queue thatFindableProcessorQueue
will process.Spring can also define whether the queue should start straight away by setting
startImmediately
totrue
. Thestart()
method withinFindableProcessorQueue
is called and a new Thread is started. This thread is managed byFindableProcessorQueue
through the methodsendCommandToManagerThread()
.
Running the queue processor
The
run()
method is called within FindableProcessorQueue. This will enter a loop to wait for a change to the queue or a change of the running status. It then calls the methodremoveQueueHead()
which calls the CommandQueueremoveHead()
method. This takes the top command off of the queue and returns it toFindableProcessorQueue
as a Command. TheCommand
returned is aJythonCommand
that contains a script file to run along with other attributes such as a description and status. E.g.JythonCommand [scriptFile=/tmp/JythonCommandRunnerCommand_7389848664369882100.py, description=Data collection. Prefix= test2_0_:NOT_STARTED]
Next
FindableProcessorQueue
callsrun()
on theCommand
. This calls therun()
method from JythonScriptFileRunnerCommand which gets the ICommandRunner (the JythonServerFacade) and calls therunScript()
from it with the script defined in the original command as the argument.The
JythonServerFacade
(which implementsICommandRunner
) then uses the JythonServer to run the script which will run the data collection/ grid scan etc.FindableProcessorQueue
will monitor the progress of the progress and notify any observers.FindableProcessorQueue
continues to remove commands from theCommandQueue
via theremoveQueueHead()
method until theCommandQueue
returnsnull
indicating that there are no more commands in the queue.
Class diagram
Managing the queue
FindableProcessorQueue also has control over the
queue and tasks. This allows the user to interact with the scan/queue, for
example, abort it, pause it, resume it etc. The calls made to the processor
FindableProcessorQueue
are initiated from the client side (see client
side for more details).
Pause:
The PauseCommandQueueHandler on the client side calls the
pause()
method in FindableProcessorQueue which, via thesendCommandToManagerThread()
method, will pause the task by callingpause()
on the Command (i.e. the JythonScriptFileRunnerCommand) which then gets the script and scan controllers to pauses the scan/script. This callspausesCurrentScan()
andpauseCurrentScript()
in JythonServerFacade which tells the JythonServer to pause. TheFindableProcessorQueue
will also pause the queue.
Stop:
The StopCommandQueueHandler on the client side calls the
stop()
method in FindableProcessorQueue which, via thesendCommandToManagerThread()
method, will stop the current task by sending a call toabort()
on the Command (i.e. the JythonScriptFileRunnerCommand) which then callsabortCommands()
on an ICommandAborter implemented by the JythonServerFacade which tells the JythonServer toabortCommands()
. TheFindableProcessorQueue
will also stop the queue.
Start:
The StartCommandQueueHandler on the client side calls the
start()
method in FindableProcessorQueue which, via thesendCommandToManagerThread()
method, will send a call toresume()
on theCommand
(i.e. theJythonScriptFileRunnerCommand
) if it had been in a paused state. The JythonScriptFileRunnerCommand callsresumeCurrentScan()
andresumeCurrentScript
on the ICurrentScanController and the IScriptController implemented by the JythonServerFacade which tells the JythonServer toresumeCurrentScan()
andresumeCurrentScript()
.
Skip the scan:
The SkipCommandQueueHandler on the client side calls the
skip()
method in FindableProcessorQueue which callsabort()
on the current Command being run. The does not disrupt the thread running so the nextCommand
will be taken from the CommandQueue.
Stop after current scan:
The StopAfterCurrentCommandQueueHandler on the client side calls the
stopAfterCurrent()
method in FindableProcessorQueue which creates a new StopCommand which is added to the CommandQueue and then moved to the top of the queue. This command will stop the processor.
20.2.2.2. Client side¶
The client side uses CommandQueueViewFactory to
display the contents and status of the queue. A client side manager must get
the queue from the CommandQueueViewFactory
and call addToTail()
on the
queue (CommandQueue method) providing a
CommandProvider as an argument. It uses the
JythonCommandCommandProvider (which implements
CommandProvider
) to create a command to run with a description and the other
elements needed. The CommandQueue.addToTail(CommandProvider)
method then
calls getCommand()
from the JythonCommandCommandProvider
to add the
Command to the CommandQueue
. These commands will subsequently be
picked off the queue by the server as described above.
An example of this implementation can be found in the MX DataCollectionScanManager.
The client side has a number of AbstractHandlers to manage user interactions:
The
execute()
method is triggered when the ‘pause queue’ button is pressed. It gets the processor from the CommandQueueViewFactory (FindableProcessorQueue) and calls thepause()
method.
The
execute()
method is triggered when the ‘stop queue’ button is pressed. It gets the processor from the CommandQueueViewFactory (FindableProcessorQueue) and calls thestop()
method.
The
execute()
method is triggered when the ‘skip scan’ button is pressed. It gets the processor from the CommandQueueViewFactory (FindableProcessorQueue) and calls theskip()
method.
The
execute()
method is triggered when the ‘Resume scan’ button is pressed. It gets the processor from the CommandQueueViewFactory (FindableProcessorQueue) and calls the start() method.
The
execute()
method is triggered when the ‘Stop after current scan’ button is pressed. It gets the processor from the CommandQueueViewFactory (FindableProcessorQueue) and calls thestopAfterCurrent()
method.
20.3. Asynchronous Threads - The Async Class¶
If tasks need to be run in asynchronously within a process then GDA developers
are strongly encouraged to use the Async.java
class. This is
available from the gda-core.git
repo and located in the
uk.ac.diamond.daq.concurrent
package within.
The Async class runs tasks in their own thread. Existing idle threads will be reused if possible to minimise the overhead of creating new threads. It manages an internal thread pool to do this. It uses an ExecutorService to manage asynchronous tasks which returns a ListenableFuture so that callbacks can be added. It can also run and manage repeating tasks using ScheduleExectuorServices.
More specifically the Async
class provides the ability to:
execute()
- run aRunnable
in its own thread.submit()
- submit a task to be run in its own thread and return aFuture
giving access to the completion or error state of the task. This means that task can be cancelled if necessary. TheFuture
also gives access to the return value of the Callable target. See below for more details or alternatively an explanation of the use ofFuture
s is available here.submitAll()
- submit a list of callables to be executed concurrently.executeAll()
- submit a collection ofRunnable
s to be executed concurrently.scheduleAtFixedRate()
- schedule a task to be run repeatedly at a fixed rate after an initial delay.schedule()
- schedule a task to run after a given delay.scheduleWithFixedDelay()
- schedule a task to be run repeatedly with a fixed delay between executions.
There are variants of each of the base methods given above which allow threads to be renamed or allow varying periods and/or times between executions.
Aside: A Future is a future result of an asynchronous computation. A result will eventually appear in the
Future
after the processing is complete. A FutureTask is a cancelable asynchronous computation and upon running it will execute the given callable. The use of theExecutorService
means that a reference to the running thread is maintained and so it can be cancelled and it can wait for it to finish before getting the result. Submitting aCallable
to the pool will return the value of the call method whenFuture.get()
is implemented.
20.3.1. Usage¶
Use the Async
class to manage your asynchronous threads/tasks as such:
Async.execute(() -> {
// do something
});
Users are strongly discouraged from creating their own threads. i.e. DO NOT USE
new Thread(new Runnable()) {
public void run() {
// do something
}
}
In the above example there is no callbacks available so the thread is simply started and then lost.
20.3.2. Examples¶
EXAMPLE 1: Readout of detectors
The following extract is taken from the ConcurrentScanChild()
class
(available in gda-core
) which contains the readoutDetectorsAndPublish()
method. This method creates a FutureTask
to readout the detectors using
parallel threads (asynchronously) into a ScanDataPoint
(during scanning). The
subtleties of running parallel threads so that each detector is readout in its
own thread are skipped here but see the actual for details on this if
interested.
Create the
FutureTask
and override thecall()
method with the tasks that should be carried out in the thread. The call method will run when the task is submitted.detectorReadoutTask = new FutureTask<Void>(new Callable<Void>() { List<Future<Object>> readoutTasks; /** * Readout each detector in a thread, add the resulting data to the * ScanDataPoint and publish. */ @Override public Void call() throws Exception { try { // do something } catch (Exception e) { // catch it } throw e; } return null; } });
Then execute this using
Async
:Async.execute(detectorReadoutTask);
EXAMPLE 2: Update the state of a motor (using the ScheduleExectorService
)
This is an example of how to schedule a command to run periodically so that the status of a motor can be reported on throughout a scan.
Create the target task to run. This task will log the motor positions and status.
private void runUpdates() { logger.trace("Updating motor {} - position: {}, target: {}, status: {}", getName(), position, targetPosition, getStatus()); updatePosition(); }
Run at a fixed rate using the
Async
scheduleAtFixedRate()
method:updateProcess = Async.scheduleAtFixedRate( this::runUpdates, 200, 500, MILLISECONDS, "%s - Motor update" getName());
Here the arguments are: the task, delay to starting, interval, unit of time for delay and interval, name format for the thread, argument object to format the name with (i.e. the name of the motor). This task will return a value which can be used to determine whether the motor has reached its final position or if there has been an issue.
EXAMPLE 3: Run sequential continuous moves
This example create a Runnable
to move a mechanism sequentially given a list
of moves.
Create the
Runnable
which defines what actually needs to happen by overriding the run() method.private class MoveRunner implements Runnable { private final List<Move> moves; public MoveRunner(final List<Move> moves) { this.moves = moves; } @Override public void run() { performingSequentialMoves = true; try { do something to move } catch (DeviceException e) { // catch response } } }
Execute the
Runnable
giving the list of moves as an argument to the constructor.Async.execute(new MoveRunner(sequentialMoves));
20.4. Scan Messaging Service¶
The MessagingService
provided an interface that allows simple objects to be
sent as messages for any non-GDA application listening to it. A ScanMessage
can be constructed during a scan which incorporates information about the scan
and current progress. It can be used to update listeners on when the scan
starts and finishes.
If scan progress information is required by some piece of code or application it is strongly recommended that this Messaging Service is used as a means of listening and obtaining the information as opposed to trying to query or interact with the actual scan process.
20.5. Finding Services¶
Much of the functionality of GDA is accessible via
OSGi services.
This is especially true for new scanning, new nexus writing and the parts of
DAWN that are used in GDA. Typically a service will be defined by an interface and
implemented - often in a different plug-in by an implementation class,
e.g. IRunnableDeviceService
vs RunnableDeviceServiceImpl
.
Services can be accessed via the ServiceProvider
by calling the method
getService
with the required service class object, as below:
final IEventService eventService = ServiceProvider.getService(IEventService.class)
The getService
method will throw an IllegalArgumentException
if no service
is registered for the given service class object.
When running GDA, this class looks up the requested service in the OSGi context. When
writing JUnit tests you may need to set up services in the ServiceProvider
by
calling the setService
method. Any test that uses ServiceProvider
should always
call ServiceProvider.reset()
in its teardown method (e.g. typically the method
annotated with @AfterAll
in JUnit 5). This will clear the ServiceProvider
for
the next test. If this is not done this may cause a build failure in Jenkins, as
registering a service with an interface for which the ServiceProvider has a service
will cause an exception to be thrown. This exception will include the stack trace
of the call that initially registered the service, to make it easier to find.
In Jython scripts the service provider can be accessed by first importing it as below:
from uk.ac.diamond.osgi.services import ServiceProvider
from org.eclipse.scanning.api.device import IRunnableDeviceService
runDeviceService = ServiceProvider.getService(IRunnableDeviceService)
20.6. Scripting Framework¶
In general it would be good practice to have a more clear cut separation between the scripts created and used by beamline scientists to run their experiments and GDA. While each beamline has its own unique requirements, there are many tasks that are common to most and so some sort of UI framework to accommodate these task could be developed with extra scripting ability available on top, e.g. a gateway script where specifics are added by the beamline staff.
One such example of a task that this could be applied to is the changing of the
beam energy. The beam energy delivered to the beamline is fixed but can be
altered using mirrors on the actual beamline. In the current environment
scientists have Python scripts that will move the mirrors or other Scannable
s
to the correct place to achieve the desired energy.
20.6.1. Tomography¶
A tomography experiment will carry out a series of scans which are then sent off for processing to build up a 3D image. Before the experiment a set of dark and flat fields are taken (the former involving no beam and the latter with the beam but no sample in position). These may also be taken midway through and/or at the end of the experiment.
All tomography experiments will take N x number of darks and M x number of flats, then take actual data and then N2 x number of darks and M2 number of flats (where N2 and M2 could be 0). All experiments will also involve the rotation of the sample. What will differ across beamlines is the actual number of positions and the angles to rotate at. Equally the format of the scan may vary as they may want to can multiple times, restart from the beginning or continue round.
Currently each beamline will create and use their own Python scripts to describe the scans that they want to do. While certain parts of the scan will be unique to each beamline/experiment, there are many parts which are generic for all (as detailed above). A basic scan can be performed from the Tomography UI screen where users can input specific parameters and click submit to run the scan. Behind the scenes this calls a selection of Python scripts that carry out the scan.
However there should be a more clear cut separation between the scientists working on the beamline and GDA. A framework of scripts and plugins could be created to perform the core tasks common to all Tomography experiments, which beamline users have access to and can wrap then into their own Python scripts should they require additional functionality.
20.7. Scannables and Monitors¶
The GDA Monitor
provides feedback on an element of the scan. For example, the
temperature of a device might have to be monitored during the scan. They
observe a value from a device but have no control over the underlying device.
The Monitor class extends the Scannable
class (although note that a Monitor
actually has less functionality than a Scannable
). This is because the
Scannable
class already contains the functionality to getPosition() for the
device in question. In the above example, the temperature of a device is not
what is scanned meaning it is not a Scannable
, but this variable may well
change during the scan and as a result of it, hence the need for the Monitor
class.
A Monitor
can be added to scan and configured to monitor items per scan (i.e.
read back a value once for each scan) or per point (i.e. for each scan
point/step). In New scanning this is distinguished be the terms ‘per point’ and
‘per scan’, whereas in Old scanning the monitor is added at the end of the scan
description (for per point) or via the scan metadata (for per scan).
Other such examples include using EpicsMonitors
(extends MonitorBase
which
implements Monitor
) to monitor the ring energy, ring current, topup etc.
These can also be grouped together as a ScannableGroup
and defined as such in
Spring.
Monitor
s and Scannable
s can be used as simple in and out processors
(Scannable
), where information can be fed in and read out. Because of this
they are often used on beamlines as a ‘mutator’.
An example in gda-core
is the ScaledScannable
(gda.devices.scannable
)
which extends ScannableBase
(which itself implements Scannable
). In this
Scannable
the position to move the device to is scaled by some factor, e.g.
@Override
public void asynchronousMoveTo(Object externalPosition) throws DeviceException {
final Double[] convertedPos = ScannableUtils.objectToArray(externalPosition);
final double actualPos = convertedPos[0] / scalingFactor;
scannable.asynchronousMoveTo(actualPos);
}
and equally the value returned from the getPosition()
method is also mutated
in a similar way:
@Override
public Object getPosition() throws DeviceException {
final double actualPos = ScannableUtils.getCurrentPositionArray(scannable)[0];
final double convertedPos = actualPos * scalingFactor;
return convertedPos;
}
20.7.1. Diffraction Scannables¶
Another commonly used example of using Scannable
s as mutators is in
diffraction experiments where scientist want to work in reciprocal space
(h,k,l) instead of Cartesian x, y, z which are the coordinates that describe
the devices movements. In this case a ScannableGroup
can be defined (which is
itself a Scannable
). This provides composite motion so that a move in x, y
and z can be defined by a single projection. Specific maths can then be
inserted into the defined ScannableGroup
so that scientist can specify the
movement they want in the hkl reciprocal space which is then converted to a
movement in xyz which is sent to the devices.
Diffcalc
is used to perform the calculations for controlling diffractometers within reciprocal lattice space. It is a Python/Jython based calculator. For more details see here.
20.7.2. Adding metadata and entries to a NeXus file¶
Once other example of using Scannable
a mutators has been demonstrated by the
i15-1 beamline who have created Scannable
s to add metadata information to
their scans and unique entries to the NeXus file that is consequently created.
The section on
NeXus File Writing
covers this implementation.
20.8. Devices¶
20.8.1. Usage within Beamlines¶
A list of all devices specified within beamline configurations is available here. The list is ordered by usage (i.e. how many times they are specified). The beamlines where each are in use is also displayed. Devices that have been labelled as the preferred implementation are also highlighted (see below).
This list can be used to identify both the most commonly used Devices (hence recommended) and those which have very few usages which perhaps could be removed or merged with another. Devices labelled with ‘Preferred Method’ have been identified as the recommended Device to use for a given implementation by a specific comment that has been added to their javadoc (see below).
Low usage
There are essentially two reasons that the Device might have a very small use count.
They provide a very specific task that is really only a feature of one beamline and not used by others. If this is the case then obviously this Device and its count is warranted.
The Device was created for a task that may be similar to that carried out by another Device but differs in a few small areas that provide specific functionality for that beamline. In this case it may be worth seeing how the two could be combined to cover all required functionality into one Device.
The Device was created for a task that may be similar to that carried out by a pre-exisitng Device and the developer did not realise that it existed. In this case it is essentially a duplicate and if it has no specific reason for use over another existing Device then it should be removed.
Further investigation in these Devices is required.
In many cases there are Devices that should be used to carry out certain tasks
i.e. ‘preferred’ methods of implementation. These will often be the Devices with the highest counts (although not always if a new implementation has yet to be fully adopted). One example is the
RMIAutomatedExporter
used to export objects over RMI and the recommended approach as it does not require having to list all of the objects asbeans
.
If a Device is identified as being the recommended/preferred implementation then a note should be made in the JavaDoc to reflect this. The standard template for the ‘preferred’ method note should take the form:
* <b><font color="green"> This is the PREFERRED method for <job this performs>.</font></b>
A link to a section in the GDA Developers Guide can also be added as such:
* <b><font color="green"> This is the PREFERRED method for <job this performs>. For more details on usage see
* <a href="https://alfred.diamond.ac.uk/documentation/manuals/GDA_Developer_Guide/master/<page>.html#<section>"> here.
* </a></font></b>
IMPORTANT: In order for the Device to be labelled in the Device list generate above as the ‘Preferred Method’, it MUST contain the following string in the javadoc:
“This is the PREFERRED method”
For example take the RMIAutomatedExporter
. Javadoc source code:
<b><font color="green"> This is the PREFERRED method for exporting objects over
RMI. For more details on usage see
<a href="https://alfred.diamond.ac.uk/documentation/manuals/GDA_Developer_Guide/master/remoting.html#automated-rmi-exporting"> here.</a></font></b>
Output:
This is the PREFERRED method for exporting objects over RMI. For more details on usage see here.
20.8.2. EnumPositioners¶
Below is a list of existing EnumPositioners currently in use on beamlines (as of August 2019). The list is in order of usage. For the exact usage value see the Device list attached to the section above.
EnumPositioner |
Beamlines |
---|---|
–> High usage (81 - 500+) |
|
b07-1, b16, b18, b21, i02-1, i02-2, i04-1, i05, i05-1, i06, i06-1, i07, i08, i08-1, i09, i09-1, i09-2, i10, i11, i11-1, i12, i13, i13-1, i14, i18, i19-1, i19-2, i19-shared, i20, i20-1, i21, i22, i23, i24, k11, lab44, mx, optics |
|
b07-1, b18, b21, i02-2, i04-1, i05-1, i06, i06-1, i07, i08, i09, i09-1, i09-2, i10, i11, i11-1, i12, i13, i13-1, i14, i15, i19-1, i19-2, i19-shared, i20, i21, i22, i23, i24, mx |
|
b07-1, b18, i06-1, i07, i08, i09, i09-1, i09-2, i11, i11-1, i13, i18, i20, i20-1, i21, i23, i24 |
|
i02-2, i04-1, i05, i05-1, i06-1, i07, i08-1, i10, i12, i13, i13-1, i14, i19-2, i20-1, i23, i24, k11, optics |
|
b18, b21, i02-1, i06, i09, i09-1, i09-2, i11, i11-1, i13, i13-1, i14, i18, i19-2, i20, i21, i22, i23, i24, lab44, mx |
|
–> Medium usage (11 - 80) |
|
b18, i18, i20, i20-1, i23, i24, k11, mx |
|
b07-1, b18, i02-1, i04-1, i05-1, i09, i09-1, i09-2, i19-1, i19-2, i23, i24, mx |
|
i09, i09-1, i09-2, i11, i21 |
|
i02-1, i06, i10, i11, i11-1, i12, i13, i13-1, i21, i22, mx |
|
b21, b24, i05, i05-1, i22 |
|
i06, i07, i10, i11, i11-1, i24 |
|
–> Low usage (1 - 10) |
|
i06-1 |
|
i02-2 |
|
i20-1, i21 |
|
i06-1 |
|
i20, i20-1 |
|
i13, i13-1 |
|
i13, i13-1 |
|
i05, i20, i20-1 |
|
i20 |
|
i18, i20, i20-1 |
|
gda.device.enumpositioner.DummyPersistentNamedEnumPositioner |
p45 |