14. New Scanning¶

14.1. Introduction¶

The New Scanning framework (also known as ‘Solstice Scanning’) was introduced around 2016 as an improvement on existing scanning (sometimes called classic scanning). It was introduced intially for mapping scans as part of the mapping project, together with Malcolm. Scanning using this new framework can be launched from the Mapping Perspective within the GDA Client, or via an mscan command (see GDA User guide).

The new scanning framework makes extensive use of OSGi services, instead of singletons or static methods, as well as using beans to represent data objects separate from classes that perform complex tasks. An introduction to OSGi services can be found here. An instance of an OSGi service can be accessed via dependency injection (common in UI code), via the BundleContext of the current plug-in, or from a ‘service holder’ class.

14.2. Devices¶

New scanning, just like classic scanning, has two types of devices: scannables and detectors. A scan consists of moving scannables to a series of positions, and at each position taking an exposure with the detectors. In the new scanning framework, the interfaces that define scannables and detectors are IScannable and IRunnableDevice respectively.

Both these interfaces extend the interfaces ILevel and INameable which provide devices with the properties level and name respectively. The name of a device is just a String that can be used to look up the device from the the appropriate service. The level property is used to indicate the order in which an action should be performed on a group of devices. These concepts are discussed further later in this document.

14.2.1. Scannables¶

A scannable has a position which can be any type, but is typically a Java Double. IScannable defines the methods getPosition() to get the current position of the scannable, and setPosition to move the scannable to a particular position. Scannables can also optionally have units and minimum and maximum positions, as returned by getUnits(), getMinimum() and getMaxiumum().

Scannables should be registered with the IScannableDeviceService by calling the register method. This is typically done via spring XML configuration, e.g.:

	<bean id="beam" class="uk.ac.diamond.daq.scanning.BeamScannable" init-method="register">
		<property name="name" value="beam" />
		<property name="beamSize" value="0.1"/>
	</bean>

The scannable in the example above can be looked by calling IScannableDeviceService.getScannable("beam").

In practice it is rare to declare a scannable this way. This is because new scanning supports the gda.device.Scannables used in old scanning. IScannableDeviceService is implemented by the class ScannableDeviceConnectorService, which is declared as an OSGi service. This class is actually within the uk.ac.gda.core plug-in project and so has access to classic Scannables.

If getScannable() is called for a particular scannable name, and no scannable is registered with the service with that name, then the getScannable() method will search first the Finder and then the jython namespace for a scannable with that name. If one is found, it is wrapped in a ScannableDeviceWrapper (a wrapper class for the scannable that implements IScannable) and placed in a cache for future use.

14.2.2. Detectors¶

The IRunnableDevice interface that represents a detector has a run() method (as you might expect). This is the method that is called in a scan to expose the detector. An IRunnableDevice can also represent a Malcolm Device (see below), in which case the run() method causes the malcolm device to run the scan it has been configured with.

It also has an associated model object. This is of some class that implements IDetectorModel. The model is used to configure those properties of a detector that can be configured. The actual model class depends on the type of the detector. The interface IDetectorModel defines method to get and set exposure time.

Detectors should be registered with the IRunnableDeviceService by calling the register() method. This is typically done via spring XML:

	<bean id="pixis_add" class="uk.ac.diamond.daq.detectors.addetector.AreaDetectorRunnableDevice" init-method="register">
		<property name="name" value="pixis_addetector" />
		<property name="model" ref="pixisModel" />
		<property name="deviceInformation" ref="pixisInfo" />
	</bean>

The detector declared above can be looked up at runtime by calling IRunnableDeviceService.getRunnableDevice("pixis_adddetector")

IRunnableDeviceService is implemented by the class RunnableDeviceServiceImpl, which is declared as OSGi service. Unlike the implementation of IScannableDeviceService, this class is fully part of the new scanning framework. It is not aware of classic detectors, so unlike classic scannable, then are not automatically wrapped in the new interface. To use a detector in a new scanning scan it must be explicitly registered with the service (except when used via malcolm, in which case it is the MalcolmDevice must be explicitly registered with the service).

The IRunnableDeviceService defines the method getDeviceInformation(String) which can be used to get device information for a particular detector as a DeviceInformation object. The method getDeviceInformation() returns a collection of the DeviceInformation objects for all registered devices. A DeviceInformation has methods to get the value the name, level, label, description, icon, as well as properties such as the state that are particularly important for malcolm devices.

14.2.3. Malcolm¶

New scanning was designed to support Malcolm. This is a layer of software, written in python, that sits between GDA and the EPICS layer that controls the hardware (hence the name Malcolm, from ‘middle-layer’). Malcolm can perform fast scans, moving motors and exposing the detectors.

In new scanning, a type of scan defined in Malcolm is called a ‘malcolm device’. A malcolm device has a name, a collection of detectors and a number of available ‘axes’ (i.e. motors).
If a malcolm device is included in a scan (see later), the scanning framework will delegate to malcolm to perform the whole scan. If an outer non-malcolm controlled axis is included, it will perform a scan for each position of that axis within the overall scan. Each sub-scan performed by malcolm is referred to as an inner scan, while the scan over the outer axis is called an outer scan.

A malcolm device is represented by an instance of the class MalcolmDevice. This class extends AbstractRunnableDevice implements the interface IMalcolmDevice which itself extends IRunnableDevice. Since a malcolm device is a type of runnable device, the rest of the new scanning framework outside of the MalcolmDevice class largely treats it the same as other runnable devices (i.e. detectors), with special-case treatment kept to a minimum.

As for detectors, malcolm devices must be registered with the IRunnableDeviceService. This is typically done via spring XML configuration, e.g.:

	<bean id="malcolmDeviceXmap" class="org.eclipse.scanning.malcolm.core.MalcolmDevice" init-method="register">
		<property name="name" value="BL08I-ML-SCAN-02" />
		<property name="model" ref="malcolmModelXmap" />
		<property name="deviceInformation" ref="malcolmInfoXmap" />
	</bean>

A MalcolmDevice object deals with communicating with the actual malcolm device, which it does using an EPICS v4 connection layer. It has methods to run a malcolm scan (the inner scan), (the run method inherited from IRunnableDevice), as well as to pause, resume, abort a malcolm scan, etc.

See Malcolm Device for more details on malcolm devices.

14.2.4. Annotations¶

A device may wish to be notified and take action at some other point in the scan that that defined by the two interfaces above, i.e. at the start or end of a point, at the end of the scan, when a scan is paused or resumed. A number of annotation types are defined in the package org.eclipse.scanning.api.annotation.scan that a method can be annotated with. This will cause the method to be called at the appropriate point in the scan. For example, if a device class has a method annotated with @PointStart it will be called at the start of a new scan point.

Annotated method can take arguments that describe the current context. These will be matched depending on the argument type. For example if the method has an argument of type ScanInformation, it will be passed the ScanInformation object describing the scan. The context here includes the OSGi context as well as additional object related to the current scan.

The code below gives an example of using scan annotations for a scannable>

public class Fred implements IScannable {
     @ScanStart
     public final void prepareScan(ScanInformation scanInfo, IPointGenerator<?> pointGen) throws Exception {
     	// by declaring arguments of type ScanInformation and IPointGenerator, we can
     	// get access to these scan 
        ...
     }
     
     @ScanFinally
     public void dispose() {
        ...
     }
     ...
 }

Annotations are managed and triggered by an AnnotationManager object created as part of running as scan. How AnnotationManager is invoked and some of the annotations it supports are discussed further as part of the Running a scan section below. The class DeviceAnnotations includes references to all available annotations. See that class as well as the referenced annotation types and their Javadoc for more details.

14.3. Point Generation¶

14.3.1. Overview¶

One of the most important parts of performing a scan, is generating the positions (also called points) to move the Scannables (e.g. motors) to before each detector exposure. In new scanning this is done by using a service called the point generator service to create a point generator from a model that describes the range of positions to generate. This model is called a points model, or sometimes a scan path model.

The new scanning framework contains an interface IPointGeneratorService that can generate points from a points model. The service is implemented by the class PointGeneratorService as is an OSGi service. The method IPointGeneratorService.createGenerator() can be called with a model class representing a scan path, and will return an IPointGenerator object that can be used to iterate through the points of the scan.

The IPointGenerator interface extends Iterable<IPosition>, where an IPosition represents an n-dimensional position, i.e. the a position over n axes, where each axis is the name of a Scannable. It encapsulates a Java Map<String, Object> where the key is the name the scannable and the value is the position of that scannable for this point.

The scan path model classes are simple beans with getter/setter methods. This architecture allows the new scanning framework to have a single scanning algorithm and therefore a single instance of a single class that implements that algorithm, independent of the type of scan or how many dimensions the scan has (i.e. scan axes).

A scan path model (also known as a points model) defines the path the scan should take over zero or more axes. All scan path model classes extends the interfaces IScanPathModel and IScanPointGeneratorModel. IScanPathModel.getScannableNames() can be called on any model to get the list of scan axes (scannable names) that will be moved by the path that the model describes. IScanPathModel also defines methods to add and remove property change listeners, useful for manipulating the models in a UI.

The interface IScanPointGeneratorModel defines getter/setter methods for the properties continuous and alternating and units.

In a continuous scan, the scan motors continue to move while the detector are exposed, if possible. In a non-continuous scan, the motors stop completely at each scan point. Continous scanning is not possible for all scan models.

If a scan path model defines a scan path within another one, i.e. a grid scan within an outer axis (see CompoundModel later), then the path described by this model is run backwards in every alternate position of the outer axis. This prevents the motors having to return to the starting point of this model for each inner step.

The units property is of type List<String>. The default units are ‘mm’. This property does not need to be set and rarely is in practice.

These properties are implemented in the abstract class AbstractPointsModel, which all concrete points model classes extend. However, they do not make sense for all model types. For example, the properties continuous and alternating do not make sense for a StaticModel as no motors are moved in the path described by this kind of model (the types of points model are discussed further below). The class AbstractPointsModel defines static methods, such as supportsContinuous() that return whether a particular model class supports a particular property.

Another property supported by many types of model is boundsToFit. This is defined by the interface IBoundsToFit. By default, this is false, meaning that the first and last points of a path are exactly the boundary of the region being scanned. If set, then the start and end points will be half a step inside the bounding region. For continuous scans, where the motors are always moving, this means the whole exposure for the first and final points are inside the bounding region.

14.3.2. Types of Scan Path Model¶

Most scan paths model classes described a specific type of scan path in a particular number of axes. These are best categorised by the number of axes that they move: 0, 1 or 2. The only model class without axes is StaticModel. Models which describe movement over a single axes implement IAxialModel, and those which describe movement over two axes implement IMapPathModel.

In addition there are model classes for models that can contain other models in order to combine the scan paths in some way.

StaticModel is the only scan path model class which has no axes, getScannableNames() returns an empty list of strings. It can represent a single point or multiple points, this can be set by calling setSize(). The IPosition(s) created for a StaticModel contain an empty map. Within a scan, when point is reached, it results in exposing the detector(s) without moving any motors. A type of scan that does this is called an acquisition scan.

14.3.2.1. Axial Models¶

All models classes which describe a path on a single axis implement the interface IAxialModel. Such models describe a path where a single scannable is moved at each point of the scan. All axial model classes extends AbstractAxialModel which implements the interface IBoundsToFit to include the boundsToFit property described above. In a mapping scan, an IAxialModel will describe the path of an outer scannable.

AxialStepModel A path defined by start and stop positions, and a step size; AxialPointsModel A path defined by start and stop positions, and a number of points; AxialArrayModel A path defined by an array of positions along a single axis; AxialMultiStepModel A path defined by multiple defined by multiple AxialStepModel`s whose resulting paths are to be iterated across sequentially. Note: all `AxialStepModel`s must be in the same axis. `OneAxisPointRepeatedModel A path consisting of a single position on an axis, to be repeated a set number of times;

14.3.2.2. Map Path Models¶

IMapPathModel is the super interface of all paths that have two axes. In other words, they define a path in two dimensional space. Note that this path itself may itself be 0, 1, or 2 dimensional, i.e. it may be defined by a point or line in two-dimensional path, as well as by some two-dimensional shape. Paths of this type are those that can be created in the mapping view within the mapping perpective, often these correspond to the axis of a sample stage.

The class AbstractMapModel is the abstract superclass of all map path models. It defines the properties xAxisName, yAxisName, xAxisUnits and yAxisUnits.

Paths that represent a one dimensional path in two dimensional space, i.e. a line from (x1, y1) to (x2, y2) are represented by an instance of a class that extends AbstractBoundingLineModel, which implements the interface IBoundingLineModel. This interface defines a single property: boundingLine, an instance of the class BoundingLine. That class defines the properties xStart, yStart, angle and length.

Paths that represent a two dimensional path in two dimensional space are represented by a instance of a class that extends AbstractBoundingBoxModel which implements the interface IBoundingBoxModel. This interface defines a single property: boundingBox and instance of the class BoundingBox. The class defines the properties xAxisStart, xAxisLength, yAxisStart and yAxisLength.

This represents a rectangular region that encompases all the points of the path defined by the model. Note that whatever 2d mapping region shape is selected in the mapping UI, this has been replaced by such a rectangular region by the time the ScanRequest containing the points model is created. The actual region drawn has been converted into a ScanRegion object added to the CompoundModel (see below).

TwoAxisPointSingleModel (Point) A path consisting of a single point in 2d space, defined by x and y positions TwoAxisLineStepModel BoundingLine A line path defined by a BoundingLine and a step size TwoAxisLinePointsModel BoundingLine A line path defined by a BoundingLine and a number of points TwoAxisGridStepModel BoundingBox A grid/raster path defined by a BoundingBox and step sizes in each of the x-axis and y-axis TwoAxisGridPointsModel BoundingBox A grid/raster path defined by a BoundingBox and number of points in each of the x-axis and y-axis TwoAxisGridPointsRandomOffsetModel BoundingBox As above, but with each points offset by a random amount, as defined by an offset property, defining the maximum offset TwoAxisSpiralModel BoundingBox A spiral path defined by a BoundingBox and a scale property which determines the distance between arcs of the spiral TwoAxisLissajousModel BoundingBox A path defining a Lissajous curve within a BoundingBox, defined by the properties a and b defining angular frequencies in each axis, and the number of points. See [Lissajous Curve](https://en.wikipedia.org/wiki/Lissajous_curve) for more details.

The two grid models above, TwoAxisGridStepModel and TwoAxisGridPointsModel extend the abstract class AbstractTwoAxisGridModel, which defines two additional properties, orientation and boundsToFit.

The property orientation is of the type Orientation, an enum type with the values HORIZONTAL and VERTICAL. This determines which of the x or y axes (i.e. the horizontal and vertical axes as plotted) is the fast axis and which is the slow axis. The fast axis is the axis that is scanned first. The default orientation is HORIZONTAL meaing that the x-axis is scanned first. If the orientation is set to VERTICAL then the y-axis is scanned first. The property boundsToFit is inherited from the interface IBoundsToFit described above.

14.3.2.3. Multi Models¶

Some types of path model define path models that contains other path models. These classes extend the abstract superclass AbstractMultiModel. AxialMultiStepModel, an axial model that contains multiple AxialStepModels has already been described above. The other are summarized in the table below.

ConcurrentMultiModel The paths defined by each contained model are run concurrently, i.e. the nth position of each point (conceptually a map) combined the nth position of each of the contained models, each of which must have the same length and have mutually exclusive axes. ConsecutiveMultiModel The paths defined by each contained model are run consecutively. All contained models must have the same axes. This is similar to AxialMultiStepModel, except that it can contain any type of child model. InterpolatedMultiScanModel A special subclass of ConsecutiveMultiModel that, but in between the paths for each model will move to an interpolation position. This is a position, likely in different axes that is not considered part of the scan. It can be used to open shutters, for example. CompoundModel The path defined by each inner model is repeated for each point of each outer model. Inner models are those that occur later in the list of models.

The most important type of multi-model is CompoundModel. This is the type of model that is used to describe a scan path in a ScanRequest (see below). Under the hood, any other type of model in a scan is replaced by a CompoundModel containing only that model.

A CompoundModel combines models so that the path of an inner model is performed for each point of an outer model. A CompoundModel can contain an arbitrary (non-zero) number of models, stored in a Java List where the last model in the list describes the innermost path, and the first model describes the outermost path. The overall path defined by the model is built up from by traversing the list backwards where the path described so far is repeated again for each point of the next outer model.

The CompoundModel class defines a number of additional properties besides the list of contained models. These are:

duration double The exposure time to be used for each point in the scan. regions Collection<ScanRegion> Regions define a geometric region of interest. Points outside of any region are excluded from the scan mutators List<IMutator> A mutator can modify the points of the scan in an artibrary way.

The duration property belongs to CompoundModel as this is the type of model used in a scan request. For malcolm scans, this is set to the exposure time of the malcolm device.

The regions property consists of a List of ScanRegions, where a ScanRegion encapsulates a ROI defining a region of interest together with a List<String> defining the axes names (i.e. scannable names) that the region of interest applies to. There are many types of ROI representing different geometrics shapes, e.g. CircularROI, PolygonalROI, RectangularROI. When a non-rectangular region is chosen for a mapping scan, this is converted to a ScanRegion containing the appropriate type of IROI and added here. This is necessary as an IMapPathModel only includes a BoundingBox or BoundingLine to define the region by default.

14.3.3. IPointGeneratorService and IPointGenerator¶

The IPointGeneratorService is an OSGi service that is used to create the IPointGenerator that iterates through the points. The simplest IPointGeneratorService method is createGenerator(T model) that takes a model and returns an IPointGenerator that can be used to iterate through the points of the model. IPointGeneratorService also contains a method that take a model class together with a IROI and return an IPointGenerator that will exclude points not within the region defined by the IROI. The method IPointGeneratorService.createCompoundGenerator(CompoundModel) create an IPointGenerator for the CompoundModel.

The IPointGeneratorService is implemented by the class PointGeneratorService. This class contructs the appropriate type of point generator for the model passed in. It contains a static map, Map<Class<? extends IScanPathModel>, Class<? extends IPointGenerator<?>>> modelToGenerator that is used to determine the point generator class to create. For example, for a AxialArrayModel and AxialArrayGenerator is created.

The interface IPointGenerator extends Iterable<IPosition> to enable iteration through the points that are generated. It also defines the method createPoints() that returns a List<IPosition> that can be used to return all the positions at once - this method is used in unit tests. The method getSize() returns the number of points the IPointGenerator will generate, while getShape() returns an int array specifying the number of positions for each dimension of the scan and getRank() returns the number dimensions. The method getNames() returns the names of all axes of the scan (as a List<String>), while getDimensionNames() returns a List<List<String>> of the axes names for each dimension index.

The dimensions of a scan indicate the dimensions to use for the datasets to be written to the nexus file for the scan. Note that the number of dimensions in a point generator does not necessarily equal the number of axes. Some dimensions may have multiple axes. For example, the point generator for a SpiralModel will have a single dimension consisting of both x- and y-axes. A grid generator may also be flattened into a single dimension if it has a non-rectangular region, or even if the detector is only capable of writing a single stack of images.

The actual implementation of points generators is written in jython. In GDA these classes are location in the folder scripts/scanpointgenerator in the org.eclipse.scanning.points plug-in project. For example spiralgenerator.py. Each Java generator class simply creates a jython generator and acts as a wrapper to it. The class ScanPointGeneratorFactory contains static methods that return a JythonObjectFactoryobject to create the appropriate Jython generator, e.g. ScanPointGenerator.JOneAxisArrayGeneratorFactory(). This is class that is implemented in Jython. It is then called with the appropriate arguments, depending on the type of object.

14.3.4. IPosition¶

An IPosition object describes the position of one or more named axes (i.e. Scannables). Conceptually it can be thought of as holding a map from axis name to a position for that axis. The method get(String), which returns an Object can be used to get the value for a particular axis name, As double is by far the most common type for an axis value, the method getDouble(String) can be used for convenience, this returns the value as a double, removing the need for a cast.

Besides having a value for each axis name, an IPosition also has an index for each axis name, of type int. This is the index of this position within the positions generated by the model for that axis, i.e. the indices for inner axis will reset to 0 when the outer axis moves on to the next position. For a (horizontal) grid scan, the axis for the x axis will reset to 0 each time the scan moves to a new line, unless the grid model is alternating, in which case the index for each point on the new line will decrement for each point starting from the maximum value back to 0. The method getIndex(String) can be used to get the index for an axis name, and getIndex(int) to get the index for a dimension index. The methods getStepIndex() returns the overall index of the position within those generated.

There are four implementations of IPosition. The classes StaticPosition, Scalar and IPoint are used for 0, 1 and 2 dimensional positions respectively. The class MapPosition covers the general case.

14.4. Defining a Scan¶

A scan is defined by a ScanBean, which contains a ScanRequest describing the scan. The other properties of a ScanBean describe the status of the scan.

A ScanRequest contains a CompoundModel describing the scan path for the scan. A CompoundModel contains one or more model objects describing the path for a particular scan axis or multiple axes. For a mapping scan, the inner most model (i.e. the last element of the map), will be an IMapPathModel, e.g. TwoAxisGridPointsModel. Any outer axes will be of some class that implements IAxialModel.

A ScanRequest also includes a Map<String, IDetectorModel> of detectors models keyed by detector name. This keys in the map are the names of the detectors to be included in the scan. The IDetectorModel value for each detector name is the model with which that detector will be configured before the scan starts. This will set the exposure time of the detector, since this is defined in IMalcolmModel, and any other properties depending on the model class (see Detectors above. For a malcolm scan, the detector name will be the name of a malcolm device, and the model will be a MalcolmModel.

The ScanRequest also includes numerous optional properties describing additional tasks to before before or after the main scan. For example a property to determine the path of the nexus file to write (see NeXus File Writing - this is determined automatically if not present). Other optional properties can be used to specify nexus templates to run (see NeXus Template Engine), the names of scripts to run before or after the scan, IPositions to move to before or after the scan, and a ProcessingRequest describing processing to perform. The ability to support live processing is a new feature that new scanning supports which classic scanning does not.

Another optional property of a ScanRequest is a list of ScanMetadata objects. These describe static metadata that can be added to the main nexus groups in the nexus file. Each contains a Map<String, Object> describing the metadata to add, as well as a type property of the enum type MetadataType. This property determines the group that metadata will be added to. It has the possible values ENTRY, SAMPLE, INSTRUMENT and USER.

14.5. Submitting a scan¶

The new scanning framework has its own implementation of a queue, defined by the interface IJobQueue and implemented by the class JobQueueImpl. The queue implementation can support multiple named queues, but for scanning we use a queue with the name org.eclipse.scanning.submission.queue. These queues are maintained within the event service, defined by the interface IEventService, and implemented by the class EventServiceImpl. The method IEvent.getJobQueue(String) can be called with name of the scanning queue (defined by the constant EventConstants.SUBMISSION_QUEUE) to get the IJobQueue object containing the scanning queue.

To submit a scan to the queue, call the submit method with a ScanBean containing the ScanRequest defining the scan. An ISubmitter can also be created by calling IEventService.createSubmitter with the URL of the activeMQ broker (which can be obtained by calling LocalProperties.getActiveMQBrokerURI()) and the name of the scanning queue as above. This also has a submit method, as well as a blockingSubmit method that will block until a scan is complete.

The queue is held in memory within the JobQueueImpl instance. In fact it has two ‘queues’: one for scans that have yet to be run, this is called the scanning submission queue; and one for scans that have started or completed, this is called the status queue. Scans (or rather the ScanBean objects that represent them) are added to the submission queue. They are moved to the status queue once they have started running.

The IJobQueue interface defines methods to run, stop, pause and resume the queue. It also has methods to manipulate the beans on the queue, such as remove, replace, moveForward and moveBackward. For more about the new scanning queue system, see here. Note as of writing the linked document is out of date.

14.6. Running a scan¶

The scanning IJobQueue instance is created by an instance of the class ScanServlet, which is declared in spring. Note that despite the name, this class is unrelated to javax.servlet.Servlet. This creates an IPublisher that can be used to publish ScanBeans to the status topic when they change. This is an ActiveMQ (JMS) queue named org.eclipse.scanning.queue.status.topic.

The IJobQueue runs a thread which takes the next ScanBean from the queue when one is available, and creates a new ScanProcess for that bean. The ScanProcess is also passed the IPublisher described above. The ScanProcess.execute() method is then called.

14.6.1. ScanProcess¶

An instance of ScanProcess is responsible for converting the ScanRequest bean to a ScanModel. This is then then given to an IScanDevice object to run. It also runs some additional tasks before and after the main scan. The difference between a ScanRequest and a ScanModel is that a ScanRequest is a bean containing primitive data types and their wrapper objects, Strings, collections and arrays of such types, as well as instances of other such bean classes. For example the ScanRequest contains the names of detector as well as models to configure them with, a CompoundModel to describe the scan path, and names of per-point and per-scan monitors. A ScanModel contains the actual IScannable and IRunnableDevice objects for such devices, as well as the IPointGenerator to use to create the IPositions for the scan.

The ScanProcess.execute() method first calls prepareScan. This is the method that creates and returns the ScanModel for the scan. First it decides the path for the nexus file and sets this property on the ScanBean. If the ScanRequest already specifies a file path, this is used directly, otherwise the method getNexusFilePath() it calls IFilePathService.getNexusPath(null) to get the next file path to use. IFilePathService is implemented by the class FilePathService and accessed as an OSGi service. This class is located in the uk.ac.gda.core plug-in project, i.e. among classic GDA classes rather than new scanning. It uses the value of the property gda.data.scan.datawriter.datadir as the directory to use, and a NumTracker to determine the scan number. The file name used is then determined by the pattern <beamlineName>-<scanNumber>.nxs.

14.6.1.1. Preparing the Scan¶

The ScanProcess.prepareScan() method then gets the CompoundModel from the ScanRequest that describes the scan path and uses the IPointGeneratorService to create the IPointGenerator that will be used to generate the points for the scan. The method checkAndFixMonitors(IPointGenerator) checks the lists of per-point and per-scan monitor names in the ScanRequest, removing any that appear as axes in the point generator. prepareScan()then calls createScanModel(IPointGenerator) with the point generator to create the ScanModel that will be used to run the scan.

The ScanModel is created by setting properties for the file path, the IPointGenerator and the CompoundModel. The scanMetadata property, List<ScanMetadata> is simply copied from the ScanRequest, as are the list of nexus template file paths.

The createScanModel method gets the actual scannable and detector objects that are only referenced by name in the ScanRequest and adds them to the ScanModel. These are are the IScannable objects for the per-point and per-scan monitors named in the ScanRequest and the IRunnableDevices for the detectors named. An exception is thrown if a device named in the ScanRequest cannot be found. createScanModel(IPointGenerator) then returns the ScanModel which prepareScan() then also returns and control is returned to the ScanProcess.execute() method.

The execute() method then creates the AnnotationManager, adding the devices in the scan to it, and calls its invoke(Class<? extends Annotation>, Object...) method with the PrepareScan annotation. This is the class responsible for calling methods on the device objects in the scan that are annotated with the given annotation, and is discussed further below.

The execute() method then move scannables to the start position, if specified by the ScanRequest.startPosition property, an IPosition. This is done by calling IPositioner.setPosition(IPosition) on an IPositioner object that was created by calling IRunnableDeviceService.createPositioner(String). IPositioner is implemented by ScannablePositioner, a type of LevelRunner runs devices in order of their levels. It the beforeScript property is specified, this is then run, by creating a ScriptRequest and calling IScriptService.execute(ScriptRequest). The only type of script supported is Jython (this is unlikely to change) and the class GDAJythonScriptService implements IScriptService for this purpose, using classic GDA API to do so.

The ScanProcess.execute() method then calls runScan(ScanModel) to actually run the scan. This calls the method createRunnableDevice(ScanModel) to create an IDeviceController, which wraps an IScanDevice which actually runs the scan. IDeviceController is a mechanism which mediates between the user and watchdogs to determine when to pause and resume the scan.

The method createRunnableDevice calls the method configureDetectors. This calls the configure(M) method on each detector, where M is the model type for the detector. Any @PreConfigure and @PostConfigure annotations are fired on this device before and after (respectively) this method is called. This can be used by the device to gain access to any scan objects that the device needs. For example this is how a malcolm device gets the IPointGenerator. It then creates the IScanDevice by using the IScanDeviceService OSGi service (which is actually implemented by the same RunnableDeviceServiceImpl that also implements IRunnableDeviceService). The IScanDevice is passed a reference to the IPublisher so that it can also publish updates to the ScanBean to the status topic.

The method createRunnableDevice also creates an IDeviceController (described above) that wraps the IScanDevice. It then calls configure(ScanModel) on the IScanDevice to prepare it to run the scan. This is described in further detail below.

The method runScanBlocking(IDeviceController, ScanModel) is then called which finally calls run() on the IDeviceController which in turns calls run() on the IScanDevice. Assuming no exception is thrown, the runScanBlocking method then runs the after script and moves to the scan end position, if either of these are specified.

If runScan() completes normally then returns back to ScanProcess.execute() which updates the ScanBean to indicate the scan has finished by setting percentage complete to 100 and the status to Status.COMPLETE.

The class ScanProcess also defines the methods pause, resume and terminate. These methods essentially call the methods on the IDeviceController with the same name. Depending on how this is mediated by the watchdogs, these will then be passed down to the IScanDevice that runs the scan.

14.6.1.2. AcquisitionDevice¶

The class that actually implements IScanDevice and runs the scan is AcquistionDevice. Is it an instance of this class that, during a software scan, iterates through the points of the scan, exposing the detectors. Note that in most JUnit tests that run scans, an instance of AcquistitionDevice is created and run directly without an instance of ScanProcess being involved. AcquisitionDevice is sufficient to run these scans, as the tests create a ScanModel directly, rather than creating a ScanRequest first, and they do not need to perform any of the additional tasks that ScanProcess runs before or after a scan, such as before/after scripts, or moving to start/end positions, etc.

14.6.1.2.1. Configuring the AcquisitionDevice¶

The configure(ScanModel) method is called as soon as the AcquistionDevice is created. The AcquistionDevice uses this method to prepare itself for the scan. After storing the ScanModel in a field, this method performs a number of tasks to configure itself according to the model. First, it looks up the IMalcolmDevice for the scan by name, if one is involved, and stores that in a field. Next, it creates an instance of AnnotationManager to use to call annotated methods, then an instance of LocationManager, a class that contains logic to do with calculating scan locations when there is an inner scan). Next it create an instance of IPositioner. This is the interface, implemented by ScannablePositioner that is use to move Scannables to their each position. It also creates the DeviceRunner and DeviceWriter instances used to run (expose) and write detectors. ScannablePositioner, DeviceRunner and DeviceWriter are all subclasses of the abstract class LevelRunner, which manages performing a particular task (i.e. moving to position, exposing detector and writing detector data, respectively), across the devices involved in order of the level of the devices.

The most important task of the configure method is to write the nexus structure for the scan. This is delegated to the configure method of NexusScanFileManager which up an in-memory representation of the nexus file called a nexus tree. Devices that implement INexusDevice will provide nexus objects (instances of some subclass of NXObject) that will be added to this tree, which is then saved to disk.

These nexus objects typically contain ILazyWritableDatasets that the devices keep a reference to in order to write to them at each point in the scan. All important device classes implement device INexusDevice or are wrapped by an object that does. A reference to the NexusScanFileManager is cached in order to call its close() method at the end of the scan.

Nexus writing is discussed in more detail here.

14.6.1.2.2. Running the AcquistionDevice¶

Finally, the run method of the AcquisitionDevice is called to run the scan. This is arguably the most important method of the whole of the new scanning framework. First, two pieces of housekeeping are performed: the ScanBean is updated with the start time, the status of Status.RUNNING and published to the status topic.

A LocationManager is used to create an Iterator<IPosition> from the IPointGenerator for the outer scan. The outer scan is that part of a malcolm scan that is performed in software. For example, for scan defined by a CompoundModel consisting of an IAxialModel for a software controlled scannable, e.g. energy, and a TwoAxisGridStepModel for a malcolm controlled x- and y-axes, the outer scan is defined by the IAxialModel and the inner scan by the TwoAxisGridStepModel.

The LocationManager uses a SubscanModerator to create an IPointGenerator for each of the inner scan, outer scan and the overall scan. This compares the axes of each model with the axes of the malcolm device to determine which models are controlled by malcolm and which are not. The LocationManager maintains this information as well as the current scan position and count.

We then begin to iterate through the positions of the scan using the Iterator<IPosition>. For each position, pos we perform the following tasks:

Call checkPausedOrAborted in case a request has been made to pause or abort the scan, see below;
Move the scannables to that position by calling IPositioner.setPosition(pos);
Call checkPausedOrAborted a second time in case a request has been made while the scannables were moving;
Wait for the DeviceWriter to complete writing for previous step (if applicable);
Call DeviceRunner.run(pos) to expose the detectors. This calls run(pos) on the IRunnableDevices;
Call DeviceWriters.run(pos, false), for detectors to write their data This calls write(pos) on any detector that implements IWriteableDetector;
Update and publish the ScanBean to reflect the completed position.

For a malcolm scan, the run(IPosition) method called by the DeviceRunner runs the inner malcolm scan, writing data into its external datasets as it goes along. For this reason MalcolmDevice does not implement IWriteableDetector and no action is taken when DeviceWriters.run(pos, false) is called.

The overall loop is wrapped in a try/catch/finally block. If an exception is thrown, processException is called to update the ScanBean accordingly, with a status of TERMINATED if the scan was aborted, and FAILED otherwise.

In the finally block, we close all the LevelRunners to release their threads, call fireEnd if the scan has finished normally to update and publish the ScanBean to reflect that, and use the AnnotationManager to fire any ScanFinally annotated device methods.

14.6.1.2.3. Pausing, Resuming and Aborting the AcquistionDevice¶

The methods AcquisitionDevice.abort(), pause() and resume() can be called to abort, pause and resume the scan being performed by the AcquisitionDevice respectively. The seek(int) method moves the scan to a particular step number, also pausing the scan.

As these methods will be called from (potentially multiple) threads other than the thread that is running the scan, they are performed while holding the stateChangeLock, a ReentrantLock. The pause() and seek() methods each set the awaitPause flag to true to signal that the scan should pause at the next possible opportunity, while the resume() and abort() methods call Condition.signalAll() on the shouldResumeCondition to cause any paused scan to wake up.

The abort method sets the aborted flag. It also calls abort() on all LevelRunners to interrupt any tasks they may be performing. The pause(), resume() and seek() methods will also call the method with the same name on the MalcolmDevice so that it can perform the appropriate action.

All these methods update the status of the ScanBean as appropriate and publish it to the status topic.

As mentioned above, for each scan position, checkPausedOrAborted() is called within the run() method. This first calls checkAborted() which simply throws an InterruptedException if the aborted flag has been set. It then calls checkPaused(). This method holds the stateChangeLock while it checks if the awaitPaused flag has been set. If so, it updates and published the ScanBean to show the status DeviceState.PAUSED, then calls Condition.wait() on the shouldResumeCondition. This will cause the main scan thread to wait until resume or abort are called from another thread.

Once this has happened, this method will continue. It will first call checkAborted in case the scan was aborted while it was paused. It then updates and publishes the ScanBean to show the status DeviceState.RESUMED.

These methods are called by the DeviceController if either the user (via ScanProcess) or a watchdog, such as the TopupWatchdog asks for the scan to be paused. The DeviceController class mediates between the watchdogs and ScanProcess to decide when the AcquistionDevice should be paused or resumed. For more information about DeviceController and the watchdogs, see here <https://confluence.diamond.ac.uk/display/CT/Watchdogs>.