14. New Scanning¶
14.1. Introduction¶
The New Scanning framework (also known as ‘Solstice Scanning’) was
introduced around 2016 as an improvement on existing scanning
(sometimes called classic scanning). It was introduced intially for mapping scans
as part of the mapping project, together with Malcolm.
Scanning using this new framework can be launched from the Mapping Perspective
within the GDA Client, or via an mscan
command (see GDA User guide).
The new scanning framework makes extensive use of OSGi services, instead
of singletons or static methods, as well as using beans to
represent data objects separate from classes that perform complex tasks.
An introduction to OSGi services can be found here.
An instance of an OSGi service can be accessed via dependency injection
(common in UI code), via the BundleContext
of the current plug-in,
or from a ‘service holder’ class.
14.2. Devices¶
New scanning, just like classic scanning, has two types of devices: scannables
and detectors. A scan consists of moving scannables to
a series of positions, and at each position taking an exposure with the
detectors. In the new scanning framework, the interfaces that define
scannables and detectors are IScannable
and IRunnableDevice
respectively.
Both these interfaces extend the interfaces ILevel
and INameable
which
provide devices with the properties level
and name
respectively.
The name of a device is just a String
that can be used to look up the device
from the the appropriate service. The level property is used
to indicate the order in which an action should be performed on a group of
devices. These concepts are discussed further later in this document.
14.2.1. Scannables¶
A scannable has a position which can be any type, but is typically a
Java Double
. IScannable
defines the methods getPosition()
to
get the current position of the scannable, and setPosition
to move the
scannable to a particular position. Scannables can also optionally have
units and minimum and maximum positions, as returned by getUnits()
,
getMinimum()
and getMaxiumum()
.
Scannables should be registered with the IScannableDeviceService
by
calling the register
method. This is typically done via spring XML
configuration, e.g.:
<bean id="beam" class="uk.ac.diamond.daq.scanning.BeamScannable" init-method="register">
<property name="name" value="beam" />
<property name="beamSize" value="0.1"/>
</bean>
The scannable in the example above can be looked by calling
IScannableDeviceService.getScannable("beam")
.
In practice it is rare to declare a scannable this way. This is because new
scanning supports the gda.device.Scannable
s used in old scanning.
IScannableDeviceService
is implemented by the class
ScannableDeviceConnectorService
, which is declared as an OSGi service.
This class is actually within the uk.ac.gda.core
plug-in
project and so has access to classic Scannable
s.
If getScannable()
is called for a particular scannable name, and no scannable is
registered with the service with that name, then the getScannable()
method
will search first the Finder
and then the jython namespace for a
scannable with that name. If one is found, it is wrapped in a
ScannableDeviceWrapper
(a wrapper class for the scannable that implements
IScannable
) and placed in a cache for future use.
14.2.2. Detectors¶
The IRunnableDevice
interface that represents a detector has a run()
method
(as you might expect). This is the method that is called in a scan to expose the
detector. An IRunnableDevice
can also represent a Malcolm Device
(see below), in which case the run()
method causes the malcolm device to run the
scan it has been configured with.
It also has an associated model object. This is of some class that implements
IDetectorModel
. The model is used to configure those properties of a detector
that can be configured. The actual model class depends on the type of the detector.
The interface IDetectorModel
defines method to get and set
exposure time.
Detectors should be registered with the IRunnableDeviceService
by
calling the register()
method. This is typically done via spring XML:
<bean id="pixis_add" class="uk.ac.diamond.daq.detectors.addetector.AreaDetectorRunnableDevice" init-method="register">
<property name="name" value="pixis_addetector" />
<property name="model" ref="pixisModel" />
<property name="deviceInformation" ref="pixisInfo" />
</bean>
The detector declared above can be looked up at runtime by calling
IRunnableDeviceService.getRunnableDevice("pixis_adddetector")
IRunnableDeviceService
is implemented by the class
RunnableDeviceServiceImpl
, which is declared as OSGi service.
Unlike the implementation of IScannableDeviceService
, this class is
fully part of the new scanning framework. It is not aware of classic
detectors, so unlike classic scannable, then are not automatically wrapped
in the new interface. To use a detector in
a new scanning scan it must be explicitly registered with the service
(except when used via malcolm, in which case it is the MalcolmDevice
must be
explicitly registered with the service).
The IRunnableDeviceService
defines the method getDeviceInformation(String)
which can be used to get device information for a particular detector as a
DeviceInformation
object. The method getDeviceInformation()
returns
a collection of the DeviceInformation
objects for all registered devices.
A DeviceInformation
has methods to get the value the name, level, label,
description, icon, as well as properties such as the state that are
particularly important for malcolm devices.
14.2.3. Malcolm¶
New scanning was designed to support Malcolm. This is a layer of software, written in python, that sits between GDA and the EPICS layer that controls the hardware (hence the name Malcolm, from ‘middle-layer’). Malcolm can perform fast scans, moving motors and exposing the detectors.
In new scanning, a type of scan defined in Malcolm is called a
‘malcolm device’. A malcolm device has a name, a collection of detectors and
a number of available ‘axes’ (i.e. motors).
If a malcolm device is included in a scan (see later), the scanning framework
will delegate to malcolm to perform the whole scan. If an outer non-malcolm
controlled axis is included, it will perform a scan for each position of that
axis within the overall scan. Each sub-scan performed by malcolm is referred to
as an inner scan, while the scan over the outer axis is called an outer scan.
A malcolm device is represented by an instance of the class MalcolmDevice
.
This class extends AbstractRunnableDevice
implements the interface
IMalcolmDevice
which itself extends IRunnableDevice
.
Since a malcolm device is a type of runnable device, the rest of the new
scanning framework outside of the MalcolmDevice
class largely treats it
the same as other runnable devices (i.e. detectors), with special-case treatment
kept to a minimum.
As for detectors, malcolm devices must be registered with the
IRunnableDeviceService
. This is typically done via spring XML configuration,
e.g.:
<bean id="malcolmDeviceXmap" class="org.eclipse.scanning.malcolm.core.MalcolmDevice" init-method="register">
<property name="name" value="BL08I-ML-SCAN-02" />
<property name="model" ref="malcolmModelXmap" />
<property name="deviceInformation" ref="malcolmInfoXmap" />
</bean>
A MalcolmDevice
object deals with communicating with the actual malcolm
device, which it does using an EPICS v4 connection layer.
It has methods to run a malcolm scan (the inner scan),
(the run
method inherited from IRunnableDevice
), as well as to pause,
resume, abort a malcolm scan, etc.
See Malcolm Device for more details on malcolm devices.
14.2.4. Annotations¶
A device may wish to be notified and take action at some other point in the
scan that that defined by the two interfaces above, i.e. at the start or
end of a point, at the end of the scan, when a scan is paused or resumed.
A number of annotation types are defined in the package
org.eclipse.scanning.api.annotation.scan
that a method can be annotated
with. This will cause the method to be called at the appropriate point in
the scan. For example, if a device class has a method annotated with
@PointStart
it will be called at the start of a new scan point.
Annotated method can take arguments that describe the current context. These
will be matched depending on the argument type. For example
if the method has an argument of type ScanInformation
, it will be passed
the ScanInformation
object describing the scan. The context here includes the
OSGi context as well as additional object related to the current scan.
The code below gives an example of using scan annotations for a scannable>
public class Fred implements IScannable {
@ScanStart
public final void prepareScan(ScanInformation scanInfo, IPointGenerator<?> pointGen) throws Exception {
// by declaring arguments of type ScanInformation and IPointGenerator, we can
// get access to these scan
...
}
@ScanFinally
public void dispose() {
...
}
...
}
Annotations are managed and triggered by an AnnotationManager
object
created as part of running as scan. How AnnotationManager
is invoked
and some of the annotations it supports are discussed further as part
of the Running a scan section below. The class
DeviceAnnotations
includes references to all available annotations.
See that class as well as the referenced annotation types and their
Javadoc for more details.
14.3. Point Generation¶
14.3.1. Overview¶
One of the most important parts of performing a scan, is generating the
positions (also called points) to move the Scannable
s (e.g. motors)
to before each detector exposure. In new scanning this is done by using
a service called the point generator service to create a point generator
from a model that describes the range of positions to generate. This
model is called a points model, or sometimes a scan path model.
The new scanning framework contains an interface IPointGeneratorService
that can generate points from a points model. The service is implemented by
the class PointGeneratorService
as is an OSGi service. The method
IPointGeneratorService.createGenerator()
can be called with a model
class representing a scan path, and will return an IPointGenerator
object
that can be used to iterate through the points of the scan.
The IPointGenerator
interface extends Iterable<IPosition>
, where an
IPosition
represents an n-dimensional position, i.e. the a position
over n axes, where each axis is the name of a Scannable
. It encapsulates
a Java Map<String, Object>
where the key is the name the
scannable and the value is the position of that scannable for this point.
The scan path model classes are simple beans with getter/setter methods. This architecture allows the new scanning framework to have a single scanning algorithm and therefore a single instance of a single class that implements that algorithm, independent of the type of scan or how many dimensions the scan has (i.e. scan axes).
A scan path model (also known as a points model) defines the path the scan
should take over zero or more axes.
All scan path model classes extends the interfaces IScanPathModel
and
IScanPointGeneratorModel
. IScanPathModel.getScannableNames()
can be called
on any model to get the list of scan axes (scannable names) that will be moved
by the path that the model describes. IScanPathModel
also defines methods to
add and remove property change listeners, useful for manipulating the models
in a UI.
The interface IScanPointGeneratorModel
defines getter/setter methods for the
properties continuous and alternating and units.
In a continuous scan, the scan motors continue to move while the detector are exposed, if possible. In a non-continuous scan, the motors stop completely at each scan point. Continous scanning is not possible for all scan models.
If a scan path model defines a scan path within another one, i.e. a grid scan
within an outer axis (see CompoundModel
later), then the path described by
this model is run backwards in every alternate position of the outer axis.
This prevents the motors having to return to the starting point of this model
for each inner step.
The units property is of type List<String>
. The default units are ‘mm’. This
property does not need to be set and rarely is in practice.
These properties are implemented in the abstract class AbstractPointsModel
,
which all concrete points model classes extend. However, they do not make sense
for all model types. For example, the properties continuous
and alternating
do not make sense for a StaticModel
as no motors are moved in the path
described by this kind of model (the types of points model are discussed further
below). The class AbstractPointsModel
defines static methods, such as
supportsContinuous()
that return whether a particular model class supports
a particular property.
Another property supported by many types of model is boundsToFit
. This is
defined by the interface IBoundsToFit
. By default, this is false, meaning
that the first and last points of a path are exactly the boundary of the
region being scanned. If set, then the start and end points will be half a
step inside the bounding region. For continuous scans, where the motors are
always moving, this means the whole exposure for the first and final points
are inside the bounding region.
14.3.2. Types of Scan Path Model¶
Most scan paths model classes described a specific type of scan path in a
particular number of axes. These are best categorised by the number of axes
that they move: 0, 1 or 2. The only model class without axes is StaticModel
.
Models which describe movement over a single axes implement IAxialModel
,
and those which describe movement over two axes implement IMapPathModel
.
In addition there are model classes for models that can contain other models in order to combine the scan paths in some way.
StaticModel
is the only scan path model class which has no axes,
getScannableNames()
returns an empty list of strings. It can represent a
single point or multiple points, this can be set by calling setSize()
.
The IPosition
(s) created for a StaticModel
contain an empty map. Within a scan,
when point is reached, it results in exposing the detector(s) without moving any
motors. A type of scan that does this is called an acquisition scan.
14.3.2.1. Axial Models¶
All models classes which describe a path on a single axis implement the
interface IAxialModel
. Such models describe a path where a single scannable
is moved at each point of the scan. All axial model classes extends
AbstractAxialModel
which implements the interface IBoundsToFit
to include the boundsToFit
property described above.
In a mapping scan, an IAxialModel
will describe the path of an outer scannable.
AxialStepModel A path defined by start and stop positions, and a step size; AxialPointsModel A path defined by start and stop positions, and a number of points; AxialArrayModel A path defined by an array of positions along a single axis; AxialMultiStepModel A path defined by multiple defined by multiple AxialStepModel`s whose resulting paths are to be iterated across sequentially. Note: all `AxialStepModel`s must be in the same axis. `OneAxisPointRepeatedModel A path consisting of a single position on an axis, to be repeated a set number of times;
14.3.2.2. Map Path Models¶
IMapPathModel
is the super interface of all paths that have two axes. In
other words, they define a path in two dimensional space. Note that this path
itself may itself be 0, 1, or 2 dimensional, i.e. it may be defined by a
point or line in two-dimensional path, as well as by some two-dimensional
shape. Paths of this type are those that can be created in the mapping
view within the mapping perpective, often these correspond to the axis of
a sample stage.
The class AbstractMapModel
is the abstract superclass of all map path
models. It defines the properties xAxisName
, yAxisName
, xAxisUnits
and yAxisUnits
.
Paths that represent a one dimensional path in two dimensional space,
i.e. a line from (x1, y1)
to (x2, y2)
are represented by an instance of a
class that extends AbstractBoundingLineModel
, which implements
the interface IBoundingLineModel
. This interface defines a single
property: boundingLine
, an instance of the class BoundingLine
. That
class defines the properties xStart
, yStart
, angle
and length
.
Paths that represent a two dimensional path in two dimensional space are
represented by a instance of a class that extends AbstractBoundingBoxModel
which implements the interface IBoundingBoxModel
. This interface
defines a single property: boundingBox
and instance of the class
BoundingBox
. The class defines the properties
xAxisStart
, xAxisLength
, yAxisStart
and yAxisLength
.
This represents a rectangular region that encompases all the
points of the path defined by the model.
Note that whatever 2d mapping region shape is selected in the mapping
UI, this has been replaced by such a rectangular region by the time the
ScanRequest
containing the points model is created. The actual region
drawn has been converted into a ScanRegion
object added to the
CompoundModel
(see below).
TwoAxisPointSingleModel (Point) A path consisting of a single point in 2d space, defined by x and y positions TwoAxisLineStepModel BoundingLine A line path defined by a BoundingLine and a step size TwoAxisLinePointsModel BoundingLine A line path defined by a BoundingLine and a number of points TwoAxisGridStepModel BoundingBox A grid/raster path defined by a BoundingBox and step sizes in each of the x-axis and y-axis TwoAxisGridPointsModel BoundingBox A grid/raster path defined by a BoundingBox and number of points in each of the x-axis and y-axis TwoAxisGridPointsRandomOffsetModel BoundingBox As above, but with each points offset by a random amount, as defined by an offset property, defining the maximum offset TwoAxisSpiralModel BoundingBox A spiral path defined by a BoundingBox and a scale property which determines the distance between arcs of the spiral TwoAxisLissajousModel BoundingBox A path defining a Lissajous curve within a BoundingBox, defined by the properties a and b defining angular frequencies in each axis, and the number of points. See [Lissajous Curve](https://en.wikipedia.org/wiki/Lissajous_curve) for more details.
The two grid models above, TwoAxisGridStepModel
and TwoAxisGridPointsModel
extend the abstract class AbstractTwoAxisGridModel
, which defines two additional
properties, orientation
and boundsToFit
.
The property orientation
is of the type Orientation
, an enum type
with the values HORIZONTAL
and VERTICAL
. This
determines which of the x or y axes (i.e. the horizontal and vertical axes as
plotted) is the fast axis and which is the slow axis. The fast axis is the axis
that is scanned first. The default orientation is HORIZONTAL
meaing that the
x-axis is scanned first. If the orientation is set to VERTICAL
then the y-axis
is scanned first. The property boundsToFit
is inherited from the interface
IBoundsToFit
described above.
14.3.2.3. Multi Models¶
Some types of path model define path models that contains other path models.
These classes extend the abstract superclass AbstractMultiModel
.
AxialMultiStepModel
, an axial model that contains multiple AxialStepModel
s
has already been described above. The other are summarized in the table below.
ConcurrentMultiModel The paths defined by each contained model are run concurrently, i.e. the nth position of each point (conceptually a map) combined the nth position of each of the contained models, each of which must have the same length and have mutually exclusive axes. ConsecutiveMultiModel The paths defined by each contained model are run consecutively. All contained models must have the same axes. This is similar to AxialMultiStepModel, except that it can contain any type of child model. InterpolatedMultiScanModel A special subclass of ConsecutiveMultiModel that, but in between the paths for each model will move to an interpolation position. This is a position, likely in different axes that is not considered part of the scan. It can be used to open shutters, for example. CompoundModel The path defined by each inner model is repeated for each point of each outer model. Inner models are those that occur later in the list of models.
The most important type of multi-model is CompoundModel
. This is the type
of model that is used to describe a scan path in a ScanRequest
(see below). Under the hood, any other type of model in a scan is replaced by
a CompoundModel
containing only that model.
A CompoundModel
combines models so that the path of an inner model is
performed for each point of an outer model. A CompoundModel
can contain
an arbitrary (non-zero) number of models, stored in a Java List
where the
last model in the list describes the innermost path, and the first model
describes the outermost path. The overall path defined by the model is
built up from by traversing the list backwards where the path described so
far is repeated again for each point of the next outer model.
The CompoundModel
class defines a number of additional properties besides
the list of contained models. These are:
duration double The exposure time to be used for each point in the scan. regions Collection<ScanRegion> Regions define a geometric region of interest. Points outside of any region are excluded from the scan mutators List<IMutator> A mutator can modify the points of the scan in an artibrary way.
The duration
property belongs to CompoundModel
as this is the type of model
used in a scan request. For malcolm scans, this is set to the exposure time of the
malcolm device.
The regions property consists of a List
of ScanRegion
s, where a ScanRegion
encapsulates a ROI
defining a region of interest together with a List<String>
defining the axes names (i.e. scannable names) that the region of interest
applies to. There are many types of ROI representing different geometrics shapes,
e.g. CircularROI
, PolygonalROI
, RectangularROI
. When a non-rectangular region
is chosen for a mapping scan, this is
converted to a ScanRegion
containing the appropriate type of IROI
and added
here. This is necessary as an IMapPathModel
only includes a BoundingBox
or
BoundingLine
to define the region by default.
14.3.3. IPointGeneratorService and IPointGenerator¶
The IPointGeneratorService
is an OSGi service that is used to create the
IPointGenerator
that iterates through the points. The simplest
IPointGeneratorService
method is createGenerator(T model)
that takes
a model and returns an IPointGenerator
that can be used to iterate through the points
of the model. IPointGeneratorService
also contains a method that take a model
class together with a IROI
and return an IPointGenerator
that will exclude
points not within the region defined by the IROI
. The method
IPointGeneratorService.createCompoundGenerator(CompoundModel)
create an
IPointGenerator
for the CompoundModel
.
The IPointGeneratorService
is implemented by the class
PointGeneratorService
. This class contructs the appropriate type of
point generator for the model passed in. It contains a static
map, Map<Class<? extends IScanPathModel>, Class<? extends IPointGenerator<?>>> modelToGenerator
that is used to determine the point generator class to create. For
example, for a AxialArrayModel
and AxialArrayGenerator
is created.
The interface IPointGenerator
extends Iterable<IPosition>
to enable
iteration through the points that are generated. It also defines
the method createPoints()
that returns a List<IPosition>
that can be used
to return all the positions at once - this method is used in unit tests.
The method getSize()
returns the number of points the IPointGenerator
will
generate, while getShape()
returns an int
array specifying the number of
positions for each dimension of the scan and getRank()
returns the number
dimensions. The method getNames()
returns the names of all axes of the
scan (as a List<String>
), while getDimensionNames()
returns a
List<List<String>>
of the axes names for each dimension index.
The dimensions of a scan indicate the dimensions to use for the datasets
to be written to the nexus file for the scan.
Note that the number of dimensions in a point generator does not necessarily
equal the number of axes. Some dimensions may have multiple axes.
For example, the point generator for a SpiralModel
will have a single
dimension consisting of both x- and y-axes. A grid generator may also be
flattened into a single dimension if it has a non-rectangular region,
or even if the detector is only capable of writing a single stack of images.
The actual implementation of points generators is written in jython. In GDA
these classes are location in the folder scripts/scanpointgenerator
in
the org.eclipse.scanning.points
plug-in project. For example
spiralgenerator.py
. Each Java generator class simply creates a
jython generator and acts as a wrapper to it. The class ScanPointGeneratorFactory
contains static methods that return a JythonObjectFactory
object
to create the appropriate Jython generator,
e.g. ScanPointGenerator.JOneAxisArrayGeneratorFactory()
. This is class
that is implemented in Jython. It is then called with the appropriate
arguments, depending on the type of object.
14.3.4. IPosition¶
An IPosition
object describes the position of one or more named axes
(i.e. Scannable
s). Conceptually it can be thought of as holding a map from
axis name to a position for that axis. The method get(String)
, which
returns an Object
can be used to get the value for a particular axis name,
As double
is by far the most common type for an axis value, the method
getDouble(String)
can be used for convenience, this returns the value as a
double, removing the need for a cast.
Besides having a value for each axis name, an IPosition
also has an index for
each axis name, of type int
. This is the index of this position within the
positions generated by the model for that axis, i.e. the indices for inner
axis will reset to 0 when the outer axis moves on to the next position.
For a (horizontal) grid scan, the axis for the x axis will reset to 0 each time
the scan moves to a new line, unless the grid model is alternating, in which
case the index for each point on the new line will decrement for each point
starting from the maximum value back to 0. The method getIndex(String)
can be
used to get the index for an axis name, and getIndex(int)
to get the index
for a dimension index. The methods getStepIndex()
returns the overall index
of the position within those generated.
There are four implementations of IPosition
. The classes StaticPosition
,
Scalar
and IPoint
are used for 0, 1 and 2 dimensional positions
respectively. The class MapPosition
covers the general case.
14.4. Defining a Scan¶
A scan is defined by a ScanBean
, which contains a ScanRequest
describing
the scan. The other properties of a ScanBean
describe the status of
the scan.
A ScanRequest
contains a CompoundModel
describing the scan path for the scan.
A CompoundModel
contains one or more model objects describing the path for
a particular scan axis or multiple axes. For a mapping scan, the inner most model
(i.e. the last element of the map), will be an IMapPathModel
, e.g.
TwoAxisGridPointsModel
. Any outer axes will be of some class that implements
IAxialModel
.
A ScanRequest
also includes a Map<String, IDetectorModel>
of detectors models
keyed by detector name. This keys in the map are the names of the detectors to be
included in the scan. The IDetectorModel
value for each detector name is the
model with which that detector will be configured before the scan starts. This will
set the exposure time of the detector, since this is defined in IMalcolmModel
,
and any other properties depending on the model class (see Detectors above.
For a malcolm scan, the detector name will be the name of a malcolm device, and the
model will be a MalcolmModel
.
The ScanRequest
also includes numerous optional properties describing
additional tasks to before before or after the main scan. For example
a property to determine the path of the nexus file to write
(see NeXus File Writing -
this is determined automatically if not present).
Other optional properties can be used to specify nexus templates to run
(see NeXus Template Engine),
the names of scripts to run before or after the scan, IPosition
s to move
to before or after the scan, and a ProcessingRequest
describing processing
to perform. The ability to support live processing is a new feature that new
scanning supports which classic scanning does not.
Another optional property of a ScanRequest
is a list of
ScanMetadata
objects. These describe static metadata that can be added to
the main nexus groups in the nexus file. Each contains a
Map<String, Object>
describing the metadata to add, as well as a type
property of the enum type MetadataType
. This property determines the
group that metadata will be added to. It has the possible values
ENTRY
, SAMPLE
, INSTRUMENT
and USER
.
14.5. Submitting a scan¶
The new scanning framework has its own implementation of a queue, defined by
the interface IJobQueue
and implemented by the class JobQueueImpl
. The
queue implementation can support multiple named queues, but for scanning we
use a queue with the name org.eclipse.scanning.submission.queue
. These
queues are maintained within the event service, defined by the interface
IEventService
, and implemented by the class EventServiceImpl
.
The method IEvent.getJobQueue(String)
can be called with name of the scanning
queue (defined by the constant EventConstants.SUBMISSION_QUEUE)
to get
the IJobQueue
object containing the scanning queue.
To submit a scan to the queue, call the submit
method with a ScanBean
containing the ScanRequest
defining the scan.
An ISubmitter
can also be created by calling
IEventService.createSubmitter
with the URL of the activeMQ broker
(which can be obtained by calling LocalProperties.getActiveMQBrokerURI()
)
and the name of the scanning queue as above. This also has a submit
method,
as well as a blockingSubmit
method that will block until a scan is complete.
The queue is held in memory within the JobQueueImpl
instance. In fact it has
two ‘queues’: one for scans that have yet to be run, this is called the
scanning submission queue; and one for scans that have started or completed,
this is called the status queue. Scans (or rather the ScanBean
objects that
represent them) are added to the submission queue. They are moved to the status
queue once they have started running.
The IJobQueue
interface defines methods to run
, stop
, pause
and resume
the queue. It also has methods to manipulate the beans on the queue,
such as remove
, replace
, moveForward
and moveBackward
. For more
about the new scanning queue system, see here.
Note as of writing the linked document is out of date.
14.6. Running a scan¶
The scanning IJobQueue
instance is created by an instance of the class
ScanServlet
, which is declared in spring. Note that despite the name, this
class is unrelated to javax.servlet.Servlet
. This creates an
IPublisher
that can be used to publish ScanBean
s to the status topic when
they change. This is an ActiveMQ (JMS) queue named
org.eclipse.scanning.queue.status.topic
.
The IJobQueue
runs a thread which takes the next ScanBean
from the queue when
one is available, and creates a new ScanProcess
for that bean. The
ScanProcess
is also passed the IPublisher
described above.
The ScanProcess.execute()
method is then called.
14.6.1. ScanProcess¶
An instance of ScanProcess
is responsible for converting the ScanRequest
bean to a ScanModel
. This is then then given to an IScanDevice
object to
run. It also runs some additional tasks before and after the main scan.
The difference between a ScanRequest
and a ScanModel
is that a ScanRequest
is a bean containing primitive data types and their wrapper objects, Strings,
collections and arrays of such types, as well as instances of other such bean
classes. For example the ScanRequest
contains the names of detector as well
as models to configure them with, a CompoundModel
to describe the scan path,
and names of per-point and per-scan monitors. A ScanModel
contains the
actual IScannable
and IRunnableDevice
objects for such devices, as well
as the IPointGenerator
to use to create the IPosition
s for the scan.
The ScanProcess.execute()
method first calls prepareScan
. This is the
method that creates and returns the ScanModel
for the scan. First it
decides the path for the nexus file and sets this property on the ScanBean
.
If the ScanRequest
already specifies a file path, this is used directly,
otherwise the method getNexusFilePath()
it calls
IFilePathService.getNexusPath(null)
to get the next file path to use.
IFilePathService
is implemented by the class FilePathService
and
accessed as an OSGi service. This class is located in the
uk.ac.gda.core
plug-in project, i.e. among classic GDA classes rather than
new scanning. It uses the value of the property gda.data.scan.datawriter.datadir
as the directory to use, and a NumTracker
to determine the scan number.
The file name used is then determined by the pattern
<beamlineName>-<scanNumber>.nxs
.
14.6.1.1. Preparing the Scan¶
The ScanProcess.prepareScan()
method then gets the
CompoundModel
from the ScanRequest
that describes the scan path and uses
the IPointGeneratorService
to create the IPointGenerator
that will be
used to generate the points for the scan. The method
checkAndFixMonitors(IPointGenerator)
checks the lists of
per-point and per-scan monitor names in the ScanRequest
, removing
any that appear as axes in the point generator. prepareScan()
then calls
createScanModel(IPointGenerator)
with the point generator to create
the ScanModel
that will be used to run the scan.
The ScanModel
is created by setting properties for the file path,
the IPointGenerator
and the CompoundModel
. The scanMetadata
property,
List<ScanMetadata>
is simply copied from the ScanRequest
,
as are the list of nexus template file paths.
The createScanModel
method gets the actual scannable and detector
objects that are only referenced by name in the ScanRequest
and
adds them to the ScanModel
. These are are the IScannable
objects
for the per-point and per-scan monitors named in the ScanRequest
and
the IRunnableDevice
s
for the detectors named. An exception is thrown if a device named in
the ScanRequest
cannot be found.
createScanModel(IPointGenerator)
then returns the ScanModel
which
prepareScan()
then also returns and control is returned to the
ScanProcess.execute()
method.
The execute()
method then creates the AnnotationManager
,
adding the devices in the scan to it, and calls
its invoke(Class<? extends Annotation>, Object...)
method with the
PrepareScan
annotation. This is the class responsible for calling
methods on the device objects in the scan that are annotated with
the given annotation, and is discussed further below.
The execute()
method then move scannables to the start position, if
specified by the ScanRequest.startPosition
property, an IPosition
.
This is done by calling IPositioner.setPosition(IPosition)
on
an IPositioner
object that was created by calling
IRunnableDeviceService.createPositioner(String)
. IPositioner
is
implemented by ScannablePositioner
, a type of
LevelRunner
runs devices in order of their
levels. It the beforeScript
property is specified, this is then
run, by creating a ScriptRequest
and calling
IScriptService.execute(ScriptRequest)
. The only type of script
supported is Jython (this is unlikely to change) and the
class GDAJythonScriptService
implements IScriptService
for this purpose,
using classic GDA API to do so.
The ScanProcess.execute()
method then calls runScan(ScanModel)
to
actually run the scan. This calls the method createRunnableDevice(ScanModel)
to create an IDeviceController
, which wraps an IScanDevice
which
actually runs the scan. IDeviceController
is a mechanism which mediates
between the user and watchdogs to determine when to pause and resume the scan.
The method createRunnableDevice
calls the method configureDetectors
.
This calls the configure(M)
method on each detector, where M
is the model
type for the detector. Any @PreConfigure
and @PostConfigure
annotations
are fired on this device before and after (respectively) this method is called.
This can be used by the device to gain access to any scan objects that the
device needs. For example this is how a malcolm device gets the
IPointGenerator
. It then creates the IScanDevice
by using the
IScanDeviceService
OSGi service
(which is actually implemented by the same RunnableDeviceServiceImpl
that
also implements IRunnableDeviceService
). The IScanDevice
is passed a
reference to the IPublisher
so that it can also publish updates to the
ScanBean
to the status topic.
The method createRunnableDevice
also creates an IDeviceController
(described above) that wraps the IScanDevice
. It then calls
configure(ScanModel)
on the IScanDevice
to
prepare it to run the scan. This is described in further detail below.
The method runScanBlocking(IDeviceController, ScanModel)
is then called which finally calls run()
on the IDeviceController
which in
turns calls run()
on the IScanDevice
. Assuming no exception is thrown,
the runScanBlocking
method then runs the after script and moves to the scan
end position, if either of these are specified.
If runScan()
completes normally then returns back to ScanProcess.execute()
which updates the ScanBean
to indicate the scan has finished by setting
percentage complete to 100 and the status to Status.COMPLETE
.
The class ScanProcess
also defines the methods pause
, resume
and
terminate
. These methods essentially call the methods on the
IDeviceController
with the same name. Depending on how this is mediated by
the watchdogs, these will then be passed down to the IScanDevice
that
runs the scan.
14.6.1.2. AcquisitionDevice¶
The class that actually implements IScanDevice
and runs the scan is
AcquistionDevice
. Is it an instance of this class that, during a software
scan, iterates through the points of the scan, exposing the detectors.
Note that in most JUnit tests that run scans, an instance of
AcquistitionDevice
is created and run directly without an instance
of ScanProcess
being involved. AcquisitionDevice
is sufficient to
run these scans, as the tests create a ScanModel
directly, rather than
creating a ScanRequest
first, and they do not need to perform any of
the additional tasks that ScanProcess
runs before or after a scan,
such as before/after scripts, or moving to start/end positions, etc.
14.6.1.2.1. Configuring the AcquisitionDevice¶
The configure(ScanModel)
method is called as soon as the AcquistionDevice
is created. The AcquistionDevice
uses this method to prepare itself for the
scan. After storing the ScanModel
in a field, this method performs a number
of tasks to configure itself according to the model. First, it looks up
the IMalcolmDevice
for the scan by name, if one is involved, and stores that
in a field. Next, it creates an instance of AnnotationManager
to use to call
annotated methods, then an instance of
LocationManager
, a class that contains logic to do with calculating
scan locations when there is an inner scan). Next it create
an instance of IPositioner
. This is the interface, implemented by
ScannablePositioner
that is use to move Scannable
s to their each
position. It also creates the DeviceRunner
and DeviceWriter
instances used to run (expose) and write detectors. ScannablePositioner
,
DeviceRunner
and DeviceWriter
are all subclasses of the abstract
class LevelRunner
, which manages performing a particular task
(i.e. moving to position, exposing detector and writing detector data, respectively),
across the devices involved in order of the level of the devices.
The most important task of the configure
method is to write the nexus
structure for the scan. This is delegated to the configure
method of
NexusScanFileManager
which up an in-memory representation
of the nexus file called a nexus tree. Devices that implement INexusDevice
will provide nexus objects (instances of some subclass of NXObject
) that
will be added to this tree, which is then saved to disk.
These nexus objects typically contain ILazyWritableDataset
s that the
devices keep a reference to in order to write to them at each point in the scan.
All important device classes implement device INexusDevice
or are wrapped by
an object that does. A reference to the NexusScanFileManager
is cached in
order to call its close()
method at the end of the scan.
Nexus writing is discussed in more detail here.
14.6.1.2.2. Running the AcquistionDevice¶
Finally, the run
method of the AcquisitionDevice
is called to run the scan.
This is arguably the most important method of the whole of the new scanning framework.
First, two pieces of housekeeping are performed: the ScanBean
is updated with the
start time, the status of Status.RUNNING
and published to the status topic.
A LocationManager
is used to create an Iterator<IPosition>
from the
IPointGenerator
for the outer scan. The outer scan is that part of a malcolm
scan that is performed in software. For example, for scan defined by a CompoundModel
consisting of an IAxialModel
for a software controlled scannable, e.g.
energy, and a TwoAxisGridStepModel
for a malcolm controlled x- and y-axes,
the outer scan is defined by the IAxialModel
and the inner scan by the
TwoAxisGridStepModel
.
The LocationManager
uses a SubscanModerator
to create an IPointGenerator
for each of the inner scan, outer scan and the overall scan. This compares
the axes of each model with the axes of the malcolm device to determine
which models are controlled by malcolm and which are not.
The LocationManager
maintains this information as well as the current scan
position and count.
We then begin to iterate through the positions of the scan using the
Iterator<IPosition>
. For each position, pos
we perform the following tasks:
Call
checkPausedOrAborted
in case a request has been made to pause or abort the scan, see below;Move the scannables to that position by calling
IPositioner.setPosition(pos)
;Call
checkPausedOrAborted
a second time in case a request has been made while the scannables were moving;Wait for the
DeviceWriter
to complete writing for previous step (if applicable);Call
DeviceRunner.run(pos)
to expose the detectors. This callsrun(pos)
on theIRunnableDevice
s;Call
DeviceWriters.run(pos, false)
, for detectors to write their data This callswrite(pos)
on any detector that implementsIWriteableDetector
;Update and publish the
ScanBean
to reflect the completed position.
For a malcolm scan, the run(IPosition)
method called by the DeviceRunner
runs
the inner malcolm scan, writing data into its external datasets as it goes along.
For this reason MalcolmDevice
does not implement IWriteableDetector
and no
action is taken when DeviceWriters.run(pos, false)
is called.
The overall loop is wrapped in a try/catch/finally
block. If an exception is
thrown, processException
is called to update the ScanBean
accordingly, with
a status of TERMINATED
if the scan was aborted, and FAILED
otherwise.
In the finally
block, we close all the LevelRunner
s to release their threads,
call fireEnd
if the scan has finished normally to update and publish the
ScanBean
to reflect that, and use the AnnotationManager
to fire any
ScanFinally
annotated device methods.
14.6.1.2.3. Pausing, Resuming and Aborting the AcquistionDevice¶
The methods AcquisitionDevice.abort()
, pause()
and resume()
can be called
to abort, pause and resume the scan being performed by the AcquisitionDevice
respectively. The seek(int)
method moves the scan to a particular step number,
also pausing the scan.
As these methods will be called from (potentially multiple)
threads other than the thread that is running the scan, they are performed
while holding the stateChangeLock
, a ReentrantLock
. The pause()
and seek()
methods each set the awaitPause
flag to true to signal
that the scan should pause at the next possible opportunity, while the
resume()
and abort()
methods call Condition.signalAll()
on the
shouldResumeCondition
to cause any paused scan to wake up.
The abort
method sets the aborted
flag. It also calls abort()
on all
LevelRunner
s to interrupt any tasks they may be performing. The pause()
,
resume()
and seek()
methods will also call the method with the same name
on the MalcolmDevice
so that it can perform the appropriate action.
All these methods update the status of the ScanBean
as appropriate and
publish it to the status topic.
As mentioned above, for each scan position, checkPausedOrAborted()
is called
within the run()
method. This first calls checkAborted()
which simply
throws an InterruptedException
if the aborted
flag has been set. It
then calls checkPaused()
. This method holds the stateChangeLock
while
it checks if the awaitPaused
flag has been set. If so, it updates and
published the ScanBean
to show the status DeviceState.PAUSED
, then calls
Condition.wait()
on the shouldResumeCondition
. This will cause the
main scan thread to wait until resume
or abort
are called from another
thread.
Once this has happened, this method will continue. It will first call
checkAborted
in case the scan was aborted while it was paused. It then
updates and publishes the ScanBean
to show the status DeviceState.RESUMED
.
These methods are called by the DeviceController
if either the user
(via ScanProcess
) or a watchdog, such as the TopupWatchdog
asks for the
scan to be paused. The DeviceController
class mediates between
the watchdogs and ScanProcess
to decide when the AcquistionDevice
should be paused or resumed. For more information about DeviceController
and the watchdogs, see here <https://confluence.diamond.ac.uk/display/CT/Watchdogs>
.