16.1. How NeXus files are written in scanning

16.1.1. New scanning

During a scan, a ScanProcess object is created to process the scan (see new scanning use case for more details). This creates a ScanModel object and an AcquisitionDevice object (this object’s class is package-private in package org.eclipse.scanning.sequencer, i.e. no Javadoc available). It is the latter object that essentially performs the scan. The AcquisitionDevice delegates the task of creating the NeXus file to the NexusScanFileManager. It is this class that will essentially build and save the NeXus tree.

Before any devices are added, a new NeXus tree is created. The tree exists only in memory at this point and is only saved to disk when the whole tree has been built. A single entry is added to the NeXus tree and then the default groups are added:

• instrument: NXinstrument

• sample: NXsample

• user: NXuser

These groups will initially be completely empty with no fields or child groups. During the scan, these groups can be populated and other groups added in two sorts of ways.

Scan metadata can be added to these groups or to the NXentry group if the ScanRequest contains one or more ScanMetadata objects, which are added by calling scanRequest.addScanMetadata(). The ScanMetadata object contains a map of metadata field values, keyed by name and a type. The type is a constant from the enum type ScanMetadata.MetadataType which can take the value: ENTRY, SAMPLE, INSTRUMENT and USER. For each entry in a map a field is added to the group corresponding to the enum type with the name and value set accordingly.

At this point, the default groups and scan metadata have been added to the NeXus tree (still only in memory). Next the NeXus object for the devices can be added. There are several types of devices in a scan and the information they are asked to write can be specified in different ways (see below):

Device Roles	ScanRequest property specified by	Asked to
Detectors	scanRequest.detectors	Write their data at each position in the scan.
Scannables	scanRequest.compoundModel.models.scannableNames	Move to a position and write their data at each point in the scan.
Monitors	scanRequest.monitorNamesPerPoint	Write their data at each position in the scan.
MetadataScannables	scanRequest.monitorNamesPerScan	Write their data at the start of the scan.

The devices will implement the IScannable device (true for all types of devices). In order for the device to add a NeXus object to the NeXus file during the scan it must also implement the interface INexusDevice. Note that GDA8 Scannables (involved in the GDA9 scanning) are wrapped by a ScannableNexusWrapper which implements both these interfaces (IScannable and INexusDevice).

The interface INexusDevice defines the method getNexusProvider() which returns an instance of the interface NexusObjectProvider. This is what will get the NeXus object and add it to the tree. Detectors, scannables and monitors are asked to write their data at each point during the scan (metadata scannables are not). For detectors (which implement IWritableDetector) this is done during the write(IPosition) method, whereas for scannables and monitors it is done in the IScannable.setPosition() method.

One or more NXdata groups are created within the NeXus file. A NXdata group is created for each primary data field of each detector in the scan. If there are no detectors in the scan then a single NXdata group will be created for the first monitor in the scan and if there are no monitors then for the first scannable.

• The AcqusitionDevice object first creates the NexuScanFileManager

  • The NexusScanFileManager gets all of the INexusDevices in the scan and calls getNexusProvider() and saves all the NexusObjectProviders to the SolsticeScanMonitor object that it holds.

  • Next it calls the method createNexusFile() which subsequently calls createEntry().

    • This method calls DefaultNexusEntryBuilder.addDefaultGroups() to add the default groups mentioned above (i.e. nxInstrument, nxSample, nxUser).

  • Next it adds scan metadata from the ScanRequest. An example of this is the metadata added to the ScanRequest by the UI to add the sample information. The entry in the ScanRequest takes the form:

    "scanMetadata":[{"@type":"ScanMetadata","type":"SAMPLE","fields":{"name":"Unnamed Sample","description":"No description provided."}}]
    

  As this is done before getting the other NeXus providers, the scanMetadata must be added to one of the default groups (i.e. nxInstrument, nxSample, nxUser).

  • Next it gets the NexusProviderObjects from the solsticeScanMonitor, which it saved them to earlier and adds these to the NeXus tree entry.

  • Finally, it creates the NXData group (which is created for each detector as described above).

16.1.2. Old Scanning

The NexusDataWriter is in charge of producing the NeXus file. It implements the DataWriter interface. It supports the ‘location map’ which is a way to define the structure that the user would like the files to take. It has no capabilities for implementing application definitions. In old scanning the NeXus file is open throughout the scan and data is iteratively written to it as it is taken.

The DefaultDataWriterFactory (implements the DataWriterFactory interface which provides pre-configured DataWriters for scans) creates a new DataWriter for each scan. The ScanBase class creates a MultithreadedScanDataPointPipeline (for more details see old scanning use case) from a ScanDataPointPublisher (which broadcasts ScanDataPoints and writes them to the DataWriter) and this pipeline will start to accept points. This is all done during the ScanBase.run() phase, specifically in the ‘prepare for collection’. The ScanDataPointPublisher will call on the NexusDataWriter to addData() passing it the data point which will incrementally add data for each point. This will happen every time the detector is instructed to readout().

If it is the first time that addData() is called on the NexusDataWriter then it performs a prepareFileAndStructure(). This will get the list of scannables that have ‘one off’ metadata to add to the scan and return a collection of these scannables and monitors. The location map is also taken into account here. Another method will write the metadata information to the NeXus file as NXpositioners or NXmonitors. Following this the NXdetector entry will be created for each detector.

16.1.3. Old Scanning With New Nexus Writing

In order to consolidate nexus writing and make it more consistent across both old and new scanning frameworks, a new DataWriter implementation has been developed, NexusScanDataWriter. DataWriter is the interface used by old scanning mechanism that is called at each point in the scan ScanDataPoint in order to write data in a manner specific to that implementation.

NexusScanDataWriter implements DataWriter to create an adapter implementing INexusDevice for each device in the scan. These are then added to a NexusScanModel which is then provided to the NexusScanFileService used in order to build the Nexus file. This is the same service used by new scanning to write Nexus files.

Note that old scanning, (a.k.a. GDA 8 scanning) is the scanning framework that is invoked by the scan command, whereas new scanning (a.k.a. GDA 9 scanning) is the scanning framework that is scanning framework used by the mscan command. Each framework also has multiple UIs within the GDA client that make use of it, e.g. the Mapping UI makes use of new scanning.

As with NexusDataWriter the addData method is called at each point of the scan with a ScanDataPoint describing that point of the scan. The structure for the nexus file is created when this method is called for the first time (i.e. for the first ScanDataPoint of the scan).

The class GDANexusDeviceAdapterFactory is used to create an adapter for each device, i.e. some object that wraps the Scannable or Detector and implements INexusDevice. These classes create the same structure as NexusDataWriter does for each device. However, this will be created as an in-memory structure and returned as the nexus object (a GroupNode) for the INexusDevice, rather than the nexus structure being created directly on disk at that time. This results in the overall nexus file having as a similar structure as possible as written by NexusDataWriter. The biggest difference is that the NexusScanFileService, which is responsible for creating the NXdata groups will use the new (2014) tagging format, as described here.

For scannables, by adapter class is ScannableNexusDevice, the same class that is used for GDA8 Scannables in for nexus writing with new scanning. For detectors an instance of one of four different adapter classes is created depending on some property of the detector. These are the same categories of detector for which NexusDataWriter has separate methods for, so the structure of the NXdetector group created is the same as that which would have been created on disk by NexusDataWriter. In particular NexusDetectorNexusDevice is used to adapt an NexusDetector, translating the INexusTree structure for the detector to create the NXdetector group for the detector.

Note: In order to correctly tag the NXdata group according to the new format, it is necessary for the nexus writing framework to know the rank of the dataset. In most cases this can easily be found from the dataset itself. The exception is where an NexusDetector where the INexusTree produced contains external links, i.e. where NXclass property has the value ExternalSDSLink. A new field externalDataRank has been added to the class NexusGroupData This must be set in order for an NXdata group to be created for the given dataset. The addExternalFileLink methods of NXDetectorData have been updated to include the parameter dataRank which sets this field. Client code may need to be updated (this has already been done in some cases).

Besides the INexusDevices, the NexusScanModel also has a number of other fields, e.g. the entry name, file path, nexus templates, etc. These fields are configured by the NexusScanDataWriter based on the information contained in the ScanDataPoint, as well as from other places. A new instance of NexusScanModel is created for each scan - it cannot be configured directly by client code or from your spring configuration.

The table below shows some relevant property of NexusScanModel and how NexusScanDataWriter configures them.

A field of particular note is dimensionNamesByIndex, a map from each scan index, an integer, to the collection of scannable names for that index. The ScanDataPoint does not provide a way to explicitly determine this information. These fields are populated via

Note: At present an assumption is made that for each index of the scan dimension array as returned by ScanDataPoint.getScanDimensions(), the single scannable for that index is the scannable at that index in the array returned by the method ScanDataPoint.getScannables(). Any remaining scannables are treated as (per-point) monitors. This information is used to correctly tag the NXdata groups with the axes information as required by the new tagging format.