16.2. NeXus Template Engine

The idea is to define what will be written by a GDA scan (new and old) in a standard way. This is to then allow a template to be used to create links back into this standard structure which can work for old and new scanning. Another prime motivation for the template engine is to provide a mechanism to add NeXus application definitions to an existing NeXus file (or tree), however the potential uses are even more general.

The NeXus Template Engine is a framework for applying templates to either a NeXus file or, an in-memory NeXus tree. The framework lives in the Eclipse plug-in org.eclipse.dawnsci.nexus.templates. There is also a product org.eclipse.dawnsci.nexus.templates.product which can be exported to create standalone template application (see below).

Important note: The template format can add static data, but it cannot refer directly to devices so any dynamic data (i.e. the value of a scannable) must already be present in the NeXus file. The template provides a way to add links from new NeXus groups to the existing data.

16.2.1. Template format

YAML is used as the format for the template. An example of what you can do is below.

myGroup/:
    NX_class@: NXentry
    attribueOnMyGroup@: "something"
    my_inline_static_data: 123.45
    my_obj_static_data:
         value: james
         myAttribute@: hello
         linkedAttr@: /link/to/attr@
    my_obj_link: thing # NOTE: adding attributes to linked nodes is not yet 
supported!
         link: /target/help
         james@: foo
    my_inline_link: /must/start/with/slash
    data*:  # the asterisk denotes copying an NXdata group with substituted axis 
names
      nodePath: /path/to/data
      axisSubstitutions:
        stagex: x
        stagey: y

The rules for the template are as follows:

  • Groups are defined by their name and a trailing slash. All groups must have an NX_class attribute specifying the NeXus class;

  • For static data you can define it inline or multiline which is required if you want to add attributes. If multiline is used, “value” must be specified which will contain the data to be written into the dataset;

  • For links they can be defined inline using a leading slash (also representing absolute) or multiline in which case “link” must be specified to point to the target. Multiline link - allowing the addition of attributes - will not be supported initially;

  • Where possible type coercion will be used to attempt ‘int’ then ‘double’ then ‘String’ for datasets and attributes;

  • Defining both a value and link is an error and it should fail.

  • A special syntax can be used to copy an NXdata group with substituted axis names. This is done by added a trailing asterisk to the new group name. The mapping for the group must contain ‘nodePath’ specifying the path of the NXdata group to be copied, and ‘axisSubstitutions’, a map specifying the axis names to be substituted where the key is the axis name in the NXdata group to be copied, and the value is the new name.

16.2.2. Standalone Template Application

The Eclipse plug-in org.eclipse.dawnsci.nexus.templates defines an Eclipse product, org.eclipse.dawnsci.nexus.templates.product. When exported this produces a stand-alone application, that can apply a template to an existing NeXus file. As of writing (2019-07-05) an early version of this is available at /dls_sw/apps/templates. The usage is:

./apply-template <templateFile.yaml> <nexusFile.nxs>

E.g

./apply-template test-template.yaml p45-example.nxs

16.2.3. Usage in new scanning

During a scan the NeXus file is constructed and saved in memory, therefore the NeXus templating system can also be applied to an ‘in-memory’ NeXus tree. The template will be applied to the NeXus tree before it is saved to disk. For this to happen it must be attached to the ScanModel, which is created from the ScanRequest. A new field has been added to the ScanRequest/ScanModel (called getTemplate()) meaning that the template will exist in the model and can be applied to the NeXus tree. Just note that the ScanModel used within the code is created as mentioned from the ScanRequest and is not for the client code to modify.

Implementation

During the ScanProcess.execute() phase of new scanning (see new scanning use case) the ScanModel is created during ScanModel.prepareScan().

  • This calls createScanModel() where the ScanModel is created from the ScanRequest:

    final ScanRequest<?> req = bean.getScanRequest();

    And the template file is set:

    scanModel.setTemplateFilePath(req.getTemplateFilePaths());

  • Following the preparation of the scan, ScanProcess.execute() calls runScan():

    • This calls createRunnableDevice() with the model as input, which subsequently calls configureDetectors() -> device.configure().

One of the key devices in new scanning is the AcquistionDevice which essentially runs the scan. So during the above step, AcquisitionDevice.configure() method is called.

  • AcqusitionDevice.configure() sets up the NexusScanFileManager and calls createNexusFile() on the manager to create the NeXus file.

    • The createNexusFile() method calls createEntry(). createEntry() essentially begins to build the NeXus file creating the NeXus tree. It:

      • adds the default groups to the NeXus file (see above) - addDefaultGroups().

      • adds the scan metadata attached in the ScanRequest (and added from there to the ScanModel) - addScanMetaData()).

      • adds devices.

      • creates an NXdata entry.

    • Next createNexusFile() calls applyTemplates() using the NeXus tree created above. This:

      • gets the template file set in the ScanModel.

      • loads this as a NexusTemplateImpl object (which implements the NexusTemplate interface) from the NexusTemplateServiceImpl (which implements the NexusTemplateService interface).

      • applied the template to the supplied in-memory NeXus tree - apply().

    • Finally, the file is created and opened ready to write to.

When apply() method is called on the NexusTemplate object, it carries out the following:

  • It creates a new InMemoryNexusContext object which implements the NexusContext interface.

  • Calls applyTemplate() with the above object:

    • creates an ApplyNexusTemplateTask - essentially just a task to apply a NeXus template to a NeXus tree.

    • calls run() on the above task. This then goes through each entry in the yaml mapping, gets each node type (GROUP, DATA, LINK, ATTRIBUTE) and adds the mapping accordingly.

Setting a template

Essentially the template file just need so be defined in the ScanRequest which is submitted.

1. A template can be added to the ScanRequest directly if the ScanRequest is created programmatically (for example in a Jython script).

2. Alternatively it can be added via the DefaultScanConfiguration. The DefaultScanConfiguration object contains scan defaults that will be merged with a ScanRequest on the server. This can be defined in the server Spring configuration, e.g:

<bean id="default_scan" class="org.eclipse.scanning.server.servlet.DefaultScanConfiguration">
  <property name="templateFilePaths">
    <set>
     <value type="java.lang.String">/path/to/templates/template1.yaml</value>
    </set>
  </property>
</bean>

After the scan is submitted, the ScanServlet will begin with createProcess() using the ScanBean. This subsequently calls preprocess() which checks if there are any ‘preprocessors’ and calls preprocess() on them. With the inclusion of DefaultScanConfiguration, the DefaultScanPreprocessor will be registered. Its job is to merge the monitors defined in its DefaultScanConfiguration into a ScanRequest. The DefaultScanPreprocessor.preprocess() call gets the template file path from the DefaultScanConfiguration and sets it in the ScanRequest (ScanRequest.setTemplateFilePath()).

3. In the case of ‘mapping’ scans, templates can simply be defined and added in the UI. Below is the field that allows this.

scale80

Templates can be added or removed:

scale60

Note that in order for this field to be available in the Mapping UI the MappingExperimentBean on the client side must define the templateFilePaths parameter. For example in the client side xml files, define:

<bean id="mapping_experiment_bean" class="uk.ac.diamond.daq.mapping.impl.MappingExperimentBean">
   <property name="templateFilePaths">
     <list>
     </list>
   </property>
</bean>

After submitting the scan, the UI converts the MappingExperimentBean to a ScanRequest using the ScanRequestConverter. This uses the same ScanRequest.setTemplateFilePath() method mentioned above to set the template file path in the ScanRequest.