23.4. Data Collection

23.4.1. Overview

The Data Collection Process is a continually running process that manages the creation and execution of a Directed Acyclic Graph (DAG) of tasks to perform the requested data collections for the plate currently onthe Goniometer.

The process may be in one of several states, on every iteration of its execution loop it may be doing one of the following:

• monitoring the execution of a current task graph

• creating and starting a new task graph for the current plate

• checking the system for a new plate on the Goniometer

Actual management of the task graph (dependency management, input and output handling, and asynchronous execution) is delegated to a separate Task Executor.

The main steps of data collection are broadly:

• Take optical image of plate’s well on the beamline

• Given an image match for the beamline’s OAV view of the plate’s well against the Formulatrix image, generate a scan description and trajectory

• Change beamline energy and perform any necessary alignment

• Move to the start position for the scan

• Prepare devices (arm detector, zebra, setup trajectory controller)

• Execute the scan (i.e. run the trajectory collecting data)

• Reset the hardware

• Write out the NeXus file (and _master.h5 file)

• Record data collection into ISPyB.

Additionally, when handling a data collection in a new well of a plate, the following additional steps must be performed before the scan can be described:

• Prepare the endstation for imaging (move the OAV into beam)

• Collect a z-stack for auto-focusing and image matching

• Run CrystalMatch (process the z-stack, determining focus positions for marked crystals and giving an overall alignment correction)

• Prepare the endstation for data collection (move the OAV and backlight out of the way, insert the scatterguard and apertures).

The desire is to move away from a large scripts approach towards a task based approach, in order to exploit potential for parallel execution of tasks.

23.4.2. Data Collection Task Graph

The Directed Acyclic Graph (DAG) used for data collection is shown belown:

23.4.3. Data Collection Process Loop

An overview of the processing of the Data Collection loop is given below:

while True:
    if running_dc and not executor.completed:
        executor.step_loop()

    elif running_dc and executor.completed:
        if executor.errored:
            raise
        running_dc = False

    elif plate_dcs:
        next_dc = plate_dcs.pop()
        graph = create_task_graph()
        executor.assign_graph(graph)
        running_dc = True

    elif not plate_dcs and have_plate:
        ispyb_set_plate_shot(plate)
        have_plate = False

    elif not have_plate and ispyb_plate_on_gonio_not_shot():
        plate = ispyb_get_gonio_plate()
        have_plate = True
        plate_dcs = ispyb_get_experiments(plate)

23.4.4. Data Collection Tasks

23.4.4.1. Overview

The steps required to collect data are held in purpose built Task objects, which are in turn organised into a DAG (see above), where tasks must wait for others to be completed before they can be executed. In short a Task represents a packet of work that must be done as part of data collection. A Task does the following:

1. Takes input

2. Does some work

3. Returns an output, which may be used by other tasks.

Tasks may move devices, write to ISPyB, or perform some calculations. Tasks are organised by their dependencies into a Directed Acyclic Graph (DAG) to perform data collections. A fresh DAG is produced for every data collection. Not all tasks need to be included for every data collection; well imaging tasks do not run if we have not changed well. Tasks with no inter-dependencies may be executed in parallel.

The data collection process holds a collection of variables for the current well and data collection, elements of which are passed to tasks as their input, and updates it with the outputs of completed tasks.

The high-level interface is given by the Task Java interface (similar to the Sample Handling Action interface), a Python version of the interface is given below:

class Task(Task):
    
    def execute(self, *args):
        ...

    def stop(self):
        ...

    def getName(self):
        ...

    def getStatus(self):
        ...

    def getResult(self):
        ...
    
    def getRunTime(self):
        ...

The task interface supports early termination, though this need not be respected, and multiple non-success conditions aborted, failed, error are discriminated.

Tasks are single use objects - they only support execution once during their lifetime.

The execute method is a blocking call that takes a list of arguments, has no return and should raise no exceptions (doing so will terminate the data collection process). Outputs are given by the getResult method, and success/errors are given by getStatus.

23.4.4.2. VMXi Task - Python Implementation

A pythonic implementation of the Task Java interface is provided by data_collection.tasks.vmxi_task.VmxiTask (gda-mx.git/configurations/i02-2-config/scripts/data_collection/tasks/vmxi_task.py).

This provides a run_task method that allows a more natural way of implementing task logic, allowing the use of return statements to provide results (including multi-value returns) and exceptions to indicate failure conditions.

The exceptions data_collection.tasks.vmxi_task.TaskFail and data_collection.tasks.vmxi_task.TaskStop are provided to indicate task failure and abortion respectively; the class will handle these exceptions and set the status of the task appropriately. Any other exceptions will result in getStatus() returning TaskStatus.ERROR.

Updates to the status field (described above) are handled automatically.

The default behaviour of VmxiTask.stop() is to set self.stop_flag = True, this flag can then be checked by code in run_task.

The utility methods VmxiTask.sleep, VmxiTask.move_to_check , and VmxiTask.move_to_limit are blocking calls that check for the stop flag and raise the appropriate exception when it is set. VmxiTask.move_to_check and VmxiTask.move_to_limit are for motor moves; move_to_check checks a motor reaches the target position to within a specified delta and move_to_limit verifies that a motor limit was hit, both raise exceptions when their respective conditions fail.

An exception raised by run_task is not lost, the resulting exc_info is stored under self.exc_info, which the Data Collection process will check for and report.

Because the expected return of getResult is a list of objects (one for each output name to be mapped to) and the automatic mapping that occurs when handling Python’s multi-value returns (i.e. return a, b is the same as return (a, b)), if a list/tuple should be regarded as a single output (to be mapped to a single output name) and it is the only output, then it must be returned as a nested list (e.g. return ((a, b),)). This is an easy trap to fall into when dealing with collections.namedtuple types.

23.4.4.3. Task Status

The task interface getStatus returns an instance of uk.ac.gda.vmxi.datacollection.process.TaskStatus , that is a Java enum that lists the following:

• PENDING - the task has not yet started. It may have been submitted to an executor, but the task itself has not marked itself as RUNNING.

• RUNNING - the task is running.

• COMPLETE - the task completed successfully, getResult will return a valid output and tasks that have this task as a dependency will be eligible to start (assuming other dependencies similarly completed).

• FAILED - the task failed to produce the output required for reasons that are anticipated. Further data collections may be attempted, but this one cannot continue.

• STOPPED - the task has safely stopped after the result of a stop request.

• ERROR - the task has stopped after encountering an error condition. No further data collections should be attempted automatically, and hardware may be in an indeterminate state.

Tasks are responsible for maintaining their own status conditions and providing an appropriate return for getStatus. This includes the transition from PENDING to RUNNING.

Once a task enters the RUNNING state, the only valid transitions are:

• COMPLETE

• FAILED

• STOPPED

• ERROR

These are considered to be termination states; a task in one of these states can make no futher changes to its state and and no further actions can undertaken.

It is not invalid for a task to skip the RUNNING state entirely, as the Data Collection process does not make decisions based on it (it tracks task submissions to the executor independently), but it is used for display purposes on the GDA client.

A task must only set its status to SUCCESS on true completion, when all outputs can be provided.

23.4.4.4. Task Failures

Tasks will sometimes fail and common failures must be handled.

Failures and errors are distinguised:

• Failure is non-leathal:

  • Some failures are expected without anything being in an error state.

  • The current data collection is skipped and the process continues.

  • Image matching sometimes fails without faults.

• Errors are fatal:

  • ERROR is he default failure case for a task.

  • All running tasks are stopped, and the process then halts.

  • Typically result of hardware problems.

Some tasks are known to fail, even when there is nothing wrong with hardware or devices. This may be because of unreasonable user-input (for instance, a requested resolution that is just out of reach) or, most likely, a failed image match. Image matching may fail without there being any fault on the beamline that would prevent data collections. We do not want to halt data collections in these cases, so the data collection process will simply skip over the data collection if a task reports FAILED. If the task in question is a well task (as image matching is), then all data collections in that well are skipped.

A side effect of this skipping is that the diffraction plans never get marked as COMPLETE (since there is no data collected against them), so if the plate comes to the endstation again (or the process were stopped and restarted), the skipped data collections would be attempted again. This may be, or may not be, undesirable, depending on the nature of the failure.

In addition, the data collection process will only tolerate task failures so much - if the same task fails for three sequential data collections, the process will stop and report an ERROR, on the assumption that something is wrong with the beamline. This is liable to interact badly in the face of poor user input.

23.4.4.5. Task Retries

There are instances where a task may encounter an error state for which there is a safe recovery and retry procedure, often relating to devices that don’t involve motion. Specific examples of this case are the OAV camera (the IOC is known to sometimes crash and needs to auto-restart) and reinitializing of the Eiger when arming.

A facility is provided to allow a task to make a single attempt to retry it’s run_task method after executing a recovery procedure - data_collection.error_handling includes the with_retry_action decorator. This will execute the provided recovery function when the task raises the specified exception (it is expected for a task to raise a specific exception for the known error case, and general exceptions for unhandled cases). The recovery function will be given the task instance as well as the arguments passed to the run_task method.

Example use of the retry decorator:

from data_collection.error_handling import with_retry_action

def task_recovery(task, *args, **kwargs):
    # run recovery procedure

class KnownErrorException(Exception): pass

class ErrorProneTask(VmxiTask):
    @with_retry_action(task_recovery, KnownErrorException)
    def run_task(self, arg1, arg2):
        # run normal code
        # if known_error_situation:
        #   raise KnownErrorException("that thing fell over again")

The OAV use of the above can be found in gda-mx.git/configurations/i02-2-config/scripts/data_collection/tasks/imaging/oav.py.

23.4.4.6. Interrupts

In general, it is preferred for a task to facilitate early termination, but it is not required. Short tasks that only do computation need not do so, but tasks that move hardware or are otherwise long running should do so.

The VmxiTask base class provides some default behaviour in this area, but a task should not leave the beamline in an indeterminate state (assuming it were the only thing running). It is not expected that a task stop immediately, some clean-up is anticipated, and devices may need to reach a specific state before the task can stop executing.

The stop facility is not a safety feature, but a convenience - it is completely valid for a task to do nothing in the face of a stop request, and run to completion.

23.4.4.7. Task Graph Creation

Tasks themselves hold no knowledge of their relationships with other tasks, or how their inputs and outputs will be managed. That information is held in data_collection.tasks.generation.

The dependencies between tasks are recorded as a dictionary of {task : [dependencies...] }. A task is only eligible for execution once all the tasks in its dependency list have run successfully.

The names of inputs and outputs to tasks are also held in two dictionaries of the form {task : [variable_names...]}. The order of names (for both inputs and outputs) must match the order of the arguments to, and return values of, run_task.

There is no association between argument names in the task definitions and the names given in these dictionaries, beyond their ordering.

23.4.4.8. Task Execution

The Task Executor is responsible for coodinating the execution of the Task Graph. The class TaskExecutor is implemented in gda-mx.git/configurations/i02-2-config/scripts/data_collection/tasks/executor.py. It receives a DAG from the Data Collection Controller and asychronously executes tasks when dependencies are satisfied, passing them the expected inputs and receiving outputs. The executor handles task failure and errors, stopping running tasks on faults.

The executor control loop runs as part of the Data Collection Process loop.

23.4.4.9. Task Concurrent Execution

Tasks that have no dependencies on each other (after considering transitive closure, i.e. are mutually unreachable from each other) may be executed concurrently. Therefore, care must be taken to ensure that this is both logically valid, and that it does not pose a risk to the beamline (notably with respect to collisions).

There is no mechanism to ensure that tasks do not compete for resources (i.e. hardware), however there are tests to ensure the graph produced by the functions in generation.py produce a truly acyclic graph and that all task inputs will be satisfied under any legal task execution order.

Just because concurrent execution is allowed, a task may not assume concurrent execution. There is no guarantee that any two tasks will ever run concurrently (even if they have the exact same dependencies and no other tasks can run). A valid (if undesirable) task “executor” is simply the blocking function def submit(task): task.execute().

23.4.5. Data Collection View

The following series of three screenshots show the progress through a data collection.

23.4.6. Data Collection throughput and timings

• ∼2 minutes to image and align a well.

• One ∼1 second rotation data collection every ∼8 seconds.

• High throughput achieved when many data collections done per well.

• Peak >200 data collections per hour.

• Grid scans (typically only one per well) only at 14 per hour.

• High variance in timings; difficult to say how long an average plate takes.