23.7. 4°C Plates

The solution detailed in this section to provide GDA support of 4°C plates on VMXi is, of end-of 2019, awaiting testing on the beamline and is therefore subject to change based upon test outcome.

For context regarding contents of this section please see previous sections Data Collection (DC) and Sample Handling.

23.7.1. Background

For data collection to be supported at four degrees, sleeves are used that sit on top of the plate holder while it is on the goniometer. These sleeves contain ice to keep the plate cool for the duration of data collection.

Data collection on a plate may take several hours, whereas the ice in a plate will begin to melt after around 30 minutes. Once turned to water, its temperature will rapidly rise and will no longer cool the plate sufficiently.

Requirement: GDA must be able to interrupt data collection to remove a plate from the goniometer when required, without halting the DC process.

Requirement: The data collection process must auto-resume when given a new plate.

Sleeves are stored in a freezer section of the Local Storage area (see beamline overview diagram in overview section of VMXi Introduction).

It takes sometime for the water in a sleeve to refreeze, and a sleeve is not be used for data collection until it has completely frozen.

Requirement: GDA must be able to determine if a sleeve is eligible for use and only use such sleeves for data collection.

The implemented solution must take care that data collection does not attempt to run during a Load/Unload action on the goniometer.

Requirement: GDA must never start data collection during a robot Load/Unload action, or start such an action while the data collection process is running, even when users press Start/Stop at unsuitable times. I.e. there are to be no race conditions.

Requirement: Sample handling must be able to unload even if data collection is not running, or there are errors.

23.7.2. Solution Overview

The Sample Handling Process reads sleeve locations and timestamps from ISPyB, if a sleeve on the goniometer is determined to be warming, a Plate Timeout flag is set on the LIMS Agent, and if it’s critical an interrupt is sent to the Data Collection Process. This interrupt will abort the current data collection, but not halt the process.

The Data Collection Process promises not to start a new data collection if the LIMS Agent Plate Timeout flag is set (this represents an addition to the contract), and will enter a waiting state.

The Sample Handling Process will check the running state of the Data Collection Process and is allowed to unload a plate if the relevant sleeve is not cold and the Data Collection Process is not running.

23.7.3. Sleeves

23.7.3.1. Description

Information regarding sleeves is stored in ISPyB, and timestamps are used to determine if a sleeve can be used, or has to be returned (for details see VMXI-497 - Work out how to store sleeve information for 4C plates Active).

A sleeve is fully described by the triple (sleeve_id, location, timestamp)

  • sleeve_id - is a unique identifier for the sleeve and refers to its reserved slot in the freezer

  • location - refers to the current location (the plate holder it is currently on), or is NULL (meaning it is in the freezer)

  • timestamp - time sleeve was moved to its current location

The single timestamp can be used to decide the state of the sleeve - if it was moved out of the freezer 30 minutes ago, then it is close to melting, and it must have been put into the freezer three hours ago (for example) before it may be reused.

The LIMS Agent has been updated with a getSleeves method that will return all sleeves in the database (expected to be ~10).

23.7.3.2. Sensor

A simple SenseSleeves has been added to sample_handling.sense. This sensor calls Lims::getSleeves and determines their state by checking against durations from Beamline Parameters.

The states returned by the sensor are:

  • SLEEVE_COLD - Sleeve is allowed to be used (has been in freezer long enough, or not outside for very long)

  • SLEEVE_WARMING - Sleeve is close to threshold where it must be returned to the freezer

  • SLEEVE_WARM - Sleeve must be returned ASAP (interrupt data collection), or has not been in the freezer long enough to be used.

It was decided not to implement Sophisticated heuristics (calculating how long a sleeve must be in the freezer based on how long it was outside).

23.7.4. Data Collection Interrupts

The contract between Data Collection and Sample Handling is handled by the Plate Shot status (Lims::setPlateCompleted). Data Collection will never do anything with a shot plate, and Sample Handling will only ever unload a shot plate, so DC can mark a plate completed after it has finished and there is no issues.

The additional requirement that a plate be unloadable without being complete adds complications. An explicit state that is checked and reset is now needed.

23.7.4.1. Plate Timeout Flag

A global interrupt flag has been added in the LimsAgent (see LIMS Agent), called the Plate Timeout (Lims::timeoutPlate, Lims::isPlateTimedout, Lims::clearPlateTimeout). Data Collection may only start data collections if this flag is clear (if it is set, it should enter the data_collection_processor.ProcessRunState.WAITING state).

Therefore, if this flag is set, any running data collection will finish, but no more will start, and the processor will enter a WAITING state, allowing safe plate unloads.

23.7.4.2. Data Collection Interruptions

It may be required that a data collection be interrupted if a sleeve reaches the critical threshold (SLEEVE_WARM state). A new interrupt method has been added to the Data Collection Processor (data_collection_processor.VmxiDataCollectionProcess::timeout_current_plate). This method will trigger an abort on the TaskExecutor, but will not result in termination of the control loop.

The combination of both the LimsAgent timeout flag and this new interrupt allow early termination of data collections and prevention of new ones until the flag is reset.

23.7.5. Sample Handling

Since it is the Sample Handling Process that actually removes a plate when the sleeve’s ice has turned to water, it makes sense for this process to be responsible for interrupting data collection.

This adds a degree of interactions that were not present before, as now Sample Handling must both interrupt, and check the state of the Data Collection Process.

23.7.5.1. Data Collection Sensor

A data collection process sensor has been added, that returns if data collection is running or not (using the new data_collection_processor.VmxiDataCollectionProcess::run_state method).

A plate unload is added to the list of allowed actions (in sample_handling.actions.selection.ActionSelector) if a sleeve is present on the goniometer, its status is not SLEEVE_COLD, and the Data Collection Process is not running, even if it isn’t shot.

23.7.5.2. Data Collection Interrupt

A data collection interrupt check is added as part of the Sample Handling control loop. After the sensors are checked, the state of any active sleeves are checked.

  • If a sleeve on the goniometer has the status SLEEVE_WARMING, the LimsAgent timeout flag is set.

  • If a sleeve on the goniometer has the status SLEEVE_WARM, a timeout_plate interrupt is made on the Data Collection Process.

These actions are only executed once between any successive actions of the Sample Handling loop (to reduce log-message spam). It is, however, safe to spam interrupts to the controllers.

23.7.5.3. Clearing Timeout

The LimsAgent timeout flag is cleared by the Sample Handling actions LocalStorageToGonio and GonioToLocalStorage. As these actions refresh the state of plate on the goniometer, it makes sense for them to reset the flag.

Beamline staff may need reminding of this if they ever perform manual interventions (placing a plate and sleeve on the goniometer manually) after a timeout.