Generic Runsheet Generation Guide

Many instruments require, or can intake, a “runsheet” or “sample manifest” file. Often, these files are simple plain-text files that just need a few values plugged in dynamically. The “RunSheet2” API in the client/content is designed to facilitate this process in a way that is flexible but still acceptably performant.

Although it is possible to create a basic runsheet using python and espclient directly, the generic runsheet generation facilities were written to:

The general purpose runsheet support consists of three pieces:

The esp.data.access module provides a functional API for flexible and reasonably efficient¹ random access data retrieval. Under the hood, it uses the standard espclient OO models for much of the heavy data lifting, but also makes use of server-side expressions and queries for batched data lookups.

The two primary entry points into the module are the build_accessor function and the esp.data.access.EntityDataAccessor protocol². For full details, see the python documentation for that module, but briefly:

The get method may also be called with a single entity as accessor.get(my_entity) and will return data in the same shape as the input arguments (ie: pass a single entity, receive a single value; pass a list of values of the same length as the input list). The data accessor API is used heavily by the general runsheet machinery but has many additional uses. For instance, the default bartender integration uses it to fill in the csv “database” of entity data needed for a given label template.

import esp.data.access as access
from esp.models import Entity # 3.0. Use Sample if you need 2.4 + 3.0 compatibility.

@access.dataaccessor('divider')
class ExampleDataAccessor(access.EntityDataAccessor):
    """ Simple data accessor that makes a divider string. """
    def __init__(self, char, mult, *args, **kwargs):
        self.char = char
        self.mult = mult
        # IMPORTANT - YOU MUST CALL THE SUPER init! (it's also just good practice,
        # but for data accessors, it is mandatory)
        super(ExampleDataAccessor, self).__init__(*args, **kwargs)
    
    @classmethod
    @access.strip_specid
    def from_string(cls, string, label=None)
       char, mult = string.split(',')
       mult = int(mult)
       return cls(char, mult, label)

    def _get_list(self, samples, params=None):
      val = char * mult
      return [val]*len(samples)
      
example = access.build_accessor('divider:=,5')
example.get([Entity('one'), Entity('two')]) # returns ['=====', '=====']

A runsheet configuration is a simple yaml (or json) configuration that can be processed by the RunSheet2 class of the espclient.runsheet module. A runsheet configuration is a yaml or JSON key-value mapping file with at least the sections key. The sections detail how to map information from L7|ESP to the output runsheet file using the data accessor strings above. The espclient.runsheet module is then used to transform the configuration to the output file, filling in the L7|ESP data along the way. Although specific instrument/application connectors may require additional keys in the configuration (e.g.: Illumina support uses a params section), no additional keys are required in the general case. A fully-specified section (ie not relying on defaults) has the following keys:

- name: My Section
  type: table
  samples: primary
  supress_name: False
  show_headers: True
  name_format: "[{}]"
  values:
    - Source: sheet:Current.Well
  prepad_section: False
  postpad_section: False

OR

{
    "type": "table",  
    "name": "My Section",
    "samples": "primary",
    "values": [
        {
            "Source": "sheet:Current.Well"
        }
    ]    
    "name_format": "[{}]",
    "postpad_section": false,
    "prepad_section": false,
    "show_headers": true,
    "supress_name": false,
}

with meanings as detailed below. Unless otherwise specified, the keys are optional with defaults as indicated.

    values:
        - Sample Name: sampleinfo:name
        - File Name Convention: fixed:GlobalFiler
        - Results Group: fixed:GlobalFiler
        - Sample Type: fixed:Sample
        - Field 1: 'sampleinfo:name@@Individual|null_to_empty'
        - Field 2: 'sampleinfo:name@@Family|null_to_empty'

Sample Name,File Name Convention,Results Group,Sample Type,Field 1,Field 2
Sample 1,GlobalFiler,GlobalFiler,Sample,Individual 1,Family 1
Sample 2,GlobalFiler,GlobalFiler,Sample,Individual 2,Family 1

Sample Name,Sample 1
File Name Convention,GlobalFiler
Results Group,GlobalFiler
Sample Type,Sample
Field 1,Individual 1
Field 2,Family 1

Sample 1
GlobalFiler
GlobalFiler
Sample
Individual 1
Family 1

The espclient.runsheet module holds the classes used to generate run sheets given the configuration. The entry point to this functionality is the RunSheetTemplate class. A RunSheetTemplate is a programmatic representation of the configuration, where all data accessors have been resolved. Templates can be configured programmatically, but if you already have the configuration as yaml or json, you can easily load those into python and use the from_config class method as:

from esp.data.access import build_accessor
from esp.models import Configuration
from espclient.runsheet import RunSheetTemplate

conf = Configuration('My Runsheet Configuration')
template = RunSheetTemplate.from_config(conf.config, accessor=build_accessor)

from_config supports up to three arguments: the configuration, an optional visitor, and the optional accessor argument. For backwards compatibility, accessor defaults to an older, deprecated implementation of data accessors, so build_accessor should normally be passed to accessor. The visitor is for advanced usage scenarios and is an implementation of the visitor pattern (see also https://en.wikipedia.org/wiki/Visitor_pattern ). Once you have a template, it can be used with a list of entities to create a runsheet object:

# for ESP < 3.0, use Sample instead of Entity.
# Sample also works for 3.0 for backwards compatibility.
from esp.models import Entity, SampleSheet 

entities = Entity.search(names=['ESP000001', 'ESP000002])
ss = SampleSheet('My Sheet')
runsheet = template.resolve_runsheet(
    {'primary': entities},
    ss,
    {'worksheet': ss}, # Not required
    mode='v2'
)

In the example above, we also provided an L7|ESP SampleSheet object. This isn’t strictly necessary, but is required for accessors such as SheetInfo, ExperimentInfo, and SheetValue. As with the accessor=build_accessor, mode should be set to v2 to use the new implementation of data accessors (which you should use because it is generally much faster with cleaner internals than the deprecated implementation).

Once you have the runsheet object, there are a variety of operations that can be performed. The object is a wrapper around a list of “RunSheetSection” objects, each of which have section-type-dependent capabilities.

You can:

The ability to interact with the RunSheet in an object-oriented way makes it possible to combine the configuration-driven approach of generic runsheet with programatic manipulation of the resulting structure for advanced use-cases. For instance, the runsheet configuration might specify a table of values, so given the the starting code:

from esp.models import Entity, SampleSheet # for ESP < 3.0, use Sample instead of Entity. Sample also works for 3.0 for backwards compatibility.
entities = Entity.search(names=['ESP000001', 'ESP000002])
ss = SampleSheet('My Sheet')
runsheet = template.resolve_runsheet({'primary': entities}, ss, {'worksheet': ss}, mode='v2')

a simple application might do:

runsheet.to_csv('output_file.csv')

Where a more complex application might produce a custom output:

resolved_data = runsheet.sections[0].to_data_frame()
# use pandas functions for additional data manipulations.
resolved_data['Complex Column'] = resolved_data.apply(lambda row: float(row.A)*float(row.B), axis=1)
with open('output_file.txt', 'w') as f:
    f.write('PREAMBLE\n')
    f.write('========\n')
    resolved_data.to_csv(f)
    f.write('========\n')
    f.write('POSTFIX\n')

Although data accessors usually resolve one value for each entity provided, they may also resolve a list of values for each entity. That is, given the list of entities [s1, s2, s3], a call to accessor.get(entities) would usually return [v1, v2, v3], but may also return: [[v1.1, v1.2], [v2.1], [v3.1, v3.2, v.3.3]]. When constructing a runsheet, all data accessors used in table generation must either return the same number of results for each entity, or return a single result for each entity. That is, given entities s1, s2, and s3 and data accessors “x1” and “x2”, the following returns are valid or not as indicated in the table.

Many liquid handlers accept a “run sheet” which is a listing of source wells, source volumes, dest wells, and dest values. The following configuration shows how you might quickly write a configuration for a liquid handler pooling operation: putting all entities into a single target well. For the sake of simplicity, this example assumes a worksheet with protocol named “Current” that has fields “Well” (the source well) and “Volume” (volume to transfer).

Configuration:

Liquid Handler:
  sections:
    - name: Main Table
      suppress_name: True
      type: table
      samples: primary
      values:
        - Source: sheet:Current.Well
        - Source Volume: sheet:Current.Volume
        - Dest: fixed:A12

With the script written as:

import argparse

from esp.data.access import build_accessor
import espclient.runsheet.RunSheetTemplate as rs
from esp.models import Configuration
from esp.models.analysis import LimsAnalysis

parser = argparse.ArgumentParser(description='Generate a simple liquid handler picklist')
parser.add_argument("output_path", help="path to write the output file to")
opts = parser.parse_args()
la = LimsAnalysis.current()
template = rs.from_config(Configuration('Liquid Handler').config, accessor=build_accessor)
runsheet = template.resolve_runsheet(
  {'primary': la.samples},
  la.sample_sheet,
  {'worksheet': la.sample_sheet},
  mode='v2').to_csv()
with open(opts.output_path) as f:
  f.write(runsheet)

¹The dataaccess API is on part with SampleSheet save performance in that it will perform operations such as: caching certain client-side objects and (re)-using them from cache and bulk fetch of data values for all entities for a given accessor. Formally, it’s possible to write custom code or a custom query to fetch all needed data values in a single API call with greater computational efficiency, but at greater implementation time and maintenance cost.

²In python, a protocol defines the set of methods an object is expected to support along with arguments those methods accept and the type of values that must be returned. In some languages, this is called an Interface. . Note that prior to python 3.8, protocol was more of an abstract concept in Python, either implemented via abstract base classes, or purely via documentation. As of python 3.8, there is dedicated support language support for protocols, but the client supports python >= 3.5 (ESP < 3.0) and python >= 3.7 (ESP >= 3.0), so we haven't updated the code with this formal support for protocols yet.