Examples¶

Below detail code-heavy examples for how to use the client API. They’re not meant to be comprehensive, but rather give users a feel for how to traverse and interact with the API, particularly for filling data in LIMS workflows.

Create and Submit Experiment¶

Similarly to how you can create and submit an experiment via YAML configuration (see below), you can also control how experiments are managed in pure python.

from esp.models import Experiment

# create experiment with 2 new illumina library samples
exp = Experiment.create(
  name='My Experiment',
  workflow='Example Workflow',
  project='My Project',
  samples={
    'count': 3,
    'type': 'Illumina Library'
  }
)

# submit experiment and create sample sheet
exp.submit(sheet=True)

To submit a sample sheet with a specific name, you can use:

exp.subit(sheet='My Sheet Name')

Update Data in a SampleSheet¶

You can also use the client to update data in a sample sheet programmatically. For instance, here’s how you fill in data for a specific column in a sample sheet from an experiment.

from esp.models import Experiment

# get submitted experiment
exp = Experiment('My Experiment')

# get first protocol from experiment
sheet = exp.protocols[0]

# you can alternatively query via protocol name
sheet = exp.protocol('My First Protocol')

# show first few rows of data
sheet.head()

# set all values for a column in a worksheet
sheet['Column 1'] = 'test'

# save values in the application
sheet.save()

Create a Sample Sheet from Multiple Experiments Programmatically¶

In some contexts, it’s useful to combine experiments into a single sample sheet for data recording. You can do this in the client using the SampleSheet object.

from esp.models import Experiment, SampleSheet

# get experiment objects
exp1 = Experiment('Experiment 1')
exp2 = Experiment('Experiment 2')

# create sample sheet with experiments
sheet = SampleSheet.create(name='Combined Sheet', experiments=[exp1, exp2])

Change the Name of a Sample and add Tags¶

from esp.models import Sample

# get sample and make sure it exists
sample = Sample('ESP000001')
if not sample.exists():
  raise AssertionError('Sample does not exist!')

# change name and add tags
sample.name = 'ESP-1'
sample.tags.extend(['foo', 'bar'])

# save updates
sample.push()

Complete a Workflow Programmatically¶

In contexts where data entry can be automated, you can use the client to populate data for workflows. This can be particularly useful if users are migrating data from a legacy system and want to maintain data structures created throughout workflows and workflow chains. Here is a YAML config example for creating an experiment and filling out data:

name: MS001
workflow: Illumina Sequencing
project: Miseq Sequencing
samples:
    count: 2
    type: Extract
protocols:
  - Set Samples:
      complete: true
      data:
        Attachment: ${CWD}/runsheet.txt
        Note: ['Ready to go!', 'Ready!']

  - Create Illumina Library:
      complete: true
      load: ${CWD}/spreadsheet.txt'

  - Analyze Sequencing Results:
      run: true
      complete: true
      data:
        Container: 'MiSeq: Lane 1'
        Reference: ['GRCh38', 'GRCh37']

You can run this config with the client using the python client CLI:

~$ esp import experiment path/to/experiment.yaml

Or, you can run it in a python shell using Experiment.create():

from esp.models import Experiment

exp = Experiment.create('path/to/experiment.yaml')

You can also use the python client to control the process programmatically in a configuration-free way. Here is matching Python code to fill data in data via code:

from esp.models import Experiment

# create experiment with samples
exp = Experiment.create(
  name='MS001',
  workflow='Illumina Sequencing',
  project='Miseq Sequencing',
  samples={
    'count': 2,
    'type': 'Extract'
  }
)

# submit experiment to sheet
exp.submit()

# fill in data for first protocol
sheet = exp.protocol('Set Samples')
sheet['Attachment'] = 'runsheet.txt'
sheet.loc[:, 'Note'] = ['Ready to go!', 'Ready!']

# save and complete sheet
sheet.save()
sheet.complete()

# fill in data for second protocol
sheet = exp.protocol('Create Illumina Library')
sheet.load('spreadsheet.txt', sep='\t')
sheet.save()
sheet.complete()

# fill in data for final (pipeline) protocol
sheet = exp.protocol('Analyze Sequencing Results')
sheet['Container'] = 'Miseq: Lane 1'
sheet.loc[:, 'Reference'] = ['GRCh38', 'GRCh37']
sheet.save()

# run pipeline for sheet
sheet.run()

# complete sheet
sheet.complete()

Complete a Workflow Chain Programmatically¶

Just like with workflows, it’s possible to complete a workflow chain programmatically or with configuration. Here is an example config for filling workflow chain data:

name: MF Chain 1
chain: Manufacturing Chain
project: Manufacturing
samples:
    count: 3
    type: Tissue
workflows:
    - Initiate Manufacturing
        submit: true
        transition: true
        protocols:
            - Initiate Manufacturing:
                complete: true
                data:
                  Start Date: 01Jan18
                  Signature: true
    - Finish Manufacturing:
        submit: true
        protocols:
            - Process A:
                data:
                  Completion Date: 01Jan18
                  Signature: true

You can run this config with the client using the python client CLI:

~$ esp import experiment_chain path/to/experiment_chain.yaml

Or, you can run it in a python shell using ExperimentChain.create():

from esp.models import ExperimentChain

exp = ExperimentChain.create('path/to/experiment_chain.yaml')

You can also use the python client to control the process programmatically in a configuration-free way. Here is matching Python code to fill data in data via code:

from esp.models import ExperimentChain

# create experiment with samples
chain = ExperimentChain.create(
  name='MF Chain 1',
  workflow=' Manufacturing Chain',
  project='Manufacturing',
  samples={
    'count': 3,
    'type': 'Tissue'
  }
)

# get first node in chain and submit
exp = chain.experiments[0]
exp.submit()

# fill data for first experiment sample sheet
sheet = exp.protocols[0]
sheet['Start Date'] = '2018-01-01'
sheet['Signature'] = True

# submit sheet and transition experiment chain
sheet.save()
sheet.complete()
sheet.transition()

# get second node in chain and submit
exp = chain.experiments[1]
exp.submit()

# fill in data for second node in chain
sheet = exp.protocols[0]
sheet['Completion Date'] = '2018-01-01'
sheet['Signature'] = True
sheet.save()
sheet.complete()

Ingest Data from an External Service¶

The following code details how to ingest data from an arbitrary process into ESP via the LIMS application.

# Make sure protocol and workflow exist (ESP requirement for adding data)
pc = Protocol.create(name='Ingest Protocol')
wf = Workflow.create(name='Ingest Workflow', protocols=[pc])

# Read Ingest data from File
import pandas
df = pandas.read_csv('iris.txt', sep='\t')

# Configure ingest protocol to take new columns
# (accounting for column differences across files)
pc.variables = [{'name': col, 'var_type': 'text'} for col in df.columns]
pc.push()

# Create experiment and push data
Experiment.create(
    name='My Ingest',
    workflow=wf,
    samples=dict(count=len(df)),
    submit=True,
    protocols=[dict(name='Ingest Protocol', data=df)])

Find and print the creation date/creator of all samples with the tag “test”¶

Sometimes it’s useful to query all samples in the application and filter by some criteria. Here’s how to do that using Sample tags:

for sample in Sample.all():
    if 'test' in sample.tags:
        print('{}: {} by {}'.format(sample.name, sample.created_at, sample.owner.name))

Alternatively, you can search by tag using the Sample.search classmethod:

samples = Sample.search(tags=['test'])

Note

The .search() method works for other models as well.

Find and Submit all Experiments with >10 Samples from a Specific Project¶

You can also use the client for managing project data. For instance, here’s an example of searching for experiments using specific criteria, and then submitting them:

from exp.models import Project
obj = Project('My Project')
for exp in obj.experiments:
    if len(exp.samples) > 10 and not exp.submitted:
        exp.submit()

Do a PCA with data from a specific Experiment¶

Here’s a more comprehensive example of how one might use the client for data science applications. Below, the client is used to pull all metadata within a project to perform a Principal Component Analysis:

# imports
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.pipeline import make_pipeline

# load the project
project = Project('My Project')
project.experiments

# get the data
df = pd.concat(map(
    lambda x: x.protocols[0].df,
    project.experiments
)).drop(['Complete'], axis=1)

# fit model on the data
X, Y = df.drop(['Species'], axis=1).values, df.Species.values
model = make_pipeline(
    StandardScaler(),
    PCA(n_components=3),
)
tX = model.fit_transform(X)

Note

More examples will be included in future versions of the documentation.

User¶

class esp.models.User(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with users from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create user:

name: New User
email: user@localhost
password: password

Create user with roles:

name: New Role User
email: user-role@localhost
password: password
roles:
    - Lab Manager
    - Admin

Examples

>>> from esp.models import User
>>> user = User('New User')
>>> user.name, user.created_at
('User', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> user.roles
[<Role(name=Lab Manager)>, <Role(name=Admin)>]
>>> user.workgroups
[<Workgroup(name=Lab A)>]

>>> # log in as specific user
>>> User('New User').login(password='password')

>>> # get current user
>>> user = User.current()

Parameters:: ident (str) – Name or uuid for object.

Role¶

class esp.models.Role(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with users from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create role:

name: my-role
desc: My Role
permissions:
    Project: [create, update]
    Sample: [create, read, update, delete]

Create role with default users:

name: my-role
desc: My Role
members:
    - User 1
    - User 2
applications:
    - admin
    - lims

Create role with specific app permissions:

name: my-role
desc: My Role
applications:
    - admin
    - lims

Configuration Notes:

Default role permissions are access to all actions for each type of model. Permissions need to be overridden to restrict access.

Examples

>>> from esp.models import Role
>>> role = Role('my-role')
>>> role.name, role.created_at
('my-role, '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> role.users
[<User(name=User 1)>, <User(name=User 2)>]

Parameters:: ident (str) – Name or uuid for object.

ParamGroup¶

class esp.models.ParamGroup(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with ParamGroups from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create param group:

illumina:
    miseq-1: /path/to/miseq-1
    miseq-2: /path/to/miseq-2

Create multiple param groups:

illumina:
    miseq-1: /path/to/miseq-1
    miseq-2: /path/to/miseq-2

integrations:
    clarity: http://clarity-url:1000
    internal: http://internal-server

Examples

>>> from esp.models import ParamGroup
>>> pg = ParamGroup('apps')
>>> pg.name, group.created_at
('apps, '2019-06-21T16:04:01.199076Z')

>>> # show params
>>> pg.params.client
'/path/to/client/bin/esp'

Parameters:: ident (str) – Name or uuid for object.

Workgroup¶

class esp.models.Workgroup(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with workgroups from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create workgroup:

name: Lab A
desc: Workgroup for all users in Lab A

Create workgroup with default users:

name: Lab A
desc: Workgroup for all users in Lab A
members:
    - User 1
    - User 2

Examples

>>> from esp.models import Workgroup
>>> group = Workgroup('Lab A')
>>> group.name, group.created_at
('Lab A, '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> group.users
[<User(name=User 1)>, <User(name=User 2)>]

Parameters:: ident (str) – Name or uuid for object.

Config¶

class esp.models.Config(*args, **kwargs)[source]

Object for interacting with instance configuration within ESP. This model allows developers to directly interact with the /api/config endpoint in ESP.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

That is, this configuraiton is always updaitng a DB row, never truly creating it, but uses the same interface as all other models for consistency.

This model is deprecated and will be removed in the future. Use Configuration instead.

Configuration¶

class esp.models.Configuration(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with database-backed instance configuration within ESP. This model allows developers to directly interact with the /api/configuration endpoint in ESP.

Model to interact with ESP’s DB-backed configuration system.

A configuration consists of two top-level dictionaries: one for “public” configuration and one for “secret” configuration. Generally, this model masks the distinction and makes the subkeys of both available as top-level properties, but also supports direct access of the public and secret dictionaries.

Examples

>>> from esp.models import Configuration
>>> conf = Configuration('esp') # get the ESP system config.
>>> conf.executor.num_workers
1
>>> conf = Configuration.create(name='CustomConfig', key1='value1', key2={'subkey1': 'value1.1', 'subkey2': 'value2.2'})
>>> conf.key2
{'subkey1': 'value2.1', 'subkey2': 'value2.2'}
>>> conf.key2.subkey1
'value2.1'
>>> conf.config
{'key1': 'value2', 'key2': {'subkey1': 'value2.1', 'subkey2': 'value2.2'}}
>>> conf.add_secret('secretkey', 'this value is secret')
>>> conf.secrets
{'secretkey': 'this value is secret'}
>>> conf.push()
>>> conf.refresh()
>>> conf.secrets
{'secretkey': 'XXXXXXX'}
>>> conf.secretkey
'XXXXXXX'
>>> conf.newkey = 'new value'
>>> conf.config
{'key1': 'value2', 'key2': {'subkey1': 'value2.1', 'subkey2': 'value2.2'}, 'newkey': 'new value'}
>>> conf.add_secret('newkey', 'conflicting secret')
>>> conf.newkey
'conflicting secret'
>>> conf.config.newkey
'newkey'

File¶

class esp.models.File(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with files from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create file:

name: My Protocol SOP
desc: An SOP file
tags: [sop, protocol]
uri: relative/path/to/instructions.pdf
upload: true
type: PDF

Create file object with reference to existing file:

name: Large Sequencing File
desc: An SOP file
tags: [sop, protocol]
uri: file:///path/to/local/file/instructions.bam
upload: true
type: raw

Create file object and register as task output:

name: SampleSheet.csv
desc: Illumina Run Sheet
task_file: Illumina Runsheet
uri: /path/to/SampleSheet.csv
upload: false
type: csv

Configuration Notes:

Due to current backend limitations, uri inputs can only take the formats: “relative/path”, “/absoluate/path”, and “file:///absolute/path”.

If upload is true, uri must resolve to an existing file on the local file system, otherwise an error will be raised. If upload is False, file will only be registered. Upload defaults to false.

The task_file parameter is primarily useful when registering pipeline output files (since you can pass in the task instance UUID).

“type” is used internally by ESP and is not necessary a mime type. For instance, the mime type of a bam file is generally application/gzip or application/octet-stream, but the type in ESP might normally be “bam”.

mime-type is an acceptable key for specifying the mime-type. If no mime-type is provided, the client will attempt to guess the type and pass that type along as the content type of the uploaded file.

Examples

>>> from esp.models import File
>>> fi = File('My Protocol SOP')
>>> fi.name, fi.created_at
('My Protocol SOP', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> fi.path
'/path/to/Lab7_ESP/Data/files/0000/instructions.pdf.6a523a74-6703-4474-a24e-5a463b9d9770'
>>> fi.contents
'... raw file contents ...'
>>> fi.download('local-copy.pdf')

Parameters:: ident (str) – Name or uuid for object.

Report¶

class esp.models.Report(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with Reports from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create report with embedded html:

name: My Report
desc: An example html report.
tags: [html, demo]
contents:
    <h1>My Report</h1>

Create report with local html file:

name: My Report
desc: An example html report.
tags: [html, demo]
contents: $LAB7DATA/contents/reports/my-report.html

Create applet from local html file:

name: My Report
desc: An example html applet.
tags: ['esp:applet']
contents: $LAB7DATA/contents/reports/my-report.html

Configuration Notes:

To promote an ESP report to an ESP applet, include esp:applet in the set of report tags.

The contents parameter can either take a local file or raw html contents to use as the report data. If no file is found from the string, the data are assumed to be raw html data to include as contents.

Examples

>>> from esp.models import Report
>>> report = Report('My Report')
>>> report.name, report.created_at
('My Report', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> report.contents
'<h1>My Report</h1>'

Parameters:: ident (str) – Name or uuid for object.

Pipeline¶

class esp.models.Pipeline(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with Pipelines from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create pipeline with single task:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
tasks:
    - Create Illumina Runsheet

Create pipeline with nested tasks:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
tasks:
    - Create Illumina Runsheet:
        desc: Run script to generate illumina runsheet.
        cmd: /path/to/script.py {{project}} > runsheet.xml
        files:
            - Illumina Runsheet:
                file_type: xml
                filename_template: '{{ "runsheet.xml" }}'

Create pipeline with task dependencies:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Create pipeline with report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - []
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Create pipeline with report and failure report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - []
failure_report:
    Runsheet Failure Report:
        desc: Report showing details of failed runsheet generation
        elements
        - - type: file_details
            depends:
                file: stderr
                tasknumber:1
        - []
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Configuration Notes:

Current implementation requires special syntax around creating report elements - - and - []. This is due to payload requirements and will be updated in future versions to be more user-friendly.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')
>>> pipe.name, pipe.created_at
('Generate Illumina Runsheet', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> pipe.tasks
[<Task(name=Create Illumina Runsheet)>, <Task(name=Upload Illumina Runsheet)>]
>>> pipe.report
<ReportTemplate(name=Runsheet Report)>

>>> # run pipeline (wait to finish)
>>> analysis = pipe.run(project='My Project')
>>> analysis.task[0].stdout
'stdout from the first analysis task'
>>> analysis.task[0].files
[<File(name=Illumina Runsheet)>]

>>> # run pipeline (in background)
>>> analysis = pipe.run(project='My Project', background=True)
>>> analysis.pause()
>>> analysis.restart()
>>> analysis.kill()

Parameters:: ident (str) – Name or uuid for object.

Task¶

class esp.models.Task(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with Tasks from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple task (no variables):

name: Ping Internal Server
desc: Run command to ping internal server.
cmd: curl http://internal-server

Simple task with variables:

name: Ping Internal Server
desc: Run command to ping internal server.
cmd: curl '{{ server }}'

Create task and track outputs:

name: Create Illumina Runsheet
desc: Run script to generate illumina runsheet.
cmd: /path/to/generate_runsheet.py {{project}} > runsheet.xml
files:
    - Illumina Runsheet:
        file_type: xml
        filename_template: '{{ "runsheet.xml" }}'

Create task with inline code:

name: Report DNA Type
desc: Run bash script to generate simple report.
cmd: |+
   # Simple Task that determines if the specified 'ratio' is in the proper range
   if [ "{{ type }}" = "DNA" ]; then
       echo "<b>This is a <font color='green'>DNA</font> sample</b>" > result.html
   elif "{{ type }}" = "DNA" ]; then
       echo "<b>This is a <font color='green'>RNA</font> sample</b>" > result.html
   fi

files:
    - Type Report:
        file_type: html
        filename_template: '{{ "report.html" }}'

Examples

>>> from esp.models import  Task
>>> task  = Task('Create Illumina Runsheet')
>>> task.name, task.created_at
('Create Illumina Runsheet', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> task.variables
['project']
>>> task.cmd
'/path/to/generate_runsheet.py {{project}} > runsheet.xml'

Parameters:: ident (str) – Name or uuid for object.

PipelineReport¶

class esp.models.PipelineReport(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with PipelineReports from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create pipeline with report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - []
tasks:
    - Create Illumina Runsheet

Create multi-step pipeline with report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements:
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - - type: raw_file
            depends:
                file: Runsheet Upload Report
                tasknumber:2
        - - type: html
            contents: |+
                <h1>Report Header</h1>
                <p>Report Body</p>
        - []
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Configuration Notes:

Report configuration will generally happen in the context of a pipeline, Accordingly, this documentation references report generation in that context.

Current implementation requires special syntax around creating report elements - - and - []. This is due to payload requirements and will be updated in future versions to be more user-friendly.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Create Illumina Runsheet')
>>> pipe.name, pipe.created_at
('Generate Illumina Runsheet', '2019-06-21T16:04:01.199076Z')
>>> pipe.report.name, pipe.created_at
('Runsheet Report', '2019-06-21T16:04:01.199076Z')
>>> pipe.report.pipeline.name, pipe.report.pipeline.created_at
('Generate Illumina Runsheet', '2019-06-21T16:04:01.199076Z')

Parameters:: ident (str) – Name or uuid for object.

Analysis¶

class esp.models.Analysis(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with analyses from the ESP database.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')

>>> # run pipeline to return analysis object
>>> analysis = pipe.run(project='My Project')
>>> analysis.task[0].stdout
'stdout from the first analysis task'
>>> analysis.task[0].files
[<File(name=Illumina Runsheet)>]

>>> # run pipeline (in background)
>>> analysis = pipe.run(project='My Project', background=True)
>>> analysis.pause()
>>> analysis.restart()
>>> analysis.kill()

>>> # get current analysis (within script)
>>> analysis = Analysis.current()
>>> analysis.pause()

Parameters:: ident (str) – Name or uuid for object.

AnalysisTask¶

class esp.models.AnalysisTask(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with task instances from the ESP database.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')

>>> # run pipeline to return analysis object
>>> analysis = pipe.run(project='My Project')

>>> # get first task object from analysis
>>> atask = analysis.task[0]
>>> atask.stdout
'stdout from the first analysis task'
>>> atask.stderr
'stderr from the first analysis task'
>>> atask.files
[<File(name=Illumina Runsheet)>]

Parameters:: ident (str) – Name or uuid for object.

AnalysisPool¶

class esp.models.AnalysisPool(analyses)[source]

Object for interacting with analyses from the ESP database.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')

>>> # run multiple scripts to gather in pool
>>> foo_analysis = pipe.run(project='Foo Project')
>>> bar_analysis = pipe.run(project='Bar Project')

>>> # create analysis pool to control execution
>>> pool = AnalysisPool([foo_analysis, bar_analysis])
>>> pool.state
'running'

>>> # wait for all analyses to finish in pool
>>> pool.join()
>>> pool.state
'done'

Parameters:: analyses (str) – List of analyses to include in pool.

ContainerType¶

class esp.models.ContainerType(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with container types from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Single-element container:

name: 96-Well Plate
desc: 96-Well Plate for sample aliquots
tags: [plate, lab]
label_format: '%s%s'
slot_capacity: element
axes:
    - Rows: [A, B, C, D, E, F, G, H]
    - Cols: [01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12]
contains:
    - Sample

Multi-element container:

name: Freezer
desc: Freezer for samples and racks
tags: [freezer, lab]
label_format: 'Shelf %s'
slot_capacity: list
axes:
    - Shelf: [1, 2, 3, 4, 5]
contains:
    - Sample
    - Container

Type with default containers:

name: Freezer
desc: Freezer for samples and racks
tags: [freezer, lab]
label_format: 'Shelf %s'
slot_capacity: list
axes:
    - Shelf: [1, 2, 3, 4, 5]
contains:
    - Sample
    - Container
create:
    - Freezer 1
    - Freezer 2

Nested config with containers and samples to fill:

name: Freezer
desc: Freezer for samples and racks
tags: [freezer, lab]
label_format: 'Shelf %s'
slot_capacity: list
axes:
    - Shelf: [1, 2, 3, 4, 5]
contains:
    - Sample
    - Container
create:
    - Freezer 1
    - name: Freezer 2
      barcode: 1234
      fill:
        Shelf 1: [ESP0001, ESP0002]

Examples

>>> from esp.models import ContainerType
>>> ct = ContainerType('Freezer')
>>> ct.name, ct.created_at
('Freezer', '2019-06-21T16:04:01.199076Z')
>>> ct.containers
[<Container(name=Freezer 1)>, <Container(name=Freezer 2)>]

Parameters:: ident (str) – Name or uuid for object.

Container¶

class esp.models.Container(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with containers from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create container:

name: Freezer 1
desc: Freezer in lab A
tags: [freezer, lab]
type: Freezer
barcode: 12345

Container with samples to fill:

name: Freezer 1
desc: Freezer in lab A
tags: [freezer, lab]
type: Freezer
barcode: 12345
fill:
    Shelf 1:
        - ESP0001
        - name: ESP0002
          desc: special new sample
          tags: [one, two]

Configuration Notes:

Upon creation, samples can be used to fill container slots via the fill parameter.

Default container values can be set for samples using the variables parameter.

Examples

>>> from esp.models import Container
>>> ct = Container('Freezer 1')
>>> ct.name, ct.created_at
('Freezer 1', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> ct.slots
['Shelf 1', 'Shelf 2', 'Shelf 3', 'Shelf 4', 'Shelf 5']
>>> ct.samples
[<Sample(name=ESP0001)>, <Sample(name=ESP0002)>]
>>> ct.slot_samples
{
    'Shelf 1': [<Sample(name=ESP0001)>, <Sample(name=ESP0002)>],
    'Shelf 2': [],
    'Shelf 3': [],
    ...
}

>>> # put items in specific slots
>>> ct.put_items_in_slots(
>>>     items=[Sample('ESP0001'), Sample('ESP0002')],
>>>     slots=['Shelf 1', 'Shelf 2']
>>> )

Parameters:: ident (str) – Name or uuid for object.

Customer¶

class esp.models.Customer(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with Customers from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple customer:

name: Customer 1
desc: Customer 1 from the North Pole.
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Customer without address:

name: Customer 2
desc: Customer 2 without address.
tags: [demo, north]

Customer with billing and shipping address:

name: Customer 3
desc: Customer 3 from the North Pole
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole
shipping_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Configuration Notes:

Customer addresses are optional, and the minimal config for a customer is one with only a specified name parameter.

Parameters:: ident (str) – Name or uuid for object.

ItemType¶

class esp.models.ItemType(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with ItemTypes from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Item type for stock reagent:

name: Demo Reagent
consumption_type: stock reagent
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent

Item type for consumable:

name: Demo Consumable
consumption_type: consumable
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent

Item type for kit component:

name: Demo Kit Component
consumption_type: kit component
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent

Create item type with associated items:

name: Consumable
consumption_type: consumable
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent
create:
    - CNS001
    - CNS002

Examples

>>> from esp.models import ItemType
>>> it = ItemType('Demo Consumable')
>>> it.name, it.created_at
('Demo Consumable', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> it.vendors
[<Vendor(name=Agilent)>]

Parameters:: ident (str) – Name or uuid for object.

Item¶

class esp.models.Item(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with ItemTypes from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple item:

name: Example Item
tags: [agilent, perishable]
barcode: 45262341
barcode_type: QR
item_type: Demo Consumable
vendor: Agilent
lot_id: Lot # 1
serial_id: 12345
initial_qty: 10
expires: 12/25/2019
status: Quarantined

Item with different barcode type:

name: Example Item
tags: [agilent, perishable]
barcode: 1234
barcode_type: 1D
item_type: Demo Consumable
vendor: Agilent
lot_id: ABC

Examples

>>> from esp.models import Item
>>> item = Item('Example Item')
>>> item.name, item.created_at
('Example Item', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> it.vendor
<Vendor(name=Agilent)>

>>> # useful functions
>>> item.expired()
False
>>> item.consumed()
True
>>> item.consumed()
True

>>> # interacting with item
>>> item.add_quantity(10)
>>> item.remove_quantity(5)
>>> item.empty()

>>> # interacting with variables
>>> item = Item('Example Item', params='["resource_vals"]')
>>> item.variables
>>> item.variables['Example Variable']

>>> # adding item with variables
>>> item_to_add = {
>>>     'name': 'The Name of the Item',
>>>     'description': 'Item Description',
>>>     'item_type': 'Item Type Name',
>>>     'initial_qty': 22,
>>>     'status': 'Quarantined',
>>>     'resource_vals': [{'name': 'The name of custom field', 'value': 'The Value'}]
>>> }
>>> Item.create(item_to_add)

Parameters:: ident (str) – Name or uuid for object.

ServiceType¶

class esp.models.ServiceType(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with ServiceTypes from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Sample service type:

name: Sample Service Type
desc: Service type for a demo service.
tags: [service, demo]
price_type: sample
base_price: 350

Time service type:

name: Time Service Type
desc: Service type for a time service.
tags: [service, demo]
price_type: time
base_price: 350
scale_price: 25
scale_units: minutes

Configuration Notes:

Available options for ServiceType price_type parameters are: sample, protocol, workflow, time, or volume. However, only sample and time are currently supported in the UI.

Available options for ServiceType scale_units parameters are: hour, minutes, and seconds.

Parameters:: ident (str) – Name or uuid for object.

Service¶

class esp.models.Service(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with Service from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple service:

name: My Service
service_type: My Service Type
customer: Lab1

Service with embedded type and customer:

name: My Service
service_type:
    My Service Type:
        price_type: time
        base_price: 350
        scale_price: 25
        scale_units: minutes
customer:
    Lab1:
        billing_address:
            street1: 1234 Northword Lane
            street2: Suite # 123
            city: North Pole
            state: North Pole
            country: North Pole
            postal_code: North Pole

Examples

>>> from esp.models import Service
>>> service = Service('My Protocol SOP')
>>> service.name, service.created_at
('My Service', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> service.service_type.name
'My Service Type'
>>> service.customer.name
'Customer 1'

Parameters:: ident (str) – Name or uuid for object.

Vendor¶

class esp.models.Vendor(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with Vendors from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple vendor:

name: Vendor 1
desc: Vendor 1 from the North Pole.
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Vendor without address:

name: Vendor 2
desc: Vendor 2 without address.
tags: [demo, north]

Vendor with billing and shipping address:

name: Vendor 3
desc: Vendor 3 from the North Pole
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole
shipping_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Configuration Notes:

Vendor addresses are optional, and the minimal config for a customer is one with only a specified name parameter.

Parameters:: ident (str) – Name or uuid for object.

Project¶

class esp.models.Project(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with Projects from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple project:

name: My Project
desc: Project for a series of experiments.
tags: [demo, test]

Project with nested experiments:

name: My Project
desc: Project for a series of experiments.
tags: [demo, test]
experiments:
    - My Experiment 1:
        workflow: Workflow 1
        submit: true
        samples: [ESP001, ESP002]

Project with nested experiment chains:

name: My Project
desc: Project for a series of experiments.
tags: [demo, test]
chains:
    - My Experiment Chain 1:
        chain: Workflow Chain 1
        samples: [ESP001, ESP002]

Configuration Notes:

Experiments and Experiment Chains can be nested within project configuration, which is particularly useful for seeding data in bulk or creating test resources.

Examples

>>> from esp.models import Project
>>> project = Project('Project 1')
>>> project.name, project.created_at
('Project 1', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> project.experiments
[<Experiment(name=Experiment 1)>, <Experiment(name=Experiment 2)>]
>>> project.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]

Parameters:: ident (str) – Name or uuid for object.

ExperimentChain¶

class esp.models.ExperimentChain(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with ExperimentChains from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create simple chain:

name: MF Chain 1
chain: Manufacturing Chain
project: Manufacturing
samples:
    count: 3
    type: Tissue

Create chain with nested experiments:

name: MF Chain 1
chain: Manufacturing Chain
project: Manufacturing
samples:
    count: 3
    type: Tissue
workflows:
    - Initiate Manufacturing
        submit: true
        transition: true
        protocols:
            - Initiate Manufacturing:
                complete: true
                data:
                  Start Date: 01Jan18
                  Signature: true
    - Finish Manufacturing:
        submit: true
        protocols:
            - Process A:
                data:
                  Completion Date: 01Jan18
                  Signature: true

Configuration Notes:

The workflows parameter in the config definition references nested configuration for specific Experiment objects.

To automatically transition across workflows via configuration, nodes with possible transitions need to include the transition: true configuration (seen above).

Experiment Chain configuration must include a referenced project, per constraints on Experiment Chains in ESP.

Examples

>>> from esp.models import ExperimentChain
>>> chain = ExperimentChain('MF Chain 1')
>>> chain.name, chain.created_at
('MF Chain 1', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> chain.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]
>>> chain.project
<Project(name=Project 1)>
>>> chain.experiments
[<Experiment(name=Experiment 1)>, <Experiment(name=Experiment 2)>]
>>> chain.chain
<WorkflowChain(name=Manufacturing Chain)>
>>> chain.workflows
[<Workflow(name=Initiate Manufacturing)>, <Workflow(name=Finish Manufacturing>]

>>> # helper methods
>>> chain.active
True
>>> chain.submit()  # submit workflow chain
>>> chain.fill(...) # fill with workflows config above

Parameters:: ident (str) – Name or uuid for object.

Experiment¶

class esp.models.Experiment(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with Experiments from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create simple experiment:

name: MS001
workflow: Illumina Sequencing
project: Miseq Sequencing
samples:
    count: 3
    type: Extract

Create experiment and fill in data:

name: MS001
workflow: Illumina Sequencing
project: Miseq Sequencing
samples:
    count: 2
    type: Extract
protocols:
  - Set Samples:
      complete: true
      data:
        Attachment: ${CWD}/runsheet.txt
        Note: ['Ready to go!', 'Ready!']

  - Create Illumina Library:
      complete: true
      load: ${CWD}/spreadsheet.txt'

  - Analyze Sequencing Results:
      run: true
      complete: true
      data:
        Container: 'MiSeq: Lane 1'
        Reference: ['GRCh38', 'GRCh37']

Create experiment and fill in data with verification:

name: MS001
workflow: Illumina Sequencing
project: Miseq Sequencing
samples:
    count: 2
    type: Extract
protocols:
  - Set Samples:
      complete: true
      before:
        verify:
          Attachment: '{{ value is None }}'
      data:
        Attachment: ${CWD}/runsheet.txt
        Note: ['Ready to go!', 'Ready!']
      after:
        verify:
          Attachment: '{{ value is not None }}'

  - Create Illumina Library:
      complete: true
      cols:
        - Index Type
        - I7 Index ID
        - I5 Index ID
      actions:
        - data:
            - ['TruSeq Amplicon', 'A701', 'A501']
            - ['TruSeq Amplicon', 'A702', 'A502']
          verify:
            Index Type: 'TruSeq Amplicon'

  - Analyze Sequencing Results:
      run: true
      complete: true
      data:
        Amount: 10
        Reference: ['GRCh38', 'GRCh37']
      verify:
        Amount: '{{ value >= 10 }}'
        Reference: ['GRCh38', 'GRCh37']

Configuration Notes:

There are several hooks for entering data in workflows, and each of them are useful in different contexts. At a high-level, data are entered in order of the block they’re specified in. An overview of the blocks are as follows:

before - Perform actions before any other actions are perform.

actions - List of action blocks to perform (as a list).

after - Perform actions after any other actions are perform.

If no before, actions, or after blocks are specified, then the parameters included in the config will be run in a single action. See the examples above for context.

Available actions that can be performed when filling in data are:

data - Fill in data specified in configuration. Data can take
the format of a dictionary of values (fill in all rows with the specified value), or a dictionary of lists (use the list indices to fill in individual rows).

load - Load tabular file into worksheet, matching file headers
with protocol columns.

user/password - Log in as specific user before entering data.

verify - Run assertions on data after data are entered into
a worksheet. This action is particularly useful for verifying that expressions work as expected.

run - Start a pipeline from a pipeline protocol. This option will
run all pipelines specified for the worksheet protocol.

wait - Wait a number of seconds before performing the next action.

To automatically create Worksheets from experiments, use the submit: true parameter.

To complete a protocol at the end of filling in data, use the complete: true parameter.

Examples

>>> from esp.models import Experiment
>>> exp = Experiment('MS001')
>>> exp.name, exp.created_at
('MS001', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> exp.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]
>>> exp.project
<Project(name=Project 1)>
>>> exp.workflow
<Workflow(name=Initiate Manufacturing)>

>>> # accessing worksheet data
>>> exp.protocols[0]
Sample ID    Column 1    Column 2
ESP001       1.2         foo
ESP002       1.3         bar
>>> exp.protocol('Set Samples')['Column 1']
[1.2, 1.3]

>>> # helper methods
>>> exp.active
True
>>> exp.submit()  # submit workflow
>>> exp.fill(...) # fill with workflow config above
>>> exp.archive()

Parameters:: ident (str) – Name or uuid for object.

SampleSheet¶

class esp.models.SampleSheet(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with SampleSheets from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create sample sheet with submitted experiments:

name: Sheet 001
desc: Sample sheet with experiments
experiments:
    - Experiment 1
    - Experiment 2

Create sample sheet nested experiments:

name: Sheet 001
desc: Sample sheet with experiments
experiments:
    - Experiment 1
    - Experiment 2:
        samples:
            count 2
        submit: True
        sheet: False

Configuration Notes:

It’s recommended to use Experiment objects for accessing worksheet data. This model is primarily useful for creating multi-experiment sample sheets.

Examples

>>> from esp.models import SampleSheet
>>> sheet = SampleSheet('Sheet 001')
>>> sheet.name, sheet.created_at
('Sheet 001', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> sheet.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]
>>> sheet.experiments
[<Experiment(name=Experiment 1)>, <Sample(name=Experiment 2)>]
>>> sheet.project
<Project(name=My Project)>

Parameters:: ident (str) – Name or uuid for object.

Worksheet¶

class esp.models.Worksheet(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for managing sample sheets within ESP. This object is not configurable, and operates like a pandas DataFrame for each protocol in a workflow. You can access these worksheet objects from an experiment via:

>>> from esp.models import Experiment
>>> exp = Experiment('My Experiment')
>>> ws = exp.protocol('My Protocol')
>>> ws = exp.protocols[0]

Also, since the Worksheet object acts as a pandas DataFrame, you can interact with it like a data frame:

>>> ws.columns
['Column 1', 'Column 2']
>>> ws['Column 1'] = ws['Column 1'].apply(lambda x: x.replace('foo', 'bar'))
>>> ws.melt(...)

After changing the data in the worksheet object, you can save it back to the L7 database and proceed to the next worksheet using:

>>> ws.save()
>>> ws.complete()
>>> ws.next()

Parameters:

ident (str) – UUID for worksheet instance.
experiment (Experiment) – Experiment object related to worksheet.

SampleType¶

esp.models.SampleType: alias of EntityType

WorkflowableClass¶

class esp.models.WorkflowableClass(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with WorkflowableClass from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple workflowable type:

name: Patient
desc: A Human Patient
tags: [patient, demo]
plural_name: Patients
view_template: <h1>My View Template</h1>
list_view_template: <h1>My View Template</h1>
detail_panel_template: <h1>My View Template</h1>
icon_svg: <svg xmlns=“http://www.w3.org/2000/svg” width=“24" height=“24” viewBox=“0 0 24 24"><g fill=“none” fill-rule=“evenodd”><path d=“M0 0h24v24H0z”/><path fill=“#550185" fill-rule=“nonzero” d=“M6 6v14h12V6h-3v1H9V6H6zm4-1a2 2 0 1 1 4 0h5v16H5V5h5zm1 5h5v1h-5v-1zm0 3h5v1h-5v-1zm-3-3h2v1H8v-1zm0 3h2v1H8v-1zm0 3h2v1H8v-1zm3 0h5v1h-5v-1z”/>

Examples

>>> from esp.models import WorkflowableClass
>>> wf = WorkflowableClass('Patient')
>>> wf.name, wf.created_at
('Library', '2019-06-21T16:04:01.199076Z')

Parameters:: ident (str) – Name or uuid for object.

Sample¶

esp.models.Sample: alias of Entity

WorkflowChain¶

class esp.models.WorkflowChain(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with WorkflowChains from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple workflow chain:

name: Manufacturing Chain
desc: Workflow chain for manufacturing process.
tags: [demo, test]
chain:
    - Initiate Manufacturing:
        to:
            - Process A
            - Process B
    - Process A:
        head: false
        workflow: My Process
        virtual: true
    - Process B:
        head: false
        workflow: My Process
        resubmit: True

Workflow chain with conditionals and strategies:

name: Manufacturing Chain
desc: Workflow chain for manufacturing process.
tags: [demo, test]
chain:
    - Initiate Manufacturing:
        to:
            - Process A:
                if: '{{ column_value("Column 1") == "foo" }}'
                sample_type: Specimen
    - Process A:
        head: false
        to: Process B
        workflow: My Process
        strategy: named_strategy
    - Process B:
        head: false
        workflow: My Process

Workflow chain with node positioning and swimlanes:

Demo WFC 3:
  chain:
    - Node 1:
        head: true
        position:
          x: 290.75
          y: 100
        startable: true
        swimlane: sl-1641460171013
        to:
          - Checkbox Test:
              if: '{{ True }}'
    - Checkbox Test:
        position:
          x: 570.8214285714287
          y: 300
        startable: true
        swimlane: sl-1641460181862
        to:
          - Approval Instructions:
              if: ''
  swimlanes:
    - color: '#3d85c6'
      height: 200
      id: sl-1641460171013
      label: lane 1
      y: 0
    - color: '#ff9900'
      height: 200
      id: sl-1641460181862
      label: lane 2
      y: 200

Configuration Notes:

See the ESP documentation for information about how to configure workflow transition strategies.

See the conditional transition logic above for an example of how to send subsets of samples to downstream nodes in a workflow chain.

More information on types of workflow configuration can be found in the Workflow object documentation.

Nesting configuration for workflows is usually an easier way of defining configuration for workflows that won’t be used across workflow chains.

Note the on the backend, rules are evaluated for transitions in the context of the worksheet that triggers the transition, except there is no “current”/”active” protocol, so expressions like column_value must specify both the column and the protocol.

Examples

>>> from esp.models import WorkflowChain
>>> chain = WorkflowChain('Illumina Sequencing')
>>> chain.name, chain.created_at
('Illumina Sequencing', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> chain.workflows
[<Workflow(name=Initiate Manufacturing)>,
 <Workflow(name=Process A)>,
 <Workflow(name=Process B)>]

Parameters:: ident (str) – Name or uuid for object.

Workflow¶

class esp.models.Workflow(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with Workflows from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple workflow:

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
protocols:
    - Set Samples
    - Create Illumina Library
    - Analyze Sequencing Results

Workflow with embedded protocols:

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
protocols:
    - Set Samples:
        protocol: standard
        variables:
          - Attachment:
              rule: attachment
          - Note:
              rule: string

    - Create Illumina Library:
        desc: Create Illumina library and assign index codes.
        protocol: sample
        sample: Illumina Library
        relation: 1-to-1

    - Analyze Sequencing Results:
        protocol: pipeline
        group: All
        pipeline: Miseq Analysis Pipeline

Workflow with sample types (embedded and referenced):

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
sample_types:
    - Generic sample
    - Illumina Library:
        desc: Sample type for illumina library.
        sequences:
            - ESP SEQUENCE
protocols:
    - Set Samples
    - Create Illumina Library
    - Analyze Sequencing Results

Workflow with data links and sample groupings:

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
links:
    Analyze Sequencing Results:
        Instrument: '{{ column_value("Instrument", "Create Illumina Library") }}'
groups:
    Analyze Sequencing Results: Set Samples
sample_types:
    - Illumina Library
protocols:
    - Set Samples
    - Create Illumina Library
    - Analyze Sequencing Results

Configuration Notes:

See the ESP documentation for information about data linking, sample groups, and available protocol configuration options.

More information on types of protocol configuration can be found in the Protocol object documentation.

Nesting configuration for protocols is usually an easier way of defining configuration for protocols that won’t be used across workflows.

Examples

>>> from esp.models import Workflow
>>> wf = Workflow('Illumina Sequencing')
>>> wf.name, wf.created_at
('Illumina Sequencing', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> wf.experiments
[<Experiment(name=Experiment 1)>, <Experiment(name=Experiment 2)>]
>>> wf.protocols
[<Protocol(name=Set Samples)>,
 <Protocol(name=Create Illumina Library)>,
 <Protocol(name=Analyze Sequencing Results)>]
>>> wf.sample_types
[<SampleType(name=Illumina Library)>]

Parameters:: ident (str) – Name or uuid for object.

Protocol¶

class esp.models.Protocol(ident, data=None, version=None, snapshot=None, **kwargs)[source]

Object for interacting with Protocols from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Standard protocol:

name: Set Samples
protocol: standard
desc: Example standard protocol with a few columns.
tags:
  - illumina
  - sequencing
variables:
  - Attachment Column:
      rule: attachment
  - Container Column:
      rule: location
      container: Freezer
  - Dropdown Column:
      rule: dropdown
      dropdown:
        - foo
        - bar
  - Date Column:
      rule: date
  - Note:
      rule: string

Sample protocol with expressions:

name: Create Illumina Library
desc: Example sample protocol with expression columns.
protocol: sample
sample: Illumina Library
relation: 1-to-1
variables:
  - Sample Type:
      rule: string
      onchange: |+
        alert('Changed Sample Type')
  - Instrument:
      rule: string
      value: MiSeq, HiSeq 2000/2500
  - Index Type:
      rule: dropdown
      required: true
      dropdown: '{{ ilmn_kits() }}'
  - I7 Index ID:
      rule: dropdown
      required: true
      dropdown: '{{ ilmn_adapter_names(column_value("Index Type"), position="i7") }}'
  - I7 Index:
      rule: string
      read_only: true
      value: '{{ ilmn_adapter_seq(column_value("Index Type"), column_value("I7 Index ID")) }}'
  - I5 Index ID:
      rule: dropdown
      required: true
      dropdown: '{{ ilmn_adapter_names(column_value("Index Type"), position="i5") }}'
  - I5 Index:
      rule: string
      read_only: true
      value: '{{ ilmn_adapter_seq(column_value("Index Type"), column_value("I5 Index ID"), position="i5") }}'

Pipeline protocol with grouping:

name: Analyze Sequencing Results
protocol: pipeline
desc: Run bioinformatics pipelines to analyze sequencing data.
group: Reference
pipeline: Miseq Analysis Pipeline
variables:
  - Sample:
      rule: string
      visible: false
      pipeline_param: true
      value: '{{ sample["name"] }}'
  - Reference:
      rule: dropdown
      pipeline_param: true
      default: GRCh38
      dropdown:
        - GRCh37
        - GRCh38

Pipeline protocol with embedded pipeline:

name: Analyze Sequencing Results
protocol: pipeline
desc: Run bioinformatics pipelines to analyze sequencing data.
group: All
pipeline:
  Miseq Analysis Pipeline:
    desc: Analyze sequenced library with MiSeq analysis pipeline.
    tasks:
      - BWA Align:
          desc: Align fastq files to reference.
          cmd: "echo 'This is a bam created with the command: bwa mem {{ Sample }}.fq.gz' > {{ Sample }}.bam"
          files:
            - Aligned BAM:
                file_type: text
                filename_template: "{{ Sample }}.bam"
      - GATK Unified Genotyper:
          desc: Use GATK to genotype bam file.
          cmd: "echo 'This is a vcf created with the command: gatk UnifiedGenotyper -I {{ Sample }}.bam -R /references/{{ Reference }}.fa' > {{ Sample }}.vcf"
          files:
            - Genotype VCF:
                file_type: text
                filename_template: "{{ Sample }}.vcf"
    deps:
      GATK Unified Genotyper: BWA Align

Protocol with inheritance:

name: Set Samples Override
inherit: Set Samples
desc: Inherited protocol with extra columns.
variables:
  - Extra Column:
      rule: numeric

Configuration Notes:

See the ESP documentation for information about type types of columns and associated metadata available to use as protocol variables.

Using the group parameter on protocol definitions allows developers to group columns by a specified column or value.

The sample protocol relation property can take either 1-to-1, fan-out, or fan-in.

Protocol inheritance is useful for creating slightly modified copies of existing protocols without needing to re-define variable configuration. All parameters are copied over from the inherited protocol.

Nesting configuration for pipelines is usually an easier way of defining configuration for pipelines that won’t be used across protocols.

Examples

>>> from esp.models import Protocol
>>> pc = Protocol('Analyze Sequencing Results')
>>> pc.name, pc.created_at
('Analyze Sequencing Results', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> pc.pipeline
<Pipeline(name=Analyze Sequencing Results)>

Parameters:: ident (str) – Name or uuid for object.