Client API

class esp.models.__base__.BaseModel(ident, data=None, version=None, snapshot=None, **kwargs)

Abstract class for managing initialization methods for common objects.

classmethod all(**kwargs)

Return all instances related to a given model.

Examples

>>> samples = Sample.all()
>>> print(samples[0].name)
'ESP0000001'

property augment

Proxy for setting Model.data.meta.augment.

classmethod create(config=None, overwrite=False, prompt=False, object_names=None, force_overwrite=False, allow_snapshot_uuid_remap=False, **kwargs)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of BaseModel for model-specific logic.

created_timestamp

Return datetime object with creation timestamp.

property data

Property containing all information about an object. This is the information typically returned from a GET request using the object endpoint and uuid.

drop(deep=False)

Issue DELETE request to remove object in ESP database.

exists() -> bool

Test if object exists in esp database.

Accesses BaseModel “data” property. If “data” has not been fetched, this access will populate it, and subsequent calls to “data” will used cached value.

export(filename: str | TextIO = None, deep: bool = False, filter_empty: bool = False, use_dump_format: bool = False, use_hub_format: bool = False, versioned: bool = False)

Export a yaml-formatted object.

classmethod from_data(data)

Instantiate object from data.

Examples

>>> import esp.base as base
>>> from esp.models import Sample
>>> response = base.SESSION.get('/api/sample?name=ESP000001')
>>> samp = Sample.from_data(response.json())
>>> print(samples[0].name)
'ESP0000001'

classmethod from_definition(uuid)

Instantiate object definition from uuid.

Examples

>>> import esp.base as base
>>> from esp.models import SampleType
>>> st = SampleType.from_definition('8552afc3-882b-41e3-b0cf-1a4cf3f28d33')
>>> print(st[0].name)
'Illumina Sample'

classmethod items_for_names(names, max_query_string_size=4000)

Fetch objects by name. This is similar to search but uses a different fetch strategy and uses generators,

so was split out for backwards compatibility.

Differences from search:

classmethod items_for_uuids(uuids, max_query_string_size=8000)

Fetch objects by UUID. This is similar to search but uses a different fetch strategy and uses generators,

so was split out for backwards compatibility.

Differences from search:

json()

Return dictionary object with object metadata.

owner

Relationship to owner (creator) of object.

classmethod parse_import(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Method for formatting import-style data to data that can be used in a backend POST/PUT request.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Method for parsing request returned from POST/PUT during object creation. This method is responsible for returning an object after create() is called. :param allow_snapshot_uuid_remap:

push(dry=False)

Run put with metadata to update entry.

classmethod search(name=None, tags=None, method='or', uuids=None, names=None)

Search for model by name wildcard or tags.

summary()

Print organized description of entity.

updated_timestamp

Return datetime object with last update timestamp.

versions

Return full history of item in ESP database as objects of this type. The order of the list is most recent (index 0) to least recent (index -1).

class esp.models.__base__.LinkedModel(ident, data=None, version=None, snapshot=None, **kwargs)

Subclass of BaseModel for decorating parent-child relationship functionality. All items in the ESP database /can/ be linked, but there needs to be more formalism around what /should/ be linked in the database. Because of this, we should restrict access to linking functionality to specific models until the pattern for provenance is more flushed out on the backend.

add_children(items)

Add specified items as children to object.

add_dependents(items, label=None)

Add resource dependencies for item or list of items.

add_parents(items)

Add specified items as parent to object.

add_priors(items, label=None)

Add resource priors for item or list of items.

children

Return sample children.

Changes to these data does not push changes to the backend. Use self.add_children for that functionality.

dependencies

Return dependency information for item.

property dependents

Proxy for accessing dependent item dependencies.

TODO: Re-map the dependencies to their related

class types in the client. This currently just returns the JSON associated with the dependency data.

generation(entity_type, which='closestup')

Efficiently return generation of a particular type.

get_dependents(label)

Method for getting dependencies tagged with specific label.

get_priors(label)

Method for getting priors tagged with specific label.

parents

Return sample parent objects.

Changes to these data does not push changes to the backend. Use self.add_parents for that functionality.

property priors

Proxy for accessing prior item dependencies.

TODO: Re-map the dependencies to their related

class types in the client. This currently just returns the JSON associated with the dependency data.

remove_children(items)

Remove specified items as children to object.

remove_dependents(items, label=None)

Remove resource dependencies for item or list of items.

remove_parents(items)

Remove specified items as parent to object.

remove_priors(items, label=None)

Remove resource priors for item or list of items.

update_link(items, link, method='add', label=None)

Helper method for managing redundancy across prior/depdendent link generation.

class esp.models.User(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with users from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create user:

name: New User
email: user@localhost
password: password

Create user with roles:

name: New Role User
email: user-role@localhost
password: password
roles:
    - Lab Manager
    - Admin

Examples

>>> from esp.models import User
>>> user = User('New User')
>>> user.name, user.created_at
('User', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> user.roles
[<Role(name=Lab Manager)>, <Role(name=Admin)>]
>>> user.workgroups
[<Workgroup(name=Lab A)>]
 
>>> # log in as specific user
>>> User('New User').login(password='password')
 
>>> # get current user
>>> user = User.current()

classmethod current()

Return user from current session.

drop(deep=False)

Overwrite drop method for current user so emails are changed on drop (allowing User re-creation to happen).

NOTE: BACKEND ARCHIVES MODEL AND MAKES IT INACCESSIBLE VIA CLIENT

THIS FUNCTIONALITY SHOULD ONLY BE USED DURING TESTING.

login(password=None)

Login as specific user.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new User in ESP database using config file or other data.

roles

Return roles associated with user.

workgroups

Return roles associated with user.

class esp.models.Role(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with users from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create role:

name: my-role
desc: My Role
permissions:
    Project: [create, update]
    Sample: [create, read, update, delete]

Create role with default users:

name: my-role
desc: My Role
members:
    - User 1
    - User 2
applications:
    - admin
    - lims

Create role with specific app permissions:

name: my-role
desc: My Role
applications:
    - admin
    - lims

Configuration Notes:

Examples

>>> from esp.models import Role
>>> role = Role('my-role')
>>> role.name, role.created_at
('my-role, '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> role.users
[<User(name=User 1)>, <User(name=User 2)>]

drop(deep=False)

Overwrite drop method for current user so emails are changed on drop (allowing User re-creation to happen).

NOTE: BACKEND ARCHIVES MODEL AND MAKES IT INACCESSIBLE VIA CLIENT

THIS FUNCTIONALITY SHOULD ONLY BE USED DURING TESTING.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new Role in ESP database using config file or other data.

users

Return users associated with roles.

class esp.models.ParamGroup(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with ParamGroups from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create param group:

illumina:
    miseq-1: /path/to/miseq-1
    miseq-2: /path/to/miseq-2

Create multiple param groups:

illumina:
    miseq-1: /path/to/miseq-1
    miseq-2: /path/to/miseq-2
 
integrations:
    clarity: http://clarity-url:1000
    internal: http://internal-server

Examples

>>> from esp.models import ParamGroup
>>> pg = ParamGroup('apps')
>>> pg.name, group.created_at
('apps, '2019-06-21T16:04:01.199076Z')
 
>>> # show params
>>> pg.params.client
'/path/to/client/bin/esp'

drop(deep=False)

Overwrite drop method so ‘apps’ param group can’t be deleted.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new Workgroup in ESP database using config file or other data.

class esp.models.Workgroup(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with workgroups from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create workgroup:

name: Lab A
desc: Workgroup for all users in Lab A

Create workgroup with default users:

name: Lab A
desc: Workgroup for all users in Lab A
members:
    - User 1
    - User 2

Examples

>>> from esp.models import Workgroup
>>> group = Workgroup('Lab A')
>>> group.name, group.created_at
('Lab A, '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> group.users
[<User(name=User 1)>, <User(name=User 2)>]

classmethod associate_resources(workgroups, resources)

Set the workgroup(s) to the set of workgroups for the resources.

drop(deep=False)

Overwrite drop method for current user so emails are changed on drop (allowing User re-creation to happen).

NOTE: BACKEND ARCHIVES MODEL AND MAKES IT INACCESSIBLE VIA CLIENT

THIS FUNCTIONALITY SHOULD ONLY BE USED DURING TESTING.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new Workgroup in ESP database using config file or other data.

users

Return users associated with roles.

class esp.models.Config(*args, **kwargs)

Object for interacting with instance configuration within ESP. This model allows developers to directly interact with the /api/config endpoint in ESP.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

That is, this configuraiton is always updaitng a DB row, never truly creating it, but uses the same interface as all other models for consistency.

This model is deprecated and will be removed in the future. Use Configuration instead.

classmethod all(**kwargs)

Return all instances related to a given model.

Examples

>>> samples = Sample.all()
>>> print(samples[0].name)
'ESP0000001'

classmethod create(config=None, overwrite=False, prompt=False, **kwargs)

Create a new config object in ESP database using config file or other data.

drop(deep=True)

Issue DELETE request to remove object in ESP database.

push(dry=False)

Run put with metadata to update entry.

class esp.models.Configuration(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with database-backed instance configuration within ESP. This model allows developers to directly interact with the /api/configuration endpoint in ESP.

Model to interact with ESP’s DB-backed configuration system.

A configuration consists of two top-level dictionaries: one for “public” configuration and one for “secret” configuration. Generally, this model masks the distinction and makes the subkeys of both available as top-level properties, but also supports direct access of the public and secret dictionaries.

Examples

>>> from esp.models import Configuration
>>> conf = Configuration('esp') # get the ESP system config.
>>> conf.executor.num_workers
1
>>> conf = Configuration.create(name='CustomConfig', key1='value1', key2={'subkey1': 'value1.1', 'subkey2': 'value2.2'})
>>> conf.key2
{'subkey1': 'value2.1', 'subkey2': 'value2.2'}
>>> conf.key2.subkey1
'value2.1'
>>> conf.config
{'key1': 'value2', 'key2': {'subkey1': 'value2.1', 'subkey2': 'value2.2'}}
>>> conf.add_secret('secretkey', 'this value is secret')
>>> conf.secrets
{'secretkey': 'this value is secret'}
>>> conf.push()
>>> conf.refresh()
>>> conf.secrets
{'secretkey': 'XXXXXXX'}
>>> conf.secretkey
'XXXXXXX'
>>> conf.newkey = 'new value'
>>> conf.config
{'key1': 'value2', 'key2': {'subkey1': 'value2.1', 'subkey2': 'value2.2'}, 'newkey': 'new value'}
>>> conf.add_secret('newkey', 'conflicting secret')
>>> conf.newkey
'conflicting secret'
>>> conf.config.newkey
'newkey'

add_secret(name, value)

Add a “secret” configuration value. :param name: The name :param value: The real value

A secret configuration value is one where the public/REST API will return ‘XXXXXX’ for the value once the value has been pushed.

classmethod all(**kwargs)

Fetch all active configurations. :param **kwargs:

Returns:

classmethod from_data(data)

Instantiate object from data.

Examples

>>> import esp.base as base
>>> from esp.models import Sample
>>> response = base.SESSION.get('/api/sample?name=ESP000001')
>>> samp = Sample.from_data(response.json())
>>> print(samples[0].name)
'ESP0000001'

get(name, default=None)

Gets a single value from configuration, supplying a default if the key doesn’t exist. :param name: Name of the value to retrieve. :param default: Default to use if the configuration key doesn’t exist in the configuration object.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new Configuration in ESP database using config file or other data.

push(dry=False, restart=False)

Update the remote configuration with local modifications.

class esp.models.File(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with files from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create file:

name: My Protocol SOP
desc: An SOP file
tags: [sop, protocol]
uri: relative/path/to/instructions.pdf
upload: true
type: PDF

Create file object with reference to existing file:

name: Large Sequencing File
desc: An SOP file
tags: [sop, protocol]
uri: file:///path/to/local/file/instructions.bam
upload: true
type: raw

Create file object and register as task output:

name: SampleSheet.csv
desc: Illumina Run Sheet
task_file: Illumina Runsheet
uri: /path/to/SampleSheet.csv
upload: false
type: csv

Configuration Notes:

Examples

>>> from esp.models import File
>>> fi = File('My Protocol SOP')
>>> fi.name, fi.created_at
('My Protocol SOP', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> fi.path
'/path/to/Lab7_ESP/Data/files/0000/instructions.pdf.6a523a74-6703-4474-a24e-5a463b9d9770'
>>> fi.contents
'... raw file contents ...'
>>> fi.download('local-copy.pdf')

contents

Return raw file contents (as string).

download(outfile, encoding='UTF-8')

Download file and store as output filename.

Fetches the contents in a single go. For large text files where only a subset may be needed, consider using the ‘contents’ endpoint instead. For large, streamable binary files, such as BAM files, esp REST APIS have support for ranged data retrieval that is not yet supported by the python client.

linkable_uri

Return a “linkable” URL.

The returned URL will be relative to the API_SERVER root, so external consumers should prepend the API_SERVER information.

Examples

>>> File.create({ 'uri': '/path/to/myfile.txt', 'name': 'myfile' }).linkable_uri()
/api/files/<:uuid>/static

This methood is provided to simplify embedding links ESP files in extensions and third-party content. Note that the URL format is subject to change in the future.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

class esp.models.Report(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Reports from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create report with embedded html:

name: My Report
desc: An example html report.
tags: [html, demo]
contents:
    <h1>My Report</h1>

Create report with local html file:

name: My Report
desc: An example html report.
tags: [html, demo]
contents: $LAB7DATA/contents/reports/my-report.html

Create applet from local html file:

name: My Report
desc: An example html applet.
tags: ['esp:applet']
contents: $LAB7DATA/contents/reports/my-report.html

Configuration Notes:

Examples

>>> from esp.models import Report
>>> report = Report('My Report')
>>> report.name, report.created_at
('My Report', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> report.contents
'<h1>My Report</h1>'

classmethod all(**kwargs)

Return all instances related to a given model.

Examples

>>> samples = Sample.all()
>>> print(samples[0].name)
'ESP0000001'

property contents

API niceness for editing internal contents, if simple html report.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

class esp.models.Pipeline(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Pipelines from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create pipeline with single task:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
tasks:
    - Create Illumina Runsheet

Create pipeline with nested tasks:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
tasks:
    - Create Illumina Runsheet:
        desc: Run script to generate illumina runsheet.
        cmd: /path/to/script.py {{project}} > runsheet.xml
        files:
            - Illumina Runsheet:
                file_type: xml
                filename_template: '{{ "runsheet.xml" }}'

Create pipeline with task dependencies:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Create pipeline with report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - []
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Create pipeline with report and failure report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - []
failure_report:
    Runsheet Failure Report:
        desc: Report showing details of failed runsheet generation
        elements
        - - type: file_details
            depends:
                file: stderr
                tasknumber:1
        - []
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Configuration Notes:

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')
>>> pipe.name, pipe.created_at
('Generate Illumina Runsheet', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> pipe.tasks
[<Task(name=Create Illumina Runsheet)>, <Task(name=Upload Illumina Runsheet)>]
>>> pipe.report
<ReportTemplate(name=Runsheet Report)>
 
>>> # run pipeline (wait to finish)
>>> analysis = pipe.run(project='My Project')
>>> analysis.task[0].stdout
'stdout from the first analysis task'
>>> analysis.task[0].files
[<File(name=Illumina Runsheet)>]
 
>>> # run pipeline (in background)
>>> analysis = pipe.run(project='My Project', background=True)
>>> analysis.pause()
>>> analysis.restart()
>>> analysis.kill()

dependencies

Show task dependency graph. Edits to this graph will be reflected when a user issues self.push().

drop(deep=False)

Drop all associated reports and pipeline.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Parse response from server when creating new pipeline. :param allow_snapshot_uuid_remap:

report

Return PipelineReport objects associated with pipeline.

reports

Return PipelineReport objects associated with pipeline.

run(**kwargs)

Run pipeline with specified parameters.

Examples

>>> pipe = Pipeline('NGS Pipeline')
>>> print(pipe.meta.tasks)
['Preprocess FASTQ', 'Align BAM', 'Call Variants']
>>> print(pipe.arguments)
'fastq'
>>> pipe.run(fastq='/path/to/file.fastq')

task_steps

Return tasks associated with pipeline along with their pipeline-specific step information.

tasks

Return tasks associated with pipeline.

class esp.models.Task(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Tasks from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple task (no variables):

name: Ping Internal Server
desc: Run command to ping internal server.
cmd: curl http://internal-server

Simple task with variables:

name: Ping Internal Server
desc: Run command to ping internal server.
cmd: curl '{{ server }}'

Create task and track outputs:

name: Create Illumina Runsheet
desc: Run script to generate illumina runsheet.
cmd: /path/to/generate_runsheet.py {{project}} > runsheet.xml
files:
    - Illumina Runsheet:
        file_type: xml
        filename_template: '{{ "runsheet.xml" }}'

Create task with inline code:

name: Report DNA Type
desc: Run bash script to generate simple report.
cmd: |+
   # Simple Task that determines if the specified 'ratio' is in the proper range
   if [ "{{ type }}" = "DNA" ]; then
       echo "<b>This is a <font color='green'>DNA</font> sample</b>" > result.html
   elif "{{ type }}" = "DNA" ]; then
       echo "<b>This is a <font color='green'>RNA</font> sample</b>" > result.html
   fi
 
files:
    - Type Report:
        file_type: html
        filename_template: '{{ "report.html" }}'

Examples

>>> from esp.models import  Task
>>> task  = Task('Create Illumina Runsheet')
>>> task.name, task.created_at
('Create Illumina Runsheet', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> task.variables
['project']
>>> task.cmd
'/path/to/generate_runsheet.py {{project}} > runsheet.xml'

class esp.models.PipelineReport(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with PipelineReports from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create pipeline with report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - []
tasks:
    - Create Illumina Runsheet

Create multi-step pipeline with report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements:
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - - type: raw_file
            depends:
                file: Runsheet Upload Report
                tasknumber:2
        - - type: html
            contents: |+
                <h1>Report Header</h1>
                <p>Report Body</p>
        - []
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Configuration Notes:

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Create Illumina Runsheet')
>>> pipe.name, pipe.created_at
('Generate Illumina Runsheet', '2019-06-21T16:04:01.199076Z')
>>> pipe.report.name, pipe.created_at
('Runsheet Report', '2019-06-21T16:04:01.199076Z')
>>> pipe.report.pipeline.name, pipe.report.pipeline.created_at
('Generate Illumina Runsheet', '2019-06-21T16:04:01.199076Z')

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

NOTE: A Pipeline with the following construct could have a corresponding config that looks like the following:

If a Pipeline (named ‘pipelineA’) with Tasks “taskB” and “taskA” is setup as such (without a temporal connection) in the UI:

and the Pipeline has a report that has the following construct:

FileDetails:

fileA_for_taskA (Step 2: taskA)

FileDetails:

fileB_for_taskB (Step 1: taskB)

the config object could have (or has) the following structure:

{

‘elements’: [

[

{

‘depends’: [

{

‘file’: ‘fileA_for_taskA’, ‘tasknumber’: 2

}

], ‘type’: ‘file_details’

}, {

‘depends’: [

{

‘file’: ‘fileB_for_taskB’, ‘tasknumber’: 1

}

], ‘type’: ‘file_details’

}

], []

], ‘name’: ‘pipelineA’, ‘uuid’: None, ‘desc’: ‘’

}

pipeline

Relationship to pipeline object.

class esp.models.Analysis(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with analyses from the ESP database.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')
 
>>> # run pipeline to return analysis object
>>> analysis = pipe.run(project='My Project')
>>> analysis.task[0].stdout
'stdout from the first analysis task'
>>> analysis.task[0].files
[<File(name=Illumina Runsheet)>]
 
>>> # run pipeline (in background)
>>> analysis = pipe.run(project='My Project', background=True)
>>> analysis.pause()
>>> analysis.restart()
>>> analysis.kill()
 
>>> # get current analysis (within script)
>>> analysis = Analysis.current()
>>> analysis.pause()

join(expect=['failed', 'done', 'paused'], timeout=1000000.0)

Wait for analysis to finish, and return when done.

tasks

Return objects for each step in pipeline.

class esp.models.AnalysisTask(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with task instances from the ESP database.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')
 
>>> # run pipeline to return analysis object
>>> analysis = pipe.run(project='My Project')
 
>>> # get first task object from analysis
>>> atask = analysis.task[0]
>>> atask.stdout
'stdout from the first analysis task'
>>> atask.stderr
'stderr from the first analysis task'
>>> atask.files
[<File(name=Illumina Runsheet)>]

files

Property for accessing all files associated with task.

script

Property to accessing task stdout.

stderr

Property to accessing task stderr.

stdout

Property to accessing task stdout.

class esp.models.AnalysisPool(analyses)

Object for interacting with analyses from the ESP database.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')
 
>>> # run multiple scripts to gather in pool
>>> foo_analysis = pipe.run(project='Foo Project')
>>> bar_analysis = pipe.run(project='Bar Project')
 
>>> # create analysis pool to control execution
>>> pool = AnalysisPool([foo_analysis, bar_analysis])
>>> pool.state
'running'
 
>>> # wait for all analyses to finish in pool
>>> pool.join()
>>> pool.state
'done'

join(period=0.25)

Wait for all analyses to finish, and return when they’re done.

property state

Return “state” of the analysis pool.

class esp.models.ContainerType(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with container types from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Single-element container:

name: 96-Well Plate
desc: 96-Well Plate for sample aliquots
tags: [plate, lab]
label_format: '%s%s'
slot_capacity: element
axes:
    - Rows: [A, B, C, D, E, F, G, H]
    - Cols: [01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12]
contains:
    - Sample

Multi-element container:

name: Freezer
desc: Freezer for samples and racks
tags: [freezer, lab]
label_format: 'Shelf %s'
slot_capacity: list
axes:
    - Shelf: [1, 2, 3, 4, 5]
contains:
    - Sample
    - Container

Type with default containers:

name: Freezer
desc: Freezer for samples and racks
tags: [freezer, lab]
label_format: 'Shelf %s'
slot_capacity: list
axes:
    - Shelf: [1, 2, 3, 4, 5]
contains:
    - Sample
    - Container
create:
    - Freezer 1
    - Freezer 2

Nested config with containers and samples to fill:

name: Freezer
desc: Freezer for samples and racks
tags: [freezer, lab]
label_format: 'Shelf %s'
slot_capacity: list
axes:
    - Shelf: [1, 2, 3, 4, 5]
contains:
    - Sample
    - Container
create:
    - Freezer 1
    - name: Freezer 2
      barcode: 1234
      fill:
        Shelf 1: [ESP0001, ESP0002]

Examples

>>> from esp.models import ContainerType
>>> ct = ContainerType('Freezer')
>>> ct.name, ct.created_at
('Freezer', '2019-06-21T16:04:01.199076Z')
>>> ct.containers
[<Container(name=Freezer 1)>, <Container(name=Freezer 2)>]

axes

Return the axes structure List[Dict[str,object]]. Each list entry is a dict of information about a single axis, including keys:

containers

Property for accessing all related containers.

property dimensions

Return the container dimensions.

drop(deep=False)

Overwrite drop to delete all corresponding containers as well.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new ContainerType in ESP database using config file or other data.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Method for parsing request and returning :param allow_snapshot_uuid_remap:

property renderer

The name of the container rendering script.

slot_for_position(position, by, recycle=False, label=True)

Convert 1d index to 1d or 2d coordinates.

Note that x may be > slot_count, so this can be used when laying out a list of samples into a list of locations.

Examples

>>> ct = ContainerType.create({'name': '96-well Plate', axes=[
... {'Row': [x for x in 'ABCDEFGH']},
... {'Col': ['{:02}'.format(x) for x in range(1, 12)]}],
... 'slot_capacity': 'element'})
>>> ct.slot_for_position(25, 0)
"C01"
>>> ct.slot_for_position(25, 1)
"A04"
>>> ct.slot_for_position(25, 0, label=False)
[3, 1]
>>> ct.slot_for_position(25, 1, label=False)
[1, 4]
>>> ct.slot_for_position(100, 0)
ValueError: 100 does not fit into 96 slots
>>> ct.slot_for_position(100, 0, recycle=True)
"A04"

slot_label(*idxdims, **kwdims)

Build individual label components.

Examples

>>> ct = ContainerType.create({'name': '96-well Plate', axes=[
... {'Row': [x for x in 'ABCDEFGH']},
... {'Col': ['{:02}'.format(x) for x in range(1, 12)]}],
... 'slot_capacity': 'element'})
>>> ct.slot_label(1, 1)
'A01'
>>> ct.slot_label(Row=1, Col=1)
'A01'

total_capacity

Return the total capacity of the container. If the slot capacity is “list”, returns “Inf”, otherwise, returns the product of the length of all dimensions.

class esp.models.Container(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with containers from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create container:

name: Freezer 1
desc: Freezer in lab A
tags: [freezer, lab]
type: Freezer
barcode: 12345

Container with samples to fill:

name: Freezer 1
desc: Freezer in lab A
tags: [freezer, lab]
type: Freezer
barcode: 12345
fill:
    Shelf 1:
        - ESP0001
        - name: ESP0002
          desc: special new sample
          tags: [one, two]

Configuration Notes:

Examples

>>> from esp.models import Container
>>> ct = Container('Freezer 1')
>>> ct.name, ct.created_at
('Freezer 1', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> ct.slots
['Shelf 1', 'Shelf 2', 'Shelf 3', 'Shelf 4', 'Shelf 5']
>>> ct.samples
[<Sample(name=ESP0001)>, <Sample(name=ESP0002)>]
>>> ct.slot_samples
{
    'Shelf 1': [<Sample(name=ESP0001)>, <Sample(name=ESP0002)>],
    'Shelf 2': [],
    'Shelf 3': [],
    ...
}
 
>>> # put items in specific slots
>>> ct.put_items_in_slots(
>>>     items=[Sample('ESP0001'), Sample('ESP0002')],
>>>     slots=['Shelf 1', 'Shelf 2']
>>> )

container_type

Return container type definition associated with item.

Using container type definition here is unintuitive and backend storage of data should be resolved.

drop(deep=False)

Remove all items from container and drop.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new Container in ESP database using config file or other data.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Parse response and put items in slots (if specified). :param allow_snapshot_uuid_remap:

put_items_in_slots(items, slots, overwrite=False, worksheet=None, fields=None, save=True)

Puts an item into the specified slot.

Items can be placed into a container inside or outside of a worksheet. Within the context of a worksheet, the backend currently stores the container information in both the lims val and the container, so synchronizing across both is required. Hence, if you’re putting an item into a slot, and that information will be recorded in a worksheet, pass the worksheet object to this method and an appropriate lims val will be created.

samples

Return array of samples associated with container.

slot_samples

Return array of container slots, filtered to only samples.

slots

Return array of container slots.

class esp.models.Customer(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Customers from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple customer:

name: Customer 1
desc: Customer 1 from the North Pole.
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Customer without address:

name: Customer 2
desc: Customer 2 without address.
tags: [demo, north]

Customer with billing and shipping address:

name: Customer 3
desc: Customer 3 from the North Pole
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole
shipping_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Configuration Notes:

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

class esp.models.ItemType(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with ItemTypes from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Item type for stock reagent:

name: Demo Reagent
consumption_type: stock reagent
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent

Item type for consumable:

name: Demo Consumable
consumption_type: consumable
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent

Item type for kit component:

name: Demo Kit Component
consumption_type: kit component
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent

Create item type with associated items:

name: Consumable
consumption_type: consumable
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent
create:
    - CNS001
    - CNS002

Examples

>>> from esp.models import ItemType
>>> it = ItemType('Demo Consumable')
>>> it.name, it.created_at
('Demo Consumable', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> it.vendors
[<Vendor(name=Agilent)>]

drop(deep=False)

Issue DELETE request to remove object in ESP database

items

Relationship to Item model.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Method for parsing request and returning :param allow_snapshot_uuid_remap:

sequences

Return sequence ID used within ESP. This id is used throughout the system in a variety of different ways, so being able to query it is useful.

vendors

Relationship to Vendor model.

class esp.models.Item(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with ItemTypes from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple item:

name: Example Item
tags: [agilent, perishable]
barcode: 45262341
barcode_type: QR
item_type: Demo Consumable
vendor: Agilent
lot_id: Lot # 1
serial_id: 12345
initial_qty: 10
expires: 12/25/2019
status: Quarantined

Item with different barcode type:

name: Example Item
tags: [agilent, perishable]
barcode: 1234
barcode_type: 1D
item_type: Demo Consumable
vendor: Agilent
lot_id: ABC

Examples

>>> from esp.models import Item
>>> item = Item('Example Item')
>>> item.name, item.created_at
('Example Item', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> it.vendor
<Vendor(name=Agilent)>
 
>>> # useful functions
>>> item.expired()
False
>>> item.consumed()
True
>>> item.consumed()
True
 
>>> # interacting with item
>>> item.add_quantity(10)
>>> item.remove_quantity(5)
>>> item.empty()
 
>>> # interacting with variables
>>> item = Item('Example Item', params='["resource_vals"]')
>>> item.variables
>>> item.variables['Example Variable']
 
>>> # adding item with variables
>>> item_to_add = {
>>>     'name': 'The Name of the Item',
>>>     'description': 'Item Description',
>>>     'item_type': 'Item Type Name',
>>>     'initial_qty': 22,
>>>     'status': 'Quarantined',
>>>     'resource_vals': [{'name': 'The name of custom field', 'value': 'The Value'}]
>>> }
>>> Item.create(item_to_add)

add_quantity(amount)

Add quantity to item.

empty()

Remove all quantity from item.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

remove_quantity(amount)

Remove quantity from item.

vendor

Relationship to Vendor model.

class esp.models.ServiceType(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with ServiceTypes from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Sample service type:

name: Sample Service Type
desc: Service type for a demo service.
tags: [service, demo]
price_type: sample
base_price: 350

Time service type:

name: Time Service Type
desc: Service type for a time service.
tags: [service, demo]
price_type: time
base_price: 350
scale_price: 25
scale_units: minutes

Configuration Notes:

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

class esp.models.Service(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Service from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple service:

name: My Service
service_type: My Service Type
customer: Lab1

Service with embedded type and customer:

name: My Service
service_type:
    My Service Type:
        price_type: time
        base_price: 350
        scale_price: 25
        scale_units: minutes
customer:
    Lab1:
        billing_address:
            street1: 1234 Northword Lane
            street2: Suite # 123
            city: North Pole
            state: North Pole
            country: North Pole
            postal_code: North Pole

Examples

>>> from esp.models import Service
>>> service = Service('My Protocol SOP')
>>> service.name, service.created_at
('My Service', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> service.service_type.name
'My Service Type'
>>> service.customer.name
'Customer 1'

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

class esp.models.Vendor(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Vendors from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple vendor:

name: Vendor 1
desc: Vendor 1 from the North Pole.
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Vendor without address:

name: Vendor 2
desc: Vendor 2 without address.
tags: [demo, north]

Vendor with billing and shipping address:

name: Vendor 3
desc: Vendor 3 from the North Pole
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole
shipping_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Configuration Notes:

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

class esp.models.Project(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Projects from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple project:

name: My Project
desc: Project for a series of experiments.
tags: [demo, test]

Project with nested experiments:

name: My Project
desc: Project for a series of experiments.
tags: [demo, test]
experiments:
    - My Experiment 1:
        workflow: Workflow 1
        submit: true
        samples: [ESP001, ESP002]

Project with nested experiment chains:

name: My Project
desc: Project for a series of experiments.
tags: [demo, test]
chains:
    - My Experiment Chain 1:
        chain: Workflow Chain 1
        samples: [ESP001, ESP002]

Configuration Notes:

Examples

>>> from esp.models import Project
>>> project = Project('Project 1')
>>> project.name, project.created_at
('Project 1', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> project.experiments
[<Experiment(name=Experiment 1)>, <Experiment(name=Experiment 2)>]
>>> project.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]

drop(deep=False)

Issue DELETE request to remove object in ESP database.

experiments

Property for fluidly accessing Experiment objects from metadata.

new_experiment(**data)

Add new Experiment to existing workflow.

Examples

>>> obj = Project('Test Workflow')
>>> print(obj.experiments)
[]
>>>
>>> exp = obj.new_experiment(name='test', samples=['one', 'two'])
>> print(obj.experiments)
['<Experiment(name=test)']
>>> print(exp.samples)
['<Sample(name=one)>',
 '<Sample(name=two)>']

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Parse response from server when creating new project. This method is primarily responsible for creating nested experiments once a project is created. :param allow_snapshot_uuid_remap:

samples

Property for fluidly accessing Sample objects from projects.

class esp.models.ExperimentChain(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with ExperimentChains from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create simple chain:

name: MF Chain 1
chain: Manufacturing Chain
project: Manufacturing
samples:
    count: 3
    type: Tissue

Create chain with nested experiments:

name: MF Chain 1
chain: Manufacturing Chain
project: Manufacturing
samples:
    count: 3
    type: Tissue
workflows:
    - Initiate Manufacturing
        submit: true
        transition: true
        protocols:
            - Initiate Manufacturing:
                complete: true
                data:
                  Start Date: 01Jan18
                  Signature: true
    - Finish Manufacturing:
        submit: true
        protocols:
            - Process A:
                data:
                  Completion Date: 01Jan18
                  Signature: true

Configuration Notes:

Examples

>>> from esp.models import ExperimentChain
>>> chain = ExperimentChain('MF Chain 1')
>>> chain.name, chain.created_at
('MF Chain 1', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> chain.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]
>>> chain.project
<Project(name=Project 1)>
>>> chain.experiments
[<Experiment(name=Experiment 1)>, <Experiment(name=Experiment 2)>]
>>> chain.chain
<WorkflowChain(name=Manufacturing Chain)>
>>> chain.workflows
[<Workflow(name=Initiate Manufacturing)>, <Workflow(name=Finish Manufacturing>]
 
>>> # helper methods
>>> chain.active
True
>>> chain.submit()  # submit workflow chain
>>> chain.fill(...) # fill with workflows config above

property active

Check if experiment is currently active.

chain

Property for accessing experiment workflow chain definition.

classmethod create(config=None, overwrite=False, prompt=False, object_names=None, **kwargs)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of BaseModel for model-specific logic.

drop(deep=False)

Issue DELETE request to remove object in ESP database.

experiments

Property for accessing experiment project.

fill(workflows)

Fill experiment with protocol data.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Parse response from experiment creation and manage submission and protocol data filling. :param allow_snapshot_uuid_remap:

project

Property for accessing experiment project.

property sample_types

Proxy for returning sample types from workflows.

samples

Property for accessing initial sample objects in experiment.

submit(sheet=True)

Submit experiment to lab for processing.

workflows

Property for accessing experiment workflow definition.

class esp.models.Experiment(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Experiments from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create simple experiment:

name: MS001
workflow: Illumina Sequencing
project: Miseq Sequencing
samples:
    count: 3
    type: Extract

Create experiment and fill in data:

name: MS001
workflow: Illumina Sequencing
project: Miseq Sequencing
samples:
    count: 2
    type: Extract
protocols:
  - Set Samples:
      complete: true
      data:
        Attachment: ${CWD}/runsheet.txt
        Note: ['Ready to go!', 'Ready!']
 
  - Create Illumina Library:
      complete: true
      load: ${CWD}/spreadsheet.txt'
 
  - Analyze Sequencing Results:
      run: true
      complete: true
      data:
        Container: 'MiSeq: Lane 1'
        Reference: ['GRCh38', 'GRCh37']

Create experiment and fill in data with verification:

name: MS001
workflow: Illumina Sequencing
project: Miseq Sequencing
samples:
    count: 2
    type: Extract
protocols:
  - Set Samples:
      complete: true
      before:
        verify:
          Attachment: '{{ value is None }}'
      data:
        Attachment: ${CWD}/runsheet.txt
        Note: ['Ready to go!', 'Ready!']
      after:
        verify:
          Attachment: '{{ value is not None }}'
 
  - Create Illumina Library:
      complete: true
      cols:
        - Index Type
        - I7 Index ID
        - I5 Index ID
      actions:
        - data:
            - ['TruSeq Amplicon', 'A701', 'A501']
            - ['TruSeq Amplicon', 'A702', 'A502']
          verify:
            Index Type: 'TruSeq Amplicon'
 
  - Analyze Sequencing Results:
      run: true
      complete: true
      data:
        Amount: 10
        Reference: ['GRCh38', 'GRCh37']
      verify:
        Amount: '{{ value >= 10 }}'
        Reference: ['GRCh38', 'GRCh37']

Configuration Notes:

Examples

>>> from esp.models import Experiment
>>> exp = Experiment('MS001')
>>> exp.name, exp.created_at
('MS001', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> exp.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]
>>> exp.project
<Project(name=Project 1)>
>>> exp.workflow
<Workflow(name=Initiate Manufacturing)>
 
>>> # accessing worksheet data
>>> exp.protocols[0]
Sample ID    Column 1    Column 2
ESP001       1.2         foo
ESP002       1.3         bar
>>> exp.protocol('Set Samples')['Column 1']
[1.2, 1.3]
 
>>> # helper methods
>>> exp.active
True
>>> exp.submit()  # submit workflow
>>> exp.fill(...) # fill with workflow config above
>>> exp.archive()

property active

Check if experiment is currently active.

add_samples(new_samples)

Add additional samples to the experiment. NOTE: If the experiment is pending, the samples will replace the current list of samples rather than add to it. :param new_samples: list of samples to add :type new_samples: List[Sample]

archive()

Archive experiment once sample sheet is finished. This provides guarantees around experiment completeness by explicitly checking for protocol completion before dropping.

classmethod create(config=None, overwrite=False, prompt=False, object_names=None, **kwargs)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of BaseModel for model-specific logic.

drop(deep=False)

Issue DELETE request to remove object in ESP database.

fill(protocols)

Fill experiment with protocol data.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Parse response from experiment creation and manage submission and protocol data filling. :param allow_snapshot_uuid_remap:

project

Property for accessing experiment project.

protocol(name, index=0)

Retrieve protocol instance by name for easier API access.

protocols

Property for accessing sample sheet related to experiment.

push()

Overwrite push method to account for inconsistencies in PUT payloads vs GET payloads.

sample_group(idx=0)

Property for accessing sample groups within experiment.

property sample_types

Proxy for returning sample types from workflows.

samples

Property for accessing initial sample objects in experiment.

sheet

Property for accessing sample sheet related to experiment.

submit(sheet=True, append_samples=False)

Submit experiment to lab for processing.

workflow

Property for accessing experiment workflow definition.

class esp.models.SampleSheet(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with SampleSheets from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create sample sheet with submitted experiments:

name: Sheet 001
desc: Sample sheet with experiments
experiments:
    - Experiment 1
    - Experiment 2

Create sample sheet nested experiments:

name: Sheet 001
desc: Sample sheet with experiments
experiments:
    - Experiment 1
    - Experiment 2:
        samples:
            count 2
        submit: True
        sheet: False

Configuration Notes:

Examples

>>> from esp.models import SampleSheet
>>> sheet = SampleSheet('Sheet 001')
>>> sheet.name, sheet.created_at
('Sheet 001', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> sheet.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]
>>> sheet.experiments
[<Experiment(name=Experiment 1)>, <Sample(name=Experiment 2)>]
>>> sheet.project
<Project(name=My Project)>

add_experiment(experiment, samples=None)

Add experiment(s) to existing sample sheet. If samples is provided, only samples in the provided list will be added. Otherwise, all samples from all experiments will be added.

classmethod all(**kwargs)

Return all instances related to a given model.

archive()

Try to gracefully archive sample sheet using DELETE request. This method make assumptions about experiment completeness and will fail if any Worksheets in SampleSheet are incomplete.

drop(deep=False)

Force drop a sample sheet by removing all samples and running super drop method.

experiment

Return experiment objects associated with sample sheet.

experiment_for_sample(sample, sample_index=None)

Superceded by experiments_for_sample(). Included for backwards compatibility.

experiments

Return experiment objects associated with sample sheet.

experiments_for_sample(sample, sample_index=None)

Get list of Experiments corresponding to a Sample row. This usually returns a list with a single Experiment, but if a fan-in Sample Protocol has fanned in Samples from multiple Experiments, these rows pass a pipe-delimited list of Experiment UUIDs for “rolled-up” rows that actually correspond to separate StepInstanceBaseSamples maintained by separate Experiments. These rows are typically updated in tandem so that they have the same contents.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Do multi-experiment adding after sample sheet creation. :param allow_snapshot_uuid_remap:

project

Return project associated with sample sheet.

projects

Return all projects associated with sample sheet.

protocol(protocol_name)

Get a protocol by name

protocols

Get all protocols in a worksheet.

remove(item)

Remove samples or experiments from existing sample sheet.

samples

Return all samples added to sample sheet. Only includes the initial sample set (ie the samples directly added by the user). Does not include child samples created by protocols within the worksheet. Samples will be returned in the order they were added to the worksheet.

transition()

Transition samples in the SampleSheet that are part of a workflow chain.

workflow

Return the workflow definition for this sample sheet.

class esp.models.Worksheet(ident, data=None, version=None, snapshot=None, **kwargs)

Object for managing sample sheets within ESP. This object is not configurable, and operates like a pandas DataFrame for each protocol in a workflow. You can access these worksheet objects from an experiment via:

>>> from esp.models import Experiment
>>> exp = Experiment('My Experiment')
>>> ws = exp.protocol('My Protocol')
>>> ws = exp.protocols[0]

Also, since the Worksheet object acts as a pandas DataFrame, you can interact with it like a data frame:

>>> ws.columns
['Column 1', 'Column 2']
>>> ws['Column 1'] = ws['Column 1'].apply(lambda x: x.replace('foo', 'bar'))
>>> ws.melt(...)

After changing the data in the worksheet object, you can save it back to the L7 database and proceed to the next worksheet using:

>>> ws.save()
>>> ws.complete()
>>> ws.next()

abort(sample, message='')

Abort specified sample or samples in sample sheet.

approve(password, sample=None, column=None)

Approve sample and column in worksheet.

complete(value=True, save_first=True, samples=None)

Mark complete column on workflow and save.

property completed

Return boolean describing whether or not sheet has been completed.

convert(col, val)

Convert column value from sheet.

convert_approval(value)

Create column value data structure for approval column.

convert_attachment(value)

Create column value data structure for attachment column.

convert_item_qty_adjust(value)

Create column value data structure for itemqtyadj column.

convert_location(value)

Create column value data structure for location column.

df

Return sheet data for data editing/fetching.

fail(sample, message='')

DEPRECATED Fail specified sample or samples in sample sheet.

grouped()

True if this worksheet tab is for a a grouped protocol

index

Return the protocol index within the workflow for this worksheet.

load(filename, sep='\t')

Update sheet with data from specified file.

next()

Return next protocol instance for editing.

refresh()

Refresh the object.

run(background=False, run_groups=None, col_name='Start')

Run pipeline for columns and wait for results to finish.

sample_group

Return the sample group for this worksheet.

sample_sheet

Return a reference to the sample sheet for this worksheet.

samples

Return the samples for this worksheet.

save(evaluate=True, refresh=True)

Set sample values that have changed

sign(signature_list: list, sample=None, column=None)

Complete signature flow in worksheet.

verify(data)

Verify that the data in the sheet matches the data specification passed in. Data can be formatted as:

// all values in colname have the same value.
colname: value
 
// all values in colname have the same value.
colname: [1, 2, 3, 4] # vector of values should match this vector
 
// Specify values for each sample.
colname:
    SAM000001: 1
    SAM000002: 2
    SAM000003: 3
    SAM000004: 4

Value is usually treated literally, but if the value is a string that starts with {{ and ends with }}, everything within {{ }} is treated as a python expression to evaluate. The expression must evaluate to a boolean and may reference the variables:

Expression can be simple, such as:

{{bool(value)}} # make sure value is specified/not null/not empty.

Or more complicated:

{{value == row['colA'] + '123' + row['colB']}}

Verify makes one utility function available to the expression: “eq”.

{{ eq(value, 123.5, 2) }}

This is equivalent to:

{{ round(float(value), 2) == round(float(123.5, 2)) }}

esp.models.SampleType

alias of EntityType

class esp.models.WorkflowableClass(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with WorkflowableClass from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple workflowable type:

name: Patient
desc: A Human Patient
tags: [patient, demo],
plural_name: Patients
view_template: <h1>My View Template</h1>
list_view_template: <h1>My View Template</h1>
detail_panel_template: <h1>My View Template</h1>
schedulability: 'on'
icon_svg: <svg xmlns=“http://www.w3.org/2000/svg” width=“24" height=“24” viewBox=“0 0 24 24"><g fill=“none” fill-rule=“evenodd”><path d=“M0 0h24v24H0z”/><path fill=“#550185" fill-rule=“nonzero” d=“M6 6v14h12V6h-3v1H9V6H6zm4-1a2 2 0 1 1 4 0h5v16H5V5h5zm1 5h5v1h-5v-1zm0 3h5v1h-5v-1zm-3-3h2v1H8v-1zm0 3h2v1H8v-1zm0 3h2v1H8v-1zm3 0h5v1h-5v-1z”/>

Examples

>>> from esp.models import WorkflowableClass
>>> wf = WorkflowableClass('Patient')
>>> wf.name, wf.created_at
('Library', '2019-06-21T16:04:01.199076Z')

drop(*args, **kwargs)

Issue DELETE request to remove object in ESP database.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

esp.models.Sample

alias of Entity

class esp.models.WorkflowChain(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with WorkflowChains from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple workflow chain:

name: Manufacturing Chain
desc: Workflow chain for manufacturing process.
tags: [demo, test]
chain:
    - Initiate Manufacturing:
        to:
            - Process A
            - Process B
    - Process A:
        head: false
        workflow: My Process
        virtual: true
    - Process B:
        head: false
        workflow: My Process
        resubmit: True

Workflow chain with conditionals and strategies:

name: Manufacturing Chain
desc: Workflow chain for manufacturing process.
tags: [demo, test]
chain:
    - Initiate Manufacturing:
        to:
            - Process A:
                if: '{{ column_value("Column 1") == "foo" }}'
                sample_type: Specimen
    - Process A:
        head: false
        to: Process B
        workflow: My Process
        strategy: named_strategy
    - Process B:
        head: false
        workflow: My Process

Workflow chain with node positioning and swimlanes:

Demo WFC 3:
  chain:
    - Node 1:
        head: true
        position:
          x: 290.75
          y: 100
        startable: true
        swimlane: sl-1641460171013
        to:
          - Checkbox Test:
              if: '{{ True }}'
    - Checkbox Test:
        position:
          x: 570.8214285714287
          y: 300
        startable: true
        swimlane: sl-1641460181862
        to:
          - Approval Instructions:
              if: ''
  swimlanes:
    - color: '#3d85c6'
      height: 200
      id: sl-1641460171013
      label: lane 1
      y: 0
    - color: '#ff9900'
      height: 200
      id: sl-1641460181862
      label: lane 2
      y: 200

Configuration Notes:

Examples

>>> from esp.models import WorkflowChain
>>> chain = WorkflowChain('Illumina Sequencing')
>>> chain.name, chain.created_at
('Illumina Sequencing', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> chain.workflows
[<Workflow(name=Initiate Manufacturing)>,
 <Workflow(name=Process A)>,
 <Workflow(name=Process B)>]

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

pin(version_name: str) -> None

Pin a workflow chain.

pipelines

Return all pipeline objects associated with pipeline button fields.

property variables

Returns the list of resource vars

class esp.models.Workflow(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Workflows from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple workflow:

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
protocols:
    - Set Samples
    - Create Illumina Library
    - Analyze Sequencing Results

Workflow with embedded protocols:

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
protocols:
    - Set Samples:
        protocol: standard
        variables:
          - Attachment:
              rule: attachment
          - Note:
              rule: string
 
    - Create Illumina Library:
        desc: Create Illumina library and assign index codes.
        protocol: sample
        sample: Illumina Library
        relation: 1-to-1
 
    - Analyze Sequencing Results:
        protocol: pipeline
        group: All
        pipeline: Miseq Analysis Pipeline

Workflow with sample types (embedded and referenced):

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
sample_types:
    - Generic sample
    - Illumina Library:
        desc: Sample type for illumina library.
        sequences:
            - ESP SEQUENCE
protocols:
    - Set Samples
    - Create Illumina Library
    - Analyze Sequencing Results

Workflow with data links and sample groupings:

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
links:
    Analyze Sequencing Results:
        Instrument: '{{ column_value("Instrument", "Create Illumina Library") }}'
groups:
    Analyze Sequencing Results: Set Samples
sample_types:
    - Illumina Library
protocols:
    - Set Samples
    - Create Illumina Library
    - Analyze Sequencing Results

Configuration Notes:

Examples

>>> from esp.models import Workflow
>>> wf = Workflow('Illumina Sequencing')
>>> wf.name, wf.created_at
('Illumina Sequencing', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> wf.experiments
[<Experiment(name=Experiment 1)>, <Experiment(name=Experiment 2)>]
>>> wf.protocols
[<Protocol(name=Set Samples)>,
 <Protocol(name=Create Illumina Library)>,
 <Protocol(name=Analyze Sequencing Results)>]
>>> wf.sample_types
[<SampleType(name=Illumina Library)>]

add(item)

Add new objects (Protocol or SampleType) to existing workflow.

Examples

>>> wf = Workflow('Test Workflow')
>>> print(wf.protocols)
['<Protocol(name=Set Samples)>',
 '<Protocol(name=Run Analysis)>']
>>>
>>> pc = Protocol('Summarize Results')
>>> wf.add(pc)
>> print(wf.protocols)
['<Protocol(name=Set Samples)',
 '<Protocol(name=Run Analysis)>',
 '<Protocol(name=Summarize Results)>']

all_experiments()

drop(deep=False)

Drop object from ESP database.

experiments

Property for fluidly accessing Experiment objects from metadata (only returns experiments associated with the current workflow version)

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

BaseModel’s overridden parse_import logic.

Create new object in ESP database using config file or other data.

protocols

Property for fluidly accessing Protocol objects from metadata.

remove(item)

Remove new objects (Protocol or SampleType) from an existing workflow.

Examples

>>> wf = Workflow('Test Workflow')
>> print(wf.protocols)
['<Protocol(name=Set Samples)',
 '<Protocol(name=Run Analysis)>',
 '<Protocol(name=Summarize Results)>']
>>>
>>> pc = Protocol('Summarize Results')
>>> wf.remove(pc)
>>> print(wf.protocols)
['<Protocol(name=Set Samples)>',
 '<Protocol(name=Run Analysis)>']
>>>

sample_type

Property for accessing single sample type object from Workflow (if only one sample type is used for the workflow).

sample_types

Property for fluidly accessing SampleType objects from metadata.

class esp.models.Protocol(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Protocols from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Standard protocol:

name: Set Samples
protocol: standard
desc: Example standard protocol with a few columns.
tags:
  - illumina
  - sequencing
variables:
  - Attachment Column:
      rule: attachment
  - Container Column:
      rule: location
      container: Freezer
  - Dropdown Column:
      rule: dropdown
      dropdown:
        - foo
        - bar
  - Date Column:
      rule: date
  - Note:
      rule: string

Sample protocol with expressions:

name: Create Illumina Library
desc: Example sample protocol with expression columns.
protocol: sample
sample: Illumina Library
relation: 1-to-1
variables:
  - Sample Type:
      rule: string
      onchange: |+
        alert('Changed Sample Type')
  - Instrument:
      rule: string
      value: MiSeq, HiSeq 2000/2500
  - Index Type:
      rule: dropdown
      required: true
      dropdown: '{{ ilmn_kits() }}'
  - I7 Index ID:
      rule: dropdown
      required: true
      dropdown: '{{ ilmn_adapter_names(column_value("Index Type"), position="i7") }}'
  - I7 Index:
      rule: string
      read_only: true
      value: '{{ ilmn_adapter_seq(column_value("Index Type"), column_value("I7 Index ID")) }}'
  - I5 Index ID:
      rule: dropdown
      required: true
      dropdown: '{{ ilmn_adapter_names(column_value("Index Type"), position="i5") }}'
  - I5 Index:
      rule: string
      read_only: true
      value: '{{ ilmn_adapter_seq(column_value("Index Type"), column_value("I5 Index ID"), position="i5") }}'

Pipeline protocol with grouping:

name: Analyze Sequencing Results
protocol: pipeline
desc: Run bioinformatics pipelines to analyze sequencing data.
group: Reference
pipeline: Miseq Analysis Pipeline
variables:
  - Sample:
      rule: string
      visible: false
      pipeline_param: true
      value: '{{ sample["name"] }}'
  - Reference:
      rule: dropdown
      pipeline_param: true
      default: GRCh38
      dropdown:
        - GRCh37
        - GRCh38

Pipeline protocol with embedded pipeline:

name: Analyze Sequencing Results
protocol: pipeline
desc: Run bioinformatics pipelines to analyze sequencing data.
group: All
pipeline:
  Miseq Analysis Pipeline:
    desc: Analyze sequenced library with MiSeq analysis pipeline.
    tasks:
      - BWA Align:
          desc: Align fastq files to reference.
          cmd: "echo 'This is a bam created with the command: bwa mem {{ Sample }}.fq.gz' > {{ Sample }}.bam"
          files:
            - Aligned BAM:
                file_type: text
                filename_template: "{{ Sample }}.bam"
      - GATK Unified Genotyper:
          desc: Use GATK to genotype bam file.
          cmd: "echo 'This is a vcf created with the command: gatk UnifiedGenotyper -I {{ Sample }}.bam -R /references/{{ Reference }}.fa' > {{ Sample }}.vcf"
          files:
            - Genotype VCF:
                file_type: text
                filename_template: "{{ Sample }}.vcf"
    deps:
      GATK Unified Genotyper: BWA Align

Protocol with inheritance:

name: Set Samples Override
inherit: Set Samples
desc: Inherited protocol with extra columns.
variables:
  - Extra Column:
      rule: numeric

Configuration Notes:

Examples

>>> from esp.models import Protocol
>>> pc = Protocol('Analyze Sequencing Results')
>>> pc.name, pc.created_at
('Analyze Sequencing Results', '2019-06-21T16:04:01.199076Z')
 
>>> # show relationships
>>> pc.pipeline
<Pipeline(name=Analyze Sequencing Results)>

drop(deep=False)

Drop object from ESP database.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

pipeline

Return associated pipeline object.

pipelines

Return all pipeline objects associated with pipeline button fields.

sample_type

Return associated sampletype object.

workflows

Query Workflows containing protocol.