Client API

Session Management

class esp.base.Session(host='127.0.0.1', port=8002, cookies=None, headers=None, username='admin@localhost', password='password', cache=False, ssl=False, token=None, email=None, tenant=None)

Object for managing connections to esp.

Parameters:

• host (str) – Host with database to connect to.

• port (int) – Port to connect to database with.

• username (str) – Username to access application with.

• password (str) – Password to be used with username.

• cookies (str) – Optional path to cookies file to store session data.

• token (str) – Optional API token to use in place of username and password for authentication. Can be a string or path to a file containing the token.

• cache (bool) – Whether or not to cache objects for faster querying.

• email (str) – Username to access application with. Deprecated but preserved through at least 2.3 for backwards compatibility. If both username and email are provided, username wins.

Variables:

session (requests.Session) – Internal requests.Session object for managing connection.

delete(endpoint, **kwargs)

Issue DELETE request to esp endpoint, and raise exception if return status code is not 200.

Parameters:

• endpoint (str) – String with relative api endpoint.

• **kwargs – Additional arguments to pass into request.delete. This will usually include the payload for the request.

get(endpoint, **kwargs)

Issue GET request to esp endpoint, and raise exception if return status code is not 200.

Parameters:

• endpoint (str) – String with relative api endpoint.

• **kwargs – Additional arguments to pass into request.get.

jar

Property containing cookie jar for use in authentication.

login()

Login to ESP system and configure authentication. This will be run implicitly if the user issues any request to any non ‘/login’ endpoint.

logout()

Logout of ESP system from the current session.

property options

Return JSON with options on the session.

post(endpoint, **kwargs)

Issue POST request to esp endpoint, and raise exception if return status code is not 200, 201, 202.

Parameters:

• endpoint (str) – String with relative api endpoint.

• **kwargs – Additional arguments to pass into request.post. This will usually include the payload for the request.

put(endpoint, **kwargs)

Issue PUT request to esp endpoint, and raise exception if return status code is not 200, 201, 202.

Parameters:

• endpoint (str) – String with relative api endpoint.

• **kwargs – Additional arguments to pass into request.put. This will usually include the payload for the request.

property user

Property for accessing User object containing current user for session.

esp.base.options(**kwargs)

Set global options for session management. You can also configure the system to use a default options file located in ~/.lab7/client.yml. Here’s an example options file:

$ cat ~/.lab7/client.yml

username: me@localhost
password: password
cache: false

Parameters:

• host (str) – Host with database to connect to.

• port (int) – Port to connect to database with.

• username (str) – Username to access application with.

• password (str) – Password to be used with username.

• cookies (str) – Optional path to cookies file to store session data.

• token (str) – Optional API token to use in place of username and password for authentication. Can be a string or path to a file containing the token.

• cache (bool) – Whether or not to cache objects for faster querying.

• email (str) – Alias for username, retained for backwards compatibility with existing scripts, but due for removal in the future.

• tenant (str) – Tenancy to use. Only for multi-tenant ESP instances. Note: this argument is only necessary for tenancies where the same username + password combination exists in multiple tenants.

Examples

>>> # set default options for connections
>>> import esp
>>> esp.options(
>>>     username='user@localhost', password='pass'
>>>     cookies='/path/to/cookies-file.txt'
>>> )
>>>
>>> # interact with esp client normally
>>> from esp.models import Protocol
>>> obj = Protocol('My Protocol')

Abstract

class esp.models.__base__.BaseModel(ident, data=None, version=None, snapshot=None, **kwargs)

Abstract class for managing initialization methods for common objects.

Parameters:

• ident (str) – Name or uuid to use for initializing object.

• data (dict) – Data to use for instantiating object. This only needs to be specified if you already have all metadata related to an object.

• **kwargs (dict) – Additional data to include in model query. For example, including sample_type=’Generic sample’ will subset the query by sample type.

classmethod all(**kwargs)

Return all instances related to a given model.

Examples

>>> samples = Sample.all()
>>> print(samples[0].name)
'ESP0000001'

property augment

Proxy for setting Model.data.meta.augment.

classmethod create(config=None, overwrite=False, prompt=False, object_names=None, force_overwrite=False, allow_snapshot_uuid_remap=False, **kwargs)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of BaseModel for model-specific logic.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete the current entry in the ESP database.

• prompt (bool) – If current entry for config exists, attempt to detect conflicts and present interactive prompt for resolving conflicts.

• object_names (list[str]) – When config refers to a list of objects to load, only load the items named by object_names. Default is to load all objects in config.

• force_overwrite (bool) – Will force an overwrite, even when no change is detected between the existing DB configuration and the configuration loaded from file.

created_timestamp

Return datetime object with creation timestamp.

property data

Property containing all information about an object. This is the information typically returned from a GET request using the object endpoint and uuid.

drop(deep=False)

Issue DELETE request to remove object in ESP database.

exists()

Test if object exists in esp database.

TODO: THINK ABOUT HOW THIS SHOULD BE USED – fill in

after usage documentation.

export(filename=None, deep=False, filter_empty=False, use_dump_format=False, use_hub_format=False, versioned=False)

Export a yaml-formatted object.

Parameters:

• filename (str) – Filename to write yaml output to

• deep (bool) – Perform a deep (True) or shallow (False) export

• filter_empty (bool) – Attributes where the value is None or empty string are filtered from export.

• use_dump_format (bool) – Whether to use the “dump” yaml format.

• use_hub_format (bool) – Whether to use the hub yaml format

• versioned (bool) – If true, attempts to export a snapshotted/pinned export. Note: not all models support pinning.

Note

A deep export embeds all nested objects (e.g.: protocols of a workflow). A deep export is therefore the yaml analog to the .lab7 file. A shallow export uses names as references to nested objects.

classmethod from_data(data)

Instantiate object from data.

Examples

>>> import esp.base as base
>>> from esp.models import Sample
>>> response = base.SESSION.get('/api/sample?name=ESP000001')
>>> samp = Sample.from_data(response.json())
>>> print(samples[0].name)
'ESP0000001'

classmethod from_definition(uuid)

Instantiate object definition from uuid.

Examples

>>> import esp.base as base
>>> from esp.models import SampleType
>>> st = SampleType.from_definition('8552afc3-882b-41e3-b0cf-1a4cf3f28d33')
>>> print(st[0].name)
'Illumina Sample'

classmethod items_for_names(names, max_query_string_size=4000)

Fetch objects by name. This is similar to search but uses a different fetch strategy and uses generators,

so was split out for backwards compatibility.

Parameters:

• names (List[str]) – List of names of objects to fetch.

• max_query_string_size (int) – Max number of characters allowed in the query string.

Differences from search:

1. Simpler interface. Only “names” is provided.

2. search does a POST to avoid long URLs; items_for_uuids does a “GET” but splits the names payload up so items_for_names may make several requests. However, items_for_names calls object-type specific endpoints that typically provide better performance than the general-purpose resource endpoint used by search.

3. items_for_names is less likely to cause an OOM exception for a web worker by asking for thousands of objects at once. In scenarios where you are simply iterating through the results and do not need to hold the entire collection of objects in memory at once, items_for_names provides better memory performance because it returns a generator so you never have to hold the entire result data in memory at once.

classmethod items_for_uuids(uuids, max_query_string_size=8000)

Fetch objects by UUID. This is similar to search but uses a different fetch strategy and uses generators,

so was split out for backwards compatibility.

Parameters:

• uuids (List[str]) – List of UUIDs of objects to fetch.

• max_query_string_size (int) – Max number of characters allowed in the query string.

Differences from search:

1. Simpler interface. Only “uuids” is provided.

2. search does a POST to avoid long URLs; items_for_uuids does a “GET” but splits the UUID payload up so items_for_uuids may make several requests. However, items_for_uuids calls object-type specific endpoints that typically provide better performance than the general-purpose resource endpoint used by search.

3. items_for_uuids is less likely to cause an OOM exception for a web worker by asking for thousands of objects at once. In scenarios where you are simply iterating through the results and do not need to hold the entire collection of objects in memory at once, items_for_uuids provides better memory performance because it returns a generator so you never have to hold the entire result data in memory at once.

json()

Return dictionary object with object metadata.

owner

Relationship to owner (creator) of object.

classmethod parse_import(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Method for formatting import-style data to data that can be used in a backend POST/PUT request.

Returns:

A data structure to use in a POST/PUT request for

creating or updating the model.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Method for parsing request returned from POST/PUT during object creation. This method is responsible for returning an object after create() is called. :param allow_snapshot_uuid_remap:

push(dry=False)

Run put with metadata to update entry.

classmethod search(name=None, tags=None, method='or', uuids=None, names=None)

Search for model by name wildcard or tags.

Parameters:

• name (str|List[str]) – Wildcard with sample name. For searching with an exact match, simply instantiate the model with the name (i.e. Sample(‘My Sample’)).

• tags (list) – List of tags to search for.

• method (str) – How to filter the search by tags. Can either be ‘and’ (intersection) or ‘or’ (union).

• uuids (List[str]) – UUID(s) search for. For fetching a single object by uuid, instantiate the object as ModelClass(<uuid>) (e.g.: Sample(<uuid>))

summary()

Print organized description of entity.

updated_timestamp

Return datetime object with last update timestamp.

versions

Return full history of item in ESP database as objects of this type. The order of the list is most recent (index 0) to least recent (index -1).

class esp.models.__base__.LinkedModel(ident, data=None, version=None, snapshot=None, **kwargs)

Subclass of BaseModel for decorating parent-child relationship functionality. All items in the ESP database /can/ be linked, but there needs to be more formalism around what /should/ be linked in the database. Because of this, we should restrict access to linking functionality to specific models until the pattern for provenance is more flushed out on the backend.

Parameters:

• ident (str) – Name or uuid to use for initializing object.

• data (dict) – Data to use for instantiating object. This only needs to be specified if you already have all metadata related to an object.

add_children(items)

Add specified items as children to object.

add_dependents(items, label=None)

Add resource dependencies for item or list of items.

add_parents(items)

Add specified items as parent to object.

add_priors(items, label=None)

Add resource priors for item or list of items.

children

Return sample children.

Note

Changes to these data does not push changes to the backend. Use self.add_children for that functionality.

dependencies

Return dependency information for item.

property dependents

Proxy for accessing dependent item dependencies.

TODO: Re-map the dependencies to their related

class types in the client. This currently just returns the JSON associated with the dependency data.

generation(entity_type, which='closestup')

Efficiently return generation of a particular type.

Parameters:

• entity_type (str) – The type of entity to search for in the hierarchy.

• which (str) – One of “closestup”, “furthestup”, “allup” for the nearest, furthest, or all ancestors of the given type, respectively, or “closestdown”, ‘furthestdown’, ‘alldown’ for the nearest, furthest, or all descendants of the given type, respectively.

Returns:

List[Sample] - List of samples of the appropriate type. May be empty. Never null.

get_dependents(label)

Method for getting dependencies tagged with specific label.

Parameters:

label (str) – Label to filter dependencies with.

get_priors(label)

Method for getting priors tagged with specific label.

Parameters:

label (str) – Label to filter priors with.

parents

Return sample parent objects.

Note

Changes to these data does not push changes to the backend. Use self.add_parents for that functionality.

property priors

Proxy for accessing prior item dependencies.

TODO: Re-map the dependencies to their related

class types in the client. This currently just returns the JSON associated with the dependency data.

remove_children(items)

Remove specified items as children to object.

remove_dependents(items, label=None)

Remove resource dependencies for item or list of items.

remove_parents(items)

Remove specified items as parent to object.

remove_priors(items, label=None)

Remove resource priors for item or list of items.

update_link(items, link, method='add', label=None)

Helper method for managing redundancy across prior/depdendent link generation.

Admin

class esp.models.User(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with users from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create user:

name: New User
email: user@localhost
password: password

Create user with roles:

name: New Role User
email: user-role@localhost
password: password
roles:
    - Lab Manager
    - Admin

Examples

>>> from esp.models import User
>>> user = User('New User')
>>> user.name, user.created_at
('User', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> user.roles
[<Role(name=Lab Manager)>, <Role(name=Admin)>]
>>> user.workgroups
[<Workgroup(name=Lab A)>]

>>> # log in as specific user
>>> User('New User').login(password='password')

>>> # get current user
>>> user = User.current()

Parameters:

ident (str) – Name or uuid for object.

classmethod current()

Return user from current session.

drop(deep=False)

Overwrite drop method for current user so emails are changed on drop (allowing User re-creation to happen).

NOTE: BACKEND ARCHIVES MODEL AND MAKES IT INACCESSIBLE VIA CLIENT

THIS FUNCTIONALITY SHOULD ONLY BE USED DURING TESTING.

login(password=None)

Login as specific user.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new User in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

roles

Return roles associated with user.

workgroups

Return roles associated with user.

class esp.models.Role(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with users from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create role:

name: my-role
desc: My Role
permissions:
    Project: [create, update]
    Sample: [create, read, update, delete]

Create role with default users:

name: my-role
desc: My Role
members:
    - User 1
    - User 2
applications:
    - admin
    - lims

Create role with specific app permissions:

name: my-role
desc: My Role
applications:
    - admin
    - lims

Configuration Notes:

• Default role permissions are access to all actions for each type of model. Permissions need to be overridden to restrict access.

Examples

>>> from esp.models import Role
>>> role = Role('my-role')
>>> role.name, role.created_at
('my-role, '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> role.users
[<User(name=User 1)>, <User(name=User 2)>]

Parameters:

ident (str) – Name or uuid for object.

drop(deep=False)

Overwrite drop method for current user so emails are changed on drop (allowing User re-creation to happen).

NOTE: BACKEND ARCHIVES MODEL AND MAKES IT INACCESSIBLE VIA CLIENT

THIS FUNCTIONALITY SHOULD ONLY BE USED DURING TESTING.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new Role in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

users

Return users associated with roles.

class esp.models.ParamGroup(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with ParamGroups from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create param group:

illumina:
    miseq-1: /path/to/miseq-1
    miseq-2: /path/to/miseq-2

Create multiple param groups:

illumina:
    miseq-1: /path/to/miseq-1
    miseq-2: /path/to/miseq-2

integrations:
    clarity: http://clarity-url:1000
    internal: http://internal-server

Examples

>>> from esp.models import ParamGroup
>>> pg = ParamGroup('apps')
>>> pg.name, group.created_at
('apps, '2019-06-21T16:04:01.199076Z')

>>> # show params
>>> pg.params.client
'/path/to/client/bin/esp'

Parameters:

ident (str) – Name or uuid for object.

drop(deep=False)

Overwrite drop method so ‘apps’ param group can’t be deleted.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new Workgroup in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

class esp.models.Workgroup(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with workgroups from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create workgroup:

name: Lab A
desc: Workgroup for all users in Lab A

Create workgroup with default users:

name: Lab A
desc: Workgroup for all users in Lab A
members:
    - User 1
    - User 2

Examples

>>> from esp.models import Workgroup
>>> group = Workgroup('Lab A')
>>> group.name, group.created_at
('Lab A, '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> group.users
[<User(name=User 1)>, <User(name=User 2)>]

Parameters:

ident (str) – Name or uuid for object.

classmethod associate_resources(workgroups, resources)

Set the workgroup(s) to the set of workgroups for the resources.

drop(deep=False)

Overwrite drop method for current user so emails are changed on drop (allowing User re-creation to happen).

NOTE: BACKEND ARCHIVES MODEL AND MAKES IT INACCESSIBLE VIA CLIENT

THIS FUNCTIONALITY SHOULD ONLY BE USED DURING TESTING.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new Workgroup in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

users

Return users associated with roles.

class esp.models.Config(*args, **kwargs)

Object for interacting with instance configuration within ESP. This model allows developers to directly interact with the /api/config endpoint in ESP.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

That is, this configuraiton is always updaitng a DB row, never truly creating it, but uses the same interface as all other models for consistency.

This model is deprecated and will be removed in the future. Use Configuration instead.

classmethod all(**kwargs)

Return all instances related to a given model.

Examples

>>> samples = Sample.all()
>>> print(samples[0].name)
'ESP0000001'

classmethod create(config=None, overwrite=False, prompt=False, **kwargs)

Create a new config object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – COnfig file or information to use in creating new object

• overwrite (bool) – Whether or not to update a current entry in the ESP database.

drop(deep=True)

Issue DELETE request to remove object in ESP database.

push(dry=False)

Run put with metadata to update entry.

class esp.models.Configuration(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with database-backed instance configuration within ESP. This model allows developers to directly interact with the /api/configuration endpoint in ESP.

Model to interact with ESP’s DB-backed configuration system.

A configuration consists of two top-level dictionaries: one for “public” configuration and one for “secret” configuration. Generally, this model masks the distinction and makes the subkeys of both available as top-level properties, but also supports direct access of the public and secret dictionaries.

Examples

>>> from esp.models import Configuration
>>> conf = Configuration('esp') # get the ESP system config.
>>> conf.executor.num_workers
1
>>> conf = Configuration.create(name='CustomConfig', key1='value1', key2={'subkey1': 'value1.1', 'subkey2': 'value2.2'})
>>> conf.key2
{'subkey1': 'value2.1', 'subkey2': 'value2.2'}
>>> conf.key2.subkey1
'value2.1'
>>> conf.config
{'key1': 'value2', 'key2': {'subkey1': 'value2.1', 'subkey2': 'value2.2'}}
>>> conf.add_secret('secretkey', 'this value is secret')
>>> conf.secrets
{'secretkey': 'this value is secret'}
>>> conf.push()
>>> conf.refresh()
>>> conf.secrets
{'secretkey': 'XXXXXXX'}
>>> conf.secretkey
'XXXXXXX'
>>> conf.newkey = 'new value'
>>> conf.config
{'key1': 'value2', 'key2': {'subkey1': 'value2.1', 'subkey2': 'value2.2'}, 'newkey': 'new value'}
>>> conf.add_secret('newkey', 'conflicting secret')
>>> conf.newkey
'conflicting secret'
>>> conf.config.newkey
'newkey'

add_secret(name, value)

Add a “secret” configuration value. :param name: The name :param value: The real value

A secret configuration value is one where the public/REST API will return ‘XXXXXX’ for the value once the value has been pushed.

classmethod all(**kwargs)

Fetch all active configurations. :param **kwargs:

Returns:

classmethod from_data(data)

Instantiate object from data.

Examples

>>> import esp.base as base
>>> from esp.models import Sample
>>> response = base.SESSION.get('/api/sample?name=ESP000001')
>>> samp = Sample.from_data(response.json())
>>> print(samples[0].name)
'ESP0000001'

get(name, default=None)

Gets a single value from configuration, supplying a default if the key doesn’t exist. :param name: Name of the value to retrieve. :param default: Default to use if the configuration key doesn’t exist in the configuration object.

Returns:

The configured value, falling back to teh povided default if necessary

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new Configuration in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

push(dry=False, restart=False)

Update the remote configuration with local modifications.

Parameters:

dry – If dry is true, returns the JSON structure that would be pushed without pushing.

Returns:

The data that was or would be pushed.

Data

class esp.models.File(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with files from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create file:

name: My Protocol SOP
desc: An SOP file
tags: [sop, protocol]
uri: relative/path/to/instructions.pdf
upload: true
type: PDF

Create file object with reference to existing file:

name: Large Sequencing File
desc: An SOP file
tags: [sop, protocol]
uri: file:///path/to/local/file/instructions.bam
upload: true
type: raw

Create file object and register as task output:

name: SampleSheet.csv
desc: Illumina Run Sheet
task_file: Illumina Runsheet
uri: /path/to/SampleSheet.csv
upload: false
type: csv

Configuration Notes:

• Due to current backend limitations, uri inputs can only take the formats: “relative/path”, “/absoluate/path”, and “file:///absolute/path”.

• If upload is true, uri must resolve to an existing file on the local file system, otherwise an error will be raised. If upload is False, file will only be registered. Upload defaults to false.

• The task_file parameter is primarily useful when registering pipeline output files (since you can pass in the task instance UUID).

• “type” is used internally by ESP and is not necessary a mime type. For instance, the mime type of a bam file is generally application/gzip or application/octet-stream, but the type in ESP might normally be “bam”.

• mime-type is an acceptable key for specifying the mime-type. If no mime-type is provided, the client will attempt to guess the type and pass that type along as the content type of the uploaded file.

Examples

>>> from esp.models import File
>>> fi = File('My Protocol SOP')
>>> fi.name, fi.created_at
('My Protocol SOP', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> fi.path
'/path/to/Lab7_ESP/Data/files/0000/instructions.pdf.6a523a74-6703-4474-a24e-5a463b9d9770'
>>> fi.contents
'... raw file contents ...'
>>> fi.download('local-copy.pdf')

Parameters:

ident (str) – Name or uuid for object.

contents

Return raw file contents (as string).

download(outfile, encoding='UTF-8')

Download file and store as output filename.

Parameters:

• outfile (str|file) – Path to downloaded file or file object to write to.

• encoding (str) – If outfile is a file object in string mode, the string encoding

• server. (to use to decode the binary data from the) –

Note

Fetches the contents in a single go. For large text files where only a subset may be needed, consider using the ‘contents’ endpoint instead. For large, streamable binary files, such as BAM files, esp REST APIS have support for ranged data retrieval that is not yet supported by the python client.

linkable_uri

Return a “linkable” URL.

The returned URL will be relative to the API_SERVER root, so external consumers should prepend the API_SERVER information.

Examples

>>> File.create({ 'uri': '/path/to/myfile.txt', 'name': 'myfile' }).linkable_uri()
/api/files/<:uuid>/static

This methood is provided to simplify embedding links ESP files in extensions and third-party content. Note that the URL format is subject to change in the future.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

class esp.models.Report(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Reports from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create report with embedded html:

name: My Report
desc: An example html report.
tags: [html, demo]
contents:
    <h1>My Report</h1>

Create report with local html file:

name: My Report
desc: An example html report.
tags: [html, demo]
contents: $LAB7DATA/contents/reports/my-report.html

Create applet from local html file:

name: My Report
desc: An example html applet.
tags: ['esp:applet']
contents: $LAB7DATA/contents/reports/my-report.html

Configuration Notes:

• To promote an ESP report to an ESP applet, include esp:applet in the set of report tags.

• The contents parameter can either take a local file or raw html contents to use as the report data. If no file is found from the string, the data are assumed to be raw html data to include as contents.

Examples

>>> from esp.models import Report
>>> report = Report('My Report')
>>> report.name, report.created_at
('My Report', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> report.contents
'<h1>My Report</h1>'

Parameters:

ident (str) – Name or uuid for object.

classmethod all(**kwargs)

Return all instances related to a given model.

Examples

>>> samples = Sample.all()
>>> print(samples[0].name)
'ESP0000001'

property contents

API niceness for editing internal contents, if simple html report.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

Analysis

class esp.models.Pipeline(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Pipelines from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create pipeline with single task:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
tasks:
    - Create Illumina Runsheet

Create pipeline with nested tasks:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
tasks:
    - Create Illumina Runsheet:
        desc: Run script to generate illumina runsheet.
        cmd: /path/to/script.py {{project}} > runsheet.xml
        files:
            - Illumina Runsheet:
                file_type: xml
                filename_template: '{{ "runsheet.xml" }}'

Create pipeline with task dependencies:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Create pipeline with report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - []
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Create pipeline with report and failure report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - []
failure_report:
    Runsheet Failure Report:
        desc: Report showing details of failed runsheet generation
        elements
        - - type: file_details
            depends:
                file: stderr
                tasknumber:1
        - []
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Configuration Notes:

• Current implementation requires special syntax around creating report elements - - and - []. This is due to payload requirements and will be updated in future versions to be more user-friendly.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')
>>> pipe.name, pipe.created_at
('Generate Illumina Runsheet', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> pipe.tasks
[<Task(name=Create Illumina Runsheet)>, <Task(name=Upload Illumina Runsheet)>]
>>> pipe.report
<ReportTemplate(name=Runsheet Report)>

>>> # run pipeline (wait to finish)
>>> analysis = pipe.run(project='My Project')
>>> analysis.task[0].stdout
'stdout from the first analysis task'
>>> analysis.task[0].files
[<File(name=Illumina Runsheet)>]

>>> # run pipeline (in background)
>>> analysis = pipe.run(project='My Project', background=True)
>>> analysis.pause()
>>> analysis.restart()
>>> analysis.kill()

Parameters:

ident (str) – Name or uuid for object.

dependencies

Show task dependency graph. Edits to this graph will be reflected when a user issues self.push().

drop(deep=False)

Drop all associated reports and pipeline.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Parse response from server when creating new pipeline. :param allow_snapshot_uuid_remap:

report

Return PipelineReport objects associated with pipeline.

reports

Return PipelineReport objects associated with pipeline.

run(**kwargs)

Run pipeline with specified parameters.

Parameters:

**kwargs – Keyword arguments to pass in for pipeline execution.

Examples

>>> pipe = Pipeline('NGS Pipeline')
>>> print(pipe.meta.tasks)
['Preprocess FASTQ', 'Align BAM', 'Call Variants']
>>> print(pipe.arguments)
'fastq'
>>> pipe.run(fastq='/path/to/file.fastq')

task_steps

Return tasks associated with pipeline along with their pipeline-specific step information.

tasks

Return tasks associated with pipeline.

class esp.models.Task(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Tasks from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple task (no variables):

name: Ping Internal Server
desc: Run command to ping internal server.
cmd: curl http://internal-server

Simple task with variables:

name: Ping Internal Server
desc: Run command to ping internal server.
cmd: curl '{{ server }}'

Create task and track outputs:

name: Create Illumina Runsheet
desc: Run script to generate illumina runsheet.
cmd: /path/to/generate_runsheet.py {{project}} > runsheet.xml
files:
    - Illumina Runsheet:
        file_type: xml
        filename_template: '{{ "runsheet.xml" }}'

Create task with inline code:

name: Report DNA Type
desc: Run bash script to generate simple report.
cmd: |+
   # Simple Task that determines if the specified 'ratio' is in the proper range
   if [ "{{ type }}" = "DNA" ]; then
       echo "<b>This is a <font color='green'>DNA</font> sample</b>" > result.html
   elif "{{ type }}" = "DNA" ]; then
       echo "<b>This is a <font color='green'>RNA</font> sample</b>" > result.html
   fi

files:
    - Type Report:
        file_type: html
        filename_template: '{{ "report.html" }}'

Examples

>>> from esp.models import  Task
>>> task  = Task('Create Illumina Runsheet')
>>> task.name, task.created_at
('Create Illumina Runsheet', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> task.variables
['project']
>>> task.cmd
'/path/to/generate_runsheet.py {{project}} > runsheet.xml'

Parameters:

ident (str) – Name or uuid for object.

class esp.models.PipelineReport(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with PipelineReports from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create pipeline with report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - []
tasks:
    - Create Illumina Runsheet

Create multi-step pipeline with report:

name: Generate Illumina Runsheet
desc: Run script to generate illumina runsheet and generate runsheet report
report:
    Runsheet Report:
        desc: Report showing details of runsheet generation
        elements:
        - - type: file_details
            depends:
                file: Illumina Runsheet
                tasknumber:1
        - - type: raw_file
            depends:
                file: Runsheet Upload Report
                tasknumber:2
        - - type: html
            contents: |+
                <h1>Report Header</h1>
                <p>Report Body</p>
        - []
tasks:
    - Create Illumina Runsheet
    - Upload Illumina Runsheet
deps:
    Upload Illumina Runsheet: Create Illumina Runsheet

Configuration Notes:

• Report configuration will generally happen in the context of a pipeline, Accordingly, this documentation references report generation in that context.

• Current implementation requires special syntax around creating report elements - - and - []. This is due to payload requirements and will be updated in future versions to be more user-friendly.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Create Illumina Runsheet')
>>> pipe.name, pipe.created_at
('Generate Illumina Runsheet', '2019-06-21T16:04:01.199076Z')
>>> pipe.report.name, pipe.created_at
('Runsheet Report', '2019-06-21T16:04:01.199076Z')
>>> pipe.report.pipeline.name, pipe.report.pipeline.created_at
('Generate Illumina Runsheet', '2019-06-21T16:04:01.199076Z')

Parameters:

ident (str) – Name or uuid for object.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Note

A Pipeline with the following construct could have a corresponding config that looks like the following:

If a Pipeline (named ‘pipelineA’) with Tasks “taskB” and “taskA” is setup as such (without a temporal connection) in the UI:

-   -   -
- taskB -
- taskA -
-   -   -

and the Pipeline has a report that has the following construct:

FileDetails:
    fileA_for_taskA (Step 2: taskA)
FileDetails:
    fileB_for_taskB (Step 1: taskB)

the config object could have (or has) the following structure:

{
    "elements": [
        [
            {
                "depends": [
                    {
                        "file": "fileA_for_taskA",
                        "tasknumber": 2
                    }
                ],
                "type": "file_details"
            },
            {
                "depends": [
                    {
                        "file": "fileB_for_taskB",
                        "tasknumber": 1
                    }
                ],
                "type": "file_details"
            }
        ],
        []
    ],
    "name": "pipelineA",
    "uuid": null,
    "desc": ""
}

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

pipeline

Relationship to pipeline object.

class esp.models.Analysis(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with analyses from the ESP database.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')

>>> # run pipeline to return analysis object
>>> analysis = pipe.run(project='My Project')
>>> analysis.task[0].stdout
'stdout from the first analysis task'
>>> analysis.task[0].files
[<File(name=Illumina Runsheet)>]

>>> # run pipeline (in background)
>>> analysis = pipe.run(project='My Project', background=True)
>>> analysis.pause()
>>> analysis.restart()
>>> analysis.kill()

>>> # get current analysis (within script)
>>> analysis = Analysis.current()
>>> analysis.pause()

Parameters:

ident (str) – Name or uuid for object.

join(expect=['failed', 'done', 'paused'], timeout=1000000.0)

Wait for analysis to finish, and return when done.

tasks

Return objects for each step in pipeline.

class esp.models.AnalysisTask(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with task instances from the ESP database.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')

>>> # run pipeline to return analysis object
>>> analysis = pipe.run(project='My Project')

>>> # get first task object from analysis
>>> atask = analysis.task[0]
>>> atask.stdout
'stdout from the first analysis task'
>>> atask.stderr
'stderr from the first analysis task'
>>> atask.files
[<File(name=Illumina Runsheet)>]

Parameters:

ident (str) – Name or uuid for object.

files

Property for accessing all files associated with task.

script

Property to accessing task stdout.

stderr

Property to accessing task stderr.

stdout

Property to accessing task stdout.

class esp.models.AnalysisPool(analyses)

Object for interacting with analyses from the ESP database.

Examples

>>> from esp.models import Pipeline
>>> pipe = Pipeline('Generate Illumina Runsheet')

>>> # run multiple scripts to gather in pool
>>> foo_analysis = pipe.run(project='Foo Project')
>>> bar_analysis = pipe.run(project='Bar Project')

>>> # create analysis pool to control execution
>>> pool = AnalysisPool([foo_analysis, bar_analysis])
>>> pool.state
'running'

>>> # wait for all analyses to finish in pool
>>> pool.join()
>>> pool.state
'done'

Parameters:

analyses (str) – List of analyses to include in pool.

join(period=0.25)

Wait for all analyses to finish, and return when they’re done.

Parameters:

period (float) – Period (seconds) for polling results on a join operation.

property state

Return “state” of the analysis pool.

Container

class esp.models.ContainerType(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with container types from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Single-element container:

name: 96-Well Plate
desc: 96-Well Plate for sample aliquots
tags: [plate, lab]
label_format: '%s%s'
slot_capacity: element
axes:
    - Rows: [A, B, C, D, E, F, G, H]
    - Cols: [01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12]
contains:
    - Sample

Multi-element container:

name: Freezer
desc: Freezer for samples and racks
tags: [freezer, lab]
label_format: 'Shelf %s'
slot_capacity: list
axes:
    - Shelf: [1, 2, 3, 4, 5]
contains:
    - Sample
    - Container

Type with default containers:

name: Freezer
desc: Freezer for samples and racks
tags: [freezer, lab]
label_format: 'Shelf %s'
slot_capacity: list
axes:
    - Shelf: [1, 2, 3, 4, 5]
contains:
    - Sample
    - Container
create:
    - Freezer 1
    - Freezer 2

Nested config with containers and samples to fill:

name: Freezer
desc: Freezer for samples and racks
tags: [freezer, lab]
label_format: 'Shelf %s'
slot_capacity: list
axes:
    - Shelf: [1, 2, 3, 4, 5]
contains:
    - Sample
    - Container
create:
    - Freezer 1
    - name: Freezer 2
      barcode: 1234
      fill:
        Shelf 1: [ESP0001, ESP0002]

Examples

>>> from esp.models import ContainerType
>>> ct = ContainerType('Freezer')
>>> ct.name, ct.created_at
('Freezer', '2019-06-21T16:04:01.199076Z')
>>> ct.containers
[<Container(name=Freezer 1)>, <Container(name=Freezer 2)>]

Parameters:

ident (str) – Name or uuid for object.

axes

Return the axes structure List[Dict[str,object]]. Each list entry is a dict of information about a single axis, including keys:

• name (str): the axis name

• size (int): the axis size

• labels (List[str]): the axis labels.

containers

Property for accessing all related containers.

property dimensions

Return the container dimensions.

drop(deep=False)

Overwrite drop to delete all corresponding containers as well.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new ContainerType in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Method for parsing request and returning :param allow_snapshot_uuid_remap:

property renderer

The name of the container rendering script.

slot_for_position(position, by, recycle=False, label=True)

Convert 1d index to 1d or 2d coordinates.

Note that x may be > slot_count, so this can be used when laying out a list of samples into a list of locations.

Parameters:

• position (int) – the 1-based 1d position (e.g.: for a 96 well plate, position is 1-96)

• by (0|1) – The fastest rotating axis for a 2d container. Not used for 1d containers, where 0 is the first axis and 1 is the second.

• recycle (bool) – If true, position may be > total capacity of the container type under the assumption that the positions will be “recycled” across multiple containers of the same type.

• label (bool) – If true, the position label is returned. Otherwise, the position coordinates (0-based) are returned as a list-like object.

Examples

>>> ct = ContainerType.create({'name': '96-well Plate', axes=[
... {'Row': [x for x in 'ABCDEFGH']},
... {'Col': ['{:02}'.format(x) for x in range(1, 12)]}],
... 'slot_capacity': 'element'})
>>> ct.slot_for_position(25, 0)
"C01"
>>> ct.slot_for_position(25, 1)
"A04"
>>> ct.slot_for_position(25, 0, label=False)
[3, 1]
>>> ct.slot_for_position(25, 1, label=False)
[1, 4]
>>> ct.slot_for_position(100, 0)
ValueError: 100 does not fit into 96 slots
>>> ct.slot_for_position(100, 0, recycle=True)
"A04"

slot_label(*idxdims, **kwdims)

Build individual label components.

Examples

>>> ct = ContainerType.create({'name': '96-well Plate', axes=[
... {'Row': [x for x in 'ABCDEFGH']},
... {'Col': ['{:02}'.format(x) for x in range(1, 12)]}],
... 'slot_capacity': 'element'})
>>> ct.slot_label(1, 1)
'A01'
>>> ct.slot_label(Row=1, Col=1)
'A01'

Parameters:

• *idxdims (list) – List of 1-based dimension indices in the same order as axes or dimensions

• **kwdims (dict) – Dict of axis-name: axis-index

total_capacity

Return the total capacity of the container. If the slot capacity is “list”, returns “Inf”, otherwise, returns the product of the length of all dimensions.

class esp.models.Container(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with containers from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create container:

name: Freezer 1
desc: Freezer in lab A
tags: [freezer, lab]
type: Freezer
barcode: 12345

Container with samples to fill:

name: Freezer 1
desc: Freezer in lab A
tags: [freezer, lab]
type: Freezer
barcode: 12345
fill:
    Shelf 1:
        - ESP0001
        - name: ESP0002
          desc: special new sample
          tags: [one, two]

Configuration Notes:

• Upon creation, samples can be used to fill container slots via the fill parameter.

• Default container values can be set for samples using the variables parameter.

Examples

>>> from esp.models import Container
>>> ct = Container('Freezer 1')
>>> ct.name, ct.created_at
('Freezer 1', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> ct.slots
['Shelf 1', 'Shelf 2', 'Shelf 3', 'Shelf 4', 'Shelf 5']
>>> ct.samples
[<Sample(name=ESP0001)>, <Sample(name=ESP0002)>]
>>> ct.slot_samples
{
    'Shelf 1': [<Sample(name=ESP0001)>, <Sample(name=ESP0002)>],
    'Shelf 2': [],
    'Shelf 3': [],
    ...
}

>>> # put items in specific slots
>>> ct.put_items_in_slots(
>>>     items=[Sample('ESP0001'), Sample('ESP0002')],
>>>     slots=['Shelf 1', 'Shelf 2']
>>> )

Parameters:

ident (str) – Name or uuid for object.

container_type

Return container type definition associated with item.

Note

Using container type definition here is unintuitive and backend storage of data should be resolved.

drop(deep=False)

Remove all items from container and drop.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new Container in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Parse response and put items in slots (if specified). :param allow_snapshot_uuid_remap:

put_items_in_slots(items, slots, overwrite=False, worksheet=None, fields=None, save=True)

Puts an item into the specified slot.

Parameters:

• item (Sample|Container|List[Sample|Container]) – that may be placed in a container slot, or a list of instances.

• slots (str|List[str]) – Name(s) of slot(s) to put item in.

• overwrite (bool) – is already occupied, overwrite=True will kick that item out and put this item in; overwrite=False will raise a ValueError. If the slot is empty or allows multiple items, the value of overwrite is ignored - new items are always appended to the slot.

• worksheet (esp.models.SampleSheet) –

• fields (dict[str,str]) – for the sample. Note that fields can only be set if worksheet is specified.

• save (bool) - If True (the default) – immediately saved to the backend. Note that callers are still responsible for pushing resource vals to the backend even if worksheet is specified.

Returns:

None|Dict[UUID, List[Dict]] - If worksheet is None, None is

returned. Otherwise, a Dict is returned mapping UUIDs to location structures. The location structures can be directly converted to string via json.dumps to be used as resource val values.

Items can be placed into a container inside or outside of a worksheet. Within the context of a worksheet, the backend currently stores the container information in both the lims val and the container, so synchronizing across both is required. Hence, if you’re putting an item into a slot, and that information will be recorded in a worksheet, pass the worksheet object to this method and an appropriate lims val will be created.

samples

Return array of samples associated with container.

slot_samples

Return array of container slots, filtered to only samples.

slots

Return array of container slots.

Inventory

class esp.models.Customer(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Customers from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple customer:

name: Customer 1
desc: Customer 1 from the North Pole.
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Customer without address:

name: Customer 2
desc: Customer 2 without address.
tags: [demo, north]

Customer with billing and shipping address:

name: Customer 3
desc: Customer 3 from the North Pole
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole
shipping_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Configuration Notes:

• Customer addresses are optional, and the minimal config for a customer is one with only a specified name parameter.

Parameters:

ident (str) – Name or uuid for object.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

class esp.models.ItemType(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with ItemTypes from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Item type for stock reagent:

name: Demo Reagent
consumption_type: stock reagent
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent

Item type for consumable:

name: Demo Consumable
consumption_type: consumable
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent

Item type for kit component:

name: Demo Kit Component
consumption_type: kit component
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent

Create item type with associated items:

name: Consumable
consumption_type: consumable
units: uL
reorder_threshold_qty: 100
reorder_amount_qty: 50
vendors:
    - Agilent
create:
    - CNS001
    - CNS002

Examples

>>> from esp.models import ItemType
>>> it = ItemType('Demo Consumable')
>>> it.name, it.created_at
('Demo Consumable', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> it.vendors
[<Vendor(name=Agilent)>]

Parameters:

ident (str) – Name or uuid for object.

drop(deep=False)

Issue DELETE request to remove object in ESP database

items

Relationship to Item model.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Method for parsing request and returning :param allow_snapshot_uuid_remap:

sequences

Return sequence ID used within ESP. This id is used throughout the system in a variety of different ways, so being able to query it is useful.

vendors

Relationship to Vendor model.

class esp.models.Item(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with ItemTypes from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple item:

name: Example Item
tags: [agilent, perishable]
barcode: 45262341
barcode_type: QR
item_type: Demo Consumable
vendor: Agilent
lot_id: Lot # 1
serial_id: 12345
initial_qty: 10
expires: 12/25/2019
status: Quarantined

Item with different barcode type:

name: Example Item
tags: [agilent, perishable]
barcode: 1234
barcode_type: 1D
item_type: Demo Consumable
vendor: Agilent
lot_id: ABC

Examples

>>> from esp.models import Item
>>> item = Item('Example Item')
>>> item.name, item.created_at
('Example Item', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> it.vendor
<Vendor(name=Agilent)>

>>> # useful functions
>>> item.expired()
False
>>> item.consumed()
True
>>> item.consumed()
True

>>> # interacting with item
>>> item.add_quantity(10)
>>> item.remove_quantity(5)
>>> item.empty()

>>> # interacting with variables
>>> item = Item('Example Item', params='["resource_vals"]')
>>> item.variables
>>> item.variables['Example Variable']

>>> # adding item with variables
>>> item_to_add = {
>>>     'name': 'The Name of the Item',
>>>     'description': 'Item Description',
>>>     'item_type': 'Item Type Name',
>>>     'initial_qty': 22,
>>>     'status': 'Quarantined',
>>>     'resource_vals': [{'name': 'The name of custom field', 'value': 'The Value'}]
>>> }
>>> Item.create(item_to_add)

Parameters:

ident (str) – Name or uuid for object.

add_quantity(amount)

Add quantity to item.

empty()

Remove all quantity from item.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

remove_quantity(amount)

Remove quantity from item.

vendor

Relationship to Vendor model.

class esp.models.ServiceType(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with ServiceTypes from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Sample service type:

name: Sample Service Type
desc: Service type for a demo service.
tags: [service, demo]
price_type: sample
base_price: 350

Time service type:

name: Time Service Type
desc: Service type for a time service.
tags: [service, demo]
price_type: time
base_price: 350
scale_price: 25
scale_units: minutes

Configuration Notes:

• Available options for ServiceType price_type parameters are: sample, protocol, workflow, time, or volume. However, only sample and time are currently supported in the UI.

• Available options for ServiceType scale_units parameters are: hour, minutes, and seconds.

Parameters:

ident (str) – Name or uuid for object.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

class esp.models.Service(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Service from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple service:

name: My Service
service_type: My Service Type
customer: Lab1

Service with embedded type and customer:

name: My Service
service_type:
    My Service Type:
        price_type: time
        base_price: 350
        scale_price: 25
        scale_units: minutes
customer:
    Lab1:
        billing_address:
            street1: 1234 Northword Lane
            street2: Suite # 123
            city: North Pole
            state: North Pole
            country: North Pole
            postal_code: North Pole

Examples

>>> from esp.models import Service
>>> service = Service('My Protocol SOP')
>>> service.name, service.created_at
('My Service', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> service.service_type.name
'My Service Type'
>>> service.customer.name
'Customer 1'

Parameters:

ident (str) – Name or uuid for object.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

class esp.models.Vendor(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Vendors from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple vendor:

name: Vendor 1
desc: Vendor 1 from the North Pole.
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Vendor without address:

name: Vendor 2
desc: Vendor 2 without address.
tags: [demo, north]

Vendor with billing and shipping address:

name: Vendor 3
desc: Vendor 3 from the North Pole
tags: [demo, north]
billing_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole
shipping_address:
    street1: 1234 Northword Lane
    street2: Suite # 123
    city: North Pole
    state: North Pole
    country: North Pole
    postal_code: North Pole

Configuration Notes:

• Vendor addresses are optional, and the minimal config for a customer is one with only a specified name parameter.

Parameters:

ident (str) – Name or uuid for object.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

Project

class esp.models.Project(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Projects from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple project:

name: My Project
desc: Project for a series of experiments.
tags: [demo, test]

Project with nested experiments:

name: My Project
desc: Project for a series of experiments.
tags: [demo, test]
experiments:
    - My Experiment 1:
        workflow: Workflow 1
        submit: true
        samples: [ESP001, ESP002]

Project with nested experiment chains:

name: My Project
desc: Project for a series of experiments.
tags: [demo, test]
chains:
    - My Experiment Chain 1:
        chain: Workflow Chain 1
        samples: [ESP001, ESP002]

Configuration Notes:

• Experiments and Experiment Chains can be nested within project configuration, which is particularly useful for seeding data in bulk or creating test resources.

Examples

>>> from esp.models import Project
>>> project = Project('Project 1')
>>> project.name, project.created_at
('Project 1', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> project.experiments
[<Experiment(name=Experiment 1)>, <Experiment(name=Experiment 2)>]
>>> project.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]

Parameters:

ident (str) – Name or uuid for object.

drop(deep=False)

Issue DELETE request to remove object in ESP database.

experiments

Property for fluidly accessing Experiment objects from metadata.

new_experiment(**data)

Add new Experiment to existing workflow.

Parameters:

**data (object) – Keyword arguments for creating experiment.

Examples

>>> obj = Project('Test Workflow')
>>> print(obj.experiments)
[]
>>>
>>> exp = obj.new_experiment(name='test', samples=['one', 'two'])
>> print(obj.experiments)
['<Experiment(name=test)']
>>> print(exp.samples)
['<Sample(name=one)>',
 '<Sample(name=two)>']

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Parse response from server when creating new project. This method is primarily responsible for creating nested experiments once a project is created. :param allow_snapshot_uuid_remap:

samples

Property for fluidly accessing Sample objects from projects.

class esp.models.ExperimentChain(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with ExperimentChains from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create simple chain:

name: MF Chain 1
chain: Manufacturing Chain
project: Manufacturing
samples:
    count: 3
    type: Tissue

Create chain with nested experiments:

name: MF Chain 1
chain: Manufacturing Chain
project: Manufacturing
samples:
    count: 3
    type: Tissue
workflows:
    - Initiate Manufacturing
        submit: true
        transition: true
        protocols:
            - Initiate Manufacturing:
                complete: true
                data:
                  Start Date: 01Jan18
                  Signature: true
    - Finish Manufacturing:
        submit: true
        protocols:
            - Process A:
                data:
                  Completion Date: 01Jan18
                  Signature: true

Configuration Notes:

• The workflows parameter in the config definition references nested configuration for specific Experiment objects.

• To automatically transition across workflows via configuration, nodes with possible transitions need to include the transition: true configuration (seen above).

• Experiment Chain configuration must include a referenced project, per constraints on Experiment Chains in ESP.

Examples

>>> from esp.models import ExperimentChain
>>> chain = ExperimentChain('MF Chain 1')
>>> chain.name, chain.created_at
('MF Chain 1', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> chain.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]
>>> chain.project
<Project(name=Project 1)>
>>> chain.experiments
[<Experiment(name=Experiment 1)>, <Experiment(name=Experiment 2)>]
>>> chain.chain
<WorkflowChain(name=Manufacturing Chain)>
>>> chain.workflows
[<Workflow(name=Initiate Manufacturing)>, <Workflow(name=Finish Manufacturing>]

>>> # helper methods
>>> chain.active
True
>>> chain.submit()  # submit workflow chain
>>> chain.fill(...) # fill with workflows config above

Parameters:

ident (str) – Name or uuid for object.

property active

Check if experiment is currently active.

chain

Property for accessing experiment workflow chain definition.

classmethod create(config=None, overwrite=False, prompt=False, object_names=None, **kwargs)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of BaseModel for model-specific logic.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete the current entry in the ESP database.

• prompt (bool) – If current entry for config exists, attempt to detect conflicts and present interactive prompt for resolving conflicts.

• object_names (list[str]) – When config refers to a list of objects to load, only load the items named by object_names. Default is to load all objects in config.

• force_overwrite (bool) – Will force an overwrite, even when no change is detected between the existing DB configuration and the configuration loaded from file.

drop(deep=False)

Issue DELETE request to remove object in ESP database.

experiments

Property for accessing experiment project.

fill(workflows)

Fill experiment with protocol data.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Parse response from experiment creation and manage submission and protocol data filling. :param allow_snapshot_uuid_remap:

project

Property for accessing experiment project.

property sample_types

Proxy for returning sample types from workflows.

samples

Property for accessing initial sample objects in experiment.

submit(sheet=True)

Submit experiment to lab for processing.

workflows

Property for accessing experiment workflow definition.

class esp.models.Experiment(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Experiments from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create simple experiment:

name: MS001
workflow: Illumina Sequencing
project: Miseq Sequencing
samples:
    count: 3
    type: Extract

Create experiment and fill in data:

name: MS001
workflow: Illumina Sequencing
project: Miseq Sequencing
samples:
    count: 2
    type: Extract
protocols:
  - Set Samples:
      complete: true
      data:
        Attachment: ${CWD}/runsheet.txt
        Note: ['Ready to go!', 'Ready!']

  - Create Illumina Library:
      complete: true
      load: ${CWD}/spreadsheet.txt'

  - Analyze Sequencing Results:
      run: true
      complete: true
      data:
        Container: 'MiSeq: Lane 1'
        Reference: ['GRCh38', 'GRCh37']

Create experiment and fill in data with verification:

name: MS001
workflow: Illumina Sequencing
project: Miseq Sequencing
samples:
    count: 2
    type: Extract
protocols:
  - Set Samples:
      complete: true
      before:
        verify:
          Attachment: '{{ value is None }}'
      data:
        Attachment: ${CWD}/runsheet.txt
        Note: ['Ready to go!', 'Ready!']
      after:
        verify:
          Attachment: '{{ value is not None }}'

  - Create Illumina Library:
      complete: true
      cols:
        - Index Type
        - I7 Index ID
        - I5 Index ID
      actions:
        - data:
            - ['TruSeq Amplicon', 'A701', 'A501']
            - ['TruSeq Amplicon', 'A702', 'A502']
          verify:
            Index Type: 'TruSeq Amplicon'

  - Analyze Sequencing Results:
      run: true
      complete: true
      data:
        Amount: 10
        Reference: ['GRCh38', 'GRCh37']
      verify:
        Amount: '{{ value >= 10 }}'
        Reference: ['GRCh38', 'GRCh37']

Configuration Notes:

• There are several hooks for entering data in workflows, and each of them are useful in different contexts. At a high-level, data are entered in order of the block they’re specified in. An overview of the blocks are as follows:

  • before - Perform actions before any other actions are perform.

  • actions - List of action blocks to perform (as a list).

  • after - Perform actions after any other actions are perform.

  If no before, actions, or after blocks are specified, then the parameters included in the config will be run in a single action. See the examples above for context.

• Available actions that can be performed when filling in data are:

  • data - Fill in data specified in configuration. Data can take

    the format of a dictionary of values (fill in all rows with the specified value), or a dictionary of lists (use the list indices to fill in individual rows).

  • load - Load tabular file into worksheet, matching file headers

    with protocol columns.

  • user/password - Log in as specific user before entering data.

  • verify - Run assertions on data after data are entered into

    a worksheet. This action is particularly useful for verifying that expressions work as expected.

  • run - Start a pipeline from a pipeline protocol. This option will

    run all pipelines specified for the worksheet protocol.

  • wait - Wait a number of seconds before performing the next action.

• To automatically create Worksheets from experiments, use the submit: true parameter.

• To complete a protocol at the end of filling in data, use the complete: true parameter.

Examples

>>> from esp.models import Experiment
>>> exp = Experiment('MS001')
>>> exp.name, exp.created_at
('MS001', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> exp.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]
>>> exp.project
<Project(name=Project 1)>
>>> exp.workflow
<Workflow(name=Initiate Manufacturing)>

>>> # accessing worksheet data
>>> exp.protocols[0]
Sample ID    Column 1    Column 2
ESP001       1.2         foo
ESP002       1.3         bar
>>> exp.protocol('Set Samples')['Column 1']
[1.2, 1.3]

>>> # helper methods
>>> exp.active
True
>>> exp.submit()  # submit workflow
>>> exp.fill(...) # fill with workflow config above
>>> exp.archive()

Parameters:

ident (str) – Name or uuid for object.

property active

Check if experiment is currently active.

add_samples(new_samples)

Add additional samples to the experiment. NOTE: If the experiment is pending, the samples will replace the current list of samples rather than add to it. :param new_samples: list of samples to add :type new_samples: List[Sample]

archive()

Archive experiment once sample sheet is finished. This provides guarantees around experiment completeness by explicitly checking for protocol completion before dropping.

classmethod create(config=None, overwrite=False, prompt=False, object_names=None, **kwargs)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of BaseModel for model-specific logic.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete the current entry in the ESP database.

• prompt (bool) – If current entry for config exists, attempt to detect conflicts and present interactive prompt for resolving conflicts.

• object_names (list[str]) – When config refers to a list of objects to load, only load the items named by object_names. Default is to load all objects in config.

• force_overwrite (bool) – Will force an overwrite, even when no change is detected between the existing DB configuration and the configuration loaded from file.

drop(deep=False)

Issue DELETE request to remove object in ESP database.

fill(protocols)

Fill experiment with protocol data.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Parse response from experiment creation and manage submission and protocol data filling. :param allow_snapshot_uuid_remap:

project

Property for accessing experiment project.

protocol(name, index=0)

Retrieve protocol instance by name for easier API access.

protocols

Property for accessing sample sheet related to experiment.

push()

Overwrite push method to account for inconsistencies in PUT payloads vs GET payloads.

sample_group(idx=0)

Property for accessing sample groups within experiment.

property sample_types

Proxy for returning sample types from workflows.

samples

Property for accessing initial sample objects in experiment.

sheet

Property for accessing sample sheet related to experiment.

submit(sheet=True, append_samples=False)

Submit experiment to lab for processing.

workflow

Property for accessing experiment workflow definition.

class esp.models.SampleSheet(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with SampleSheets from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Create sample sheet with submitted experiments:

name: Sheet 001
desc: Sample sheet with experiments
experiments:
    - Experiment 1
    - Experiment 2

Create sample sheet nested experiments:

name: Sheet 001
desc: Sample sheet with experiments
experiments:
    - Experiment 1
    - Experiment 2:
        samples:
            count 2
        submit: True
        sheet: False

Configuration Notes:

• It’s recommended to use Experiment objects for accessing worksheet data. This model is primarily useful for creating multi-experiment sample sheets.

Examples

>>> from esp.models import SampleSheet
>>> sheet = SampleSheet('Sheet 001')
>>> sheet.name, sheet.created_at
('Sheet 001', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> sheet.samples
[<Sample(name=ESP001)>, <Sample(name=ESP001)>, ...]
>>> sheet.experiments
[<Experiment(name=Experiment 1)>, <Sample(name=Experiment 2)>]
>>> sheet.project
<Project(name=My Project)>

Parameters:

ident (str) – Name or uuid for object.

add_experiment(experiment, samples=None)

Add experiment(s) to existing sample sheet. If samples is provided, only samples in the provided list will be added. Otherwise, all samples from all experiments will be added.

classmethod all(**kwargs)

Return all instances related to a given model.

archive()

Try to gracefully archive sample sheet using DELETE request. This method make assumptions about experiment completeness and will fail if any Worksheets in SampleSheet are incomplete.

drop(deep=False)

Force drop a sample sheet by removing all samples and running super drop method.

experiment

Return experiment objects associated with sample sheet.

experiment_for_sample(sample, sample_index=None)

Superceded by experiments_for_sample(). Included for backwards compatibility.

experiments

Return experiment objects associated with sample sheet.

experiments_for_sample(sample, sample_index=None)

Get list of Experiments corresponding to a Sample row. This usually returns a list with a single Experiment, but if a fan-in Sample Protocol has fanned in Samples from multiple Experiments, these rows pass a pipe-delimited list of Experiment UUIDs for “rolled-up” rows that actually correspond to separate StepInstanceBaseSamples maintained by separate Experiments. These rows are typically updated in tandem so that they have the same contents.

Parameters:

• sample (str) – Sample UUID.

• sample_index (int) – Index of Sample row. Only needed for first (submitted) SampleGroup which may have the same Sample included from more than one Experiment.

Returns:

List of Experiments.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

classmethod parse_response(data, overwrite=False, prompt=False, allow_snapshot_uuid_remap=False)

Do multi-experiment adding after sample sheet creation. :param allow_snapshot_uuid_remap:

project

Return project associated with sample sheet.

projects

Return all projects associated with sample sheet.

protocol(protocol_name)

Get a protocol by name

protocols

Get all protocols in a worksheet.

remove(item)

Remove samples or experiments from existing sample sheet.

samples

Return all samples added to sample sheet. Only includes the initial sample set (ie the samples directly added by the user). Does not include child samples created by protocols within the worksheet. Samples will be returned in the order they were added to the worksheet.

transition()

Transition samples in the SampleSheet that are part of a workflow chain.

workflow

Return the workflow definition for this sample sheet.

class esp.models.Worksheet(ident, data=None, version=None, snapshot=None, **kwargs)

Object for managing sample sheets within ESP. This object is not configurable, and operates like a pandas DataFrame for each protocol in a workflow. You can access these worksheet objects from an experiment via:

>>> from esp.models import Experiment
>>> exp = Experiment('My Experiment')
>>> ws = exp.protocol('My Protocol')
>>> ws = exp.protocols[0]

Also, since the Worksheet object acts as a pandas DataFrame, you can interact with it like a data frame:

>>> ws.columns
['Column 1', 'Column 2']
>>> ws['Column 1'] = ws['Column 1'].apply(lambda x: x.replace('foo', 'bar'))
>>> ws.melt(...)

After changing the data in the worksheet object, you can save it back to the L7 database and proceed to the next worksheet using:

>>> ws.save()
>>> ws.complete()
>>> ws.next()

Parameters:

• ident (str) – UUID for worksheet instance.

• experiment (Experiment) – Experiment object related to worksheet.

approve(password, sample=None, column=None)

Approve sample and column in worksheet.

Parameters:

• password (str) – Password to approve with.

• sample (str) – Sample name to approve (default is all samples)

• column (str) – Column name to approve (default is all columns)

complete(value=True, save_first=True, samples=None)

Mark complete column on workflow and save.

property completed

Return boolean describing whether or not sheet has been completed.

convert(col, val)

Convert column value from sheet.

convert_approval(value)

Create column value data structure for approval column.

convert_attachment(value)

Create column value data structure for attachment column.

convert_item_qty_adjust(value)

Create column value data structure for itemqtyadj column.

convert_location(value)

Create column value data structure for location column.

df

Return sheet data for data editing/fetching.

fail(sample, message='')

Fail specified sample or samples in sample sheet.

grouped()

True if this worksheet tab is for a a grouped protocol

index

Return the protocol index within the workflow for this worksheet.

load(filename, sep='\t')

Update sheet with data from specified file.

next()

Return next protocol instance for editing.

refresh()

Refresh the object.

run(background=False, run_groups=None, col_name='Start')

Run pipeline for columns and wait for results to finish.

Parameters:

• background (bool) – When true, return immediately after submitting the pipelines. Otherwise, wait for pipelines to complete before returning.

• run_groups (optional[List|Dict|Set]) – If not none, only samples with a group value included in run_groups will be executed. For ungrouped pipeline protocols, run_groups should use the sample names.

Returns:

AnalysisPool for submitted analyses, or None if no analyses were submitted.

sample_group

Return the sample group for this worksheet.

sample_sheet

Return a reference to the sample sheet for this worksheet.

samples

Return the samples for this worksheet.

save(evaluate=True, refresh=True)

Set sample values that have changed

verify(data)

Verify that the data in the sheet matches the data specification passed in. Data can be formatted as:

// all values in colname have the same value.
colname: value

// all values in colname have the same value.
colname: [1, 2, 3, 4] # vector of values should match this vector

// Specify values for each sample.
colname:
    SAM000001: 1
    SAM000002: 2
    SAM000003: 3
    SAM000004: 4

Value is usually treated literally, but if the value is a string that starts with {{ and ends with }}, everything within {{ }} is treated as a python expression to evaluate. The expression must evaluate to a boolean and may reference the variables:

• value - the observed value for one row/sample for the column being verified.

• row - the complete (pandas) row under evaluation.

Expression can be simple, such as:

{{bool(value)}} # make sure value is specified/not null/not empty.

Or more complicated:

{{value == row['colA'] + '123' + row['colB']}}

Verify makes one utility function available to the expression: “eq”.

{{ eq(value, 123.5, 2) }}

This is equivalent to:

{{ round(float(value), 2) == round(float(123.5, 2)) }}

Entity

esp.models.SampleType

alias of EntityType

class esp.models.WorkflowableClass(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with WorkflowableClass from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple workflowable type:

name: Patient
desc: A Human Patient
tags: [patient, demo]
plural_name: Patients
view_template: <h1>My View Template</h1>
list_view_template: <h1>My View Template</h1>
detail_panel_template: <h1>My View Template</h1>
icon_svg: <svg xmlns=“http://www.w3.org/2000/svg” width=“24" height=“24” viewBox=“0 0 24 24"><g fill=“none” fill-rule=“evenodd”><path d=“M0 0h24v24H0z”/><path fill=“#550185" fill-rule=“nonzero” d=“M6 6v14h12V6h-3v1H9V6H6zm4-1a2 2 0 1 1 4 0h5v16H5V5h5zm1 5h5v1h-5v-1zm0 3h5v1h-5v-1zm-3-3h2v1H8v-1zm0 3h2v1H8v-1zm0 3h2v1H8v-1zm3 0h5v1h-5v-1z”/>

Examples

>>> from esp.models import WorkflowableClass
>>> wf = WorkflowableClass('Patient')
>>> wf.name, wf.created_at
('Library', '2019-06-21T16:04:01.199076Z')

Parameters:

ident (str) – Name or uuid for object.

drop(*args, **kwargs)

Issue DELETE request to remove object in ESP database.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

esp.models.Sample

alias of Entity

Workflow

class esp.models.WorkflowChain(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with WorkflowChains from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple workflow chain:

name: Manufacturing Chain
desc: Workflow chain for manufacturing process.
tags: [demo, test]
chain:
    - Initiate Manufacturing:
        to:
            - Process A
            - Process B
    - Process A:
        head: false
        workflow: My Process
        virtual: true
    - Process B:
        head: false
        workflow: My Process
        resubmit: True

Workflow chain with conditionals and strategies:

name: Manufacturing Chain
desc: Workflow chain for manufacturing process.
tags: [demo, test]
chain:
    - Initiate Manufacturing:
        to:
            - Process A:
                if: '{{ column_value("Column 1") == "foo" }}'
                sample_type: Specimen
    - Process A:
        head: false
        to: Process B
        workflow: My Process
        strategy: named_strategy
    - Process B:
        head: false
        workflow: My Process

Workflow chain with node positioning and swimlanes:

Demo WFC 3:
  chain:
    - Node 1:
        head: true
        position:
          x: 290.75
          y: 100
        startable: true
        swimlane: sl-1641460171013
        to:
          - Checkbox Test:
              if: '{{ True }}'
    - Checkbox Test:
        position:
          x: 570.8214285714287
          y: 300
        startable: true
        swimlane: sl-1641460181862
        to:
          - Approval Instructions:
              if: ''
  swimlanes:
    - color: '#3d85c6'
      height: 200
      id: sl-1641460171013
      label: lane 1
      y: 0
    - color: '#ff9900'
      height: 200
      id: sl-1641460181862
      label: lane 2
      y: 200

Configuration Notes:

• See the ESP documentation for information about how to configure workflow transition strategies.

• See the conditional transition logic above for an example of how to send subsets of samples to downstream nodes in a workflow chain.

• More information on types of workflow configuration can be found in the Workflow object documentation.

• Nesting configuration for workflows is usually an easier way of defining configuration for workflows that won’t be used across workflow chains.

• Note the on the backend, rules are evaluated for transitions in the context of the worksheet that triggers the transition, except there is no “current”/”active” protocol, so expressions like column_value must specify both the column and the protocol.

Examples

>>> from esp.models import WorkflowChain
>>> chain = WorkflowChain('Illumina Sequencing')
>>> chain.name, chain.created_at
('Illumina Sequencing', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> chain.workflows
[<Workflow(name=Initiate Manufacturing)>,
 <Workflow(name=Process A)>,
 <Workflow(name=Process B)>]

Parameters:

ident (str) – Name or uuid for object.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

pin(version_name: str) → None

Pin a workflow chain.

Parameters:

version_name – The name to give the pinned version

pipelines

Return all pipeline objects associated with pipeline button fields.

property variables

Returns the list of resource vars

class esp.models.Workflow(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Workflows from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Simple workflow:

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
protocols:
    - Set Samples
    - Create Illumina Library
    - Analyze Sequencing Results

Workflow with embedded protocols:

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
protocols:
    - Set Samples:
        protocol: standard
        variables:
          - Attachment:
              rule: attachment
          - Note:
              rule: string

    - Create Illumina Library:
        desc: Create Illumina library and assign index codes.
        protocol: sample
        sample: Illumina Library
        relation: 1-to-1

    - Analyze Sequencing Results:
        protocol: pipeline
        group: All
        pipeline: Miseq Analysis Pipeline

Workflow with sample types (embedded and referenced):

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
sample_types:
    - Generic sample
    - Illumina Library:
        desc: Sample type for illumina library.
        sequences:
            - ESP SEQUENCE
protocols:
    - Set Samples
    - Create Illumina Library
    - Analyze Sequencing Results

Workflow with data links and sample groupings:

name: Illumina Sequencing
desc: Workflow for filling in data during illumina sequencing.
tags: [demo, test]
links:
    Analyze Sequencing Results:
        Instrument: '{{ column_value("Instrument", "Create Illumina Library") }}'
groups:
    Analyze Sequencing Results: Set Samples
sample_types:
    - Illumina Library
protocols:
    - Set Samples
    - Create Illumina Library
    - Analyze Sequencing Results

Configuration Notes:

• See the ESP documentation for information about data linking, sample groups, and available protocol configuration options.

• More information on types of protocol configuration can be found in the Protocol object documentation.

• Nesting configuration for protocols is usually an easier way of defining configuration for protocols that won’t be used across workflows.

Examples

>>> from esp.models import Workflow
>>> wf = Workflow('Illumina Sequencing')
>>> wf.name, wf.created_at
('Illumina Sequencing', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> wf.experiments
[<Experiment(name=Experiment 1)>, <Experiment(name=Experiment 2)>]
>>> wf.protocols
[<Protocol(name=Set Samples)>,
 <Protocol(name=Create Illumina Library)>,
 <Protocol(name=Analyze Sequencing Results)>]
>>> wf.sample_types
[<SampleType(name=Illumina Library)>]

Parameters:

ident (str) – Name or uuid for object.

add(item)

Add new objects (Protocol or SampleType) to existing workflow.

Parameters:

item (object) – exsiting Protocol or SampleType object to add to workflow.

Examples

>>> wf = Workflow('Test Workflow')
>>> print(wf.protocols)
['<Protocol(name=Set Samples)>',
 '<Protocol(name=Run Analysis)>']
>>>
>>> pc = Protocol('Summarize Results')
>>> wf.add(pc)
>> print(wf.protocols)
['<Protocol(name=Set Samples)',
 '<Protocol(name=Run Analysis)>',
 '<Protocol(name=Summarize Results)>']

all_experiments()

Returns:

All experiments for this workflow, regardless of workflow version

drop(deep=False)

Drop object from ESP database.

experiments

Property for fluidly accessing Experiment objects from metadata (only returns experiments associated with the current workflow version)

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

protocols

Property for fluidly accessing Protocol objects from metadata.

remove(item)

Remove new objects (Protocol or SampleType) from an existing workflow.

Parameters:

item (object) – exsiting Protocol or SampleType object to remove from workflow.

Examples

>>> wf = Workflow('Test Workflow')
>> print(wf.protocols)
['<Protocol(name=Set Samples)',
 '<Protocol(name=Run Analysis)>',
 '<Protocol(name=Summarize Results)>']
>>>
>>> pc = Protocol('Summarize Results')
>>> wf.remove(pc)
>>> print(wf.protocols)
['<Protocol(name=Set Samples)>',
 '<Protocol(name=Run Analysis)>']
>>>

sample_type

Property for accessing single sample type object from Workflow (if only one sample type is used for the workflow).

sample_types

Property for fluidly accessing SampleType objects from metadata.

class esp.models.Protocol(ident, data=None, version=None, snapshot=None, **kwargs)

Object for interacting with Protocols from the ESP database.

See the Usage and Examples pages of the documentation for more context and comprehensive examples of how to create and use this type of objects.

Configuration:

Standard protocol:

name: Set Samples
protocol: standard
desc: Example standard protocol with a few columns.
tags:
  - illumina
  - sequencing
variables:
  - Attachment Column:
      rule: attachment
  - Container Column:
      rule: location
      container: Freezer
  - Dropdown Column:
      rule: dropdown
      dropdown:
        - foo
        - bar
  - Date Column:
      rule: date
  - Note:
      rule: string

Sample protocol with expressions:

name: Create Illumina Library
desc: Example sample protocol with expression columns.
protocol: sample
sample: Illumina Library
relation: 1-to-1
variables:
  - Sample Type:
      rule: string
      onchange: |+
        alert('Changed Sample Type')
  - Instrument:
      rule: string
      value: MiSeq, HiSeq 2000/2500
  - Index Type:
      rule: dropdown
      required: true
      dropdown: '{{ ilmn_kits() }}'
  - I7 Index ID:
      rule: dropdown
      required: true
      dropdown: '{{ ilmn_adapter_names(column_value("Index Type"), position="i7") }}'
  - I7 Index:
      rule: string
      read_only: true
      value: '{{ ilmn_adapter_seq(column_value("Index Type"), column_value("I7 Index ID")) }}'
  - I5 Index ID:
      rule: dropdown
      required: true
      dropdown: '{{ ilmn_adapter_names(column_value("Index Type"), position="i5") }}'
  - I5 Index:
      rule: string
      read_only: true
      value: '{{ ilmn_adapter_seq(column_value("Index Type"), column_value("I5 Index ID"), position="i5") }}'

Pipeline protocol with grouping:

name: Analyze Sequencing Results
protocol: pipeline
desc: Run bioinformatics pipelines to analyze sequencing data.
group: Reference
pipeline: Miseq Analysis Pipeline
variables:
  - Sample:
      rule: string
      visible: false
      pipeline_param: true
      value: '{{ sample["name"] }}'
  - Reference:
      rule: dropdown
      pipeline_param: true
      default: GRCh38
      dropdown:
        - GRCh37
        - GRCh38

Pipeline protocol with embedded pipeline:

name: Analyze Sequencing Results
protocol: pipeline
desc: Run bioinformatics pipelines to analyze sequencing data.
group: All
pipeline:
  Miseq Analysis Pipeline:
    desc: Analyze sequenced library with MiSeq analysis pipeline.
    tasks:
      - BWA Align:
          desc: Align fastq files to reference.
          cmd: "echo 'This is a bam created with the command: bwa mem {{ Sample }}.fq.gz' > {{ Sample }}.bam"
          files:
            - Aligned BAM:
                file_type: text
                filename_template: "{{ Sample }}.bam"
      - GATK Unified Genotyper:
          desc: Use GATK to genotype bam file.
          cmd: "echo 'This is a vcf created with the command: gatk UnifiedGenotyper -I {{ Sample }}.bam -R /references/{{ Reference }}.fa' > {{ Sample }}.vcf"
          files:
            - Genotype VCF:
                file_type: text
                filename_template: "{{ Sample }}.vcf"
    deps:
      GATK Unified Genotyper: BWA Align

Protocol with inheritance:

name: Set Samples Override
inherit: Set Samples
desc: Inherited protocol with extra columns.
variables:
  - Extra Column:
      rule: numeric

Configuration Notes:

• See the ESP documentation for information about type types of columns and associated metadata available to use as protocol variables.

• Using the group parameter on protocol definitions allows developers to group columns by a specified column or value.

• The sample protocol relation property can take either 1-to-1, fan-out, or fan-in.

• Protocol inheritance is useful for creating slightly modified copies of existing protocols without needing to re-define variable configuration. All parameters are copied over from the inherited protocol.

• Nesting configuration for pipelines is usually an easier way of defining configuration for pipelines that won’t be used across protocols.

Examples

>>> from esp.models import Protocol
>>> pc = Protocol('Analyze Sequencing Results')
>>> pc.name, pc.created_at
('Analyze Sequencing Results', '2019-06-21T16:04:01.199076Z')

>>> # show relationships
>>> pc.pipeline
<Pipeline(name=Analyze Sequencing Results)>

Parameters:

ident (str) – Name or uuid for object.

drop(deep=False)

Drop object from ESP database.

classmethod parse_import(config, overwrite=False, allow_snapshot_uuid_remap=False)

Create new object in ESP database using config file or other data. This method should be overwritten by subclasses of LinkedModel for model-specific logic.

Parameters:

• config (str, dict, list) – Config file or information to use in creating new object.

• overwrite (bool) – Whether or not to delete current entry in the ESP database.

pipeline

Return associated pipeline object.

pipelines

Return all pipeline objects associated with pipeline button fields.

sample_type

Return associated sampletype object.

workflows

Query Workflows containing protocol.