======== Examples ======== Below detail code-heavy examples for how to use the client API. They're not meant to be comprehensive, but rather give users a feel for how to traverse and interact with the API, particularly for filling data in LIMS workflows. Create and Submit Experiment -------------------------------------------------------------------------- Similarly to how you can create and submit an experiment via YAML configuration (see below), you can also control how experiments are managed in pure python. .. code-block:: python from esp.models import Experiment # create experiment with 2 new illumina library samples exp = Experiment.create( name='My Experiment', workflow='Example Workflow', project='My Project', samples={ 'count': 3, 'type': 'Illumina Library' } ) # submit experiment and create sample sheet exp.submit(sheet=True) To submit a sample sheet with a specific name, you can use: .. code-block:: python exp.subit(sheet='My Sheet Name') Update Data in a SampleSheet -------------------------------------------------------------------------- You can also use the client to update data in a sample sheet programmatically. For instance, here's how you fill in data for a specific column in a sample sheet from an experiment. .. code-block:: python from esp.models import Experiment # get submitted experiment exp = Experiment('My Experiment') # get first protocol from experiment sheet = exp.protocols[0] # you can alternatively query via protocol name sheet = exp.protocol('My First Protocol') # show first few rows of data sheet.head() # set all values for a column in a worksheet sheet['Column 1'] = 'test' # save values in the application sheet.save() Create a Sample Sheet from Multiple Experiments Programmatically -------------------------------------------------------------------------- In some contexts, it's useful to combine experiments into a single sample sheet for data recording. You can do this in the client using the ``SampleSheet`` object. .. code-block:: python from esp.models import Experiment, SampleSheet # get experiment objects exp1 = Experiment('Experiment 1') exp2 = Experiment('Experiment 2') # create sample sheet with experiments sheet = SampleSheet.create(name='Combined Sheet', experiments=[exp1, exp2]) Change the Name of a Sample and add Tags -------------------------------------------------------------------------- .. code-block:: python from esp.models import Sample # get sample and make sure it exists sample = Sample('ESP000001') if not sample.exists(): raise AssertionError('Sample does not exist!') # change name and add tags sample.name = 'ESP-1' sample.tags.extend(['foo', 'bar']) # save updates sample.push() Complete a Workflow Programmatically -------------------------------------------------------------------------- In contexts where data entry can be automated, you can use the client to populate data for workflows. This can be particularly useful if users are migrating data from a legacy system and want to maintain data structures created throughout workflows and workflow chains. Here is a YAML config example for creating an experiment and filling out data: .. code-block:: yaml name: MS001 workflow: Illumina Sequencing project: Miseq Sequencing samples: count: 2 type: Extract protocols: - Set Samples: complete: true data: Attachment: ${CWD}/runsheet.txt Note: ['Ready to go!', 'Ready!'] - Create Illumina Library: complete: true load: ${CWD}/spreadsheet.txt' - Analyze Sequencing Results: run: true complete: true data: Container: 'MiSeq: Lane 1' Reference: ['GRCh38', 'GRCh37'] You can run this config with the client using the python client CLI: .. code-block:: bash ~$ esp import experiment path/to/experiment.yaml Or, you can run it in a python shell using ``Experiment.create()``: .. code-block:: python from esp.models import Experiment exp = Experiment.create('path/to/experiment.yaml') You can also use the python client to control the process programmatically in a configuration-free way. Here is matching Python code to fill data in data via code: .. code-block:: python from esp.models import Experiment # create experiment with samples exp = Experiment.create( name='MS001', workflow='Illumina Sequencing', project='Miseq Sequencing', samples={ 'count': 2, 'type': 'Extract' } ) # submit experiment to sheet exp.submit() # fill in data for first protocol sheet = exp.protocol('Set Samples') sheet['Attachment'] = 'runsheet.txt' sheet.loc[:, 'Note'] = ['Ready to go!', 'Ready!'] # save and complete sheet sheet.save() sheet.complete() # fill in data for second protocol sheet = exp.protocol('Create Illumina Library') sheet.load('spreadsheet.txt', sep='\t') sheet.save() sheet.complete() # fill in data for final (pipeline) protocol sheet = exp.protocol('Analyze Sequencing Results') sheet['Container'] = 'Miseq: Lane 1' sheet.loc[:, 'Reference'] = ['GRCh38', 'GRCh37'] sheet.save() # run pipeline for sheet sheet.run() # complete sheet sheet.complete() Complete a Workflow Chain Programmatically -------------------------------------------------------------------------- Just like with workflows, it's possible to complete a workflow chain programmatically or with configuration. Here is an example config for filling workflow chain data: .. code-block:: yaml name: MF Chain 1 chain: Manufacturing Chain project: Manufacturing samples: count: 3 type: Tissue workflows: - Initiate Manufacturing submit: true transition: true protocols: - Initiate Manufacturing: complete: true data: Start Date: 01Jan18 Signature: true - Finish Manufacturing: submit: true protocols: - Process A: data: Completion Date: 01Jan18 Signature: true You can run this config with the client using the python client CLI: .. code-block:: bash ~$ esp import experiment_chain path/to/experiment_chain.yaml Or, you can run it in a python shell using ``ExperimentChain.create()``: .. code-block:: python from esp.models import ExperimentChain exp = ExperimentChain.create('path/to/experiment_chain.yaml') You can also use the python client to control the process programmatically in a configuration-free way. Here is matching Python code to fill data in data via code: .. code-block:: python from esp.models import ExperimentChain # create experiment with samples chain = ExperimentChain.create( name='MF Chain 1', workflow=' Manufacturing Chain', project='Manufacturing', samples={ 'count': 3, 'type': 'Tissue' } ) # get first node in chain and submit exp = chain.experiments[0] exp.submit() # fill data for first experiment sample sheet sheet = exp.protocols[0] sheet['Start Date'] = '2018-01-01' sheet['Signature'] = True # submit sheet and transition experiment chain sheet.save() sheet.complete() sheet.transition() # get second node in chain and submit exp = chain.experiments[1] exp.submit() # fill in data for second node in chain sheet = exp.protocols[0] sheet['Completion Date'] = '2018-01-01' sheet['Signature'] = True sheet.save() sheet.complete() Ingest Data from an External Service -------------------------------------------------------------------------- The following code details how to ingest data from an arbitrary process into ESP via the LIMS application. .. code-block:: python # Make sure protocol and workflow exist (ESP requirement for adding data) pc = Protocol.create(name='Ingest Protocol') wf = Workflow.create(name='Ingest Workflow', protocols=[pc]) # Read Ingest data from File import pandas df = pandas.read_csv('iris.txt', sep='\t') # Configure ingest protocol to take new columns # (accounting for column differences across files) pc.variables = [{'name': col, 'var_type': 'text'} for col in df.columns] pc.push() # Create experiment and push data Experiment.create( name='My Ingest', workflow=wf, samples=dict(count=len(df)), submit=True, protocols=[dict(name='Ingest Protocol', data=df)]) Find and print the creation date/creator of all samples with the tag "test" --------------------------------------------------------------------------- Sometimes it's useful to query all samples in the application and filter by some criteria. Here's how to do that using Sample tags: .. code-block:: python for sample in Sample.all(): if 'test' in sample.tags: print('{}: {} by {}'.format(sample.name, sample.created_at, sample.owner.name)) Alternatively, you can search by tag using the ``Sample.search`` classmethod: .. code-block:: python samples = Sample.search(tags=['test']) .. note:: The ``.search()`` method works for other models as well. Find and Submit all Experiments with >10 Samples from a Specific Project ------------------------------------------------------------------------ You can also use the client for managing project data. For instance, here's an example of searching for experiments using specific criteria, and then submitting them: .. code-block:: python from exp.models import Project obj = Project('My Project') for exp in obj.experiments: if len(exp.samples) > 10 and not exp.submitted: exp.submit() Do a PCA with data from a specific Experiment --------------------------------------------- Here's a more comprehensive example of how one might use the client for data science applications. Below, the client is used to pull all metadata within a project to perform a Principal Component Analysis: .. code-block:: python # imports import pandas as pd from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA from sklearn.pipeline import make_pipeline # load the project project = Project('My Project') project.experiments # get the data df = pd.concat(map( lambda x: x.protocols[0].df, project.experiments )).drop(['Complete'], axis=1) # fit model on the data X, Y = df.drop(['Species'], axis=1).values, df.Species.values model = make_pipeline( StandardScaler(), PCA(n_components=3), ) tX = model.fit_transform(X) .. note:: More examples will be included in future versions of the documentation. User ---- .. autoclass:: esp.models.User :noindex: Role ---- .. autoclass:: esp.models.Role :noindex: ParamGroup ---------- .. autoclass:: esp.models.ParamGroup :noindex: Workgroup --------- .. autoclass:: esp.models.Workgroup :noindex: Config ------ .. autoclass:: esp.models.Config :noindex: Configuration ------------- .. autoclass:: esp.models.Configuration :noindex: File ---- .. autoclass:: esp.models.File :noindex: Report ------ .. autoclass:: esp.models.Report :noindex: Pipeline -------- .. autoclass:: esp.models.Pipeline :noindex: Task ---- .. autoclass:: esp.models.Task :noindex: PipelineReport -------------- .. autoclass:: esp.models.PipelineReport :noindex: Analysis -------- .. autoclass:: esp.models.Analysis :noindex: AnalysisTask ------------ .. autoclass:: esp.models.AnalysisTask :noindex: AnalysisPool ------------ .. autoclass:: esp.models.AnalysisPool :noindex: ContainerType ------------- .. autoclass:: esp.models.ContainerType :noindex: Container --------- .. autoclass:: esp.models.Container :noindex: Customer -------- .. autoclass:: esp.models.Customer :noindex: ItemType -------- .. autoclass:: esp.models.ItemType :noindex: Item ---- .. autoclass:: esp.models.Item :noindex: ServiceType ----------- .. autoclass:: esp.models.ServiceType :noindex: Service ------- .. autoclass:: esp.models.Service :noindex: Vendor ------ .. autoclass:: esp.models.Vendor :noindex: Project ------- .. autoclass:: esp.models.Project :noindex: ExperimentChain --------------- .. autoclass:: esp.models.ExperimentChain :noindex: Experiment ---------- .. autoclass:: esp.models.Experiment :noindex: SampleSheet ----------- .. autoclass:: esp.models.SampleSheet :noindex: Worksheet --------- .. autoclass:: esp.models.Worksheet :noindex: SampleType ---------- .. autoclass:: esp.models.SampleType :noindex: WorkflowableClass ----------------- .. autoclass:: esp.models.WorkflowableClass :noindex: Sample ------ .. autoclass:: esp.models.Sample :noindex: WorkflowChain ------------- .. autoclass:: esp.models.WorkflowChain :noindex: Workflow -------- .. autoclass:: esp.models.Workflow :noindex: Protocol -------- .. autoclass:: esp.models.Protocol :noindex: