Automated Content Testing

A standard customer repository layout should have a “tests” directory at the top level. Within this directory are several files of note:

A content test is a python object that follows pytest nomenclature that extends from both the standard python library object unittest.TestCase and from esp.testing.ModelLoaderMixin. For instance:

from unittest import TestCase
from esp.models import Workflow
from esp.testing import ModelLoaderMixin
class TestIlluminaLibraryPrepWorkflow(ModelLoaderMixin, TestCase):
    def test_workflow_exists(self):
        """
        Test Case ID
        Tests:
          * Requirement ID
        Setup:
          # None
        Steps:
          # Request the Illumina Library Prep workflow from ESP
          # Check whether the server returned the workflow
        Expectation:
          # The Illumina Library Prep workflow exists
        """
        assert Workflow('Illumina Library Prep').exists()

Although the above test case could have been written without using ModelLoaderMixin. ModelLoaderMixin makes several facilities available to the test class for convenience.

from unittest import TestCase
from esp.models import Workflow
from esp.testing import ModelLoaderMixin
from . import RESOURCES
class TestIlluminaLibraryPrepWorkflow(ModelLoaderMixin, TestCase):
  DATA = dict(
      Experiment=[
          os.path.join(RESOURCES, 'IlluminaLibraryPrep_001.yml'),
          os.path.join(RESOURCES, 'IlluminaLibraryPrep_002.yml'),
          os.path.join(RESOURCES, 'IlluminaLibraryPrep_003.yml')
      ]
  )

Which would load three “Illumina Library Prep” experiments.

Any content loaded via these mechanisms are “tracked” and can be removed on test end or on next test execution. By default, content tests do not remove created content on test-end. This ensures content authors have an opportunity to review the experiments, worksheets, etc. after the automated tests complete.

“fixtures” allow pytest authors to establish test dependencies in a way that is re-usable across tests. ESP’s testing facilities leverage this capability to make it easy to set up ESP-specific testing dependencies. Using fixtures is the preferred way to set up any pre-conditions for your test, such as:

Using fixtures, you can even create test-specific workflows on-the-fly, which can be useful for testing workflows that have external dependencies (e.g. that use tagged_value or cell to pull data from other workflows) without having to run samples through an entire workflow chain to test the one workflow of interest. Fixtures are generally preferred to using the SEED and IMPORT properties of tests because they are more efficient when running multiple tests that have the same set of dependencies. ESP pytest fixtures are declared using the fixtures function from the esp.testing module. The tests/fixtures.py file should already have this function imported so in practice, most content tests simply add fixtures to that file, but it is also possible to add fixtures directly to test script files.

The fixtures function takes the following arguments:

Within the fixture dependency section, the keys are the names of esp client model objects, including content models:

And data models:

The dependency values are always lists. The list items maybe either the name of an object declared in a content file referenced by the seed file(s), or may be a dictionary that will be passed to the create class method of the corresponding model type, as was done in the “Verification Tests” project test above. This allows for complex creation of test-specific objects. Because the fixture file is a standard python file, any valid python expression can be used, including variables and function calls. A few dependency examples help illustrate the possibilities.

fixtures(
  [os.path.join(ROLES, 'seed', 'content.yml')],
  myfixture=dict(
      sample=[
        {
          'name': 'TestIndividual5',
          'sample_type': 'Individual',
          'children': [{
              'name': 'TestSubject5',
              'sample_type': 'Subject',
              'children': [
                  {'name': 'TestWB000005',
                   'sample_type': 'Whole Blood',
                   'children': [
                      {'name': 'TestBS000005', 'sample_type': 'Blood Spot'}
                    ]},
                    {'name': 'TestDNA00005', 'sample_type': 'Extracted DNA'}
                ]
          }]
        },
        {
          'name': 'TestOrder000005',
          'sample_type': 'Test Order',
          'variables': {
              'Test type': 'STR',
              'Test Code': 'T1021',
              'Test Status': 'Active',
              'Subject': 'TestSubject5'
            }
        })
  )
)

The above fixture declaration creates a new fixture “my fixture” which depends on the “TestIndividual5” entity and the ‘TestOrder000005’ entity. The TestIndividual5 entity will be created once for any tests that depend on the myfixture fixture and that creation process will create the sample hierarchy:

Note that the “Subject” property of the “Test Order” type is a resource link. Because the samples will be created in the order specified in the list, TestIndividual5 (and the hierarchy) will be created and present when TestOrder000005 is created, so the declaration may referencing it. When creating data models, all appropriate model configurations must be loaded. In the example above, the fixture is assuming the sample types “Test Order”, “Whole Blood”, “Extracted DNA”, “Blood Spot”, “Individual”, and “Subject” all exist already. These may exist by virtual of the seed file contents being pre-loaded into the target test system (preferred, see best practices), or they may themselves be declared as dependencies of the fixture. Fixture dependencies are resolved in a logical order that ensures, for instance, sample_type dependencies are resolved prior to sample dependencies that might require them.

In addition to the dependencies above, a fixture’s dependencies may also include the key “fixtures” which is a list of other fixture names. For instance:

fixtures(
  [os.path.join(ROLES, 'seed', 'content.yml')],
  dict(
    myfixture=dict(
      fixtures=['common']
    )
  )
)

The “myfixture” fixture now depends on the “common” fixture, so any tests that use the “myfixture” fixture will also have all of the “common” dependencies loaded. For additional information and examples, see the SDK documentation and the "Fixtures for Pytests" document.

A very common paradigm in ESP testing is to craft experiment data models as yaml files and load the data model via the DATA attribute. Any additional testing or verification activities are then handled via standard python test methods. Experiment yaml files have a powerful assortment of capabilities. A basic example of an Experiment file is:

GlobalFiler STR Blood Spot 001:
    description: |+
        Test standarad paths through global filer str BloodSpot workflow
    submit: true
    workflow: GlobalFiler STR Blood Spot
    project: Verification Tests
    tags:
        - verificationtest
    samples:
        - {name: TestBS000001, type: Blood Spot}
        - {name: TestBS000002, type: Blood Spot}
        - {name: 'BSC # 1', type: Inventory Item}
        - {name: 'NTC # 1', type: Inventory Item}
    protocols:

The above creates an experiment “GlobalFiler STR Blood Spot 001” of type “GlobalFiler STR Blood Spot” in the “Verification Tests” project with four samples: “TestBS000001”, “TestBS000002” of type “Blood Spot” and “BSC # 1” “NTC # 1” of type “Inventory Item”. It fills the data in the first two protocols of the workflow and checks the calculated values. Anything supported by Experiment.create is supported within the yaml files. Formally, an experiment creation definition is:

Experiment Name:
  description: optional description.
  submit: boolean - must be true to fill protocol data
  workflow: name of workflow (required)
  project: name of project (required)
  tags: optional experiment tags
  samples: required. Value is any valid sample declaration that can be passed to Sample.create.
  protocols: list of protocol declarations.

Valid sample declarations include:

Protocol declarations follow the form:

protocol name:
  complete: boolean (default: false)
  run: boolean (default: false)
  actions:
      - action block

Each action block can contain any of the following keys:

Action blocks are executed in the order they appear, allowing the experiment author to perform actions like change worksheet values (data), save and verify computed columns multiple times in succession, with a single experiment. Within a single action block, actions are executed in the order listed above. Note that the ESP content test infrastructure will save the sheet after filling in data.

Often times, you only need a single action block. In these cases, the structure can be flattened:

        - GlobalFiler Blood Spot Sample Setup:
            complete: true
            # for now, fill in concentration manually.
            data:
                Blood Spot Strg: ['Box 1', 'Box 1', 'NA', 'NA']
            verify:
                Individual ID: ['{{ valid_expression(value) }}', '{{ valid_expression(value) }}', '{{ value is None}}', '{{ value is None}}']
                Subject ID: ['{{ valid_expression(value) }}', '{{ valid_expression(value) }}', '{{ value is None}}', '{{ value is None}}']
                Sample Type: [Blood Spot, Blood Spot, Bloodspot STR PCR Control, No Template Control]
                Batch ID: '{{ valid_expression(value) }}'

When using the flattened structure, the special keys “before” and “after” can also be used. For instance:

- My protocol:
    complete: true
    before:
      verify:
        My Column: My Value
    data:
      My Column: Set to a new value
    verify:
      My Other column: a different value
    after:
      data:
        My Other column: value 3

The above is exactly equivalent to:

- My protocol:
    complete: true
    actions:
      - verify:
          My Column: My Value
      - data:
          My Column: Set to a new value
        verify:
          My Other column: a different value
      - data:
          My Other column: value 3

The before and after keys predate the actions key are are maintained for backwards compatibility. Of the available action keys, verify , data, and approve deserve additional discussion.

data:
   Concentration: '{{ random(minval=0, maxval=5, n=5) + random(minval=5.1, maxval=10, n=5)}}

verify:
  <columnname>: value # for instance: Batch ID: Batch1
  <columnname>: [list, of values] # for instance: QC Status: ['Pass', 'Pass', 'Fail']
  <columnname>__options: [list, of, options] # for instance: QC Status__options: ['Pass', 'Fail']
  <columnname>__options: [[first, list], [second, list], [third, list]]

approve:
  Lab Manager Approval: password