Extensions

The extensions folder in the L7|ESP SDK allows developers to create custom expressions and endpoints in the application. Here’s an example of the structure of that directory:

└── extensions/
    ├── server/
    │   ├── expressions.py
    │   ├── invokables.py
    │   └── requirements.txt
    └── client/
        └── expressions.js

L7|ESP offers a number of extension mechanisms for cases when the out-of-the-box functionality is insufficient to meet a particular need, both for server-side and client-side functionality. On the client-side, the invokables.js, renderers.js, and fas-overrides.js files provide entry to augment and customize the user experience. On the server-side are custom Expressions, Invokables, Transition Strategies, Data Souce Loaders, and API Callbacks.

Server Side Extensions

L7|ESP provides a number of server-side extension points detailed below, in order from “most-used” to “least-used”.

Custom Expression Functions

Custom server-side expressions available in L7|ESP Worksheets can be created in the extensions/server/expressions.py file.

Users may contribute new Expression functions to L7|ESP. Expressions must be callable or subclasses of lab7.expression.APICall. They are contributed using the lab7.extensions.expression decorator.

@expression Arguments:

name - the name of the Expression. If not specified, the callable object is examined for name or __name__ attributes, in order. If found, the value is used for the name, otherwise an error is raised.
context - the Expression evaluation context to add to. Currently, only Lab7ExpressionContext.ALL is supported.

Expressions may accept any arguments and return any value, although in most contexts, the return value will ultimately be coerced to a string. For example, contributing a quantile function might look like:

from esp.extensions import expression

@expression
def quantile(array, quant):
    import numpy as np
    return np.percentile([float(x) for x in array], float(quant*100))

After registering the extension with L7|ESP (see “Registering server-side-extensions”), the quantile function will be available in all contexts.

Here are some additional examples of using the @expression decorator:

@expression
def myfunc(a, a):
    return int(a) + int(b)

Now myfunc is accessible from, e.g., LIMS default values:

{{ myfunc(1, 2) }}

To override the name by which the Expression is registered, supply the name argument to the decorator.

@expression(name='add2')
def myfunc(a, a):
    return int(a) + int(b)

Now the expression is:

{{ add2(1, 2) }}

By default, Expressions are set to available in every evaluation context. To limit to a specific context, supply the context argument.

from lab7.extensions.expressions import Lab7ExpressionContext

@expression(name='add2', context=Lab7ExpressionContext.ALL)
def myfunc(a, a):
    return int(a) + int(b)

The Expression is available in every context. If no context is provided, the context is set to Lab7ExpressionContext.ALL.

Note

Currently, the only context used in L7|ESP is Lab7ExpressionContext.ALL, but support for context-specific functions will be added in a future release of L7|ESP.

In advanced Expression scenarios, users may need access to the database. In these cases, Expressions should be objects that extend from lab7.expressions.APICall. Objects that extend from APICall have access to several useful attributes, including

self.ctx - the Expression evaluation context. See “Using Expressions in L7|ESP” for information on what is available in various Expression evaluation contexts.
self.session - the SQL Alchemy database session
self.agent - The active L7|ESP user
self.cache - A dictionary shared across Expressions and Expression evaluations for a single evaluation event. For instance, when evaluating a LIMS worksheet, the cache dictionary is created at the beginning of evaluation and retained until the entire worksheet is finished. This allows Expression authors to lookup a value 1x/in-bulk, and cache the results. For database-backed Expressions, it is highly encouraged to use the cache to help ensure sample sheet saving stays fast. (In one example at L7, a sample sheet save with 100 samples went from 30+ seconds down to 3 seconds by adding caching to one Expression).

The following example creates a “service_types” Expression function that looks up the list of service types from the DB, but only if the list has not already been fetched before in this evaluation context.

@expression(name="service_types")
class ServiceTypes(APICall):
    """Return a list of service types

    Args:
        names_only (bool): If true (the default), the returned list is only the service type names. If false, the
            returned list is the full service type dictionary.
    """

    def api_func(self, names_only=True):
        key = '__service_type_list__{}'.format(names_only)
        if key not in self.cache:
            service_types = query_service_types(params=['name'], agent=self.agent, session=self.session)
            if names_only:
                self.cache[key] = [x['name'] for x in service_types]
            else:
                self.cache[key] = service_types
        return self.cache[key]

Custom Queries

Custom queries can be contributed to the backend by defining the query in a yaml file and placing the yaml file in /opt/l7esp/data/content/queries. Queries added this way are available through API call from /api/v2/queries/{query_file_name}. The YAML file MUST have the suffix .yaml and the filename must not have any spaces. The structure of the YAML file is as follows:

name: {query name, spaces ok}
description: |+
  QUERY DESCRIPTION
query: SQL
parameters:
  param1:
    description: param1 description
    type: list # list of values, such as what might be used in an IN clause.
  singleton_param:
    description: param 2 description. Param 2 is a single value.

For instance, a simple query to fetch project names might be:

name: Project Name List
description: |+
    Fetch the name for all projects in L7|ESP, even ``archived`` projects.
query: |+
    SELECT name from resource where cls=540
parameters:

If the file was placed in /opt/l7esp/data/content/queries/project_name_list.yaml, it would then be possible to call /api/v2/queries/project_name_list. The API call returns a JSON object with various keys, including name (the value of name in the yaml file) and one of either results or error depending on if the query ran successfully. It not, error will contain the error message. If successful, results will contain the results as a list of JSON objects. For instance, /api/v2/queries/project_name_list from above might return:

{
    "name": "Project Name List",
    "results": [
        {"name": "project 1"},
        {"name": "project 2"}
    ]
}

Custom Endpoints

L7|ESP allows users to contribute new backend endpoints to the application without modifying the core server. These endpoints are called Invokables and are contributed using the lab7.extensions.invokable decorator.

Decorator arguments:

Name - Name of the Invokable. This will appear in the API as /api/invoke/<name>. If name is not supplied, the callable will be examined in turn for a “name” or “__name__” attribute and use the attribute value if found. If no attribute is found, a value error will be raised.
Session - The value of session is the name of the callable argument that should receive the SQLAlchemy session. If no session is required, the argument may be omitted.
Files - The value of files is the name of the callable argument that should receive the uploaded files data structure. (For details on the value, see https://www.tornadoweb.org/en/stable/guide/structure.html.

The decorated Invokable objects must be callable. The first argument of an Invokable MUST be “agent”, which receives the authenticated user making the request. Note that Invokable endpoints may only be requested by authenticated users. Aside from session and files argument (see arguments above and examples below), all other arguments are specific to the Invokable and supplied by requesting agents by posting {"kwargs": {"arg1": "value1", ...}}. The return value of the callable must be JSON-serializable, the request should be made with the HTTP header Content-Type: application/json and requestors should expect an returned content type of application/json.

Note

In a future release of L7|ESP, Invokables will be reworked so the kwargs key may be omitted so JSON object keys are mapped directly to function arguments. i.e. {"kwargs": {"arg1": "value1", ..}} would simplify to just {"arg1": "value1", ...}. Support for kwargs will be maintained, however, for backwards compatibility.

If you need an Invokable for non-JSON returns, contact your L7 field application scientist.

Custom invokables (endpoints) available through the application can also be created in the extensions/server/invokables.py file. Here’s an example of creating a custom endpoint to return an ‘ok’ status on GET request:

# contents of extensions/server/expressions.py

# endpoint definitions
class Ping(object):
    """
    Ping server and return OK response.
    """

    def __call__(self, agent, *args, **kwargs):
        return {'status': 'ok'}

# export (boilerplate)
INVOKABLES = {
    'ping': Ping,
}

Once this custom invokable has been defined, you can access the /api/invoke/ping URL via authenticated request:

>>> from esp import base
>>> base.SESSION.get('/api/invoke/ping')
{'status': 'ok'}

More Examples:

from lab7.extensions import invokable
@invokable
def custom_api(agent, a, b):
    return {'result': int(a)+int(b)}

The above function can now be “invoked” by issuing a request like POST /api/invoke/custom_api

With a content body of {"kwargs": {"a": 1, "b": 2}}

And will return an application/json response of {'result': 3}

Supply the name argument to the Invokable decorator to override the name:

@invokable(name='add2')
def custom_api(agent, a, b):
    return {'result': int(a)+int(b)}

Now the API call is to POST /api/invoke/add2

The first argument to Invokable must be agent. The Invokable machinery supplies the agent as the resolved User object corresponding to the authenticated user who issued the request.

Supply the session and files argument to the decorator to signal that your Invokable should receive the request session and uploaded files, respectively.

@invokable(session='sessionarg', files='filesarg')
def myapi(agent, a, b, sessionarg, filesarg):
    pass

Now the multipart POST API call to /api/invoke/myapi will pass the arguments a and b from the kwargs, the agent as the standard first argument, the request session to the sessionarg argument, and the request files (Tornado’s data structure) to the filesarg argument of myapi.

Custom Transition Strategies

Users may contribute their own Transition Strategies to L7|ESP. Transition strategies are used when routing samples from one workflow to the the next. A custom Transition Strategy might be used to send a notification when samples route to a particular workflow; or to adjust the routing so the sample going into the next workflow is a relative of the sample ending the previous workflow (for instance, routing a tissue sample for processing when a fluid sample fails).

Transition strategies are contributed using the lab7.extensions.transition_strategy decorator.

Arguments:

Name - Name of the strategy as used in, e.g., Workflow Chain configurations. If not specified, the strategy object is examined for name and __name__ attributes in-turn. If found, the value is used, otherwise, an error is raised.
Virtual - if True, the transition is “virtual”. The default is False. Currently, the only different between virtual and non-virtual transitions in L7|ESP is how they are rendered in Workflow Chain views. Virtual transitions render as dashed orange curves. Non-virtual transitions are blue and, typically, straight. Non-virtual transitions are used to inform the initial graph layout; virtual transitions are rendered after the graph structure is defined.

Strategies must be callable and must accept the following two parameters:

sample_uuids - The list of sample_uuids being transitioned. The list only contains UUIDs that pass the transition rule.
context - an instance of WFCTransitionContext which simplifies passing various useful data to the Transition Strategies, such as the incoming workflow instance (from_experiment property), the outgoing workflow instance (to_experiment property), the active transition, the active SQLAlchemy session, etc.

Strategies must return a list of sample_uuids that will be put into the downstream to_experiment.

The following example Transition Strategy transitions the parents of the samples in the final workflow sample set to the next workflow. An example use-case: sequencing libraries are pooled into a “library pool”. The library pool is sequenced. After sequencing, the libraries need to be pushed to bioinformatics, but the sequencing workflow has the library pool. You could use submit_parents to move the pool parents, the libraries, into bioinformatics if the sequencing passes QC.

@transition_strategy
def submit_parents(sample_uuids, context):
    import lab7.sample.api as sapi
    parents = []
    found = set()
    for uuid in sample_uuids:
        deps = sapi.get_sample_dependencies(
            uuid, parents=True, children=False, uuids_only=True,
            agent=context.agent, session=context.session)
        for parent in deps['parents'][::-1]:
            if parent not in found:
                found.add(parent)
                parents.append(parent)
    return parents

Custom Protocol Actions

You can write your own protocol actions like other server side extensions. You will need to use the @protocol_action decorator, provide a name, and any params that it needs. For example:

@protocol_action(
    name="Sync Item Field",
    desc="Copies a LIMS value into an inventory item field."
    parameters=[
        ExtensionParameter(
            name="prot_field", type_="string", label="Protocol Source Field",
            help="The ID of the protocol field to copy from"
        ),
        ExtensionParameter(
            name="item_field", type_="string", label="Item Target Field",
            help="The ID of the protocol field to copy from"
        )
    ]
)
def sync_field_value_with_items(rows, context):
    """
    Copies a LIMS value into an inventory item field. The target field must be a field on the item entity (eg: initial_qty, lot_id, status, expiration_timestamp)
    The user will need to specify both the source (the protocol column name) and target item field.
    """
    from lab7.resource.models import ResourceActionType
    source_id = context.action["parameters"]["prot_field"]
    target_id = context.action["parameters"]["item_field"]

    msg_template = (
        "Updated item field `{target_id}` from `{{}}` to `{{}}` " +
        "via protocol action `{context.action_name}` " +
        "of protocol `{context.protocol.name}` " +
        "in sheet `{context.sheet.name}` ({context.sheet.uuid})"
    )

    actions = []

    for row in rows:
        try:
            old_value = getattr(row.sample, target_id)
            new_value = row.get_value(source_id)

            setattr(row.sample, target_id, new_value)
            actions.append((
                msg_template.format(old_value, new_value),
                ResourceActionType.property_change(target_id, old_value, new_value)
            ))
        except ValueError:
            pass

    return actions

In the above example, we define a function and apply the @protocol_action decorator such that the function will be called when a protocol executes the ‘Sync Item Field’ action.

The @protocol_action decorator accepts the following parameters:

name - The name of the action.
desc - A description of the action.
parameters - A list of ExtensionParameter objects.

Since our example copies a value from a protocol field to an item field, we pass in two ExtensionParameter objects - one for the protocol field name (source) and another for the item field name (target).

An ExtensionParameter object accepts the following parameters:

name - The name of the parameter.
type_ - The type of the parameter (one of: string, sample_type, sample_id).
label - The label for the parameter.
default - A default value for the parameter (optional).
required (bool) - Whether the parameter is required or not (default: True).
help - A help text for the parameter.

The function will receive a list of rows (rows) and a context object.

The context object contains the following properties:

action - The action object, containing a parameters dict where the keys are the parameter names and the values are the parameter values.
action_name - The name of the action.
action_def - The action definition object.
agent - The agent making the request.
condition - The condition object.
executed_actions - A list of executed actions.
final_protocol_state - The final protocol state.
final_row_state - The final row state.
initial_protocol_state - The initial protocol state.
initial_row_state - The initial row state.
newly_approved_rows - A list of newly approved rows.
newly_completed_rows - A list of newly completed rows.
newly_failed_rows - A list of newly failed rows.
newly_unapproved_rows - A list of newly unapproved rows.
newly_uncompleted_rows - A list of newly uncompleted rows.
newly_unfailed_rows - A list of newly unfailed rows.
protocol - The protocol object.
protocol_ended_active - The protocol ended active.
protocol_started_active - The protocol started active.
session - The current database session
sheet - The sheet object.
sheet_env - The sheet environment object.
tab_idx - The tab index.

API Callbacks

API Callbacks are a powerful mechanism for augmenting the behavior of any existing L7|ESP endpoint.

Note

API Callbacks provide a lot of power, but can also easily contribute performance problems into a running L7|ESP instance if care is not taken. Use with caution. At L7, API Callbacks are used in only a handful of implementations where customers asked for functionality and the following criteria were met:

There was no other mechanism to accommodate the request in a manner that would meet the user needs
The request was specific enough to one customer that adding it as a core feature would not benefit any other customers and would add additional complexity to the codebase and UI.

API Callbacks are contributing using the lab7.extensions.api_callback decorator

Callback arguments:

callback (callable function): The callable object to register. The call must accept four arguments.

agent (User): The resolved esp User object for the requesting user

request_details (RequestDetails): A RequestDetails object with information about the request including:

endpoint (string): The endpoint being called (e.g. /api/samples)
For GraphQL endpoints, the api_callback machinery automatically adds the operationName to the endpoint, so the endpoint will be /graphql/<operationName> For instance: /graphql/AddItemToContainer. Note also that GraphQL endpoints are always POSTed to, even for data retrieval.

when (string): Either "before" or "after"

method (string): The request method (POST, PUT, GET, or DELETE)

status (string|int): Request status. If when is “before”, it will be an empty string.
If when is “after” it is the integer status code of the request.

data (dict): All request parameters, including query and body parameters.

session (Session): Database session, which is optional and may be None

name (string): The name of the callback. If no name is provided, the Callable is checked in order for a __name__ or name property and, if found, this property is used. Otherwise, an error is raised.

endpoints (string|list[string]): An API endpoint this callback is valid for. The string or list of strings are treated as exact matches except that '*' is a valid wildcard.

methods (string|list[string]): The HTTP method the callback is valid for. The special string '*' (default) indicates the callback is valid for all methods. A list of methods may be supplied such as method=['PUT', 'POST'].

when (string|list[string]): When the callback is valid. May be 'before', 'after', the list ['before', 'after'], or '*' (shortcut for ['before', 'after'])

stop_on_failure (Boolean): If we should stop processing and rollback on a failure

Examples:

from lab7.extensions import api_callback

@api_callback
def global_callback(agent, session, request_details, data):
    with open('mylog', 'a') as log:
        log.write("Called {} ({})\n".format(request_details.endpoint, request_details.when))

@api_callback(name='specific_callback', endpoints='/api/samples', methods='POST', when='before'):
def specific_callback(agent, session, request_details, data):
    if 'name' not in data:
        raise ValueError('This ESP installation does not support auto-sample-name generation')

The first example above would append to a file mylog before and after every API call in L7|ESP.

The second example is only called POSTing to the /api/samples endpoint and is called just before the standard L7|ESP handler. In this example, the callback is used to add a customer-specific constraint that sample POST always supplies a name. This would have the effect that samples could only be added by API call since the UI for sample creation relies on samples having names.

Note that callback registration is a two-step process. The first step is to declare and register the callback for invokation, as above. The second step is to add the name of the callback to the system configuration. This allows extension modules to make callbacks available for use while still requiring explicit configuration to enable them for a given installation. This is accomplished by adding an api_callbacks section to the system configuration:

api_callbacks:
    before:
        - global_callback
        - specific_callback
    after:
        - global_callback

Note that it is an error to add a callback to a section that is not supported by the callback registration. For instance, adding specific_callback to the after block is an error since specific_callback declares when as “before”. This check ensures callbacks are used in the manner intended by callback authors.

Data Source Loaders

One of L7|ESP’s configuration extensions is Data Source Loaders. Data Source Loaders offer a mechanism for flexibly loading relatively static data into the server memory space. Examples include assay configurations, Illumina index ID to index sequencing mappings, and more. Although many uses of Data Source Loaders have been supplanted by the newer database-backed dynamic configuration, they are still useful for any data source that can not be mapped to JSON, or which would be undesirable to map to JSON. For instance, ontology files in *.obo format, which can be read by such programs as obonet into a networkx graph. Out of the box, Data Source Loaders support YAML, JSON, CSV, and TSV file formats, but users can contribute their own Data Source Loader to augment, or even replace the out-of-box functionality. For instance, if a YAML or JSON file should be parsed into a specific Python object instead of generic dict and list. Data Source Loader logic follows the “chain of command” pattern, wherein a list of Data Source Loaders is traversed and the first Data Source Loader that can properly parse the data is used. Built-in Data Source Loaders can therefore be overridden just by inserting a custom Data Source Loader ahead of the built-in in the chain. Data Source Loaders are contributed using the lab7.extensions.datasource_loader decorator.

datasource_loader arguments:

where (callable) - “where” in the chain of command to place this Data Source Loader. This is a callable that receives the Data Source Loader and positions it in the list. lab7.extensions.datasource_loaders provides the following out-of-box locators:
- beginning - Position the Data Source Loader as the first Data Source Loader in the list. If two or more Data Source Loaders all specify beginning, they will be inserted in the beginning in LIFO order.
- end - Position the Data Source Loader as the last Data Source Loader in the list. If two or more Data Source Loaders all specify end, they will be appended to the end in LILO order.
- after(prior) - Position the Data Source Loader after Data Source Loader given by prior where prior is the class of the other Data Source Loader.

The Data Source Loader is expected to be an object conforming to the DataSourceLoader protocol. The DataSourceLoader protocol consists of two methods:

can_load(datasource) - returns True if the Data Source Loader is capable of handling datasource, where datasource is the configuration for the data source, including the url to load and, optionally, the data type.
load(name, datasource) - returns the loaded object structure. Name is the name of the data source in the configuration.

Data source loading then works by searching the list for a Data Source Loader that can load the current data source and, when found, loading it. Custom Data Source Loaders are checked before out of the box Data Source Loaders, allowing for overriding default loading behavior for out-of-the-box formats.

Note that the url key of datasource is guaranteed to exist and to be an already stripped string value when calls to can_load or load are made.

Registering server-side extensions

Server side extensions are added to L7|ESP by means of Python entry points.

Users new to Python entry points should review the official Entry Points Specification.

For a less formal example of using entry points, see Python Entry Points Explained.

L7|ESP’s extension entry point is l7.esp.extensions. To add your extensions to L7|ESP, create a Python package to hold the extensions (but see below for SDK users), make sure all modules with extensions are loaded when the package is loaded, and add the appropriate entry_point configuration to the setup.py file, For example, if the extensions are in an acme Python package with a structure:

acme
├── acme
│   ├── __init__.py
│   ├── api_callbacks.py
│   ├── dataloaders.py
│   ├── expressions.py
│   ├── invokables.py
│   └── transition_strategies.py
└── setup.py

Then the __init__.py file would have:

import acme.api_callbacks
import acme.dataloaders
import acme.expressions
import acme.invokables
import acme.transition_strategies

And the setup.py file entry point configuration would look like:

setup(
    name='acme',
    ...,
    entry_points={
        'l7.esp.extensions': [
            'acme=acme'
    })

Users taking advantage of the standard SDK layout and tooling for developing their L7|ESP instance can place extensions in the extensions/server directory. Files in this directory are auto-packaged into the l7ext python package at build time. Within the extensions/server directory, the auto-packaging ensures expressions.py, invokables.py, transition_strategies.py, api_callbacks.py, and dataloaders.py are auto-loaded when the l7ext package is loaded, so users can add to those files without worrying about anything else.

User-interface extensions

The user interface supports the following extensions: invokables.js, renderers.js, and fas-overrides.js, in descending order of usage likelihood.

`invokables.js`

invokables.js should be placed in Lab7_ESP/current/lib/www/static/js. It is loaded in LIMS, Reports, and Applet apps. Therefore, any functions, objects, or variables placed in this file are available to protocol event handlers (onrender, onchange) as well as to reports and applets. Other than loading symbols from this file, L7|ESP does not do anything special with this file.

`renderers.js`

renderers.js should be placed in Lab7_ESP/current/lib/www/static/js. It is loaded by the LIMS app. Unlike invokables.js, L7|ESP will examine renderers.js for specific symbols and process them if found.

`manipulateGridOptions`

manipulateGridOptions allows users to customize the behavior of LIMS grids. It is called with the argument gridOptions and should return a grid options object (usually by modifying the passed-in object, then returning it). The list of options is extensive, but the most common use is to register additional column rendering and/or editing components. For instance:

manipulateGridOptions: function(gridOpts) {
  // Any custom renderers MUST be added to gridOpts.components in order for transpose to work.
  // Other custom logic can be added, but default is no-op.
  var renderers = this.getRenderers()
  gridOpts.components = {
    ...gridOpts.components,
    ...renderers
  }
  return gridOpts
},

`manipulateColumnDefs`

manipulateColumnDefs receives a columnDefs argument and is expected to return the same. columnDefs is a list of column definition dictionaries. Using this function, users can alter the behavior of specific columns in a worksheet. For example - overriding the component used to render or edit a particular column value. For instance:

manipulateColumnDefs: function(columnDefs) {
  var renderers = this.getRenderers()
  console.log(columnDefs)
  columnDefs.forEach(x => {
    if (!x.meta) {
      return
    }
    let meta = null
    try {
      meta = JSON.parse(x.meta)
    } catch(err) {
      meta = x.meta
    }
    if (meta && meta.augment && meta.augment.renderer && renderers[meta.augment.renderer]) {
      x.cellRendererSelector = function() {
        return {component: meta.augment.renderer}
      }
      x.transposeRendererName = meta.augment.renderer
    }
  })
  return columnDefs
}

The above block examines the meta property of each column definition to see if the path meta.argument.renderer matches a known custom renderers. If so, it uses the custom renderer in place of the standard renderer for that column.

Note

The functions in renderers.js are called for protocol for every workflow in the system. Operations in this file should therefore be lightweight to avoid performance penalties.

`fas-overrides.js`

fas-overrides.js is a file loaded in global scope on every page. As with api_callback on the server-side, this file should only be used in circumstances where other techniques fail to adequately address user-needs and the functionality required is so specific to a single use-case that others would not benefit from the functionality as a more general feature.

Extensions

Server Side Extensions

Custom Expression Functions

Custom Queries

Custom Endpoints

Custom Transition Strategies

Custom Protocol Actions

API Callbacks

Data Source Loaders

Registering server-side extensions

User-interface extensions

invokables.js

renderers.js

manipulateGridOptions

manipulateColumnDefs

fas-overrides.js

`invokables.js`

`renderers.js`

`manipulateGridOptions`

`manipulateColumnDefs`

`fas-overrides.js`