Extensions
The extensions
folder in the L7|ESP SDK allows developers to create custom expressions and endpoints in the application. Here’s an example of the structure of that directory:
└── extensions/ ├── server/ │ ├── expressions.py │ ├── invokables.py │ └── requirements.txt └── client/ └── expressions.js
L7|ESP offers a number of extension mechanisms for cases when the out-of-the-box functionality is insufficient to meet a particular need, both for server-side and client-side functionality. On the client-side, the invokables.js
, renderers.js
, and fas-overrides.js
files provide entry to augment and customize the user experience. On the server-side are custom Expressions, Invokables, Transition Strategies, Data Souce Loaders, and API Callbacks.
Server Side Extensions
L7|ESP provides a number of server-side extension points detailed below, in order from “most-used” to “least-used”.
Custom Expression Functions
Custom server-side expressions available in L7|ESP Worksheets can be created in the extensions/server/expressions.py
file.
Users may contribute new Expression functions to L7|ESP. Expressions must be callable or subclasses of lab7.expression.APICall
. They are contributed using the lab7.extensions.expression
decorator.
@expression
Arguments:
name
- the name of the Expression. If not specified, the callable object is examined forname
or__name__
attributes, in order. If found, the value is used for the name, otherwise an error is raised.context
- the Expression evaluation context to add to. Currently, onlyLab7ExpressionContext.ALL
is supported.
Expressions may accept any arguments and return any value, although in most contexts, the return value will ultimately be coerced to a string. For example, contributing a quantile
function might look like:
from esp.extensions import expression @expression def quantile(array, quant): import numpy as np return np.percentile([float(x) for x in array], float(quant*100))
After registering the extension with L7|ESP (see “Registering server-side-extensions”), the quantile
function will be available in all contexts.
Here are some additional examples of using the @expression
decorator:
@expression def myfunc(a, a): return int(a) + int(b)
Now myfunc
is accessible from, e.g., LIMS default values:
{{ myfunc(1, 2) }}
To override the name by which the Expression is registered, supply the name argument to the decorator.
@expression(name='add2') def myfunc(a, a): return int(a) + int(b)
Now the expression is:
{{ add2(1, 2) }}
By default, Expressions are set to available in every evaluation context. To limit to a specific context, supply the context argument.
from lab7.extensions.expressions import Lab7ExpressionContext @expression(name='add2', context=Lab7ExpressionContext.ALL) def myfunc(a, a): return int(a) + int(b)
The Expression is available in every context. If no context is provided, the context is set to Lab7ExpressionContext.ALL
.
Note
Currently, the only context used in L7|ESP is Lab7ExpressionContext.ALL
, but support for context-specific functions will be added in a future release of L7|ESP.
In advanced Expression scenarios, users may need access to the database. In these cases, Expressions should be objects that extend from lab7.expressions.APICall
. Objects that extend from APICall have access to several useful attributes, including
self.ctx
- the Expression evaluation context. See “Using Expressions in L7|ESP” for information on what is available in various Expression evaluation contexts.self.session
- the SQL Alchemy database sessionself.agent
- The active L7|ESP userself.cache
- A dictionary shared across Expressions and Expression evaluations for a single evaluation event. For instance, when evaluating a LIMS worksheet, the cache dictionary is created at the beginning of evaluation and retained until the entire worksheet is finished. This allows Expression authors to lookup a value 1x/in-bulk, and cache the results. For database-backed Expressions, it is highly encouraged to use the cache to help ensure sample sheet saving stays fast. (In one example at L7, a sample sheet save with 100 samples went from 30+ seconds down to 3 seconds by adding caching to one Expression).
The following example creates a “service_types” Expression function that looks up the list of service types from the DB, but only if the list has not already been fetched before in this evaluation context.
@expression(name="service_types") class ServiceTypes(APICall): """Return a list of service types Args: names_only (bool): If true (the default), the returned list is only the service type names. If false, the returned list is the full service type dictionary. """ def api_func(self, names_only=True): key = '__service_type_list__{}'.format(names_only) if key not in self.cache: service_types = query_service_types(params=['name'], agent=self.agent, session=self.session) if names_only: self.cache[key] = [x['name'] for x in service_types] else: self.cache[key] = service_types return self.cache[key]
Custom Queries
Custom queries can be contributed to the backend by defining the query in a yaml file and placing the yaml file in /opt/l7esp/data/content/queries
. Queries added this way are available through API call from /api/v2/queries/{query_file_name}
. The YAML file MUST have the suffix .yaml
and the filename must not have any spaces. The structure of the YAML file is as follows:
name: {query name, spaces ok} description: |+ QUERY DESCRIPTION query: SQL parameters: param1: description: param1 description type: list # list of values, such as what might be used in an IN clause. singleton_param: description: param 2 description. Param 2 is a single value.
For instance, a simple query to fetch project names might be:
name: Project Name List description: |+ Fetch the name for all projects in L7|ESP, even ``archived`` projects. query: |+ SELECT name from resource where cls=540 parameters:
If the file was placed in /opt/l7esp/data/content/queries/project_name_list.yaml
, it would then be possible to call /api/v2/queries/project_name_list
. The API call returns a JSON object with various keys, including name
(the value of name
in the yaml file) and one of either results
or error
depending on if the query ran successfully. It not, error
will contain the error message. If successful, results
will contain the results as a list of JSON objects. For instance, /api/v2/queries/project_name_list
from above might return:
{ "name": "Project Name List", "results": [ {"name": "project 1"}, {"name": "project 2"} ] }
Custom Endpoints
L7|ESP allows users to contribute new backend endpoints to the application without modifying the core server. These endpoints are called Invokables and are contributed using the lab7.extensions.invokable
decorator.
Decorator arguments:
Name - Name of the Invokable. This will appear in the API as
/api/invoke/<name>
. If name is not supplied, the callable will be examined in turn for a “name” or “__name__” attribute and use the attribute value if found. If no attribute is found, a value error will be raised.Session - The value of session is the name of the callable argument that should receive the SQLAlchemy session. If no session is required, the argument may be omitted.
Files - The value of files is the name of the callable argument that should receive the uploaded files data structure. (For details on the value, see https://www.tornadoweb.org/en/stable/guide/structure.html.
The decorated Invokable objects must be callable. The first argument of an Invokable MUST be “agent”, which receives the authenticated user making the request. Note that Invokable endpoints may only be requested by authenticated users. Aside from session and files argument (see arguments above and examples below), all other arguments are specific to the Invokable and supplied by requesting agents by posting {"kwargs": {"arg1": "value1", ...}}
. The return value of the callable must be JSON-serializable, the request should be made with the HTTP header Content-Type: application/json
and requestors should expect an returned content type of application/json
.
Note
In a future release of L7|ESP, Invokables will be reworked so the kwargs
key may be omitted so JSON object keys are mapped directly to function arguments. i.e. {"kwargs": {"arg1": "value1", ..}}
would simplify to just {"arg1": "value1", ...}
. Support for kwargs
will be maintained, however, for backwards compatibility.
If you need an Invokable for non-JSON
returns, contact your L7 field application scientist.
Custom invokables (endpoints) available through the application can also be created in the extensions/server/invokables.py
file. Here’s an example of creating a custom endpoint to return an ‘ok’ status on GET request:
# contents of extensions/server/expressions.py # endpoint definitions class Ping(object): """ Ping server and return OK response. """ def __call__(self, agent, *args, **kwargs): return {'status': 'ok'} # export (boilerplate) INVOKABLES = { 'ping': Ping, }
Once this custom invokable has been defined, you can access the /api/invoke/ping
URL via authenticated request:
>>> from esp import base >>> base.SESSION.get('/api/invoke/ping') {'status': 'ok'}
More Examples:
from lab7.extensions import invokable @invokable def custom_api(agent, a, b): return {'result': int(a)+int(b)}
The above function can now be “invoked” by issuing a request like POST /api/invoke/custom_api
With a content body of {"kwargs": {"a": 1, "b": 2}}
And will return an application/json
response of {'result': 3}
Supply the name
argument to the Invokable decorator to override the name:
@invokable(name='add2') def custom_api(agent, a, b): return {'result': int(a)+int(b)}
Now the API call is to POST /api/invoke/add2
The first argument to Invokable must be agent
. The Invokable machinery supplies the agent
as the resolved User object corresponding to the authenticated user who issued the request.
Supply the session
and files
argument to the decorator to signal that your Invokable should receive the request session and uploaded files, respectively.
@invokable(session='sessionarg', files='filesarg') def myapi(agent, a, b, sessionarg, filesarg): pass
Now the multipart POST
API call to /api/invoke/myapi
will pass the arguments a
and b
from the kwargs
, the agent
as the standard first argument, the request session to the sessionarg
argument, and the request files (Tornado’s data structure) to the filesarg
argument of myapi
.
Custom Transition Strategies
Users may contribute their own Transition Strategies to L7|ESP. Transition strategies are used when routing samples from one workflow to the the next. A custom Transition Strategy might be used to send a notification when samples route to a particular workflow; or to adjust the routing so the sample going into the next workflow is a relative of the sample ending the previous workflow (for instance, routing a tissue sample for processing when a fluid sample fails).
Transition strategies are contributed using the lab7.extensions.transition_strategy
decorator.
Arguments:
Name - Name of the strategy as used in, e.g., Workflow Chain configurations. If not specified, the strategy object is examined for
name
and__name__
attributes in-turn. If found, the value is used, otherwise, an error is raised.Virtual - if
True
, the transition is “virtual”. The default isFalse
. Currently, the only different between virtual and non-virtual transitions in L7|ESP is how they are rendered in Workflow Chain views. Virtual transitions render as dashed orange curves. Non-virtual transitions are blue and, typically, straight. Non-virtual transitions are used to inform the initial graph layout; virtual transitions are rendered after the graph structure is defined.
Strategies must be callable
and must accept the following two parameters:
sample_uuids
- The list ofsample_uuids
being transitioned. The list only contains UUIDs that pass the transition rule.context
- an instance ofWFCTransitionContext
which simplifies passing various useful data to the Transition Strategies, such as the incoming workflow instance (from_experiment
property), the outgoing workflow instance (to_experiment
property), the active transition, the active SQLAlchemy session, etc.
Strategies must return a list of sample_uuids
that will be put into the downstream to_experiment
.
The following example Transition Strategy transitions the parents of the samples in the final workflow sample set to the next workflow. An example use-case: sequencing libraries are pooled into a “library pool”. The library pool is sequenced. After sequencing, the libraries need to be pushed to bioinformatics, but the sequencing workflow has the library pool. You could use submit_parents
to move the pool parents, the libraries, into bioinformatics if the sequencing passes QC.
@transition_strategy def submit_parents(sample_uuids, context): import lab7.sample.api as sapi parents = [] found = set() for uuid in sample_uuids: deps = sapi.get_sample_dependencies( uuid, parents=True, children=False, uuids_only=True, agent=context.agent, session=context.session) for parent in deps['parents'][::-1]: if parent not in found: found.add(parent) parents.append(parent) return parents
Custom Protocol Actions
You can write your own protocol actions like other server side extensions. You will need to use the @protocol_action
decorator, provide a name, and any params that it needs. For example:
@protocol_action( name="Sync Item Field", desc="Copies a LIMS value into an inventory item field." parameters=[ ExtensionParameter( name="prot_field", type_="string", label="Protocol Source Field", help="The ID of the protocol field to copy from" ), ExtensionParameter( name="item_field", type_="string", label="Item Target Field", help="The ID of the protocol field to copy from" ) ] ) def sync_field_value_with_items(rows, context): """ Copies a LIMS value into an inventory item field. The target field must be a field on the item entity (eg: initial_qty, lot_id, status, expiration_timestamp) The user will need to specify both the source (the protocol column name) and target item field. """ from lab7.resource.models import ResourceActionType source_id = context.action["parameters"]["prot_field"] target_id = context.action["parameters"]["item_field"] msg_template = ( "Updated item field `{target_id}` from `{{}}` to `{{}}` " + "via protocol action `{context.action_name}` " + "of protocol `{context.protocol.name}` " + "in sheet `{context.sheet.name}` ({context.sheet.uuid})" ) actions = [] for row in rows: try: old_value = getattr(row.sample, target_id) new_value = row.get_value(source_id) setattr(row.sample, target_id, new_value) actions.append(( msg_template.format(old_value, new_value), ResourceActionType.property_change(target_id, old_value, new_value) )) except ValueError: pass return actions
In the above example, we define a function and apply the @protocol_action
decorator such that the function will be called when a protocol executes the ‘Sync Item Field’ action.
The @protocol_action
decorator accepts the following parameters:
name
- The name of the action.desc
- A description of the action.parameters
- A list of ExtensionParameter objects.
Since our example copies a value from a protocol field to an item field, we pass in two ExtensionParameter
objects - one for the protocol field name (source) and another for the item field name (target).
An ExtensionParameter
object accepts the following parameters:
name
- The name of the parameter.type_
- The type of the parameter (one of:string
,sample_type
,sample_id
).label
- The label for the parameter.default
- A default value for the parameter (optional).required
(bool
) - Whether the parameter is required or not (default:True
).help
- A help text for the parameter.
The function will receive a list of rows (rows
) and a context
object.
The context
object contains the following properties:
action
- The action object, containing aparameters
dict where the keys are the parameter names and the values are the parameter values.action_name
- The name of the action.action_def
- The action definition object.agent
- The agent making the request.condition
- The condition object.executed_actions
- A list of executed actions.final_protocol_state
- The final protocol state.final_row_state
- The final row state.initial_protocol_state
- The initial protocol state.initial_row_state
- The initial row state.newly_approved_rows
- A list of newly approved rows.newly_completed_rows
- A list of newly completed rows.newly_failed_rows
- A list of newly failed rows.newly_unapproved_rows
- A list of newly unapproved rows.newly_uncompleted_rows
- A list of newly uncompleted rows.newly_unfailed_rows
- A list of newly unfailed rows.protocol
- The protocol object.protocol_ended_active
- The protocol ended active.protocol_started_active
- The protocol started active.session
- The current database sessionsheet
- The sheet object.sheet_env
- The sheet environment object.tab_idx
- The tab index.
API Callbacks
API Callbacks are a powerful mechanism for augmenting the behavior of any existing L7|ESP endpoint.
Note
API Callbacks provide a lot of power, but can also easily contribute performance problems into a running L7|ESP instance if care is not taken. Use with caution. At L7, API Callbacks are used in only a handful of implementations where customers asked for functionality and the following criteria were met:
There was no other mechanism to accommodate the request in a manner that would meet the user needs
The request was specific enough to one customer that adding it as a core feature would not benefit any other customers and would add additional complexity to the codebase and UI.
API Callbacks are contributing using the lab7.extensions.api_callback
decorator
Callback arguments:
-
callback
(callable
function): The callable object to register. The call must accept four arguments. agent (User): The resolved esp User object for the requesting userrequest_details (RequestDetails): A RequestDetails object with information about the request including:endpoint (string): The endpoint being called (e.g. /api/samples)For GraphQL endpoints, the api_callback machinery automatically adds the operationName to the endpoint, so the endpoint will be /graphql/<operationName> For instance: /graphql/AddItemToContainer. Note also that GraphQL endpoints are always POSTed to, even for data retrieval.when (string): Either "before" or "after"method (string): The request method (POST, PUT, GET, or DELETE)status (string|int): Request status. If when is “before”, it will be an empty string.If when is “after” it is the integer status code of the request.data (dict): All request parameters, including query and body parameters.session (Session): Database session, which is optional and may be None
-
name
(string
): The name of the callback. If no name is provided, the Callable is checked in order for a__name__
orname
property and, if found, this property is used. Otherwise, an error is raised.endpoints
(string|list[string]
): An API endpoint this callback is valid for. The string or list of strings are treated as exact matches except that'*'
is a valid wildcard.methods
(string|list[string]
): The HTTP method the callback is valid for. The special string'*'
(default) indicates the callback is valid for all methods. A list of methods may be supplied such asmethod=['PUT', 'POST']
.when
(string|list[string]
): When the callback is valid. May be'before'
,'after'
, the list['before', 'after']
, or'*'
(shortcut for['before', 'after']
)stop_on_failure
(Boolean): If we should stop processing and rollback on a failure
Examples:
from lab7.extensions import api_callback @api_callback def global_callback(agent, session, request_details, data): with open('mylog', 'a') as log: log.write("Called {} ({})\n".format(request_details.endpoint, request_details.when)) @api_callback(name='specific_callback', endpoints='/api/samples', methods='POST', when='before'): def specific_callback(agent, session, request_details, data): if 'name' not in data: raise ValueError('This ESP installation does not support auto-sample-name generation')
The first example above would append to a file mylog
before and after every API call in L7|ESP.
The second example is only called POSTing to the /api/samples
endpoint and is called just before the standard L7|ESP handler. In this example, the callback is used to add a customer-specific constraint that sample POST always supplies a name. This would have the effect that samples could only be added by API call since the UI for sample creation relies on samples having names.
Note that callback registration is a two-step process. The first step is to declare and register the callback for invokation, as above. The second step is to add the name of the callback to the system configuration. This allows extension modules to make callbacks available for use while still requiring explicit configuration to enable them for a given installation. This is accomplished by adding an api_callbacks
section to the system configuration:
api_callbacks: before: - global_callback - specific_callback after: - global_callback
Note that it is an error to add a callback to a section that is not supported by the callback registration. For instance, adding specific_callback
to the after
block is an error since specific_callback
declares when
as “before”. This check ensures callbacks are used in the manner intended by callback authors.
Data Source Loaders
One of L7|ESP’s configuration extensions is Data Source Loaders. Data Source Loaders offer a mechanism for flexibly loading relatively static data into the server memory space. Examples include assay configurations, Illumina index ID to index sequencing mappings, and more. Although many uses of Data Source Loaders have been supplanted by the newer database-backed dynamic configuration, they are still useful for any data source that can not be mapped to JSON, or which would be undesirable to map to JSON. For instance, ontology files in *.obo
format, which can be read by such programs as obonet
into a networkx
graph. Out of the box, Data Source Loaders support YAML, JSON, CSV, and TSV file formats, but users can contribute their own Data Source Loader to augment, or even replace the out-of-box functionality. For instance, if a YAML or JSON file should be parsed into a specific Python object instead of generic dict
and list
. Data Source Loader logic follows the “chain of command” pattern, wherein a list of Data Source Loaders is traversed and the first Data Source Loader that can properly parse the data is used. Built-in Data Source Loaders can therefore be overridden just by inserting a custom Data Source Loader ahead of the built-in in the chain. Data Source Loaders are contributed using the lab7.extensions.datasource_loader
decorator.
datasource_loader
arguments:
where
(callable
) - “where” in the chain of command to place this Data Source Loader. This is a callable that receives the Data Source Loader and positions it in the list.lab7.extensions.datasource_loaders
provides the following out-of-box locators:beginning
- Position the Data Source Loader as the first Data Source Loader in the list. If two or more Data Source Loaders all specify beginning, they will be inserted in the beginning in LIFO order.end
- Position the Data Source Loader as the last Data Source Loader in the list. If two or more Data Source Loaders all specify end, they will be appended to the end in LILO order.after(prior)
- Position the Data Source Loader after Data Source Loader given byprior
whereprior
is the class of the other Data Source Loader.
The Data Source Loader is expected to be an object conforming to the DataSourceLoader
protocol. The DataSourceLoader
protocol consists of two methods:
can_load(datasource)
- returnsTrue
if the Data Source Loader is capable of handlingdatasource
, wheredatasource
is the configuration for the data source, including theurl
to load and, optionally, the data type.load(name, datasource)
- returns the loaded object structure. Name is thename
of the data source in the configuration.
Data source loading then works by searching the list for a Data Source Loader that can load the current data source and, when found, loading it. Custom Data Source Loaders are checked before out of the box Data Source Loaders, allowing for overriding default loading behavior for out-of-the-box formats.
Note that the url
key of datasource
is guaranteed to exist and to be an already stripped string value when calls to can_load
or load
are made.
Registering server-side extensions
Server side extensions are added to L7|ESP by means of Python entry points.
Users new to Python entry points should review the official Entry Points Specification.
For a less formal example of using entry points, see Python Entry Points Explained.
L7|ESP’s extension entry point is l7.esp.extensions
. To add your extensions to L7|ESP, create a Python package to hold the extensions (but see below for SDK users), make sure all modules with extensions are loaded when the package is loaded, and add the appropriate entry_point
configuration to the setup.py
file, For example, if the extensions are in an acme
Python package with a structure:
acme ├── acme │ ├── __init__.py │ ├── api_callbacks.py │ ├── dataloaders.py │ ├── expressions.py │ ├── invokables.py │ └── transition_strategies.py └── setup.py
Then the __init__.py
file would have:
import acme.api_callbacks import acme.dataloaders import acme.expressions import acme.invokables import acme.transition_strategies
And the setup.py
file entry point configuration would look like:
setup( name='acme', ..., entry_points={ 'l7.esp.extensions': [ 'acme=acme' })
Users taking advantage of the standard SDK layout and tooling for developing their L7|ESP instance can place extensions in the extensions/server
directory. Files in this directory are auto-packaged into the l7ext
python package at build time. Within the extensions/server
directory, the auto-packaging ensures expressions.py
, invokables.py
, transition_strategies.py
, api_callbacks.py
, and dataloaders.py
are auto-loaded when the l7ext
package is loaded, so users can add to those files without worrying about anything else.
User-interface extensions
The user interface supports the following extensions: invokables.js
, renderers.js
, and fas-overrides.js
, in descending order of usage likelihood.
invokables.js
invokables.js
should be placed in Lab7_ESP/current/lib/www/static/js
. It is loaded in LIMS, Reports, and Applet apps. Therefore, any functions, objects, or variables placed in this file are available to protocol event handlers (onrender
, onchange
) as well as to reports and applets. Other than loading symbols from this file, L7|ESP does not do anything special with this file.
renderers.js
renderers.js
should be placed in Lab7_ESP/current/lib/www/static/js
. It is loaded by the LIMS app. Unlike invokables.js
, L7|ESP will examine renderers.js
for specific symbols and process them if found.
manipulateGridOptions
manipulateGridOptions
allows users to customize the behavior of LIMS grids. It is called with the argument gridOptions
and should return a grid options object (usually by modifying the passed-in object, then returning it). The list of options is extensive, but the most common use is to register additional column rendering and/or editing components. For instance:
manipulateGridOptions: function(gridOpts) { // Any custom renderers MUST be added to gridOpts.components in order for transpose to work. // Other custom logic can be added, but default is no-op. var renderers = this.getRenderers() gridOpts.components = { ...gridOpts.components, ...renderers } return gridOpts },
manipulateColumnDefs
manipulateColumnDefs
receives a columnDefs
argument and is expected to return the same. columnDefs
is a list of column definition dictionaries. Using this function, users can alter the behavior of specific columns in a worksheet. For example - overriding the component used to render or edit a particular column value. For instance:
manipulateColumnDefs: function(columnDefs) { var renderers = this.getRenderers() console.log(columnDefs) columnDefs.forEach(x => { if (!x.meta) { return } let meta = null try { meta = JSON.parse(x.meta) } catch(err) { meta = x.meta } if (meta && meta.augment && meta.augment.renderer && renderers[meta.augment.renderer]) { x.cellRendererSelector = function() { return {component: meta.augment.renderer} } x.transposeRendererName = meta.augment.renderer } }) return columnDefs }
The above block examines the meta
property of each column definition to see if the path meta.argument.renderer
matches a known custom renderers. If so, it uses the custom renderer in place of the standard renderer for that column.
Note
The functions in renderers.js
are called for protocol for every workflow in the system. Operations in this file should therefore be lightweight to avoid performance penalties.
fas-overrides.js
fas-overrides.js
is a file loaded in global scope on every page. As with api_callback
on the server-side, this file should only be used in circumstances where other techniques fail to adequately address user-needs and the functionality required is so specific to a single use-case that others would not benefit from the functionality as a more general feature.