System Tuning

As L7|ESP is cross-compiled and runs on many operating systems; we don’t currently have any specific requirements for tuning at the operating system level (apart from hardware sizing recommendations which are in another document). Depending on the operating system, it’s possible that some of the default values may not be suitable for a server environment. One such example of this pertains to the default per-user/application ulimit for open file descriptors, which can often be set to a very low value such as 1024 while a more suitable value for any server environment may be much higher (e.g. 24000) depending on the amount of traffic the server is handling. Our recommendation is to increase these limits if you find you are bumping against them, however the exact values chosen may vary widely depending on scale.

L7 recommends adjusting the following settings in a production L7|ESP installation, contingent on the server running the DB having adequate resources to do so.

services:
  server:
    image: l7esp/server
    shm_size: 1gb

L7|ESP application servers don’t store files in the database, instead writing these to disk and storing a reference to the file path in the database. This is the case for any pipeline scripts that are generated and executed, the stdout/stderr logs we persist for the runs of such pipelines, and files that are generated by these pipelines and/or user-uploaded to the system via the Data application or related API endpoints.

L7|ESP writes all such state to a directory in the server root named “data”. Depending on the installation, this may also contain logs generated by the running L7|ESP services and the PostgreSQL data directory if using the included local database. This works well for Dockerized use-cases where you may wish to make the filesystem read-only and capture all persistent state in a volume, though usually logging would be reconfigured to redirect to stdout in this case.

This can also be leveraged to create highly available architectures, where two or more L7|ESP application servers are placed behind a load balancer and are all accepting traffic, or you wish to have a hot standby that isn’t accepting traffic in the event that the primary application server experiences a failure. In such mode, you will want to mount this directory to a network storage location (such as AWS EFS) and careful consideration should be given to the supervisord configuration, particularly to where log files and PID files are written and, if written to a shared storage, if the names of such files will collide and/or cause multiple processes to write to them.

Your options here vary depending on whether you wish these services to log to disk, to syslog, to stdout (e.g. Docker) and, if logging to disk, if you wish to log to the shared storage or not (e.g. for backup purposes). The simplest recommendation in a HA setup is to adjust the LAB7LOGDIR and LAB7VARRUN environment variables to some directory outside of the Data directory, though it’s also possible to tailor logging in supervisord.conf to your exact requirements, such as including the hostname environment variable as part of the log filenames using the special %(ENV_HOSTNAME)s syntax.

In the event that the data volume should be mounted to a shared location but sharding occurs (e.g. the mount doesn’t appear on one or more L7|ESP application servers for some reason and they begin writing these files to local disk), recovery should be as simple as merging the directories and files in these folders back together as all automatically generated file and directory names have either the database UUID they reference or a timestamp in their path or filename. Files that are not dynamically generated, such as custom Python scripts that pipelines may execute should always be the same on each application server as these are usually placed there at install time.

The L7|ESP application doesn’t need any permissions outside of its own PostgreSQL database so it is recommended to create a single database schema for the application, as well as a dedicated role with matching name, and grant all permissions for that role to that schema.

Currently, we don’t separate “read-only”, “read/write”, “read/write/DDL modification permissions” users at the application level. The same user is used in l7 init (DDL migrations) as in normal application execution. The credentials stored in the database config file $LAB7DATA/conf/database.json are used in both instances.

However, it is possible to switch these credentials with a more privileged account, run l7 init to perform any database migrations and then switch back to the less privileged account credentials for day-to-day application execution. Note that database migrations will only occur if you are upgrading the underlying L7|ESP version (e.g. L7|ESP 2.3.3 to L7|ESP 2.4) but won’t take place during normal upgrades that only seed new content.

The database configuration file at $LAB7DATA/conf/database.json should take the following format when using an external hosted PostgreSQL solution such as AWS RDS:

{
    "host": "postgres",
    "port": 5432,
    "user": "l7esp",
    "pass": "password",
    "name": "l7esp",
    "start_service": false
}

Regarding encryption in transit, the PostgreSQL driver will automatically negotiate an appropriate connection based on the SSL mode defined on the database server.

L7|ESP should be installed in Linux user-space and by default will only listen on user/registered ports, rather than system/well-known ports below 1024 that require root privileges.

The installation requires some basic tools, such as make, wget, bzip2, and rsync. This is enough for the installer to use Miniconda to bootstrap a Python environment where Ansible will be installed and automate the complete L7|ESP installation.

As root, you should install the basic requirements, and create a non-privileged Linux user/group for the application, and switch to that user:

root@l7espapp:~# yum install make wget bzip2 rsync
root@l7espapp:~# groupadd l7esp
root@l7espapp:~# useradd --groups l7esp l7esp
root@l7espapp:~# su - l7esp

As the L7|ESP user, you may fetch the deployment bundle, extract it and perform the installation:

l7esp@l7espapp:~$ wget "${ESP_DEPLOYMENT_BUNDLE_URL}"
l7esp@l7espapp:~$ tar xf "${ESP_DEPLOYMENT_BUNDLE_FILENAME}"
l7esp@l7espapp:~$ cd "${CUSTOMER_NAME}"
l7esp@l7espapp:~/${CUSTOMER_NAME}$ make install

To aide in system administration tasks, you may additionally wish to add the following lines to this Linux user’s Bash profile so that common utilities will be made available via $PATH the path environment variable, as well as some other useful environment variables:

l7esp@l7espapp:~$ cat ~/.profile
source /data/ESP/Lab7_ESP/current/bin/env.sh
PATH="/data/ESP/client/bin:$PATH"

l7esp@l7espapp:~$ which l7
/data/ESP/Lab7_ESP/current/bin/l7

l7esp@l7espapp:~$ which esp
/data/ESP/client/bin/esp

l7esp@l7espapp:~$ env \| grep LAB7
LAB7LOGDIR=/data/ESP/Lab7_ESP/Data/log

l7esp@l7espapp:~$ tail -n 0 -f $LAB7LOGDIR/\*
==> /data/ESP/Lab7_ESP/Data/log/l7-esp.http.0.access.log <==

If for some reason, you need to migrate an existing L7|ESP instance to a different Linux account on the same machine (or any other machine for that matter), the steps would be as follows:

The deployment bundle has the ability to install L7|ESP as a systemd service, however this will require the application user to be able to escalate to sudo privileges and that the Ansible variable service: True is set in the roles/container.yml file in your L7|ESP deployment bundle. You may also find the template used to create the service in the file roles/esp/templates/service and the related Ansible tasks can be found in the roles/esp/tasks/run.yml file. If you wish to make changes to any of this, please let us know so that we may include and test your desired defaults in future L7|ESP deployment bundle revision you receive from us.

Simply put, all previous versions of workflows, protocols, and entities (samples) are stored in the database. In terms of old/new data values for a given field, L7|ESP captures the full history of changes in the resource_action table.

The resource_action database table provides an audit log for each resource in the system, which can typically be viewed under the History tab in the UI for a given resource.

When it comes to configuration items, such as workflow and protocol versions: a given version of a protocol (e.g.) is immutable. Each time you save a protocol or workflow, you are actually creating a new version of the protocol under the hood. The old version remains unmodified.

Currently, experiments use the most recent version of a workflow, in addition to the most recent version of the protocols nested in this workflow. Once submitted, the samples included in the experiment are “locked” into these versions - subsequent changes to the workflow or protocol definitions will not impact any “in-flight” samples.