DMPACK User Guide

First, install the build and run-time dependencies:

Instead of math/gnuplot, you may want to install package math/gnuplot-lite which does not depend on X11 (but lacks the raster graphic terminals). The web applications additionally require a web server:

Optionally, install Pygments and AsciiDoctor to generate the man pages and the User Guide:

On Debian, install GCC, GNU Fortran, and the build environment first:

The third-party dependencies have to be installed with development headers:

Instead of package gnuplot, you may prefer the no-X11 flavour gnuplot-nox if raster graphic formats are not required (essentially, SVG output only). The SQLite 3 package version must be ≥ 3.39.0. Depending on the package repository, the names of the HDF5 and Lua packages may differ.

dmapi is an HTTP-RPC API service for remote DMPACK database access. The web application has to be executed through a FastCGI-compatible web server or a FastCGI spawner. It is recommended to use lighttpd(1).

The dmapi service offers endpoints for clients to insert beats, logs, and observations into the local SQLite database, and to request data in CSV or JSON format. Authentication and encryption are independent from dmapi and have to be provided by the web server.

All POST data has to be serialised in Fortran 95 Namelist format, with optional deflate or zstd compression.

If HTTP Basic Auth is enabled, the sensor id of each beat, log, node, sensor, and observation sent to the HTTP-RPC service must match the name of the authenticated user. For example, to store an observation of a node with the id node-1, the HTTP Basic Auth user name must be node-1 as well. If the observation is sent by any other user, it will be rejected (HTTP 401).

The web application is configured through environment variables. The web server or FastCGI spawner must be able to pass environment variables to dmapi. See RPC Server for an example configuration.

The service accepts HTTP GET and POST requests. Section RPC API gives an overview of the available endpoints. The response format depends on the MIME type set in the HTTP Accept header of the request, either:

By default, responses are in CSV format. The Namelist format is available only for single records. Status messages are returned as key–value pairs, indicated by content type text/plain.

The dmbackup utility creates an online backup of a running SQLite database. By default, the SQLite backup API is used. The program is functional equivalent to running the sqlite3(1) command-line interface:

dmbackup does not replace existing backup databases.

The dmbeat program is a heartbeat emitter that sends heartbeats or handshakes via HTTP POST to a remote dmapi service. The heartbeat messages include time stamp, system uptime, and last connection error. The server may inspect this data to check if a client is still running and has network access. The RPC endpoint is expected at [http|https]://<host>:<port>/api/v1/beat.

Passing the server credentials via the command-line arguments --username and --password is insecure on multi-user operating systems and only recommended for testing.

The dmdb program collects observations from a POSIX message queue and stores them in a SQLite database. The name of the message queue equals the given dmdb name, by default dmdb. The IPC option enables process synchronisation via POSIX semaphores. The value of the semaphore is changed from 0 to 1 if a new observation has been received. The name of the semaphore equals the dmdb name. Only a single process may wait for the semaphore.

The dmdbctl utility program performs create, read, update, or delete operations (CRUD) on the observation database.

Only nodes, sensors, and targets are supported. All data attributes are passed through command-line arguments.

The dmexport program writes beats, logs, nodes, sensors, targets, observations, and data points from database to file, in ASCII block, CSV, JSON, or JSON Lines format. The ASCII block format is only available for X/Y data points. The types data point, log, and observation require a sensor id, a target id, and a time range in ISO 8601 format.

If no output file is given, the data is printed to standard output. The output file will be overwritten if it already exists. If no records are found, an empty file will be created.

The dmfeed program creates a web feed from log messages in Atom Syndication Format. The log messages are read from database and written as XML to standard output or file.

The feed id has to be a 36 characters long UUID with hyphens. News aggregators use the id to identify the feed. Therefore, the id should not be reused among different feeds. Run dmuuid to generate a valid UUID4.

The time stamp of the feed in element updated is set to the date and time of the last log message. If no logs have been added to the database since the last file modification of the feed, the output file is not updated, unless argument --force is passed. To update the feed periodically, add dmfeed to crontab.

If an XSLT style sheet is given, web browsers may be able to display the Atom feed in HTML format. Set the option to the (relative) path of the public XSL on the web server. An example style sheet feed.xsl is located in /usr/local/share/dmpack/.

The dmfs program reads observations from file system, virtual file, or named pipe. The program can be used to read sensor data from the 1-Wire File System (OWFS).

If any receivers are specified, observations are forwarded to the next receiver via POSIX message queue. dmfs can act as a sole data logger if output and format are set. If the output path is set to -, observations are written to stdout instead of file.

The requests of each observation have to contain the path of the (virtual) file in attribute request. Response values are extracted by named group from the raw response using the given regular expression pattern. Afterwards, the observation is forwarded to the next receiver via POSIX message queue.

A configuration file is mandatory to describe the jobs to perform. Each observation must have a valid target id. Node, sensor, and target have to be present in the database.

The dmgrc program creates log messages from Leica GeoCOM return codes. Observations received by POSIX message queue are searched for a GeoCOM return code (GRC) response. If the code does not equal GRC_OK, a log message is sent to the configured logger instance.

By default, observation responses of name grc are verified. For each GeoCOM error code, a custom log level may be specified in the configuration file. Otherwise, the default log level is used instead.

The dminfo utility program prints build, database, and system information to standard output. The path to the beat, log, or observation database is passed through command-line argument --database. Only one database can be specified.

The output contains compiler version and options; database PRAGMAs, tables, and number of rows; as well as system name, version, and host name.

The dmimport program reads logs, nodes, sensors, targets, and observations in CSV format from file and imports them into the database. The database inserts are transaction-based. If an error occurs, the transaction is rolled back, and no records are written to the database at all.

The database has to be a valid DMPACK database and must contain the tables required for the input records. The nodes, sensors, and targets referenced by input observations must exist in the database. The nodes referenced by input sensors must exist as well.

The dminit utility program creates beat, log, and observation databases. No action is performed if the specified database already exists. A synchronisation table is required for observation and log synchronisation with an dmapi server. The argument can be omitted if this feature is not needed. The journal mode Write-Ahead Logging (WAL) should be enabled for databases with multiple readers.

The dmlog utility forwards a log message to the message queue of a dmlogger instance. The argument --message is mandatory. The default log level is info. Pass the name of the dmlogger instance through command-line argument --logger. The program terminates after log transmission.

The following log levels are accepted:

The dmlogger program collects log messages from a POSIX message queue and writes them to a SQLite database. The name of the message queue will equal the given dmlogger name with leading /, by default /dmlogger.

If a minimum log level is selected, only logs of a level greater or equal the minimum are stored in the database. Log messages with a lower level are printed to standard output before being discarded (only if the verbose flag is enabled).

The IPC option allows an optional process synchronisation via a named POSIX semaphores. The value of the semaphore is changed from 0 to 1 whenever a new log was received. The name of the semaphore will equal the dmlogger name with leading /. Only a single process should wait for the semaphore unless round-robin passing is desired. This feature may be used to automatically synchronise incoming log messages with a remote HTTP-RPC API server. dmsync will wait for new logs before starting synchronisation if the dmlogger instance name has been passed through command-line argument --wait.

The following log levels are accepted:

The dmlua program runs a custom Lua script to process observations received from message queue. Each observation is passed as a Lua table to the function of the name given in option procedure. If the option is not set, function name process is assumed by default. The Lua function must return the (modified) observation table on exit.

The observation returned from the Lua function is forwarded to the next receiver specified in the receivers list of the observation. If no receivers are left, the observation will be discarded.

The dmpipe program reads responses from processes connected via pipe.

All requests of an observation have to contain the process in attribute request. Response values are extracted by group from the raw response using the given regular expression pattern.

If any receivers are specified, observations are forwarded to the next receiver via POSIX message queue. The program can act as a sole data logger if output and format are set. If the output path is set to -, observations are printed to stdout.

A configuration file is mandatory to configure the jobs to perform. Each observation must have a valid target id. Node id, sensor id, and observation id are added by dmpipe. Node, sensor, and target have to be present in the database for the observation to be stored.

The dmplot program is a front-end to gnuplot(1) that creates plots of observations read from database. Plots are either written to file or displayed in terminal or X11 window.

Depending on the selected terminal back-end, you may have to set the environment variable GDFONTPATH to the local font directory first:

If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, an alias has to be added to the global profile:

The output file is ignored when using the terminals sixelgd and x11. Plotting parameters passed via command-line have priority over those from configuration file.

The dmrecv program listens to the POSIX message queue of its name and writes received logs or observations to stdout, file, or named pipe; in CSV, JSON Lines, or Namelist format. By default, the serialised data is appended to the end of the output file. If argument --replace is passed, the file will be replaced consecutively.

Received observations are not forwarded to the next specified receiver unless argument --forward is set. If no receivers are defined or left, the observation will be discarded after output.

The output format block is only available for observation data and requires a response name to be set. Observations will be searched for this response name and converted to data point type if found. The data point is printed in ASCII block format.

If the JSON Lines output format is selected, logs and observations are written as JSON objects to file or stdout, separated by new line (\n). Use jq(1) to convert records in JSON Lines file input.jsonl into a valid JSON array in output.json:

The program settings are passed through command-line arguments or an optional configuration file. The arguments overwrite settings from file.

The dmreport program creates reports in HTML5 format, containing plots of observations and/or log messages selected from database. Plots are created by calling gnuplot(1) and inlining the returned image (GIF, PNG, SVG) as a base64-encoded data URI. Any style sheet file with classless CSS can be included to alter the presentation of the report. The output of dmreport is a single HTML file.

Depending on the selected plot format, the environment variable GDFONTPATH may have to be set to the local font directory first:

If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, an alias has to be added to the global profile:

A configuration file is mandatory to create reports. Only a few parameters can be set through command-line arguments. Passed command-line arguments have priority over settings in the configuration file.

The dmsend program reads observations or logs in CSV or Fortran 95 Namelist format, and sends them sequentially to the POSIX message queue of the given receiver. The data is either read from file or from standard input. If the input data is of type observ and the argument --forward is passed, each observation will be sent to its next specified receiver in the receivers list. If no receivers are declared, or if the end of the receivers list is reached, the observation will not be forwarded.

The program settings are passed through command-line arguments or an optional configuration file. The arguments overwrite settings from file.

The dmserial program sends requests to a sensor or actor connected via USB/RS-232/RS-422/RS-485. Sensor commands and responses are sent/received through a teletype (TTY) device provided by the operating system. A pseudo-terminal (PTY) may be used to connect a virtual sensor.

Each request of an observation must contains the raw request intended for the sensor in attribute request. Response values are extracted by group from the raw response using the given regular expression pattern. Each group name must match a response name. Response names are limited to eight characters.

Observations will be forwarded to the next receiver via POSIX message queue if any receiver is specified. The program can act as a sole data logger if output and format are set. If the output path is set to -, observations are printed to stdout, else to file.

A configuration file is required to configure the jobs to perform. Each observation must have a valid target id. The database must contain the specified node, sensor, and targets. Parameters and functions of the Lua API may be used in the configuration file.

The following baud rates are supported: 50, 75, 110, 134, 150, 200, 300, 600, 1200, 1800, 2400, 4800, 9600, 19200, 38400, 57600, 115200, 230400, 460800, 921600.

The dmsync program synchronises logs, nodes, observations, sensors, and targets from local database concurrently with a remote dmapi server. The synchronisation may be started only once if no interval is set (to transfer nodes, sensors, and targets from client to server), periodically as a cron job, or by waiting for a POSIX semaphore.

The nodes, sensors, and targets referenced by observations in the local database must also exist in the remote server database. They can be created either with dmdbctl or dmweb, but also synchronised with dmsync. Logs and targets do not require any additional database entries on server-side.

The client databases must contain synchronisation tables. The tables are created automatically by dminit if command-line argument --sync is passed. Alternatively, start dmsync with argument --create once.

If the RPC server uses HTTP Basic Auth for authentication, the RPC user name must match the node id of the transmitted node, sensor, observation, log, or beat record. Otherwise, the server will reject the record and return HTTP 401 (Unauthorized).

The database records are send in compressed Fortran 95 Namelist format via HTTP to the server. The program uses libcurl for data transfer. The accessed RPC API endpoints are expected under URL [http|https]://<host>:<port>/api/v1/<endpoint>.

The result of each synchronisation attempt is stored in the local database. Records are marked as synchronised only if the server returns HTTP 201 (Created).

Passing the server credentials via the command-line arguments --username and --password is insecure on multi-user operating systems and only recommended for testing.

The dmuuid program is a command-line tool to generate pseudo-random UUID4s. By default, DMPACK uses 32 characters long UUID4s in hexadecimal format (without hyphens). Hyphens can be added by a command-line flag. The option --convert expects UUID4s to be passed via standard input. Invalid UUID4s will be replaced with the default UUID4.

dmweb is a CGI-based web user interface for DMPACK database access on client and server. The web application has to be executed through a CGI-compatible web server. It is recommended to run lighttpd(1). Any other server must be able to pass environment variables to the CGI application. gnuplot(1) is required for the plotting back-end (no-X11 flavour is sufficient).

The web application provides the following pages:

The style sheet of dmweb is based on missing.css. It may be replaced with any other classless CSS theme. For best experience, the IBM Plex font family should be installed locally.

If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, an alias has to be added to the global profile:

Copy the style sheet dmpack.min.css manually to the WWW root directory, or create a symlink. Environment variables are used to configure dmweb. Transport security and authentication have to be provided by the web server. See section Web UI for an example configuration.

In the lighttpd(1) configuration file, set auth.backend.htpasswd.userfile to the path of the file that contains the HTTP Basic Auth credentials, or remove the related lines from the configuration if authentication is not desired. You can run openssl(1) to add credentials to the htpasswd user file:

Replace <user> and <password> with real values. Instead of a htpasswd file, we may select a different authentication back-end, for example, LDAP, MySQL/MariaDB, PostgreSQL, or SQLite 3. See the lighttpd(1) auth module documentation for further instructions.

If the HTTP-RPC API will be accessed by a client-side application running in the browser, the web server has to be configured to send the appropriate Cross-Origin Resource Sharing (CORS) headers. By default, asynchronous JavaScript requests are forbidden by the same-origin security policy. Refer to the documentation of the web server on how to set the Access-Control-* headers. For lighttpd(1), load the module mod_setenv and add response headers for OPTION requests:

If the web server is behind a reverse proxy, CORS headers should be set by the proxy instead.

The databases are expected to be in directory /var/dmpack/. Change the environment variables in the web server configuration to the actual paths. The observation, log, and beat databases the web applications will access must be created and initialised beforehand:

Make sure the web server has read and write access to the directory and all databases inside:

Change www:www to the user and the group the web server is running as.

The snippet in this section may be added to the lighttpd(1) configuration to run the dmapi service. The lighttpd(1) web server does not require an additional FastCGI spawner. The following server modules have to be imported:

Add the IP address of the proxy server to the list of trusted forwarders to have access to the real IP of a client.

The FastCGI socket will be written to /var/run/lighttpd/sockets/dmapi.sock. Change max-procs to the desired number of FastCGI processes. Set the environment variables to the locations of the databases. The databases must exist prior start.

On FreeBSD, add the service to the system rc file /etc/rc.conf and start the server manually:

If served locally, access the RPC API at http://127.0.0.1/api/v1/.

The lighttpd(1) web server has to be configured to run the CGI application under base path /dmpack/. The following server modules are required:

The example configuration may be appended to your lighttpd.conf:

Copy the CSS file dmpack.min.css from /usr/local/share/dmpack/ (/usr/share/dmpack/ on Linux) to the WWW root directory, in this case, /var/www/, or simply create a symlinks:

If the files have to be served from a path other than the root path, add a rewrite rule or alias to the web server configuration. On FreeBSD, add the service to the system rc file /etc/rc.conf and start the web server manually:

If served locally, access the web application at http://127.0.0.1/dmpack/.

Returns service status in API status format as text/plain.

Returns all heartbeats in CSV, JSON, or JSON Lines format from database.

Returns logs of a given node and time range in CSV, JSON, or JSON Lines format from database. Node id and time range are mandatory.

Returns all nodes in CSV, JSON, or JSON Lines format from database.

Returns observations of given node, sensor, target, and time range from database, in CSV, JSON, or JSON Lines format.

Returns all sensors in CSV, JSON, or JSON Lines format from database.

Returns all targets in CSV, JSON, or JSON Lines format from database.

Returns time series as observation views or data points (X/Y records) in CSV format from database. In comparison to the observation endpoint, the time series include only a single response, selected by name.