DMPACK User Guide

On Debian, install the compilers and the build environment first:

The third-party dependencies have to be installed with development headers:

Instead of package gnuplot, you may prefer the no-X11 flavour gnuplot-nox if raster graphic formats are not required (essentially, SVG output only). The SQLite 3 package version must be ≥ 3.39.0. If the version in the package repository is too old, like on Ubuntu 22.04 LTS, you can also build the library from source. Depending on the Linux distribution, the names of the HDF5 and Lua packages may differ.

First, install the build and run-time dependencies:

Instead of math/gnuplot, you may want to install package math/gnuplot-lite which does not depend on X11 (but lacks the raster graphic terminals). The web applications additionally require a web server:

Optionally, install Pygments and AsciiDoctor to generate the man pages and the User Guide from source:

Update the cloned source code repository and its submodules with Git:

dmapi is an HTTP-RPC API service for remote DMPACK database access. The web application has to be executed through a FastCGI-compatible web server. It is recommended to use lighttpd(1). The service is configured through environment variables. The web server or FastCGI spawner must be able to pass environment variables to dmapi.

The dmapi service offers endpoints for clients to insert beats, logs, and observations into the local SQLite database, and to request data in CSV or JSON format. Only HTTP GET and POST requests are accepted. All POST data has to be serialised in Fortran 95 Namelist format, with optional deflate or zstd compression. Section RPC API gives an overview of the available endpoints.

Authentication and encryption are independent from dmapi and have to be provided by the web server. If HTTP Basic Auth is enabled, the sensor id of each beat, log, node, sensor, and observation sent to the HTTP-RPC service must match the name of the authenticated user. For example, to store an observation of a node with the id node-1, the user name of the client must be node-1 as well. If the observation is sent by any other user, it will be rejected (HTTP 401).

The response format depends on the MIME type set in the HTTP Accept header of the request, either:

By default, responses are in CSV format. The Namelist format is available only for single records. Status messages are returned as key–value pairs, indicated by content type text/plain.

See section RPC Server for a basic lighttpd(1) configuration.

The dmbackup utility creates an online backup of a running SQLite database. By default, the SQLite backup API is used. The program is functional equivalent to running the sqlite3(1) command-line interface:

dmbackup does not replace existing backup databases.

Create an online backup of an observation database:

The dmbeat program is a heartbeat emitter that sends handshake messages via HTTP POST to a remote dmapi service. Heartbeats include the following attributes:

The server may inspect the data to check if a client is still running and has network access. The RPC endpoint on the server is expected at URL [http|https]://<host>:<port>/api/v1/beat.

Send a single heartbeat to a dmapi service on localhost:

A sensor node with id dummy-node must exist in the server database. The web application dmweb lists the beats received by the server.

The dmbot program is an XMPP bot that accepts commands via chat. Access to the bot is limited to the JIDs added to table group in the configuration file. Requests from clients whose JID is not in the table will be rejected. If table group is empty, all clients are allowed to send commands to the bot.

The XMPP resource is automatically set to the name of the bot instance. If the JID of the bot account is bot@example.com, the full JID will be set to bot@example.com/dmbot if the bot name is dmbot.

All commands start with prefix !. For an overview, send chat command !help to the bot. The following commands are supported:

Passing the XMPP credentials via the command-line arguments --jid and --password is insecure on multi-user operating systems and only recommended for testing.

Connect with JID bot@example.com to an XMPP server on port 5223 and wait for commands:

If no configuration file is used, any client may send commands to the bot without authorisation. Start a chat with the bot JID and send a command. For instance, on command !uptime the bot sends a reply like the following:

The dmdb program collects observations from a POSIX message queue and stores them in a SQLite database. The name of the message queue equals the given dmdb name and leading /. The IPC option enables process synchronisation via POSIX semaphores. The value of the semaphore is changed from 0 to 1 if a new observation has been received. Only a single process shall wait for the semaphore.

Only observation types in binary format are accepted. Log messages are stored to database by the distinct dmlogger program.

Create a message queue /dmdb, wait for incoming observations, and store them in the given database:

Log messages and observation ids are printed to stdout if argument --verbose is set.

The dmdbctl utility program performs create, read, update, or delete operations (CRUD) on the observation database.

Only nodes, sensors, and targets are supported. All data attributes are passed through command-line arguments.

Add node, sensor, and target to observation database:

Delete a target from the database:

Read attributes of sensor sensor-1:

The dmexport program writes beats, logs, nodes, sensors, targets, observations, and data points from database to file, in ASCII block, CSV, JSON, or JSON Lines format. The ASCII block format is only available for X/Y data points. The types data point, log, and observation require a sensor id, a target id, and a time range in ISO 8601 format.

If no output file is given, the data is printed to standard output. The output file will be overwritten if it already exists. If no records are found, an empty file will be created.

Export log messages from database to JSON file:

Export observations from database to CSV file:

The dmfeed program creates a web feed from log messages in Atom Syndication Format. The log messages are read from database and written as XML to standard output or file.

The feed id has to be a 36 characters long UUID with hyphens. News aggregators will use the id to identify the feed. Therefore, the id should not be reused among different feeds. Run dmuuid to generate a valid UUIDv4.

The time stamp of the feed in element updated is set to the date and time of the last log message. If no logs have been added to the database since the last file modification of the feed, the output file is not updated, unless argument --force is passed. To update the feed periodically, add dmfeed to crontab.

If an XSLT style sheet is given, web browsers may be able to display the Atom feed in HTML format. Set the option to the (relative) path of the public XSL on the web server. An example style sheet feed.xsl is located in /usr/local/share/dmpack/.

First, generate a unique feed id:

Then, write the last 50 log messages in Atom format to file feed.xml, and include a link to the XSLT style sheet feed.xsl:

Copy the XSLT style sheet to the directory of the Atom feed:

If /var/www/ is served by a web server, feed readers can subscribe to the feed. Additionally, we may translate feed and style sheet into a single HTML document feed.html, using an arbitrary XSLT processor, for instance:

The dmfs program reads observations from file system, virtual file, or named pipe. The program can be used to read sensor data from the 1-Wire File System (OWFS).

If any receivers are specified, observations are forwarded to the next receiver via POSIX message queue. dmfs can act as a sole data logger if output and format are set. If the output path is set to -, observations are written to stdout instead of file.

The requests of each observation have to contain the path of the (virtual) file in attribute request. Response values are extracted by named group from the raw response using the given regular expression pattern. Afterwards, the observation is forwarded to the next receiver via POSIX message queue.

A configuration file is mandatory to describe the jobs to perform. Each observation must have a valid target id. Node, sensor, and target have to be present in the database.

Start dmfs to execute the jobs in the configuration file:

The dmgrc program creates log messages from Leica GeoCOM return codes. Observations received by POSIX message queue are searched for a GeoCOM return code (GRC) response. If the code does not equal GRC_OK, a log message is sent to the configured logger instance.

By default, observation responses of name grc are verified. For each GeoCOM error code, a custom log level may be specified in the configuration file. Otherwise, the default log level is used instead.

A configuration file is not required, but allows to specifiy the log level of certain GeoCOM return codes. In the following example configuration, the default log level for all return codes other than GRC_OK is set to LL_WARNING. The level is further refined for specific GeoCOM codes:

See section GeoCOM API for a table of all supported return codes. Pass the path of the configuration file through the command-line argument:

The name argument must match the name of the configuration table. A logger process of name dmlogger must be running to process the generated log messages.

The dminfo utility program prints build, database, and system information to standard output. The path to the beat, log, or observation database is passed through command-line argument --database. Only one database can be specified.

The output contains compiler version and options; database PRAGMAs, tables, and number of rows; as well as system name, version, and host name.

Print build, database, and system information:

The dmimport program reads logs, nodes, sensors, targets, and observations in CSV format from file and imports them into the database. The database inserts are transaction-based. If an error occurs, the transaction is rolled back, and no records are written to the database at all.

The database has to be a valid DMPACK database and must contain the tables required for the input records. The nodes, sensors, and targets referenced by input observations must exist in the database. The nodes referenced by input sensors must exist as well.

Import observations from CSV file observ.csv into database observ.sqlite:

The dminit utility program creates beat, log, and observation databases. No action is performed if the specified database already exists. A synchronisation table is required for observation and log synchronisation with an dmapi server. The argument can be omitted if this feature is not needed. The journal mode Write-Ahead Logging (WAL) should be enabled for databases with multiple readers.

Create an observation database with remote synchronisation tables (WAL):

Create a log database with remote synchronisation tables (WAL):

Create a heartbeat database (WAL):

The dmlog utility forwards a log message to the message queue of a dmlogger or dmrecv instance. The program may be executed through a shell script to add logs to the DMPACK database. The argument --message is mandatory. The default log level is info. Pass the name of the dmlogger or dmrecv instance to send the log to through command-line argument --logger.

Logs are sent in binary format. The program terminates after log transmission. The log level may be one of the following:

Both, parameter strings and literal log level values, are accepted as command-line arguments. For level warning, set argument --level to 3 or warning.

Send a log message to the message queue of logger dmlogger:

The dmlogger process will receive the log message in real-time and store it in the log database (if the log level is ≥ the configured minimum log level):

The dmlogger program collects log messages from a POSIX message queue and writes them to a SQLite database. The name of the message queue will equal the given dmlogger name with leading /, by default /dmlogger.

If a minimum log level is selected, only logs of a level greater or equal the minimum are stored in the database. Log messages with a lower level are printed to standard output before being discarded (only if the verbose flag is enabled).

The IPC option allows an optional process synchronisation via a named POSIX semaphores. The value of the semaphore is changed from 0 to 1 whenever a new log was received. The name of the semaphore will equal the dmlogger name with leading /.

Only a single process should wait for the semaphore unless round-robin passing is desired. This feature may be used to automatically synchronise incoming log messages with a remote HTTP-RPC API server. dmsync will wait for new logs before starting synchronisation if the dmlogger instance name has been passed through command-line argument --wait.

The following log levels are accepted:

Create a message queue /dmlogger, wait for incoming logs, and store them in the given database if logs are of level error (4) or higher:

Push semaphore /dmlogger each time a log has been received:

Let dmsync wait for semaphore /dmlogger before synchronising the log database with host 192.168.1.100, then repeat:

The dmlua program runs a custom Lua script to process observations received from message queue. Each observation is passed as a Lua table to the function of the name given in option procedure. If the option is not set, function name process is assumed by default. The Lua function must return the (modified) observation table on exit.

The observation returned from the Lua function is forwarded to the next receiver specified in the receivers list of the observation. If no receivers are left, the observation will be discarded.

The following Lua script script.lua just prints observation table observ to standard output, before returning it to dmlua unmodified:

Any observation sent to receiver dmlua will be passed to the Lua function process() in script.lua, then forwarded to the next receiver (if any):

The dmpipe program reads responses from processes connected through a pipe to read sensor data from a third-party program. Requests of an observation have to contain the process to call in attribute request. Response values are extracted by group from the raw response using the given regular expression pattern.

If any receivers are specified, observations are forwarded to the next receiver via POSIX message queue. The program can act as a sole data logger if output and format are set. If the output path is set to -, observations are printed to stdout.

A configuration file is mandatory to configure the jobs to perform. Each observation must have a valid target id. Node id, sensor id, and observation id are added by dmpipe. If the observation will be stored in a database, the node, sensor and target ids have to exist in the database.

The example reads the remaining battery life returned by the sysctl(8) tool (available on FreeBSD):

On Linux, the battery life can be read with dmfs from /sys/class/power_supply/BAT0/capacity instead.

The regular expression pattern describes the response and defines the group battery for extraction. The name of one of the responses in the responses table must equal the group name. The observation will be forwarded to the message queue of a dmdb process. Backslash characters in the string values have to be escaped with \.

Pass the path of the configuration file to dmpipe:

The result returned by sysctl(8) will be formatted according to the current locale (decimal separator). You may have to change the locale first to match the regular expression pattern:

The dmplot program is a front-end to gnuplot(1) that creates plots of observations read from database. Plots are either written to file or displayed in terminal or X11 window.

Depending on the selected terminal back-end, you may have to set the environment variable GDFONTPATH to the path of the local font directory first:

If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, create a symbolic link or add an alias to the global profile:

The output file is ignored when using the terminals sixelgd and x11. Plotting parameters passed via command-line have priority over those from configuration file.

Create a plot of observations selected from database observ.sqlite in PNG format, and write the file to /tmp/plot.png:

Output the plot directly to terminal, using the configuration in dmplot.conf:

The sixelgd format requires a terminal emulator with Sixel support, such as xterm(1) or mlterm(1).

The dmrecv program listens to the POSIX message queue of its name and writes received logs or observations to stdout, file, or named pipe; in CSV, JSON Lines, or Namelist format. By default, the serialised data is appended to the end of the output file. If argument --replace is passed, the file will be replaced consecutively.

Received observations are not forwarded to the next specified receiver unless argument --forward is set. If no receivers are defined or left, the observation will be discarded after output. If the JSON Lines output format is selected, logs and observations are written as JSON objects to file or stdout, separated by new line (\n). Use jq(1) to convert records in JSON Lines file input.jsonl into a valid JSON array in output.json:

The output format block is only available for observation data and requires a response name to be set. Observations will be searched for this response name and converted to data point type if found. The data point is printed in ASCII block format.

The program settings are passed through command-line arguments or an optional configuration file. The arguments overwrite settings from file.

Write log messages received from POSIX message queue /dmrecv to file /tmp/logs.csv in CSV format:

Output observations in JSON Lines format to stdout:

Write the observations serialised in JSON Lines format to named pipe /tmp/fifo_dmrecv:

Another process can now read the observations from /tmp/fifo_dmrecv:

Responses in block format can also be piped to a graph tool like trend to update a chart in real-time. For instance, to pipe the responses of name tz0 for observations received through message queue /dmrecv to trend(1), run:

The dmreport program creates reports in HTML5 format, containing plots of observations and/or log messages selected from database. Plots are created by calling gnuplot(1) and inlining the returned image (GIF, PNG, SVG) as a base64-encoded data URI. Any style sheet file with classless CSS can be included to alter the presentation of the report. A basic style sheet dmreport.css and its minified version dmreport.min.css are provided in /usr/local/share/dmpack/dmreport/. The output of dmreport is a single HTML file with inlined CSS. Use a command-line tool like wkhtmltopdf to convert the HTML report to PDF format.

Depending on the selected plot format, the environment variable GDFONTPATH may have to be set to the local font directory containing the TrueType fonts first, for example:

Add the export statement to the global profile /etc/profile. If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, create a symbolic link or add an alias to /etc/profile:

A configuration file is mandatory to create reports. Only a few parameters can be set through command-line arguments. Passed command-line arguments have priority over settings in the configuration file.

The settings are stored in Lua table dmreport in the configuration file. The observations are read from database observ.sqlite, the log messages from log.sqlite. You might want to use absolute paths for the databases.

The sensor node dummy-node, the sensor dummy-sensor, and the target dummy-target must exist in the database, and the observations to plot need to have responses of name tz0. Write a report to file report.html based on settings in dmreport.conf. The command-line arguments overwrite the settings of the configuration file:

In order to update reports periodically, we can customise the shell script mkreport.sh in /usr/local/share/dmpack/dmreport/. The script determines the timestamps of the last and the current month (to allow observations to arrived late), which will then be passed to dmreport to create monthly reports. Modify the script according to your set-up:

The shell script writes two reports to /var/www/reports/.

The directory may be served by lighttpd(1). Add the script to your crontab to run the report generation periodically.

The dmsend program reads observations or logs in CSV and Fortran 95 Namelist format, and sends them sequentially to the POSIX message queue of a given receiver. The data is either read from file or standard input. If the input data is of type observ and the argument --forward is passed, each observation will be sent to its next specified receiver in the receivers list instead of the receiver given through argument --receiver. If no receivers are set, or if the end of the receivers list is reached, the observation will be discarded.

The program settings are passed through command-line arguments or an optional configuration file. The arguments overwrite settings from file.

Read a single observation from Namelist file observ.nml and send it to the next receiver specified by attribute next:

Send multiple logs in CSV file logs.csv sequentially to process dmrecv:

The dmserial program sends requests to a sensor or actor connected via USB/RS-232/RS-422/RS-485. Sensor commands and responses are sent/received through a teletype (TTY) device provided by the operating system. A pseudo-terminal (PTY) may be used to connect a virtual sensor.

Each request of an observation must contains the raw request intended for the sensor in attribute request. Response values are extracted by group from the raw response using the given regular expression pattern. Each group name must match a response name. Response names are limited to eight characters. Observations will be forwarded to the next receiver via POSIX message queue if any receiver is specified. The program can act as a sole data logger if output file and format are set. If the output is set to -, observations are printed to stdout.

A configuration file is mandatory to configure the jobs to perform. Each observation must have a valid target id. The database must contain the specified node, sensor, and targets. Parameters and functions of the Lua API may be used in the configuration file. The following baud rates are supported: 50, 75, 110, 134, 150, 200, 300, 600, 1200, 1800, 2400, 4800, 9600, 19200, 38400, 57600, 115200, 230400, 460800, 921600.

Read the jobs to perform from configuration file and execute them sequentially:

The dmsync program synchronises logs, nodes, observations, sensors, and targets from local databases concurrently with a remote dmapi server. The synchronisation may be started only once if no interval is set (to transfer nodes, sensors, and targets initially from client to server), periodically as a cron job, or by waiting for a POSIX semaphore.

The nodes, sensors, and targets referenced by observations in the local database must also exist in the remote server database. They can be created on the server with dmdbctl or dmweb, or sent from client to server with dmsync. Logs and targets do not require any additional database entries on the server-side.

The client databases must contain synchronisation tables. The tables are created automatically by dminit if command-line argument --sync is passed. Otherweise, start dmsync with argument --create once to add the missing tables.

If the RPC server uses HTTP Basic Auth for authentication, the RPC user name must match the node id of the transmitted node, sensor, observation, log, or beat records, or the server will reject the requests and return HTTP 401 (Unauthorized).

The database records are serialised in Fortran 95 Namelist format and optionally compressed before being sent to the server. The program uses libcurl for data transfer, and deflate or zstd for compression. The RPC API endpoints to post records to are expected at URL [http|https]://<host>:<port>/api/v1/<endpoint>.

The result of each synchronisation attempt is stored in the local database. Records are marked as synchronised only if the server returns HTTP 201 (Created).

Passing the server credentials via the command-line arguments --username and --password is insecure on multi-user operating systems and only recommended for testing.

Initially synchronise nodes, sensors, and targets in the local observation database with an HTTP-RPC server (without authentication):

Synchronise observations:

Synchronise log messages:

The dmuuid program is a command-line tool to generate pseudo-random UUIDs. By default, DMPACK uses 32 characters long UUIDv4 identifiers in hexadecimal format (without hyphens). Hyphens can be added by a command-line flag. The option --convert expects UUIDv4 identifiers to be passed via standard input. Invalid identifiers will be replaced with the default UUID. The program may be used to create a feed id for dmfeed.

Create three identifiers:

Create a UUIDv4 with hyphens:

Add hyphens to a hexadecimal UUID:

dmweb is a CGI-based web user interface for DMPACK database access on client and server. The web application has to be executed through a CGI-compatible web server. It is recommended to run lighttpd(1). Any other server must be able to pass environment variables to the CGI application. gnuplot(1) is required for the plotting back-end (no-X11 flavour is sufficient).

The web application provides the following pages:

The style sheet of dmweb is based on missing.css. It can be replaced with any other classless CSS theme. For best experience, the IBM Plex font family should be installed locally.

If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, create a symbolic link or add an alias to the global profile /etc/profile:

On FreeBSD, it might be necessary to add the environment variable GDFONTPATH to the path of the font directory:

The map view requires a URL to the tile server in environment variable DM_TILE_URL. For example, set the variable to https://tile.openstreetmap.org/{z}/{x}/{y}.png to use OpenStreetMap as the backend.

Copy the directory /usr/local/share/dmpack/dmweb manually to the WWW root directory, or create a symlink. Environment variables are used to configure dmweb. Transport security and authentication have to be managed by the web server. See section Web UI for an example configuration.

The HTTP-RPC API and the web interface will be publicly accessible if the web server is not configured to manage user authentication. HTTP Basic Auth is a simple method to authenticate users by name and password. The lighttpd(1) web server includes an auth module with various back-ends. In the web server configuration, set auth.backend.htpasswd.userfile to the path of the file that contains the credentials. You can run openssl(1) to add one or more user accounts with hashed password (SHA-512) to the htpasswd file, in this case /usr/local/etc/lighttpd/htpasswd:

Enter the user name and the associated password after the read commands. As an alternative to storing the credentials in a flat file, we can select a different authentication back-end, for example, LDAP, PAM, or database. See the documentation of the module for further instructions. In the web server configuration, load module mod_authn_file, select the back-end, and enable authentication globally or for specific routes:

If the HTTP-RPC API will be accessed by a client-side application running in the browser, the web server has to be configured to send the appropriate Cross-Origin Resource Sharing (CORS) headers. By default, asynchronous JavaScript requests are forbidden by the same-origin security policy. Refer to the documentation of the web server on how to set the Access-Control-* headers. For lighttpd(1), load the module mod_setenv and add response headers for OPTION requests:

If the web server is behind a reverse proxy, CORS headers should be set by the proxy instead.

The databases are expected to be in directory /var/dmpack/. Change the environment variables in the web server configuration to the actual paths. The observation, log, and beat databases the web applications will access must be created and initialised beforehand:

Make sure the web server has read and write access to the directory and all databases inside:

Change www:www to the user and the group the web server is running as.

The snippet in this section may be added to the lighttpd(1) configuration to run the dmapi service. The lighttpd(1) web server does not require an additional FastCGI spawner. The following server modules have to be imported:

Add the IP address of the proxy server to the list of trusted forwarders to have access to the real IP of a client.

The FastCGI socket will be written to /var/run/lighttpd/sockets/dmapi.sock. Change max-procs to the desired number of FastCGI processes. Set the environment variables to the locations of the databases. The databases must exist prior start. On FreeBSD, add the service to the system rc file /etc/rc.conf and start the server manually:

If served locally, access the RPC API at http://127.0.0.1/api/v1/.

The lighttpd(1) web server has to be configured to run the CGI application under base path /dmpack/. The following server modules are required:

The example configuration may be appended to your lighttpd.conf:

Copy the directory dmweb from /usr/local/share/dmpack/ (or /opt/share/dmpack/) to the WWW root directory, in this case, /var/www/, or simply create a symlink:

If the files have to be served from a path other than the root path, add a rewrite rule or alias to the web server configuration. On FreeBSD, add the service to the system rc file /etc/rc.conf and start the web server manually:

If served locally, access the web application at http://127.0.0.1/dmpack/.

The sqlite3(1) program is stand-alone command-line shell for SQLite database access that allows the user to execute arbitrary SQL statements. Third-party programs provide an additional graphical user interface: