6.5. Logging Messages in Links and Nodes

6.5.1. What this is about

This chapter describes methods to log and evaluate message data in LN. They are geared toward a complete ad-hoc recording of messages which can be used to analyze or debug a system.

6.5.2. Two methods: LN recorder and Daemon Logger

The two methods described here are the LN recorder and the ln daemon logger. Both allow to save message data in files, but do have technical differences.

6.5.2.1. Main differences

LN Recorder Features

The LN recorder is just an LN client which is started with a regexp pattern describing a set of topic or service names. When any of these topics sends messages, their content is stored as a time series. The same is true for service requests and responses. To do this, the LN recorder receives that data as an LN client, so it typically needs to be transmitted via a network.

Storage is done continuously in a efficient binary format along with metadata about the messages. In order to avoid stalls of the system or data loss, the location of storage needs to be on a local file system.

The data format is a special type of raw binary format which is called LNRDB, which can be easily read into memory-mapped numpy arrays, and equally easily converted into the HDF5 format which is widely used for time series and geophysical data. These data formats both support memory-mapping, which means that a computer can easily evaluate data sets which are larger than its RAM.

Because of the continuous storing operation, the LN recorder can store an almost unlimited duration and amount of data (as much as the local file system can store). However because data is transported as LN messages to the client node, there exist certain limits on the maximum bandwidth with which the data can be transmitted. These limits depend on both network hardware and connectivity.

LN Daemon Logger Features

The LN daemon logger runs in an LN daemon on a specific host. It only records data from local topics, so that data can be stored at a very high bandwidth. When doing the recording, the data is stored in a ring buffer in-memory, which means that there is a maximum recording size (and therefore, time) for the data to stay available, and if that amount is surpassed, the oldest data is erased. The limit depends on the hardware and RAM of the node. Recording is started and stopped by LN client API calls, and a further call instructs the manager to save the data to a file on the manager node. The possible storage data formats are Matlab *.mat files, which do have the practical limit that they should not be larger than the physical RAM of the machine processing the data, “ln-daemon-log-raw-data” files, “flat python pickle” or lnrecorder-pcap files.

The typical use case for recordings that use the LN daemon logger are captures of short segments of high-frequency data which is used for debugging in real-time contexts. Because no network transmission is necessary at the time of recording, the possible data rates are very high, which allows to store events and parameters, for example from hardware controllers, at a high frequency.

6.5.3. The LN Recorder

As already mentioned, the LN recorder is a client application which stores data from a set of topics and services as time series into a binary file.

This means that data can be transported across the network like any other message data.

6.5.3.1. Version Information

Both lnrecorder and the lnrdb Python library require at least LN version 2.1.0. Prior versions have bugs which, among other things, cause that metadata is not written correctly when the lnrecorder is terminated by a signal. It works together with older versions of LN.

See also

Note on Synchronization of logged Packages in the Frequently Asked Questions part.

6.5.3.2. Command Line Arguments

The lnrecorder is called with a required “command” argument and options which indicate topic and service names. The command required to record messages is “record”. This is followed by one or more patterns for topics in the format "topic:<name-pattern>[@frequency]" and for services in the format "service:<name-pattern>".

There, the name pattern is a regexp which matches the patterns which should be recorded. A star "*" matches all topics & services. The topics pattern can be followed by a frequency option, which is formed of an "@" followed by the frequency in Hertz. Using this option requires that the publishers of the message set the message time stamps, with a unit of seconds.

A further option is "-v" for “verbose”. In this mode, the LN recorder prints out any topics which have been added or removed during its run.

If new topics appear which match a pattern, they are included in the recording. Each topic recording is numbered with a global counter, this means when a topic is started and stopped multiple times, it is recorded with increasing counter values.

here the complete output of lnrecorder --help:

usage: lnrecorder [OPTIONS...] VERB [ARGS...]

VERBS can be one of:

  record OUTPUT_FILE [name-pattern...]
          start recording of matching topics and services. further arguments
          are intrepreted as glob patterns. in this case only topics or services
          with matching names are recorded. each pattern can be prefixed with 'topic:'
          or 'service:' to only match topics or services.
          prepend a '!' to exclude matching topics/patterns.
          each topic pattern can be followed by a '@'-sign and a subscription rate
          like this: 'large_topics*@3' for 3Hz subscription.

          example: 'record topics:* !topics:do_not_log.*' (would record all topics that
          do not start with 'do_not_log.')

  replay INPUT_FILE [name-pattern...]
          start replaying topics. further glob-patterns select
          which topics to replay, if not all.

  list INPUT_FILE
          list contents of given file

  idle
          do nothing, just wait for LN-service requests

  convert INPUT_FILE OUTPUT
          convert from one logging format to another

  write_msg_defs INPUT_FILE FOLDER
          (unfinished, output topic msg defs to stdout)

OPTIONS can be:

  -v      produce more verbose output (repeat up to 3 times)

  -s RELATIVE_START_TIMESTAMP
          (only for replay) start replaying at given timestamp
          this timestamp is relative to the logfile start-timestamp

  -vs
          show verbose service request/response events when they happen

  -name NAME
          use this NAME as prefix for all LN-services

  -z --gzip
          filter recorded log through gzip before writing to pcap
  -j --bzip2
          filter recorded log through bzip2 before writing to pcap
  -J --xz
          filter recorded log through xz before writing to pcap

6.5.3.3. Example

Here is an example lnrdb.lnc which can be found in the folder examples/guide/example_logging_lnrdb/:

# this is a config for using not Cissy but a local install of
# LN, using scons. It requires to install LN >= 2.1.0 !
#
# Important: The Python h5py package needs to be installed
# manually, for example in Debian by issuing
# "apt install python3-h5py"
# (make sure here to match the Python version used when
# installing LN!)

instance
name: lnrdb example for %(env USER) on %(hostname)
manager: :%(get_port_from_string "%(instance_name)")
enable_auto_groups: true

add_message_definition_dir: %(CURDIR)/ln_msg_defs

push_name_prefix: all

process publisher
pass_environment: PYTHONPATH, LD_LIBRARY_PATH
change_directory: %(CURDIR)
add flags: no_error_on_successful_stop
node: localhost
command: %(LN_PYTHON) publisher.py

# record all LN topics at a rate of max 500 Hz!
process lnrecorder
node: localhost
add flags: start_in_shell
change_directory: %(CURDIR)/logs
# lnrecorder -v record last "topic:*@500"
command: [
  FN=example_log_$(date +%Y%m%dT%H%M%S);
  echo "recording to $FN";
  (rm test_log 2>/dev/null; ln -s $FN test_log);
  exec lnrecorder -v record $FN "topic:*@500"
]

process python example read log
node: localhost
change_directory: %(CURDIR)
add flags: no_error_on_successful_stop
pass_environment: PATH, PYTHONPATH, LD_LIBRARY_PATH
add flags: use_execvpe
command: %(LN_PYTHON) read_log.py

process convert log
node: localhost
change_directory: %(CURDIR)
pass_environment: PATH, PYTHONPATH, LD_LIBRARY_PATH
add flags: no_error_on_successful_stop
command: %(LN_PYTHON) convert_logs.py "test_log.hdf5" "logs/test_log"

process python example read hdf5
node: localhost
# when running under cissy, this requires Python3
# (can also run with h5py installed from pip)
change_directory: %(CURDIR)
pass_environment: PATH, PYTHONPATH, LD_LIBRARY_PATH
add flags: no_error_on_successful_stop
command: %(LN_PYTHON) read_hdf5.py "test_log.hdf5"

The message definitions used are as follows:

lnrdb_example/two_doubles

double a
double b

lnrdb_example/two_vectors

double q[7]

# matrix, row major, 6 rows, 7 columns:
double J[42]

# counts from 0 to 4:
uint8_t flag

This example starts several processes. process “publisher” publishes some data, and the process “lnrecorder” records it.

An important point is that the lnrecorder needs to be started in a shell snippet with “exec”, or alternatively with the “use_execvpe” flag in the process section. This ensures that it will receive correct signals on termination, and this in turn makes sure that files with meta data are properly closed.

Note

todo: is this still necessary with LN >= 2.1.0?

6.5.3.4. Synchronization of logged Data and Time Stamps

The lnrecorder does not ensure that data from different topic subscribers or processes are logged in a synchronized manner. It solely records any packets that were captured during its time of operation. When different subscribers, or different processes are started, they will not start operation at the same time (the difference can easily be 0.5 seconds for the subscribe operation alone). Therefore, both the original times of recorded messages, and the number of messages are expected to differ.

However, the lnrecorder records the time stamps which are passed when any packets are sent. These refer to the system clock, and allow for a precise synchronization of different messages, and aligning related data. If recording from different hosts need to be related in terms of time, it must be ensured that their system clocks are synchronized via network time protocol (NTP) or other means.

Each lnrdb table for a recorded topic includes these two timestamp columns:

• timestamp local system time when lnrecorder got hold of this topic message. It corresponds to the _packet_log_ts in ln_daemon recordings.

• publisher_timestamp system time on publisher host when message was published. It corresponds to the _packet_source_ts in ln_daemon recordings.

Note

the timestamp is usually of less importance: it will show more jitter, drift and probably lags behind by the time needed for transfer between hosts…

See also

Note on Synchronization of logged Packages in the Frequently Asked Questions part.

Description of time stamp for the ln_read() function in Receiving Packets.

6.5.3.5. LN Service Interface aka idle-mode

when lnrecorder is started like this: lnrecorder idle, it will only react to these LN-service calls:

• <name>.record using md lnrecorder/record

• <name>.record_waiting using md lnrecorder/record_waiting

• <name>.stop using md lnrecorder/stop

• <name>.replay using md lnrecorder/replay

<name> defaults to lnrecorder but if started via ln-manager it will be the name fo the process definition which started the lnrecorder.

when calling the .record-service you need to provide these fields:

• filename name of to be created lnrdb or .pcap-file

• patterns is a list of ln/string. here you can specify exactly the same topic- / service-name-patterns as you would do on the command line (including the @-rate-limiter for topics). if this list is kept empty, all topics & services will be logged!

a service response with empty error_message field indicates that the lnrecorder is now recording. The .record service waits up to 10 seconds until topics that existed when recording started have received a first packet, but it still reports success if that maximum wait time expires.

The .record_waiting service uses the same fields as .record and adds:

• wait_timeout maximum time in seconds to wait until initially existing topics have received packets.

The .record_waiting response includes pending_topics, a list of initially existing topics that had not received packets when wait_timeout expired. The recording has already started; callers can decide whether a non-empty pending_topics list is an error for their use case.

to stop a recording, you should call the .stop LN-service. it does not need any request fields.

after the lnrecorder is again in its stopped-state, you can either start another recording or start a replay:

to replay a log you call the .replay LN-service and provide these request fields:

• filename is the name of the lnrdb or .pcap-file

• patterns is a list of ln/string and is optional. use it to only replay a selection of the recorded topics / services. if not specified, all topics / services are replayed.

• start_time start replaying from a given time-offset (seconds) into the recording. use 0 to start from the beginning.

• stop_time stop playback at given time in seconds after recording-start. set to 0 to play until end of recording.

• wait optional. if set to 1 the service call will wait until the replay is finished. otherwise the service call will return as soon as the replay was started

a replay can also be stopped via the .stop LN-service from above.

6.5.3.6. Data Accesss

6.5.3.6.1. Reading logged Data in LNRDB Format into Python

The process “python example read log” reads the generated log data, and prints some values. It uses the “lnrdb” python module from the links_and_nodes_python package, ands its listing looks like this:

import lnrdb

log = lnrdb.Db("logs/test_log")

print("logged tables / topic instances:")
for tn in log.tables.keys():
    print(" - %r %d rows" % (tn, log.tables[tn].n_rows))

print("first & last row::")
for tn in log.tables.keys():
    print(" - %r" % tn)
    table = log.tables[tn]
    print("     original topic name: %r" % table.meta["topic.name"])
    print("     original topic md: %r" % table.meta["topic.md"])
    print("     numpy dtype: %r" % (table.dtype, ))
    if table.n_rows > 0:
        
        data_mmap = table.mmap() # this numpy mmap object handles mostly like any other numpy recarray!
        
        print("     first row: %r" % (data_mmap[0], ))
        print("     last row: %r" % (data_mmap[-1], ))
        if table.meta["topic.md"] == "lnrdb_example/two_vectors":
            i = int(table.n_rows / 2)
            last = data_mmap[i]
            print("     middle idx: %r" % (i, ))
            print("     middle q: %r" % (last.q, ))
            N = last.q.shape[0]
            print("     middle J: %r" % (last.J.reshape(-1, N), ))
            print("     middle flag: %r" % (last.flag, ))
            

An important aspect of the lnrdb library is its capability to return memory-mapped numpy arrays. This means that it is possible to access time series which are far larger than a size that would fit into memory, but in the same way as a “normal” memory-backed numpy array. Data is read from disk only on-demand and can also be modified.

6.5.3.6.2. Converting Logged Data to HDF5

The process “convert log” reads the LNRDB binary data and saves it in HDF5 format. HDF5 is a scientific data format that was developed for time series of large amounts of geophysical data, for example. One advantage it has is that it can hold any relevant metadata, and also that it can be easily converted into other formats.

The converter script shown here takes at least two arguments: The first is the hdf5 file which is to be generated as output, and the following ones are the lnrdb recordings which should be included in the hdf5 file. That file acts like a kind of archive and container and can hold multiple time series as well as their meta data.

import sys
import lnrdb
import h5py


def convert_recording(f, recording_name):
    grp = f.create_group(recording_name)

    log = lnrdb.Db(recording_name)

    print("converting logged tables / topic instances:")

    for tn in log.tables.keys():
        vars_log = vars(log.tables[tn])
        # print(vars(log.tables[tn]))
        print("vars_log.keys() ", vars_log.keys())
        if "rows" in vars_log.keys():
            print("found rows")
        print(" - %r %d rows" % (tn, log.tables[tn].n_rows))

        table = log.tables[tn]
        print("     original topic name: %r" % table.meta["topic.name"])
        print("     original topic md: %r" % table.meta["topic.md"])
        print("     numpy dtype: %r" % (table.dtype,))

        if table.n_rows > 0:

            data_mmap = table.mmap()  # this numpy mmap object handles mostly like
            #                           any other numpy recarray!
            dset = grp.create_dataset(tn, data=data_mmap, dtype=table.dtype)
        else:
            dset = grp.create_dataset(tn, (0,), dtype=table.dtype)

        # keep all meta information
        for meta_name, meta_value in table.meta.items():
            dset.attrs[meta_name] = meta_value


if len(sys.argv) < 3:
    print(
        """Usage: convert_logs <hdf5_filename> <recording1> [<recording2> ...]
    converts one or more lnrecorder recordings in lnrdb format into one HDF5 file,
    conserving meta data"""
    )

print("converting lnrecorder logs (using h5py from %s)" % h5py.__file__)
print("converting lnrecorder logs (using lnrdb from %s)" % lnrdb.__file__)


with h5py.File(sys.argv[1], "w") as f:
    for recording_name in sys.argv[2:]:
        convert_recording(f, recording_name)

print("conversion finished.")

Once the conversion is done, no dependencies on LN or the real-time run-time system are needed, which means that the data can be analysed off-line if desired.

there is also a ready to use conan-packaged stand-alone converter: lnrdb_to_hdf5_converter/1.0.0@common/unstable for details check here https://rmc-github.robotic.dlr.de/common/lnrdb_to_hdf5_converter

6.5.3.6.3. Reading HDF5 Data

As with the lnrdb module, such HDF5 files can be used to access memory-mapped binary data. The script “read_hdf5.py” shows how this can be done:

from __future__ import print_function, division

import sys
import h5py
import numpy as np

list_of_datasets = []
def printshape(name):
    try:
        print(name, f[name].shape)
        list_of_datasets.append(name)
        # note that in LNRDB, the final part of the dataset name
        # (after the topic) is a process-global index that numbers the
        # order in which the topic was started to be
        # recorded. Multiple runs of a publisher generate multiple
        # datasets (and the order in which topic recording starta can
        # change if topics start in parallel!).
    except AttributeError:
        print(name, "has no shape")

def show_extract(f, dataset_name):
    dset = f[dataset_name]

    print("+" * 20, "\n\n\n")
    print("dataset %s" % dataset_name)
    print("   shape = %r" % dset.shape)

    if dset.shape[0] > 0:
        print("   initial time stamp = {:>10.3f}, publisher_timestamp = {:>10.3f}, "
              "timestamp delta = {:>5.3f}".format(dset[0, "timestamp"],
                                            dset[0, "publisher_timestamp"],
                                            dset[0, "publisher_timestamp"] - dset[0, "timestamp"]))
        print("   final time stamp   = {}, publisher_timestamp = {}, "
              "timestamp delta = {:>5.3f}".format(dset[-1, "timestamp"],
                                            dset[-1, "publisher_timestamp"],
                                            dset[-1, "publisher_timestamp"] - dset[-1, "timestamp"]))
        
        print("   first record:", dset[0])

        n_rec = dset.shape[0]
        middle_idx = n_rec // 2
        for idx in [0, middle_idx, -1]:
            for col in dset.dtype.names :                
                print("   {}({}) = {}".format(col, idx, dset[idx,col]))
                                    
        
with h5py.File(sys.argv[1], "r") as f: 
    f.visit(printshape)

    for ds in list_of_datasets:
        show_extract(f, ds)
    

6.5.3.6.4. Reading HDF5 into Matlab

The script in matlab/example_hdf5.m shows how a HDF5 file can be accessed in matlab.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% first topic
h5disp('test_log.hdf5', '/logs/test_log/topics/topic0/1')

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% second topic

h5disp('test_log.hdf5', '/logs/test_log/topics/topic1/2')
data2 = h5read('test_log.hdf5', '/logs/test_log/topics/topic1/2',1,1)

timestamp_start = data2.timestamp(1)
publisher_timestamp_start = data2.publisher_timestamp(1)
a_start = data2.a(1)
b_start = data2.b(1)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% third topic

%% shows how to read time series data from a large file that
%% dnoes not fit into memory
%% key is to do the indexing in h5read so that matlab
%% does not load the whole data set into memory

h5disp('test_log.hdf5', '/logs/test_log/topics/topic2/3')

% read first record
data3_s = h5read('test_log.hdf5', '/logs/test_log/topics/topic2/3',1,1)

q_start = data3_s.q
J_start = data3_s.J

% read last record
data3_info = h5info('test_log.hdf5', '/logs/test_log/topics/topic2/3')

data3_len = data3_info.Dataspace.Size

data3_e = h5read('test_log.hdf5', '/logs/test_log/topics/topic2/3',data3_len,1)


q_end = data3_e.q
J_end = data3_e.J

% read record in the middle

% remember that umpy uses zero-based indices, while Matlab uses
% indices starting from one
mid_idx = floor(data3_len/2) + 1

data3_mid = h5read('test_log.hdf5', '/logs/test_log/topics/topic2/3',mid_idx,1)

q_mid = data3_mid.q

J_mid = reshape(data3_mid.J, [6,7])
flag_mid = data3_mid.flag

Here, the hdf5 reader routine returns matlab matrix objects. In difference to Numpy, these objects are not memory-mapped. This means that if the time series is very large, only a selection of it can be read into memory at the same time (Matlab is capable to access memory-mapped data, but only with low-level routines).

6.5.3.6.5. Direct Access to LNRDB Data in Matlab

As a last example, we show some scripts which read binary LNRDB data directly in matlab:

%% first directly read all of first topic1 recording
data = read_topic1('logs/test_log/topics_topic1_2/data');
% this only works because read_topic1 knows the data format!
% -> lnrdb_example/two_doubles
whos data
l = length(data)
log_ts = data(:, 1);
pub_ts = data(:, 2);

disp(sprintf('first row [%f, %f]', data(1, 3), data(1, 4)));
disp(sprintf('last row [%f, %f]', data(l, 3), data(l, 4)));

%% directly read all of first topic2 recording
[log_ts, pub_ts, q, J, flag] = read_topic2('logs/test_log/topics_topic2_3/data');
% this only works because read_topic2 knows the data format!
% -> lnrdb_example/two_vectors
whos q J
middle = floor(length(q) / 2) + 1
disp(['middle q: ', sprintf('%f ', q(middle, :))]);

middle_J = reshape(J(middle, :, :), 6, 7)
middle_J_size = size(middle_J)

middle_flag = flag(middle)

function data = read_topic1(fn)
% reads lnrdb data file from topics of md `lnrdb_example/two_doubles`

% there are always two timestamp-doubles per row!
% first is logger-timestamp,
% second is publisher-timestmap

fid = fopen(fn);
data = fread(fid, [2 + 2, Inf], 'double')';
fclose(fid);

function [log_ts, pub_ts, q, J, flag] = read_topic2(fn)
% reads lnrdb data file from topics of md `lnrdb_example/two_vectors`

% there are always two timestamp-doubles per row!
% first is logger-timestamp,
% second is publisher-timestmap

% first read & interpret all double values
fid = fopen(fn);
n_doubles = 2 + 7 + 42 % 51
data = fread(fid, [n_doubles, Inf], '51*double', 1)'; % skip the flag byte at the end of each row
fclose(fid);

[l, cols] = size(data);

log_ts = data(:, 1);
pub_ts = data(:, 2);

data = data(:, 3:cols);
[l_, cols] = size(data);

q = data(:, 1:7);

J = data(:, 1+7:cols);
% restore matrices, and transpose last two dims
J = reshape(J, length(J), 7, 6);
J = permute(J, [1, 3, 2]);

% now read uint8-flag
fid = fopen(fn);
fseek(fid, n_doubles * 8, 'bof'); % skip first n_doubles in first row
flag = fread(fid, [Inf], 'uint8', n_doubles * 8)'; % after each flag skip n_doubles of next row
fclose(fid);

The offsets need to be computed from the message definitions.

As we see, this approach is more fragile, since it needs to use manually computed direct offsets into the binary files, which depend on the message definitions. Any change to the message definitions will render these offsets invalid, requiring consistent updates to avoid reading invalid binary data.

6.5.3.6.6. Troubleshooting

If any metadata files appear to be missing, check that the right LN version >= 2.1.0 is used, and that the lnrecorder program is started either with the “exec” shell command, or with the “execvpe” flag in the LNM configuration for the process!

6.5.4. The LN Daemon Logger

The LN daemon logger can be accessed by the LN client API, which allows to instruct an LN daemon to log message data to an circular buffer in RAM. It is run from an LN client, and instructs the daemon on a given node to log named topics.

Here is an example script how to use it:

#!/usr/bin/python
# -*- mode: python -*-

import sys
import time
import links_and_nodes as ln

from numpy import *

logfile = "/tmp/log_test.topics.pcap"

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("Please specify logfile and command as arguments to script!")
        print("Usage:")
        print("  %s <pickle logfile> <command>" % sys.argv[0])
        sys.exit(1)

    clnt = ln.client("ln_daemon log test", sys.argv)
    logger = clnt.get_logger("test_logger")

    logfile = sys.argv[1]
    command = sys.argv[2]

    if command == "start":
        logger.add_topic("test.topic1", 100000)
        logger.add_topic("test.topic2", 100000)
        logger.enable()
        print("Started logging...")

    elif command == "stop":
        print("Stop logging...")
        logger.disable()

        print("Saving logfile as %s at manager" % logfile)
        logger.manager_save(logfile)

    else:
        raise Exception(
            "Command '%s' not known! Only commands supported: start, stop" % command
        )

Here, logger.add_topic() configures logging for two topics: test.topic1 and test.topic2, for each topic the last 100000 samples will be kept in RAM. logger.enable() starts logging, and logger.disable() stops it. logger.manager_save(filename) instructs the LN manager to fetch the collected data from the daemons and save it to the given file.

The default is to log the “source shared memory” on the host the publisher runs on. Each shared-memory can only have one logger running at a time. If you have a subscriber on a different host or a subscriber with subscriber-rate != -1 they will get their own shared-memory. If you want to log data that arrives there, you need to specify a topic-name in the form of .add_topic("subscriber:<CLIENT-NAME>[@<SUB-RATE>]:<TOPIC-NAME>) where <CLIENT-NAME> is the unique name of the ln-client doing that subscription (check the ln_manager’s “clients”-tab). specifying a <SUB-RATE> is only useful if that client does multiple subscriptions of the same topic (not many use-cases).

In the example, the file name terminaes with “pcap”, so that PCAP format is used. The logger can also save in Matlab’s .mat format or python’s .pickle.

At the time being, recording of complex data structures is probably done best with the LN recorder with its “lnrdb” format that can be converted into HDF5 if needed.

6.5.5. LN Recorder and Daemon Logger: When to use Which One

aspect	LN recorder	Daemon logger
max duration	huge; limited by file system	limited by ram of publisher-node
max data rate	limited by network	very fast; limited by shared memory
suitable for complex data	yes	less
stores meta data	yes	no
supported file formats	LNRDB, HDF5, PCAP	PICKLE, PCAP, MAT, RAW
memory-mapped view avilable	yes	no (could be done with RAW format)
typical usage	recording motions, I/O, workflow	debugging hardware drivers, controllers