.. index:: single: topic communication; example (Python) Topic Communication in Python ============================= .. contents:: .. note:: The example uses Python 3. If you need to know more about which versions of Python and/or C++ are supported, see section :ref:`tutorial/prerequisites/language_standards` in the tutorial. Section :ref:`tutorial/prerequisites/installation` gives detailes about the required installation. This section shows a minimal example of how :term:`processes ` can exchange information in :term:`real-time`, via :term:`topics ` which exchange :term:`messages ` via a :term:`publish/subscribe` pattern. (If you want to look up what these terms mean, they are defined in the :ref:`communication ` bullet point of the :doc:`introduction` chapter). To summarize, the main points are these: * different processes can exchange information via messages, which are compound data structures a fixed data type. * messages are associated with topics, which is basically a label for a specific stream of messages * one process can send messages, it is the :term:`publisher` in respect to that topic. * other processes (one or more) can receive messages, they are :term:`subscribers `. We will show now how that works. The first thing we need to do is to define what data elements the messages we want to send will contain. This is done using a piece of text information which we call a :term:`message definition`. Creating Message Definitions ---------------------------- .. index:: single: message definitions; simple example (Python) Message definitions are usually stored in plain text files which are part of the set-up of a distributed system. So, to create a message definition with name ``counter/count_message``, we create a plain ASCII text file with this content: .. literalinclude:: examples/quickstart/python/topics/ln_msg_defs/counter/count_message and store this text file in :file:`quickstart/python/topics//ln_msg_defs/counter/count_message`. What does the content of the message definition mean? it means that we have two fields: * time * num_counts and that the "time" field has the type `double` (which is a 64-bit floating point value), and the "num_counts" field has the type `uint32_t` (which is the C type name for an unsigned integer value with 32 bits). So, we can imagine messages of this type as a packet of data which has the same structure as a C or C++ structure, or a python dictionary, or a python ``argparse.Namespace`` instance: Each name is mapped to a value, and each value has a fixed type. Except that we can use the definition in other languages such as python, that we can beam them back and forth between different processes and processes on different computers, and also that we can even share it between systems which have a different :term:`endianness`, for example. The Publisher Python Script --------------------------- .. index:: single: topic publisher; quickstart example (Python) create a python script :file:`quickstart/python/topics//python/counter_publisher.py` like this: .. literalinclude:: examples/quickstart/python/topics/python/counter_publisher.py :language: python :linenos: Let's go just through a few interesting aspects of the script: * In line 2, the ``links_and_nodes`` python module is imported as ``ln``. The short name is useful to save typing and avoid wildcard imports. We suggest to always use the name ``ln``, to make reading code easier for other people. .. only:: rmc .. note:: The ``links_and_nodes`` module needs to be provided by the package manager that you use. If you use Conan, you can provide it using a conan python environment. Please look into the Conan examples for further information about this. * in line 4, an LN client is initialized. The parameter that is passed for the initialization is the name of the client, which is used by the LN Manager to display its state. * In line 5, a publisher port is initialized, using the ``ln.client.publish()`` method. The two arguments are the names of the topic which is used (``counter1.count_message``), and the name of the message definition (``counter/count_message``). Here, the name of the topic is almost equal to the name of the message definition. However this is optional: While it is often practical and makes things simpler to keep names in sync, they can be different. The reason for this is that message definitions and topic names serve two different purposes: The topic is like the frequency of a radio broadcast, which determines which content we will get where, and under which title. (For further reading, the section on :ref:`Message definitions and topics ` in the tutorial gives a more in-depth explanation of the relationships between message definitions, topics, and their names.) The message definition is, in contrast, just the *type* of the transmitted data. This means that one can use the same message definitions in different places. For example, a message definition which defines a vector of three floating point elements could be used to signal the place of a vehicle, its speed vector, its acceleration, and perhaps also the place it should go to. * after the initialization, the program starts its main processing loop. This is interspersed with any kind of activity (like the ``sleep`` command), and then generating and sending data. The latter is done in two steps: * the assignment to ``port.packet.time`` puts data into the message buffer. The elements of the message are member values of ``ports.packet``, and the ``time`` field is defined by the message definition and generated automatically when the port was initialized in line 5. In the same way, the assignment to ``port.packet.num_counts`` generates some data which is updated within each step. This value field does not need to be initialized in Python, because the packet values are initialized to zero. * When the message data is prepared, it is sent using the ``port.write()`` command. This function transmits the message and returns after the transmission has been initiated. The Subscriber Python Script ---------------------------- .. index:: single: topic subscriber; quickstart example (Python) The script :file:`quickstart/python/topics/python/counter_subscriber.py` is an example for a matching subscriber program: .. literalinclude:: examples/quickstart/python/topics/python/counter_subscriber.py :language: python :linenos: * The initialization of the program is very similar to the initialization of the publisher, with one difference: the method ``clnt.publish(...)`` is replaced by the method ``clnt.subscribe(...)``. This means that this port will wait for data which was written by the publisher script. .. index:: pair: initializing clients; timing and performance considerations .. note:: This initialization is done only once. This is recommended, since the initialization of a publisher or subscriber client (or other communication channels) can be much more time-consuming than using the communication link, even if it requires only very little code. * The main loop uses the ``port.read()`` method in order to receive data. If there is no data available, the client will block until this is the case. * When the ``port.read()`` method has completed, it returns a ``packet`` instance, whose members ``packet.time`` and ``packet.num_counts`` are populated with the values that have been sent by the publisher. Configuring the LN Manager for Message Exchange ----------------------------------------------- .. index:: triple: quickstart (Python); message definitions; configuring the LN manager triple: quickstart (Python); message definitions; LN manager configuration Now, we add a minimal configuration for the LN manager that supports exchange of these messages. First, we will do that in a bare minimum way, without using the manager for process management. After that, we will add automatic process management again. Create a ln-manager config file :file:`quickstart/python/topics/main.lnc`: .. literalinclude:: examples/quickstart/python/topics/main.lnc :language: lnc :linenos: This creates a unique instance name and manager listening port. As in the previous `quickstart example on basic process configuration `, it sets the instance name and the port number derived from environment variables. In addition to that example, there is also a directive which tells the LN manager where it can find the message definition which we defined in the first step. This is necessary because the LN manager needs to start some background processing that supports message exchange, and for this it needs to know the message definitions which we are using. .. index:: pair: quickstart example (Python); %(CURDIR) variable The ``%(CURDIR)`` expression defines the folder in which the search path starts, which is the folder in which the configuration file which we are discussing is included. Because we have put the message definition into the folder "ln_msg_defs`, we need to add it. Once the LN manager knows this folder, it and the client programs can find the correct message definition, as the rest of the path is defined in the name of the message definition. (And, of course, it is possible to use multiple folders with message definitions, so that we can compose larger systems from smaller ones). Running Publisher and Subscriber with manual Process Management ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. index:: pair: quickstart (Python); process management; manual Start the ln_manager in one terminal: .. code:: bash ln_manager -c quickstart/python/topics/main.lnc .. on rmc-host: .. source rmc/conan_source ln_manager .. cd documentation/examples .. ../../python/links_and_nodes_manager/ln_manager -c quickstart/python/topics/main.lnc The manager will open its GUI as shown below and output a log-message similar to ``listening on ':' for ln-clients!``, we need the ````. .. figure:: images/quickstart_ln_manager_on_start.png The LN manager serving a publisher and subscriber In another terminal, start the publisher: .. code:: bash LN_MANAGER=localhost: python3 quickstart/python/topics/python/counter_publisher.py .. on rmc-host: source rmc/conan_source ln_manager export PYTHONPATH=$(pwd)/python:$PYTHONPATH cd documentation/examples If all goes right, no output should be generated. In a third terminal, now start the subscriber: .. code:: bash LN_MANAGER=localhost: python3 quickstart/python/topics/python/counter_subscriber.py The process should output lines like this:: time: 1647435215.4, num_counts: 5513 ... Within the ln-manager GUI, when clicking on the "topics" tab, you should see our ``counter`` topic published with a rate of ~10Hz as shown below: .. figure:: images/quickstart_ln_manager_with_subscriber.png The counter topic, when selecting the "topics" tab. This shows in the top half a list of topics. In the bottom half, after selecting the "counter" topic, it shows a list of the clients and the averaged frequency of exchanged messages. Delegating the Process Management to the LN Manager ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. index:: pair: quickstart (Python); process management; by LN manager pair: quickstart (Python); process management The example above is not very practical, because we need to start the processes with the right port number, so that they can reach the manager. We can do this with less manual typing, and in a more comfortable way, when we use the LN manager for process management, as shown in section :ref:`Process Management`. To do this, we can put those command lines in the ln-manager config file so that we don't have to remember them: .. literalinclude:: examples/quickstart/python/topics/topics_with_process_definitions.lnc :linenos: :language: lnc The example uses a few additional flags for the processes. The details are explained in the reference part in :doc:`config_process`. To explain and highlight the flags and attributes used here: * as before, each process has a name, which is given at the start of each process section * the ``change_directory`` directive defines the working directory of the process. This makes it easy to start the script, whose name is a parameter with a relative path to the python command. * The LN manager keeps the :term:`environment` of each process as clean as possible. However, certain environment variables need to be passed, so that the python script can find module dependencies and shared libraries. These are defined with the directive ``pass_environment``, which tells the LN Manager to pass the environment variables ``PATH, PYTHONPATH``, and ``LD_LIBRARY_PATH`` to the defined process. .. only:: rmc *Note on the RMC packaging system:* The environment could also be managed by the packaging system. Especially, Cissy/Conan pass information about library and executable paths in the environment, and pass them automatically to LN, using the ``cissy run`` command and cissy workspaces. Please refer to the cissy examples for detailed examples on this. .. index:: pair: use_execvpe flag; quickstart example pair: finding programs in PATH; quickstart example pair: passing environment variables; quickstart example * the directive ``add flags: use execvpe`` tells the manager to use the :envvar:`PATH` environment variable to find and start the process (instructing the operating system kernel to do so), and to pass it the environment values specified via the ``pass_environment`` directive. * as shown in the previous quickstart chapter :doc:`quickstart_python_process_management` in section :ref:`quickstart/basic_process_configuration`, we can (and probably also should) define dependencies between processes. To keep it short, we skip over this aspect here; however if you want to know more on dependencies in this case, this aspect is discussed in section :ref:`tutorial_python/designing-dependencies`. After starting both processes you should get something like this: .. figure:: images/quickstart_ln_manager_topics_processes_running.png Processes communicating as publisher and subscribe which are managed by the LN Manager A final note: We have simplified a lot to make the main points clearer. The :doc:`user_guide` gives a lot more information on details. One thing worth noting here is that a process can be both a subscriber and a publisher, and can of course access more than one topic. .. seealso:: See section :ref:`guide/hints_on_using_message_definitions` for some hints on how to use message definitions effectively.