.. _quickstart/python/services: .. default-domain:: py ######################################### Calling and handling Services from Python ######################################### .. contents:: .. index:: pair: LN services (Python tutorial); quickstart pair: LN services(Python tutorial); motivation see: services; LN services pair: LN services; main principle What can LN Services provide? ============================= This quickstart example shows how to define and call :term:`LN services ` from Python. Services are different from LN messages. While the latter are fire-and-forgot messages, services work much like a function call: Running code calls another piece of code with some parameters, and as a result, it gets one or more return values back. While this happens, the current flow of execution is paused, so the programmer does not need to handle the result concurrently, resulting in an easy control flow. The *differences* to normal function calls are three-fold: 1. the code providing the computation can run in another :term:`process` 2. the code can also be written in another language 3. the code can also run on another computer, as long as a :term:`network` connection exists. To achieve these things, links and nodes uses, again, :term:`messages `, which are however defined slightly different. They have slightly different message definitions, called service message definitions. Therefore, we will start out again with these. .. index:: pair: quickstart (Python); simple example services Our example: Geometric heavy lifting ==================================== .. index:: single: LN services; example Let's assume we have a process which needs to make some really complicated geometry calculations. For the purpose of this example, let's just show how to compute areas of a circle, a triangle, or an ellipse, so that we can keep the sample code short. (In practice, it can of course be any kind of computation, or object-oriented method call.) Defining message definitions for service calls =============================================== .. index:: pair: LN services; message definitions To call services for this computation, we first need to create message definitions with the parameters and return values of each computation. As mentioned, this message definition is a bit different from the former examples. Specifically, it has two distinguished sections, one under the label ``request``, and one under the name ``response``. As you might guess, the message parts under "request" are the *input* or *call parameters* of the computation, and the message parts under "response" are the *return parameters*. Create a sub-folder :file:`msg_defs/area_service/`, and create three message definitions like that: 1. the file :file:`circle_area`: .. literalinclude:: examples/quickstart/python/services/msg_defs/area_service/circle_area :language: python To compute the area of an circle, one only needs to know its radius as an input parameter. The resulting area is the return parameter. Both parameters are of type ``double``, which is a standard 64-bit floating point number (as it is used in C/C++). Because errors can happen almost everywhere in code, we add two fields which are needed for error handling: an ``error_code`` field, and an ``error_message`` field, which is basically an array of bytes with a variable length. The length is stored into ``error_message_len`` (when using the Python API, this happens automatically). We create a second file named :file:`triangle_area`: .. literalinclude:: examples/quickstart/python/services/msg_defs/area_service/triangle_area :language: python In the case of an triangle, we need to know the base width and the height, for example. For the third case, we create a file :file:`ellipse_area`: .. literalinclude:: examples/quickstart/python/services/msg_defs/area_service/ellipse_area :language: python Here, ``a`` and ``b`` mean the smaller and the larger semi-axis of the ellipse, which is a popular notation (if we wanted, we could of course also use longer variable names, to support our memory about the meaning). .. index:: single: LN services; provider example in Python Defining a service provider in Python ===================================== Now, we need to define code which actually does the computation. As before, we write it in Python, and give it the name :file:`ln_quickstart_services/provider.py`. .. index:: triple: quickstart (Python); service provider; initialization Initialization -------------- We start with the initialization of the provider class. Here is how it looks: .. literalinclude:: examples/quickstart/python/services/provider.py :language: python :linenos: :emphasize-lines: 13, 15, 21, 27, 18, 24, 30 :start-at: #!/usr/bin :end-before: def compute_circle_area This is the code which initializes the service provider class and its handlers for service message requests. To explain it very briefly: * We first define a class which provides handler methods, so-called :term:`service handlers `. * in line 13, it initializes an :py:class:`links_and_nodes.client` object, which is used to manage all the communication. * in lines 15, 21, and 27, it initializes three :term:`LN service provider` handles, using the method :py:meth:`links_and_nodes.client.get_service_provider()`. The two parameters of the method calls are each time first the name of the service call that is provided, and the name of a :term:`service message definition`. Both names need to match between service provider and service client. In our case, the message definition names are the message definitions which we defined above, and the names passed do not include the folder in which the message definition is stored. * then, in lines 18, 24 and 30, a handler method is stored for each service call that the class wants to respond, using the method :py:meth:`links_and_nodes.service_provider.set_handler()`. That handler method is a Python function, in our case it is simply a method of the class ``area_provider`` that we will define shortly. * as the final part of the initialization, the method :py:meth:`links_and_nodes.service_provider.do_register()` actually registers the service provider with the LN manager, so that service name, message definition name, and the handler function that implements it are logically joined. The single parameter of :py:meth:`links_and_nodes.service_provider.do_register()` is the name of the :term:`service group`, which is here the default service group, so all call handlers will run in the main thread of our process. .. Tip:: Note that while in many cases, names of services and message definitions will be almost identical, they can be different. Specifically, as message definitions define simple data types, these can also originate from a system library or another package which provides a unique definition. (For further reading, the section on :ref:`Message Definitions and Topics ` in the tutorial gives a more in-depth explanation of the relationships between message definitions, topics, and their names.) .. index:: triple: quickstart (Python); service provider; implementing a handler Handler Methods --------------- Now, how do the service handlers look like? They are defined this way: .. literalinclude:: examples/quickstart/python/services/provider.py :language: python :lineno-match: :linenos: :start-at: def compute_circle_area :end-before: def run(self) Each handler method has three parameters ``conn``, ``req``, and ``resp``. The first is an object that represents the service call connection. The connection is essentially an object which offers several methods that can handle (respond, abort, and so on) a service call. The second parameter, named ``req``, is an object that represents the call parameters that are transmitted with the service call - each member of ``req`` corresponds to one member or element of the "request" part of the message definition. The third parameter, called ``resp``, represents the response data structure or response buffer. Its members correspond to the "response" section of the associated service message definition. Before responding the call, the results are assigned to the members of that structure. In the three methods of our example, called ``compute_circle_area()``, ``compute_triangle_area()``, and ``compute_ellipse_area()``, the different formulas for each geometric figure are evaluated. We see that in each method, in the first part, the input parameters are checked for sanity: For example, in lines 34 to 40 of ``compute_circle_area()``, the radius of the circle is checked whether it is smaller than zero; if this is the case, the fields ``resp.error_message`` and ``resp.error_code`` are set, so that the caller of the service method can know what went wrong. Note that even in this branch of the code path, the ``respond()`` method is called, since the error result needs to be communicated back to the client. If, in turn, all parameters are correct, the computed value is assigned to ``resp.area``. When all response fields are assigned, the method :py:meth:`links_and_nodes.connection.respond()` is called, which returns the response data to the caller, and finishes the request. In the handler method ``compute_ellipse_area()``, we see another case that we need to cover: Here, the computation of the area of an ellipse is done by calling a library function, which could throw an exception (in our example, we emulate an external library function by calling into the function ``lib_compute_ellipse_area()``, but it could be of course any function defined in some external library). In that case, we need to catch any exception that could occur, as is done in the ``try.... except`` clause in lines 127 to 137, so that the error is caught and a sensible error message is returned to the client. Otherwise, the client might receive an error message that could not be comprehensible at all, which is generated by the LN service handler code when it catches the exception on the run. .. index:: pair: quickstart (Python); running service handler Running the Handlers -------------------- As the final parts of the service provider script, we need one more method definition, and a function call that runs the provider. This is done like shown here: .. literalinclude:: examples/quickstart/python/services/provider.py :language: python :linenos: :lineno-match: :start-at: def run(self) .. index:: pair: quickstart (Python); service group The function which actually performs the handling of service requests has the name :meth:`links_and_nodes.client.wait_and_handle_service_group_requests()`. Here, the first parameter is the group of service handlers which will be run to serve a request. We set it to the same :term:`service group` name as we have passed in the calls to ``do_register()`` above. The second parameter is the maximum waiting time which the call will wait for a request to arrive. In our example, the service handlers will run in the main thread of the program, which keeps things nicely simple. Avoid polling the same service group from multiple application threads. LN serializes concurrent calls internally, but one dispatcher thread per service group is the recommended structure. Use a service group thread pool if handlers should run concurrently. .. index:: pair: quickstart (Python); multi-threading pair: quickstart (Python); running service handlers in parallel threads .. note:: It is also possible to run service handlers in multiple threads. However, this requires :term:`thread safe ` programming, which becomes quickly very difficult and has a high risk of difficult-to-debug errors like :term:`race conditions `. For technical reasons, pure Python code is also unlikely to run faster in several threads. Therefore, it is strongly recommended to use simpler means unless processing in multiple threads is absolutely required. [#the-problem-with-threads]_ .. index:: single: LN services; client example in Python triple: service client; wrap_service; quickstart example .. index:: pair: quickstart (Python); service client example pair: quickstart (Python); calling an LN service Calling services from Python ============================ .. index:: pair: quickstart (Python); initializing an LN client Instantiating the Client ------------------------ As the counterpart, we need, of course, code which *calls* the computation. For this, we create a service client program :file:`ln_quickstart_services/client.py` like this: .. literalinclude:: examples/quickstart/python/services/client.py :language: python :emphasize-lines: 11-19 :linenos: Here, the method call :py:meth:`links_and_nodes.client.get_service()` has again two parameters, the first is, exactly as before, the name of the service method that is provided (which here is equal to the the function name at the provider side), and the second is the used service message definition. The :py:meth:`links_and_nodes.client.get_service()` method returns again a service handle of type :py:class:`links_and_nodes.service` for the client. .. Attention:: This retrieval of the handle should only be done once during the initialization of the client program, because it can take much longer than the ongoing actual calls! .. index:: pair: quickstart (Python); executing a service call Calling the Service from Client Methods --------------------------------------- Now, we can write methods which call the service. As an example, we can look at the definition of ``compute_circle_area()``: It has the needed parameter ``radius``. For performing the call, it first creates a shortcut name to the service client handle of type :py:class:`links_and_nodes.service`, which we call ``svc``. After that, it can assign the *request parameters* to the member fields of :py:attr:`links_and_nodes.service.req`. In our case, this is ``svc.req.radius``. Then, the service is actually called by calling :py:meth:`links_and_nodes.service.call()`, which sends the data to the service provider, waits for the result of the computation, and transports it back - including any possibly generated error messages. The response data is in the member fields of ``svc.resp``, which is of type :py:attr:`links_and_nodes.service.resp`. To check for the error, we just look whether the value ``svc.resp.error_code`` is larger than zero; if this is the case, we can use ``svc.resp.error_message`` in order to print a diagnostic message, for example. (In many cases, idiomatic Python code would raise a suitable exception here that signals a failure - doing this is however within the responsibility of the service client). .. index:: pair: quickstart (Python); configuring services in LN Configuring services support in the links and nodes manager =========================================================== .. index:: pair: LN services; LN manager configuration see: LNM configuration; LN manager configuration see: LNM; LN manager Finally, we need to write a bit of configuration for the links and nodes manager to connect all three of message definitions, service provider, and service client. This looks like this: .. literalinclude:: examples/quickstart/python/services/sample_services.lnc :language: lnc :linenos: We can now run this sample by starting the LN manager like this: .. code:: bash ln_manager -c sample_services.lnc In our service script, the different methods are called simply a number of times, before the program exits. Of course, we could do many more complex things, like doing calls in an endless processing loop, or calling computations from different threads. But in essence, the above is how service calls work. Compatibility between Client and Provider Implementations --------------------------------------------------------- .. index:: pair: quickstart (Python); testing service clients and providers pair: service clients; cross-compatibility between languages pair: quickstart (Python); compatibility between LN versions One last interesting and often very useful thing we want to mention is that you can **freely mix service clients and provider implementations written in different languages**, such as between Python 3 and Python 2, or Python 3 and C++, as long as you use exactly the same message definitions, and the same names for the services. This is often useful when we write a draft or concept implementation of an algorithm in Python, and need to re-implement it in C++: We can test, for example, the new service provider written in C++ against the old service client written in Python, and verify that it has the same result. .. index:: pair: tutorial (Python); compatibility between LN versions It is also useful to keep in mind that the messaging protocol remains fully compatible (that is, both forward-compatible and backwards-compatible) between different LN versions. This means that you can mix clients with newer and older versions in one and the same system, and even written in different language standards, such as Python 2 and Python 3, or C++98, C++11, and C++17. .. Tip:: In some situations, you might have to define a lot of services with similar code, which can result in a large amount of boilerplate code. In this case, you can use auto-generated wrapper code which is explained in the chapter :doc:`quickstart_python_services_wrapped-api`. One caveat: While this code is shorter, it is not simple to explain and define its interface concisely. This "wrapped" interface has a much larger surface, and its exact behavior is probably harder to understand, which is the reason it is not part of the default introduction presented here. ----- .. TODO: do we need more samples for service / quickstart? See below > Haben wir eigentlich ein gutes Beispiel zu den Services? Vielleicht > ist das hier der beste Ausgangspunkt? ich habe letzte woche erst diese beispiele geschrieben: https://rmc-github.robotic.dlr.de/common/verbose/tree/release/4.0.0/example_pages/ln%20examples die verwenden allerdings einen duennen wrapper um den ln.client herum: https://rmc-github.robotic.dlr.de/common/verbose/blob/release/4.0.0/stdpages/lib/links%20and%20nodes/ln%20interface.py aber fuer das service beispiel spiel das keine rolle: call set_label.py label provider.py ist ein einfaches beispiel fuer einen service client und provider. (in verbose wird nur sichergestellt das da eine "main integration" verwendet wird, damit die gtk gui nicht blockiert wird...) das ist aber evtl kein ebsonders gutes beispiel fuer eine service msg-def, weil dort das "ln2/pyobject" verwendet wird, und das hat ein bisschen "magie" im python-layer... ansonsten waer evtl etwas abgeleitet von den automatischen tests bruachbar: hier ein provider: https://rmc-github.robotic.dlr.de/common/links_and_nodes/blob/release/2.1.0/tests/console/provider.py in dem beispiel sieht mach auch einmal die "direkte" ln-service-provider api mit "get_service_provider(), .set_handler(), .do_register() und im handler dann mit den argumenten (request, req_data, resp_data)). damit sollte man wahrscheinlich anfangen, weil es die zu grunde legende api ist. es gibt dann ontop noch helfer/wrapper welche die request-felder versucht als funktions-argumente im python interpreter abzubilden, mit python-ublichen datenstrukturen wie dicts & lists. (siehe dort self.wrap_service_provider()-calls) damit kann man etwas mehr pythonic-feeling erreichen. aber ist vielleicht eher was fuer das 2. beispiel und nicht fuer das erste. ...sag bescheid wenn du fragen hast oder wir uns das mal zusammen anschauen sollen. .. rubric:: Footnotes .. [#the-problem-with-threads] For a substantial explanation why, see for example `The Problem with Threads by Edward A. Lee `_, or alternatively start reading from `typical questions on stack overflow `_.