hermes-python 0.8.1
Python bindings for Snips Hermes Protocol
License: MIT
Keywords: snips

Hermes Python
*************

.. image:: https://travis-ci.org/snipsco/hermes-protocol.svg
   :target: https://travis-ci.org/snipsco/hermes-protocol

.. image:: https://badge.fury.io/py/hermes-python.svg
   :target: https://badge.fury.io/py/hermes-python


About
*****

The ``hermes-python`` library provides python bindings for the Hermes
protocol that snips components use to communicate together over MQTT.
``hermes-python`` allows you to interface seamlessly with the Snips
platform and kickstart development of Voice applications.

``hermes-python`` abstracts away the connection to the MQTT bus and
the parsing of incoming and outcoming messages from and to the
components of the snips platform.


Requirements
************

Pre-compiled wheels are available for Python 2.7+ and Python 3.5

The pre-compiled wheels supports the following platform tags :

* ``manylinux1_x86_64``

* ``armv7l``, ``armv6``

* ``macos``

If you want to install ``hermes-python`` on another platform, you have
to build it from source.


Installation
************

The library is packaged as a pre-compiled platform wheel, available on
`PyPi <https://pypi.org/project/hermes-python/>`_.

It can be installed with : ``pip install hermes-python``.

Or you can add it to your *requirements.txt* file.


Building from source
********************

If you want to use ``hermes-python`` on platforms that are not
supported, you have to manually compile the wheel.

You need to have ``rust`` and ``cargo`` installed :

``curl https://sh.rustup.rs -sSf``

Clone, the ``hermes-protocol`` repository :

::

   git clone git@github.com:snipsco/hermes-protocol.git
   cd hermes-protocol/platforms/hermes-python

You can then build the wheel :

::

   virtualenv env
   source env/bin/activate
   python setup.py bdist_wheel

The built wheels should be in ``platforms/hermes-python/dist``

You can install those with pip : ``pip install
platforms/hermes-python/dist/<your_wheel>.whl``


Advanced wheel building
***********************

We define a new API for including pre-compiled shared objects when
building a platform wheel.

::

   python setup.py bdist_wheel

This command will compile the ``hermes-mqtt-ffi`` Rust extension, copy
them to an appropriate location, and include them in the wheel.

We introduce a new command-line argument : ``include-extension`` which
is a way to include an already compiled (in previous steps)
``hermes-mqtt-ffi`` extension in the wheel.

Its usage is the following : ``include-extension=<default |
the/path/to/your/extension.[so|dylib]>``

For instance :

::

   python setup.py bdist_wheel --include-extension=default

The default value for ``include-extension`` will look up for
pre-compiled extension in the default paths (in
``hermes-protocol/target/release/libhermes_mqtt_ffi.[dylib|so]`` and
``hermes-protocol/platforms/hermes-python/hermes_python/dylib``).

::

   python setup.py bdist_wheel --include-extension=<the/path/to/your/extension.[so|dylib]>

When doing x-compilation, you can also specify the target platform :

::

   python setup.py bdist_wheel --include-extension=<the/path/to/your/extension.[so|dylib]> --plat-name=<the_platform_tag>


Tutorial
********

The lifecycle of a script using ``hermes-python`` has the following
steps :

* Initiating a connection to the MQTT broker

* Registering callback functions to handle incoming intent parsed by
   the snips platform

* Listening to incoming intents

* Closing the connection

Let’s quickly dive into an example :

Let’s write an app for a Weather Assistant ! This code implies that
you created a weather assistant using the `Snips Console
<https://console.snips.ai/>`_, and that it has a
``searchWeatherForecast`` intent. Or you could download `this weather
Assistant
<https://resources.snips.ai/assistants/assistant-weather-EN-0.19.0-dyn-heysnipsv4.zip>`_
.

::

   from hermes_python.hermes import Hermes

   MQTT_ADDR = "localhost:1883"        # Specify host and port for the MQTT broker

   def subscribe_weather_forecast_callback(hermes, intent_message):    # Defining callback functions to handle an intent that asks for the weather.
       print("Parsed intent : {}".format(intent_message.intent.intent_name))

   with Hermes(MQTT_ADDR) as h: # Initialization of a connection to the MQTT broker
       h.subscribe_intent("searchWeatherForecast", subscribe_weather_forecast_callback) \  # Registering callback functions to handle the searchWeatherForecast intent
            .start()
       # We get out of the with block, which closes and releases the connection.

This app is a bit limited as it only prints out which intent was
detected by our assistant. Let’s add more features.


Handling the ``IntentMessage`` object
=====================================

In the previous example, we registered a callback that had this
signature.

::

   subscribe_intent_callback(hermes, intent_message)

The ``intent_message`` object contains information that was extracted
from the spoken sentence.

For instance, in the previous code snippet, we extracted the name of
the recognized intent with

::

   intent_message.intent.intent_name

We could also retrieve the associated confidence score the NLU engine
had when classifying this intent with

::

   intent_message.intent.confidence_score


Extracting slots
----------------

Here are some best practices when dealing with slots. The
``IntentMessage`` object has a ``slots`` attribute.

This ``slots`` attributes is a **container** that is empty when the
intent message doesn’t have slots :

::

   assert len(intent_message.slots) == 0

This container is a dictionary where the key is the name of the slot,
and the value is a list of all the slot values for this slot name.

You can access these values in two ways :

::

   assert len(intent_message.slots.slot1) == 0
   assert len(intent_message.slots["slot1"]) == 0

The slot values are of type ``NluSlot`` which is a deeply nested
object, we offer convenience methods to rapidly access the
*slot_value* attribute of the *NluSlot*.

To access the first ``slot_value`` of a slot called ``myslot``, you
can use :

::

   intent_message.slots.myslot.first()

You can also access all the ``slot_value`` of a slot called ``myslot``
:

::

   intent_message.slots.myslot.all()

Let’s add to our Weather assistant example.

We assume that the ``searchWeatherForecast`` has one slot called
``forecast_location``, that indicates which location the user would
like to know the weather at.

Let’s print all the ``forecast_location`` slots :

::

   for slot in intent_message.slots.forecast_location:
       name = slot.slot_name
       confidence = slot.confidence_score
       print("For slot : {}, the confidence is : {}".format(name, confidence))

The *dot* notation was used, but we can also use the dictionary
notation :

::

   for slot in intent_message.slots.forecast_location:
       name = slot["slot_name"]
       print(name)

Some convenience methods are available to easily retrieve slot values
:

*Retrieving the first slot value for a given slot name*

::

   slot_value = intent_message.slots.forecast_location.first()

*Retrieving all slot values for a given slot name*

::

   slot_values = intent_message.slots.forecast_location.all()

Coming back to our example, we can now have the app print the
``forecast_location`` slot value back to the user :

::

   def subscribe_weather_forecast_callback(hermes, intent_message):
       slot_value = intent_message.slots.forecast_location.first().value
       print("The slot was : {}".format(slot_value)


Managing sessions
=================

The Snips platform includes support for conversations with back and
forth communication between the Dialogue Manager and the client code.
Within the Snips platform, a conversation happening between a user and
her assistant is called a session.

In this document, we will go through the details of how to start,
continue and end a session.

In its default setup, you initiate a conversation with your assistant
by pronouncing the defined wake-word. You say your request out-loud,
an intent is extracted from your request, and triggers the portion of
the action code you registered to react to this intent. Under the
hood, the Dialogue Manager starts a new **session** when the wake-word
is detected. The session is then ended by the action code.


Starting a session
------------------

A session can be also be started programmatically. When you initiate a
new session, the Dialogue Manager will start the session by asking the
TTS to say the text (if any) and wait for the answer of the end user.

You can start a session in two manners :

* with an action

* with a notification

When initiating a new session with an action, it means the action code
will expect a response from the end user.

For instance: You could have an assistant that books concerts tickets
for you. The action code would start a session with an action, having
the assistant asking for what band you would like to see live.

When initiating a new session with a notification, it means the action
code only inform the user of something without expecting a response.

For instance: Instead of pronouncing your defined wake-word, you could
program a button to initiate a new session.

Let’s build up on our previous example of an assistant that book
concerts tickets for you. Here, we are going to initiate a new session
with an **action**, filtering on the intent the end-user can respond
with.

::

   from hermes_python.hermes import Hermes, MqttOptions

   with Hermes(mqtt_options=MqttOptions()) as h:
       h.publish_start_session_action(None,
           "What band would you like to see live ?",
           ["findLiveBands"],
           True, False, None)

Let’s say that we added a physical button to initiate a conversation
with our concert tickets booking assistant. We could use this button
to initiate a new session and start talking immediately after pressing
the button instead of relying on triggering a wake-word.

When the button is pressed, the following code could be ran :

::

   hermes.publish_start_session_notification("office", None, None)

This would initiate a new session on the ``office`` site id.


Ending a session
----------------

To put an end to the current interaction the action code can terminate
a started session. You can optionally terminate a session with a
session with a message that should be said out loud by the TTS.

Let’s get back to our concert tickets booking assistant, we would end
a session like this :

::

   from hermes_python.hermes import Hermes, MqttOptions


   def find_shows(band):
       pass


   def findLiveBandHandler(hermes, intent_message):
       band = intent_message.slots.band.first().value
       shows = find_shows(band)
       hermes.publish_end_session(intent_message.session_id, "I found {} shows for this band !".format(len(shows)))


   with Hermes(mqtt_options=MqttOptions()) as h:
       h\
           .subscribe_intent("findLiveBand", findLiveBandHandler)\
           .start()


Continuing a session
--------------------

You can programmatically extend the lifespan of a dialogue session,
expecting interactions from the end users. The typical use of
continuing a session is for your assistant to ask additional
information to the end user.

Let’s continue with our concert tickets booking assistant, after
starting a session, we will continue a session, expecting the user to
tell us how many tickets the assistant should buy.

::

   import json
   from hermes_python.hermes import Hermes, MqttOptions

   required_slots = {  # We are expecting these slots.
       "band": None,
       "number_of_tickets": None
   }

   def ticketShoppingHandler(hermes, intent_message):
       available_slots = json.loads(intent_message.custom_data)

       band_slot = intent_message.slots.band.first().value or available_slots["band"]
       number_of_tickets = intent_message.slots.number_of_tickets.first().value or available_slots["number_of_tickets"]

       available_slots["band"] = band_slot
       available_slots["number_of_tickets"] = number_of_tickets

       if not band_slot:
           return hermes.publish_continue_session(intent_message.session_id,
                                                  "What band would you like to see live ?",
                                                  ["ticketShopping"],
                                                  custom_data=json.dumps(available_slots))

       if not number_of_tickets:
           return hermes.publish_continue_session(intent_message.session_id,
                                                  "How many tickets should I buy ?",
                                                  ["ticketShopping"],
                                                  custom_data=json.dumps(available_slots))

       return hermes.publish_end_session(intent_message.session_id, "Ok ! Consider it booked !")


   with Hermes(mqtt_options=MqttOptions("raspi-anthal-support.local")) as h:
       h\
           .subscribe_intent("ticketShopping", ticketShoppingHandler)\
           .start()


Slot filling
------------

You can programmatically continue a session, and asking for a specific
slot. If we build on our previous example, we could continue a dialog
session by specifying which slot the assistant expects from the
end-user.

::

   import json
   from hermes_python.hermes import Hermes, MqttOptions

   required_slots_questions = {
       "band": "What band would you like to see live ?",
       "number_of_tickets": "How many tickets should I buy ?"
   }

   def ticketShoppingHandler(hermes, intent_message):
       available_slots = json.loads(intent_message.custom_data)

       band_slot = intent_message.slots.band.first().value or available_slots["band"]
       number_of_tickets = intent_message.slots.number_of_tickets.first().value or available_slots["number_of_tickets"]

       available_slots["band"] = band_slot
       available_slots["number_of_tickets"] = number_of_tickets

       missing_slots = filter(lambda slot: slot is None, [band_slot, number_of_tickets])

       if len(missing_slots):
           missing_slot = missing_slots.pop()
           return hermes.publish_continue_session(intent_message.session_id,
                                                  required_slots_questions[missing_slot],
                                                  custom_data=json.dumps(available_slots),
                                                  slot_to_fill=missing_slot)
       else:
           return hermes.publish_end_session(intent_message.session_id, "Ok ! Consider it booked !")


   with Hermes(mqtt_options=MqttOptions("raspi-anthal-support.local")) as h:
       h\
           .subscribe_intent("ticketShopping", ticketShoppingHandler)\
           .start()


Dynamic Vocabulary using Entities Injection
===========================================

Please refer to the `official documentation
<https://docs.snips.ai/articles/platform/nlu/dynamic-vocabulary>`_ for
further information.

Sometimes, you want to extend your voice assistant with new vocabulary
it hasn’t seen when it was trained. For instance, let’s say that you
have a bookstore voice assistant, that you update every week with new
book titles that came out.

The snips platform comes with the **Entities Injection** feature,
which allows you to update both the ASR and the NLU models directly on
the device to understand new vocabulary.

Each intent within an assistant may contain some slots, and each slot
has a specific type that we call an entity. If you have a book_title
entity that contains a list of book titles in the inventory of your
book store, Entities Injection lets you add new titles to this list.

To inject new entity values, you have multiple operations at your
disposal :

* ``add`` adds the list of values that you provide to the existing
   entity values.

* ``addFromVanilla`` removes all the previously injected values to
   the entity, and then, adds the list of values provided. Note that
   the entity values coming from the console will be kept.

Let’s see how an injection would be performed by the action code :

::

   from hermes_python.hermes import Hermes
   from hermes_python.ontology.injection import InjectionRequestMessage, AddInjectionRequest, AddFromVanillaInjectionRequest

   def retrieve_new_book_releases():
       return ["The Half-Blood Prince", "The Deathly Hallows"]


   def retrieve_book_inventory():
       return ["The Philosopher's Stone", "The Chamber of Secrets", "The Prisoner of Azkaban", "The Goblet of Fire",
               "The Order of the Phoenix", "The Half-Blood Prince", "The Deathly Hallows"]


   # First example : We just add weekly releases

   operations =  [
       AddInjectionRequest({"book_titles" : retrieve_new_book_releases() }),
   ]

   request1 = InjectionRequestMessage(operations)

   with Hermes("localhost:1883") as h:
       h.request_injection(request1)


   # Second example : We reset all the previously injected values of the book_title entity, and then, adds the list of values provided

   operations =  [
       AddInjectionRequest({"book_titles" : retrieve_book_inventory() }),
   ]

   request2 = InjectionRequestMessage(operations)

   with Hermes("localhost:1883") as h:
       h.request_injection(request2)

**Careful**, performing an entity injection is a CPU and memory
intensive task. You should not trigger multiple injection tasks at the
same time on devices with limited computing power.

You can register a callback so that your code knows when an injection
process is completed :

::

   def injection_completed(hermes, injection_complete_message):
       print("The injection operation with id {} completed !".format(injection_complete_message.request_id))

   with Hermes("localhost:1883") as h:
       h.subscribe_injection_complete(injection_completed).request_injection(injection_request)

You can monitor the progress of your injection request with
``snips-watch -vvv``.

You can also reset the injected vocabulary of your assistant to its
factory settings using the ``request_injection_reset` method of
``hermes``. Since the operation of resetting the injection is
asynchronous, you can register a callback to know when the injection
reset process is completed :

::

   def injection_reset_completed(hermes, injection_reset_complete_message):
       print("The injection reset operation with id {} completed !".format(injection_reset_complete_message.request_id))

   with Hermes("localhost:1883") as h:
       h.subscribe_injection_reset_complete(injection_reset_completed).request_injection_reset(request)


Configuring MQTT options
========================

The connection to your MQTT broker can be configured with the
``hermes_python.ffi.utils.MqttOptions`` class.

The ``Hermes`` client uses the options specified in the
``MqttOptions`` class when establishing the connection to the MQTT
broker.

Here is a code example :

::

   from hermes_python.hermes import Hermes
   from hermes_python.ffi.utils import MqttOptions

   mqtt_opts = MqttOptions()

   def simple_intent_callback(hermes, intent_message):
       print("I received an intent !")

   with Hermes(mqtt_options=mqtt_opts) as h:
       h.subscribe_intents().loop_forever()

Here are the options you can specify in the MqttOptions class :

* ``broker_address``: The address of the MQTT broker. It should be
   formatted as ``ip:port``.

* ``username``: Username to use on the broker. Nullable

* ``password``: Password to use on the broker. Nullable

* ``tls_hostname``: Hostname to use for the TLS configuration.
   Nullable, setting a value enables TLS

* ``tls_ca_file``: CA files to use if TLS is enabled. Nullable

* ``tls_ca_path``: CA path to use if TLS is enabled. Nullable

* ``tls_client_key``: Client key to use if TLS is enabled. Nullable

* ``tls_client_cert``: Client cert to use if TLS is enabled. Nullable

* ``tls_disable_root_store``: Boolean indicating if the root store
   should be disabled if TLS is enabled.

Let’s connect to an external MQTT broker that requires a username and
a password :

::

   from hermes_python.hermes import Hermes
   from hermes_python.ffi.utils import MqttOptions

   mqtt_opts = MqttOptions(username="user1", password="password", broker_address="my-mqtt-broker.com:18852")

   def simple_intent_callback(hermes, intent_message):
       print("I received an intent !")

   with Hermes(mqtt_options=mqtt_opts) as h:
       h.subscribe_intents().loop_forever()


Configuring Dialogue
====================

``hermes-python`` offers the possibility to configure different
aspects of the Dialogue system.


Enabling and disabling intents on the fly
-----------------------------------------

It is possible to enable and disable intents of your assistant on the
fly. Once an intent is disabled, it will not be recognized by the NLU.

Note that intents in the intent filters of started or continued
session will take precedence over intents that are enabled/disabled in
the configuration of the Dialogue.

You can disable/enable intents with the following methods :

::

   from hermes_python.ontology.dialogue import DialogueConfiguration

   dialogue_conf = DialogueConfiguration()                          \
                           .disable_intent("intent1")               \
                           .enable_intent("intent2")                \
                           .enable_intents(["intent1", "intent2"])  \
                           .disable_intents(["intent2", "intent1"])

   hermes.configure_dialogue(dialogue_conf)


Configuring Sound Feedback
==========================


Enabling and disabling sound feedback
-------------------------------------

By default, the snips platform notify the user of different events of
its lifecycle with sound. It emits a sound when the wakeword is
detected, or when the NLU engine (natural understanding engine) has
successfuly extracted an intent from a spoken sentence.

``hermes-python`` allows to disable this sound feedback
programmatically, by sending a message to the snips platform,
specifying the ``siteId`` where the sound feedback should be disabled.

::

   from hermes_python.hermes import Hermes
   from hermes_python.ontology.feedback import SiteMessage

   with Hermes("localhost:1883") as h:
       h.disable_sound_feedback(SiteMessage("kitchen"))
       h.start()


Making the TTS play custom sounds
---------------------------------

The snips-platform allows you to register custom sounds which can be
played later by the TTS engine.

``hermes-python`` allows you to register sounds on the fly, by
specifying a string identifier for the sound, and providing a ``wav``
file.

For instance, let’s say that your assistant tells a bad joke and that
you want to play a *ba dum tss* sound at the end of the punchline.

::

   from builtins import bytearray
   from hermes_python.hermes import Hermes
   from hermes_python.ontology.tts import RegisterSoundMessage

   # Step 1 : We read a wav file
   def read_wav_data():
       with open('ba_dum_tss.wav', 'rb') as f:
           read_data = f.read()
       return bytearray(read_data)


   # Step 2 : We register a sound that will be named "bad_joke"
   sound = RegisterSoundMessage("bad_joke", read_wav_data())

   def callback(hermes, intent_message):
       hermes.publish_end_session(intent_message.session_id, "A very bad joke ... [[sound:bad_joke]]")  # Step 4 : You play your registered sound

   with Hermes("localhost:1883") as h:
       h.connect()\
           .register_sound(sound)\    # Step 3 : You register your custom sound
           .subscribe_intents(callback)\
           .start()

In the TTS string, when you specify the sound you want to play, you
need to follow the syntax : ``[[sound:<your_sound_id>]]``


Enabling Debugging
==================

You can debug ``hermes-python`` if you encounter an issue and get a
better stacktrace that you can send us.

To do so, you have to set the ``rust_logs_enabled`` flag to True when
you create an instance of the ``Hermes`` class :

::

   from hermes_python.hermes import Hermes

   def callback(hermes, intent_message):
       pass

   with Hermes("localhost:1883", rust_logs_enabled=True) as h:
       h.subscribe_intent("...", callback)
       h.start()

You should then execute your script with the ``RUST_LOG`` environment
variable : ``RUST_LOG=TRACE python your_script.py``.


Release Checklist
*****************

Everytime you need to perform a release, do the following steps :

* [ ] Commit all changes to the project for said release

* [ ] Write all the changes introduced in the Changelog
   (source/HISTORY.rst file) and commit it

* [ ] Run tests

* [ ] Build the documentation and commit the README.rst

* [ ] Bump the version and commit it

* [ ] Upload to PyPI


Build details
*************


Creating macOS wheels
=====================

The build script : ``build_scripts/build_macos_wheels.sh`` uses
``pyenv`` to generate ``hermes-python`` wheels for different versions
of python.

To be able to run it, you need to :

* install pyenv : brew install pyenv. Then follow the additional
   steps detailled

* you then have to install python at different versions:

``pyenv install --list`` to list the available version to install *
Before installing and building the different python version from
sources, install the required dependencies : `Link here
<https://github.com/pyenv/pyenv/wiki/>`_

That’s it !
History
==========
0.8.1 (2019-10-03)
------------------
* Hotfix : adding back DialogueConfiguration in the main module + Conversion function for SessionTermination object

0.8.0 (2019-09-10)
------------------
* Adds subscription to injection lifecycle events : subscribe_injection_complete, subscribe_injection_reset_complete
* Adds a component field to the SessionTerminationType class
* Introduces alternatives intent resolutions
* Fixes folder creation issue when building the wheel from sources

0.7.0 (2019-05-14)
------------------
* Introduces Entities Injection API.

0.6.1 (2019-05-10)
------------------
* Introduces `register_sound` API

0.5.2 (2019-05-07)
------------------
* Fixes nullable fields in Dialogue ontology and brings more type annotations

0.5.1 (2019-05-06)
------------------
* introduces new (cli) API to build python wheel that include pre-compiled hermes-mqtt-ffi extension.

0.5.0 (2019-04-19)
-------------------
* Adds APIs to enable and disable sound feedback. 

0.4.1 (2019-03-29)
------------------
* Re-enables debugging of hermes-python with the `rust_logs_enabled` flag
* AmountOfMoneyValue, InstantTimeValue and DurationValue slot values now use Precision and Grain enumerations

0.4.0 (2019-03-20)
------------------
* Adds support to configure the Dialogue Mananger : enabling and disabling intents on the fly.
* Adds slot filling API : You can ask for a specific slot when continuing a session
* adding support for `OrdinalSlot`

0.3.3 (2019-03-06)
------------------
* Fixes a bug with `publish_start_session_notification` that didn't take the `text` parameter into account.

0.3.2 (2019-02-25)
------------------
* Fixes an important bug that gave the argument `hermes` the wrong type for every registered callback. 
* Fixes an important bug that caused the program to crash when parsing intentMessage that had no slots. 

0.3.1 (2019-02-25)
------------------
* Fixes import bug with templates, the `hermes_python.ffi.utils` module now re-exports `MqttOptions`

0.3.0 (2019-02-25)
------------------
* `IntentClassifierResult`'s `probability` field has been renamed to `confidence_score`.
* Introduces support for snips-platform `1.1.0 - 0.61.1`.

0.2.0 (2019-02-04)
------------------
* Introduces options to connect to the MQTT broker (auth + TLS are now supported).

0.1.29 (2019-01-29)
-------------------
* Fixes bug when deserializing `TimeIntervalValue` that used wrong `encode` method instead of `decode`.

0.1.28 (2019-01-14)
-------------------
* Fixes bug when the `__exit__` method was called twice on the `Hermes` class.
* Introduces two methods to the public api : `connect` and `disconnect` that should bring more flexibility

0.1.27 (2019-01-07)
-------------------
* Fixed broken API introduced in `0.1.26` with the publish_continue_session method of the Hermes class. 
* Cast any string that goes in the mqtt_server_adress parameter in the constructor of the Hermes class to be a 8-bit string.

0.1.26 (2019-01-02)
---------------------
* LICENSING : This wheel now has the same licenses as the parent project : APACHE-MIT. 
* Subscription to not recognized intent messages is added to the API. You can now write your own callbacks to handle unrecognized intents.  
* Adds send_intent_not_recognized flag to continue session : indicate whether the dialogue manager should handle non recognized intents by itself or sent them as an `IntentNotRecognizedMessage` for the client to handle.

0.1.25 (2018-12-13)
---------------------
* Better error handling : Errors from wrapped C library throw a LibException with detailled errors.