snips-nlu-metrics 0.15.0
Python package to compute NLU metrics
License: Apache 2.0
Keywords: metrics nlu nlp intent slots entity parsing

Snips NLU Metrics
=================

.. image:: https://travis-ci.org/snipsco/snips-nlu-metrics.svg?branch=master
    :target: https://travis-ci.org/snipsco/snips-nlu-metrics

.. image:: https://img.shields.io/pypi/v/snips-nlu-metrics.svg?branch=master
    :target: https://pypi.python.org/pypi/snips-nlu-metrics

.. image:: https://img.shields.io/pypi/pyversions/snips-nlu-metrics.svg?branch=master
    :target: https://pypi.python.org/pypi/snips-nlu-metrics


This tools is a python library for computing `cross-validation`_ and
`train/test`_ metrics on an NLU parsing pipeline such as the `Snips NLU`_ one.

Its purpose is to help evaluating and iterating on the tested intent parsing
pipeline.

Install
-------

.. code-block:: console

    $ pip install snips_nlu_metrics


NLU Metrics API
---------------

Snips NLU metrics API consists in the following functions:

* ``compute_train_test_metrics`` to compute `train/test`_ metrics
* ``compute_cross_val_metrics`` to compute `cross-validation`_ metrics

The metrics output (json) provides detailed information about:

* `confusion matrix`_
* `precision, recall and f1 scores`_ of intent classification
* precision, recall and f1 scores of entity extraction
* parsing errors

Data
----

Some sample datasets, that can be used to compute metrics, are available
`here <samples/>`_. Alternatively, you can create your own dataset either by
using ``snips-nlu``'s `dataset generation tool`_ or by going on the
`Snips console`_.

Examples
--------

The Snips NLU metrics library can be used with any NLU pipeline which satisfies
the ``Engine`` API:

.. code-block:: python

    class Engine:
        def fit(self, dataset):
            # Perform training ...
            return self

        def parse(self, text):
            # extract intent and slots ...
            return {
                "input": text,
                "intent": {
                    "intentName": intent_name,
                    "probability": probability
                },
                "slots": slots
            }


----------------
Snips NLU Engine
----------------

This library can be used to benchmark NLU solutions such as `Snips NLU`_. To
install the ``snips-nlu`` python library, and fetch the language resources for
english, run the following commands:

.. code-block:: bash

    $ pip install snips-nlu
    $ snips-nlu download en


Then, you can compute metrics for the ``snips-nlu`` pipeline using the metrics
API as follows:

.. code-block:: python

    from snips_nlu import SnipsNLUEngine
    from snips_nlu_metrics import compute_train_test_metrics, compute_cross_val_metrics

    tt_metrics = compute_train_test_metrics(train_dataset="samples/train_dataset.json",
                                            test_dataset="samples/test_dataset.json",
                                            engine_class=SnipsNLUEngine)

    cv_metrics = compute_cross_val_metrics(dataset="samples/cross_val_dataset.json",
                                           engine_class=SnipsNLUEngine,
                                           nb_folds=5)

-----------------
Custom NLU Engine
-----------------

You can also compute metrics on a custom NLU engine, here is a simple example:

.. code-block:: python

    import random

    from snips_nlu_metrics import compute_train_test_metrics

    class MyNLUEngine:
        def fit(self, dataset):
            self.intent_list = list(dataset["intents"])
            return self

        def parse(self, text):
            return {
                "input": text,
                "intent": {
                    "intentName": random.choice(self.intent_list),
                    "probability": 0.5
                },
                "slots": []
            }

    compute_train_test_metrics(train_dataset="samples/train_dataset.json",
                               test_dataset="samples/test_dataset.json",
                               engine_class=MyNLUEngine)

Links
-----
* `Changelog <CHANGELOG.md>`__
* `Bug tracker <https://github.com/snipsco/snips-nlu-metrics/issues>`__
* `Snips NLU <https://github.com/snipsco/snips-nlu>`__
* `Snips NLU Rust <https://github.com/snipsco/snips-nlu-rs>`__: Rust inference pipeline implementation and bindings (C, Swift, Kotlin, Python)
* `Snips <https://snips.ai/>`__

Contributing
------------
Please see the `Contribution Guidelines <CONTRIBUTING.rst>`_.

Copyright
---------
This library is provided by `Snips <https://www.snips.ai>`_ as Open Source software. See `LICENSE <LICENSE>`_ for more information.

.. _cross-validation: https://en.wikipedia.org/wiki/Cross-validation_(statistics)
.. _train/test: https://en.wikipedia.org/wiki/Training,_test,_and_validation_sets
.. _Snips NLU: https://github.com/snipsco/snips-nlu
.. _precision, recall and f1 scores: https://en.wikipedia.org/wiki/Precision_and_recall
.. _confusion matrix: https://en.wikipedia.org/wiki/Confusion_matrix
.. _dataset generation tool: http://snips-nlu.readthedocs.io/en/latest/tutorial.html#snips-dataset-format
.. _Snips console: https://console.snips.ai