neleval 3.1.1
Command-line evaluation tools for named entity linking and (cross-document) coreference resolution
License: Apache 2.0
Keywords: clustering,coreference,evaluation,entity disambiguation

Entity linking evaluation
=========================

Python command-line evaluation scripts for `TAC <http://www.nist.gov/tac/>`__
entity linking and related wikification, named entity disambiguation, and
within- and cross-document coreference tasks.

|version| |licence| |py-versions|

|issues| |build| |docs| |coverage|

It aims for **fast** and **flexible** coreference resolution and
**sophisticated** named entity recognition evaluation, such as partial scores
for partial overlap between gold and system mentions. CEAF, in particular, is
much faster to calculate here than in the `CoNLL-11/12 scorer
<https://github.com/conll/reference-coreference-scorers>`__. It boasts features
such as configurable metrics; accounting for or ignoring cross-document
coreference (see the ``evaluate --by-doc`` flag); plotting to compare
evaluation by system, measure and corpus subset; and bootstrap-based confidence
interval calculation for document-wise evaluation metrics.

Requires that ``python`` (2.7, with Py3k support experimental/partial)
be installed on your system with ``numpy`` (and preferably ``scipy`` for
fast CEAF calculation) and ``joblib``. ``matplotlib`` is required for
the ``plot-systems`` command.

See a list of commands with:

.. code:: bash

    ./nel --help

Or install onto your Python path (e.g. with
``pip install git+https://github.com/wikilinks/neleval``) then

.. code:: bash

    python -m neleval --help

TAC-KBP 2014 EDL quickstart
===========================

.. code:: bash

    ./scripts/run_tac14_evaluation.sh \
        /path/to/gold.xml \              # TAC14 gold standard queries/mentions
        /path/to/gold.tab \              # TAC14 gold standard link and nil annotations
        /system/output/directory \       # directory containing (only) TAC14 system output files
        /script/output/directory \       # directory to which results are written
        number_of_jobs                   # number of jobs for parallel mode

Each file in in the system output directory is scored against gold.tab.

Similar facility is available for TAC-KBP'15 EDL.

More details
============

See `the documentation <https://neleval.readthedocs.io>`__ for more
details.


.. |py-versions| image:: https://img.shields.io/pypi/pyversions/neleval.svg
    :alt: Python versions supported

.. |version| image:: https://badge.fury.io/py/neleval.svg
    :alt: Latest version on PyPi
    :target: https://badge.fury.io/py/neleval

.. |build| image:: https://travis-ci.org/wikilinks/neleval.svg?branch=master
    :alt: Travis CI build status
    :scale: 100%
    :target: https://travis-ci.org/wikilinks/neleval

.. |issues| image:: https://img.shields.io/github/issues/wikilinks/neleval.svg
    :alt: Issue tracker
    :target: https://github.com/wikilinks/neleval

.. |coverage| image:: https://coveralls.io/repos/github/wikilinks/neleval/badge.svg
    :alt: Test coverage
    :target: https://coveralls.io/github/wikilinks/neleval

.. |docs| image:: https://readthedocs.org/projects/neleval/badge/?version=latest
     :alt: Documentation Status
     :scale: 100%
     :target: https://neleval.readthedocs.io/en/latest/?badge=latest

.. |licence| image:: https://img.shields.io/badge/Licence-Apache%202.0-blue.svg
     :target: https://opensource.org/licenses/Apache-2.0