Improving NER Research Workflows with S eq S core (2023)

Abstract: We describe the features of SeqScore, an MIT-licensed Python toolkit for working with named entity recognition (NER) data.While SeqScore began as a tool for NER scoring, it has been expanded to help with the full lifecycle of working with NER data: validating annotation, providing at-a-glance and detailed summaries of the data, modifying annotation to support experiments, scoring system output, and aiding with error analysis.SeqScore is released via PyPI (https://pypi.org/project/seqscore/) and development occurs on GitHub (https://github.com/bltlab/seqscore).

## introduction while much attention in language technology development is focused on the creation of better models and datasets, it is essential to also have tools for understanding the output of those models and the contents of the datasets. for classification tasks, the combination of scikit-learn @xcite and pandas @xcite can provide preprocessing, data exploration, powerful modeling, and error analysis. however, chunking tasks like named entity recognition (ner) pose a challenge for data workflows and error analysis. while ner is often treated like a sequence-labeling problem like part of speech (pos) tagging, unlike pos tagging, the annotation and evaluation are performed at the chunk level, not individual tokens. for example, if a sentence begins "alan turing was...", an ner task may require that alan turing is correctly identified as a mention of a person's name. less or no credit would be received for identifying just the person name alan or separately identifying the names alan and turing without noting that they form a single unit. typically, each mention (also called an entity, phrase, or chunk) is encoded using bio encoding, so for example alan turing might receive the tags b-per i-per to reflect the beginning and continuation of a mention of type person, while non-mention tokens are tagged o. 1 the nature of chunking tasks means that every step of data processing is necessarily more complicated than traditional classification tasks. unlike a per-token classification task, looking at the individual token labels is a poor summary of the dataset. scoring is more difficult as the scorer must interpret a sequence of tagged tokens as mentions, which becomes non-trivial when the tags produced by a system do not follow the norms of the mention encoding method (e.g., bio) used @xcite . this paper describes the seqscore toolkit and its applications for validating, summarizing, and transforming ner data. a previous publication @xcite , introduced seqscore and described its value as a reproducibility-focused ner scorer. while this paper is also about seq-score, it has a different focus. we describe the development of new features for seqscore and what was needed to extend it from being just a scorer to a more complete toolkit for working with ner data. we discuss new feature development on seqscore and the process of expanding it to fill gaps common in ner workflows. in addition to discussing the details of seqscore, we discuss the challenges of trying to build open-source software for research. for decades, ner researchers have been able to be productive without any popular toolkits for working with ner data. there has been a commonlyused scorer, conlleval, which was provided for the 2002-3 conll shared tasks @xcite , available for two decades now. while standard-izing around a single scorer provides significant benefits @xcite , conlleval is not actively maintained and has not been updated as approaches to ner have changed. while common scorers and shared tasks are uniquely capable of uniting the research community, there is still a vital need for tools for tools for many stages of working with ner data. for example, given a dataset, how can we examine it? how can we determine what mention encoding it uses and whether it was used consistently? how can we modify it efficiently? how can we examine performance beyond just a few numbers? seqscore aims to provide efficient, command-line solutions to these problems. the most similar software package to seqscore is iobes @xcite . while the two projects began development concurrently, iobes was released first and has previously been published at nlp-oss. the iobes package is designed for api-level access and manipulation of spans. seqscore is focused on a command-line interface for scoring ner data and performing common manipulations on that data. both provide logic around converting chunk encodings (bio, bioes, etc.) to and from mentions. ## seqscore's features this section describes the features of seqscore, focusing on the newest features that enable it to assist in many ner data workflows. previous work @xcite has described the scoring features of seqscore, so they are not discussed in detail in this paper. seqscore is released via pypi ( https://pypi.org/project/seqscore/ ) and development occurs on github ( https:// github.com/bltlab/seqscore ). ## design challenges explicit or implicit? unlike conlleval, which will score many mention encodings without any direction from the user-even encodings it does not support @xcite , footnote 2)-seqscore requires users to be specify the mention encoding and how they want invalid transitions to be repaired. this can be confusing to new users, because if they do not specify a repair method, scoring may raise an error. seqscore has always erred on the side of making users be explicit and avoiding any silent defaults that could affect the results, but this comes at a price of some user frustration. limiting scope while seqscore is so far the most richly-featured toolkit for working with ner files, we have intentionally limited the scope of its capabilities where there is risk of misuse. one example is scoring ner in cases where the reference and system output may disagree in the tokens of a sentence, such as when performing ner on the output of speech recognition. seqscore currently insists that the reference and system output have the same text to avoid issues where sentences have become misaligned between the two, and this cannot be disabled. providing a flag to disable this check could result in users specifying it "just in case" to make sure scoring never raises an error, potentially leading to incorrect scores. @xcite forked seqscore for their study of ner on speech so they could score more flexibly. test coverage while seqscore stands apart from much research software in having a test suite and code coverage instrumentation, as the complexity of the toolkit has increased, so has the time required for tests to keep up with functionality. while very time-intensive to maintain, writing tests that exercise the command-line interface has been essential to avoiding regressions. seqscore uses click foot_2 to implement the cli, and testing is greatly aided by its clirunner class which allows direct invocation of the cli in unit tests. seqscore's test coverage stands at 95%, but it will take substantial effort to reach 100%. a handful of warnings and error cases are not exercised by the current tests due to the high time cost of developing inputs that would reach them and maintaining these inputs as the codebase changes. performance seqscore is written highly defensively to protect against user errors that could result in incorrect evaluation results. this unfortunately comes at the cost of speed; seqscore is slower at scoring and processing long files than other scorers. while we are interested in improving speed by using profiling, we are unwilling to optimize for speed at the expense of safety. ## conclusion seqscore provides a feature-rich toolkit for working with ner data, and we believe it will enable easier and more reproducible ner research. as more users adopt seqscore, we look forward to addressing their needs and the bugs they find. development of seqscore is ongoing. a major area of interest is developing a stable api for scoring. unlike iobes and seqeval @xcite , the primary use case of seqscore has been through the command line. we plan to add a stable api before a version 1.0 release.