S eq S core: Addressing Barriers to Reproducible Named Entity Recognition Evaluation (2021)

Abstract: To address a looming crisis of unreproducible evaluation for named entity recognition, we propose guidelines and introduce SeqScore, a software package to improve reproducibility. The guidelines we propose are extremely simple and center around transparency regarding how chunks are encoded and scored. We demonstrate that despite the apparent simplicity of NER evaluation, unreported differences in the scoring procedure can result in changes to scores that are both of noticeable magnitude and statistically significant. We describe SeqScore, which addresses many of the issues that cause replication failures.

## introduction there are many complex tasks in natural language processing (nlp) where current evaluation standards are based around evolving metrics designed to correlate well with human judgments, some complex and some simple. for example, every year sees the introduction and careful evaluation of new metrics for machine translation, summarization, and natural language generation. however, named entity recognition (ner) and other chunk extraction tasks have largely been evaluated the same way since the conll shared tasks of the early 2000s @xcite @xcite . following the conll chunking and ner shared tasks, a true positive prediction typically requires exact matches 1 in span (the tokens or characters in a chunk) and the type assigned to the chunk (e.g., person). with such a simple metric, it would seem that performing exact match ner evaluation would be trivially simple. precision, recall, and f1 are easy 1 while there have been efforts to promote partial matching @xcite and focusing on rarer entities @xcite , micro-averaaged exact match f1 is still the most common metric in use for ner. to compute; all that is required is to count true positives, false positives, and false negatives. but when it comes to evaluation, challenges emerge in how evaluation is actually implemented. in the case of ner, as we will demonstrate these challenges emerge in the process of converting token-level annotations and system predictions into spans. we do not think it is sufficient to point out these issues without attempting to provide a solution. inspired by successful efforts in the machine translation community to address similar issues @xcite , we began developing a toolkit and set of practices in summer 2020 to improve the replicability of experiments for ner. our toolkit, seqscore, provides researchers the necessary tools to score, validate, and examine both system outputs and annotation. seqscore is open source and has been publicly released. 2 this paper provides clear, easy-to-follow guidelines that facilitate reproducibility for ner (and other chunking task) experiments, explains them, provides a toolkit for easily following them, and then presents experiments using seqscore that shows the impact of following them. the contribution of this paper is that it introduces and justifies guidelines for ner experiment reproducibility and provides a toolkit that makes them easy to follow. ## guidelines for reproducibility we propose that in order to have sound and reproducible ner evaluation, the following guidelines should be followed: many of these will seem like obvious ideas or practices that should be taken as a given. however, we have found almost no papers provide enough information to determine whether they are compliant with all of these guidelines specifically, very few papers report what scorer was used to produce the reported scores, and many that provide accompanying code do not include any evaluation code. we examined several papers with state of the art ner results on the conll 2003 dataset considering guidelines 1, 2, and 3. of these papers @xcite follow 1, 2, and 3. yamada et al. (2020) explicitly follows guidelines 2 and 3. luoma and pyysalo (2020) met guideline 1. @xcite give details of their scoring decision for a previous paper, @xcite , mentioning they fixed an prior error in scoring, but do not explicitly detail how they fixed their scoring procedure for the baseline in @xcite . all other papers we surveyed did not explicitly satisfy guidelines 1, 2, and 3 @xcite @xcite @xcite @xcite . as an example of a common departure from these practices, many papers that perform ner experiments publish the scores produced by ncrf++ @xcite . as previously detailed by lignos and kamyab (2020), ncrf++ uses an internal scorer with an undocumented label sequence repair method, so reporting any numbers from it would be contrary to guidelines 2 and 3. as lignos and kamyab demonstrated, on a specific subset of models that produce a high number of invalid transitions, that scorer produces f1 scores approximately half a point higher than the most commonly-used external scorer. guideline 4, which requires that the annotation precisely follow the chunk encoding scheme, also seems obvious. however, it was not actually followed for 2 of the 4 datasets for the conll ner evaluations in 2002-3, as only the english and dutch data were free of errors of this type (see section 3.4). as these datasets are arguably the most famous ner datasets in existence, this is surprising. while this would only have a very minor impact on evaluation results, an evaluation cannot be reproducible if different scorers might interpret the gold standard differently due to differences in how invalid label sequences are handled (see section 3.2). when examining other ner datasets, we have found more pervasive occurrences of invalid label sequences. we will not discuss guideline 5 in any detail as practices change over time, but we will highlight the need to report a distribution of scores, rather than a single score. @xcite demonstrate this clearly for ner specifically, and seqscore supports aggregating scoring across multiple runs and reporting summary statistics. many of these rules may seem like common sense, but by enumerating them, we provide a published "checklist" for researchers to follow. ## the mechanics of ner evaluation we now turn to explaining the mechanics of ner evaluation to explain why following these guidelines is important. in this section, we explain the subtleties of working with chunk encodings, which will reinforce the importance of following the first three guidelines. ## experiments our paper so far has discussed the importance of following the proposed guidelines but has not quantified the impact of doing so. we conducted a series of experiments on ner datasets to examine the extent to which the variations in scores from different repair methods applied to system outputs could lead to different results. these experiments also demonstrate the usefulness of seqscore as a package for producing a reproducible and complete set of results for sequence labeling tasks. as we show in the following experiments, ner using large multilingual models fine-tuned on lower-resourced datasets can show significant variation due to the scoring method used. we selected lower-resourced datasets for two reasons. first, we believe that this a major frontier for innovation in ner, and many new results will be reported in this area for years to come. second, unlike higherresourced datasets, the current state of the art for these datasets involves the application of large language models, which are particularly prone to producing invalid transitions. ## comparison with other toolkits to place seqscore in the context of other work and to address the question of novelty, we compare seqscore to other similar tools. ## conclusion this paper has provided guidelines for reproducible ner research and demonstrated the importance of following them, both by describing the principles behind them and demonstrating the impact on actual scores. ultimately, researchers can choose to either accept the status quo-which for ner is non-reproducible research due to both a lack of standard practices and a lack of standard tools-or attempt to elevate the practice in the field to higher standards. we hope that by providing a software toolkit to help follow these guidelines, we have substantially reduced the barriers to performing reproducible research for this task. while we have created a software package to accompany our recommendations-one that makes following them extremely simple-we do not claim that using seqscore is necessary for reproducible results. just like sacrebleu is not the only way to produce a reproducible mt score, seqscore is not the only reproducible way to score ner output. however, as it is actively maintained, welltested, and it can handle multiple repair methods, we strongly encourage its use. by focusing on transparency and prioritizing supporting reproducible research in the design of seqscore, we believe we have produced a toolkit that can have substantial positive impact on the field. adoption of this paper's recommendations by researchers will increase transparency in the scoring process and enable standardization of scoring methods in a field we believe is approaching a reproducibility crisis.