If You Build Your Own NER Scorer, Non-replicable Results Will Come (2020)

Abstract: We attempt to replicate a named entity recognition (NER) model implemented in a popular toolkit and discover that a critical barrier to doing so is the inconsistent evaluation of improper label sequences. We define these sequences and examine how two scorers differ in their handling of them, finding that one approach produces F1 scores approximately 0.5 points higher on the CoNLL 2003 English development and test sets. We propose best practices to increase the replicability of NER evaluations by increasing transparency regarding the handling of improper label sequences.

## introduction the goal of this paper is to demonstrate an issue that complicates the comparison and replication of named entity recognition (ner) systems. standard f1-based evaluation of ner models in the manner made popular by the conll 2002-3 shared tasks @xcite requires decoding a sequence of per-token labels into entity mentions and computing precision and recall by evaluating the types and spans of the mentions. while there are popular scorer implementations and popular ner toolkits, there is little transparency regarding the exact process that scorers use to decode label sequences. in the case where all label sequences are properly formed, this lack of transparency should have no impact; all correct decoding processes should produce the same result. however, many systems can produce what we call improper label sequences (see section 3.2), and different approaches to decoding improper label sequences for evaluation produce different scores. as the processes for label decoding have not been standardized and are often undocumented, comparisons between reported scores may not be fair, and replication of those scores can prove difficult. ## our negative result we did not intend to begin a project on ner evaluation reproducibility. the discovery process for our negative result began with three separate research projects encountering the same issue: higher-thanexpected f1 scores for a certain class of models implemented in a popular ner toolkit, ncrf++ @xcite . one of the authors of this paper attempted to reimplement ncrf++'s models as a learning exercise. the reimplementation yielded similar f1 scores to ncrf++ when a crf output layer was used but produced lower f1 scores when using softmax output. after the other author found the same result in an independent reimplementation, we turned our attention to the only commonality between our implementations: an open-source external scorer. we consulted with researchers in our lab using ncrf++ for two other projects, and a consistent story began to emerge: when scoring softmax models, ncrf++'s internal scorer produced higher scores than other ner scorers. our negative result was a failure to replicate the performance of ncrf++, and the cause of this failure was that we did not understand its approach to evaluation. ncrf++'s scorer differs from others in how it treats improper label sequences, and its approach consistently produces higher f1 scores for softmax-output models. the handling of these sequences is effectively undefined behavior for an ner scorer; there is no single correct strategy. in this paper, we quantify the impact of those strategies and propose an evaluation approach that would improve reproducibility by requiring explicit, transparent handling of improper label sequences. supplemental material and the resources needed to replicate this study are available at https:// lignos.org/repro-ner. ## results the goal of our experiments is to estimate the increase in f1 scores that can be attributed to use of ncrf++'s internal decoding compared to conllstyle decoding when evaluating softmax-output models. we trained models using ncrf++ after modifying it to additionally use an external conllstyle scorer, seqeval. 3 we report scores from the internal and external scorer. we use a bidirectional lstm architecture at the word level and test multiple character-level architectures: a bi-lstm, cnn, and no character-level representation. we use the same hyperparameters, pretrained word embedding (glove 100d), and data (conll 2003 english) used by @xcite . each configuration was run ten times using different random initializations (seeds 0-9). the test set was evaluated using the model from the epoch that attained the highest development set f1 (as scored internally by ncrf++) during training. all scores reported so far have been from converged models. it is also of interest to explore what the increase in scores looks like during training. figure 1 gives the increase in development set f1 scores across all training epochs for all configurations we ran, displaying 3,000 points per encoding. early in training, ncrf++'s internal scorer can produce f1 scores several points higher than a conll-style scorer, presumably due to producing a high number of improper label sequences preconvergence. crucially, in all 6,000 development set epochs we evaluated, ncrf++'s internal scorer always reported a higher f1 than seqeval in our evaluation of softmax-output models. ## a vision for ner evaluation our study leads us to propose a vision for ner evaluation as follows: this procedure separates the process of scoring from the process of interpreting improper label sequences. it also suggests a new research avenue of designing methods for optimally converting improper label sequences into proper ones. until the tools required for our vision-perhaps something like sacrebleu @xcite -are developed, we recommend seqeval as the best solution for ner scoring. it is an easy-to-inspect, faithful reimplementation of conlleval. unlike conlleval, it can be called directly from python and does not truncate scores, avoiding systematic downward bias when aggregating them. regarding extensions to this study, while we evaluated on the conll 2003 english data to compare with @xcite , evaluating in more languages is essential. looking beyond the other conll 2002-3 languages, testing against smaller annotated data sets and in lower-resourced languages may reveal more complexity to the problem of decoding improper sequences.