TMR : Evaluating NER Recall on Tough Mentions (2021)

Abstract: We propose the Tough Mentions Recall (TMR) metrics to supplement traditional named entity recognition (NER) evaluation by examining recall on specific subsets of ”tough” mentions: unseen mentions, those whose tokens or token/type combination were not observed in training, and type-confusable mentions, token sequences with multiple entity types in the test data. We demonstrate the usefulness of these metrics by evaluating corpora of English, Spanish, and Dutch using five recent neural architectures. We identify subtle differences between the performance of BERT and Flair on two English NER corpora and identify a weak spot in the performance of current models in Spanish. We conclude that the TMR metrics enable differentiation between otherwise similar-scoring systems and identification of patterns in performance that would go unnoticed from overall precision, recall, and F1.

## introduction for decades, the standard measures of performance for named entity recognition (ner) systems have been precision, recall, and f1 computed over entity mentions. foot_0 ner systems are primarily evaluated using exact match foot_1 f1 score, micro-averaged across mentions of all entity types. while perentity-type scores available from the conlleval scorer @xcite are often reported, there are no widely-used diagnostic metrics that further analyze the performance of ner systems and allow for separation of systems close in f1. this work proposes tough mentions recall (tmr), a set of metrics that provide a fine-grained analysis of the mentions that are likely to be most challenging for a system: unseen mentions, ones that are present in the test data but not the training data, and type-confusable mentions, ones that appear with multiple types in the test set. we evaluate the performance of five recent popular neural systems on english, spanish and dutch data using these fine-grained metrics. we demonstrate that tmr metrics enable differentiation between otherwise similar-scoring systems, and the model that performs best overall might not be the best on the tough mentions. our ner evaluation tool is publicly available via a github repository. 3 2 related work previous work in ner and sequence labeling has examined performance on out-of-vocabulary (oov) tokens and rare or unseen entities. @xcite and @xcite evaluate system performance on mentions containing tokens not present in the pretrained embeddings or training data. such analysis can be used broadly-ma and hovy perform similar analyses for part of speech tagging and ner-and can guide system design around the handling of those tokens. @xcite present a thorough analysis of the generalization abilities of ner systems, quantifying the performance gap between seen and unseen mentions, among many other factors. their work predates current neural ner models; the newest model they use in their evaluation is senna @xcite . while prior work has considered evaluation on unseen mentions, it has focused on experimenting on english data, and the definition of "unseen" has focused on the tokens themselves being unseen (unseen-tokens in our ). we use the umbrella of "tough mentions" to cover a number of possible distinctions that can be made with regards to how unseen test set data is, and we experiment on multiple languages. @xcite propose an iterative approach for long-tail entity extraction, focusing on entities of two specific types in the scientific domain. @xcite propose evaluation on a set of unique mentions, which emphasizes the ability of a system to recognize rarer entities. as entities and their types change quickly @xcite , recall on emerging entities is becoming a more critical measure in evaluating progress. @xcite propose checklist, which can be applied to ner by using invariance tests; for example, replacing a mention with another one of the same entity type should not affect the output of the model. @xcite evaluate the generalization of ner models through breakdown tests, annotation errors and dataset bias. they examine the performance on subsets of entities based on the entity coverage rate between train and test set. they also release reconll, a revised version of conll-2003 english with fewer annotation errors which we use in this work. ## conclusion we have proposed tough mentions recall (tmr), a set of evaluation metrics that provide a finegrained analysis of different sets of formalized mentions that are most challenging for a ner system. by looking at recall on specific kinds of "tough" mentions-unseen and type-confusable ones-we are able to better differentiate between otherwise similar-performing systems, compare systems using dimensions beyond the overall score, and evaluate how systems are doing on the most difficult subparts of the ner task. we summarize our findings as follows. for english, the tmr metrics provide greater differentiation across systems than overall recall and are able to identify differences in performance between bert and flair, the best-performing systems in our evaluation. flair performs better on type-confusable mentions regardless of ontology, while performance on unseen mentions largely follows the overall recall, which is higher for flair on reconll and for bert on ontonotes. in spanish and dutch, the tmr metrics are not needed to differentiate systems overall, but they provide some insight into performance gaps between spanish and english related to unseen-type mentions. one challenge in applying these metrics is simply that there may be relatively few unseen mentions or tcms, especially in the case of lowerresourced languages. while we are interested in finer-grained metrics for lower-resourced settings, data sparsity issues pose great challenges. as shown in section 3.3, even in a higher-resourced setting, some subsets of tough mentions include less than 1% of the total mentions in the test set. we believe that lower-resourced ner settings can still benefit from our work by gaining information on pretraining or tuning models towards better performance on unseen and type-confusable mentions. for new corpora, these metrics can be used to guide construction and corpus splitting to make test sets as difficult as possible, making them better benchmarks for progress. we hope that this form of scoring will see wide adoption and help provide a more nuanced view of ner performance.