What do we really know about State of the Art NER ? (2022)

Abstract: Named Entity Recognition (NER) is a well researched NLP task and is widely used in real world NLP scenarios. NER research typically focuses on the creation of new ways of training NER, with relatively less emphasis on resources and evaluation. Further, state of the art (SOTA) NER models, trained on standard datasets, typically report only a single performance measure (F-score) and we don’t really know how well they do for different entity types and genres of text, or how robust are they to new, unseen entities. In this paper, we perform a broad evaluation of NER using a popular dataset, that takes into consideration various text genres and sources constituting the dataset at hand. Additionally, we generate six new adversarial test sets through small perturbations in the original test set, replacing select entities while retaining the context. We also train and test our models on randomly generated train/dev/test splits followed by an experiment where the models are trained on a select set of genres but tested genres not seen in training. These comprehensive evaluation strategies were performed using three SOTA NER models. Based on our results, we recommend some useful reporting practices for NER researchers, that could help in providing a better understanding of a SOTA model’s performance in future.

## https://paperswithcode.com/sota/ named-entity-recognition-ner-on-ontonotes-v5 developing generalizable nlp models is an open research question. ner research too focused on this aspect in the past, from a modeling perspective. for example, @xcite proposed methods develop models to account for multi-genre scenarios. @xcite recently proposed an approach that forces models to learn more contextual information and memorize less. @xcite generate counterfactual examples to enhance the original dataset while training a model. however, these approaches address only one side of the problem. the other side of generalizability is about evaluating on new test sets. performing adversarial attacks has been a recent area of research in nlp and machine learning to understand some aspects of this question. the goal of such an approach is to create minimal perturbations in the input text, which will maximize the probability of a model making wrong predictions. from simple word/character substitutions and heuristics targeting various forms of text transformations to using large pre-trained language models, a range of strategies have been proposed in the past, to generate adversarial text @xcite @xcite @xcite . in the recent past, such methods have been applied for various tasks such as co-reference resolution @xcite , question answering @xcite , morphological analysis @xcite , natural language inference @xcite , named entity linking @xcite . @xcite extended this research to ner by creating new test sets replacing some entities selectively, using lists of common names by various nationalities. @xcite 's rockner used wikidata and bert to substitute entities and contexts and generate new ner test sets. in this paper, we extend this line of enquiry further by creating six new adversarial test sets for evaluating ner (in english). to summarize, inspired by some past research into generalizability @xcite and evaluation @xcite @xcite @xcite of ner, and evaluation of nlp systems in general @xcite , we attempted to understand what we could learn about sota ner beyond a single f-score in this paper. while some of these questions were addressed separately in the past research to various degrees, to our knowledge, there is no previous research that addressed all these questions together, comparing multiple ner systems in the past.