A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained Model Lead to the Promised Land? (2020)

Abstract: Fine-tuning pretrained model has achieved promising performance on standard NER benchmarks. Generally, these benchmarks are blessed with strong name regularity, high mention coverage and sufficient context diversity. Unfortunately, when scaling NER to open situations, these advantages may no longer exist. And therefore it raises a critical question of whether previous creditable approaches can still work well when facing these challenges. As there is no currently available dataset to investigate this problem, this paper proposes to conduct randomization test on standard benchmarks. Specifically, we erase name regularity, mention coverage and context diversity respectively from the benchmarks, in order to explore their impact on the generalization ability of models. To further verify our conclusions, we also construct a new open NER dataset that focuses on entity types with weaker name regularity and lower mention coverage to verify our conclusion. From both randomization test and empirical experiments, we draw the conclusions that 1) name regularity is critical for the models to generalize to unseen mentions; 2) high mention coverage may undermine the model generalization ability and 3) context patterns may not require enormous data to capture when using pretrained encoders.

## introduction named entity recognition (ner), or more generally name tagging, aims to identify text spans pertaining to specific entity types. ner is a fundamental task of information extraction which enables many downstream nlp applications, such as relation extraction @xcite , event extraction @xcite and machine reading comprehension @xcite ., name regularity entity types with strong regularity entity types with weak or no regularity mention coverage training set with high mention coverage many new and unseen mentions context pattern with decent training instances to capture fully-annotated training data is rare examples … at [cherry street]… … go to [9 th avenue] … starting from [cherry street] … at [8 th avenue] … i watched [avatar]last night …[the matrix] is the best... wow...[joker] was great! love [inception] so much. location movie test train train test 2016). recently, neural network-based supervised models dominate the ner task. by supervised fine-tuning upon large-scale language model pretrained architectures (e.g., @xcite , bert @xcite , xlnet @xcite , etc.), we have witnessed superior performances on almost all widely-used ner benchmarks, including conll03, ace2005 and tac-kbp datasets @xcite @xcite . despite the success of recent models, there are specific advantages in current ner benchmarks which significantly facilitate supervised neural networks. first, these benchmarks focus on limited entity types, and most mentions of these types have strong name regularity. for example, nearly all person names follow the "firstname lastname" or "lastname firstname" patterns, while location names mostly end with indicator words such as "street" or "road". second, the training and test data in these benchmarks are sampled from the same corpus, and therefore the training data usually have high mention coverage on the test data, i.e., a large proportion of mentions in the test set have been observed in the training set. however, it is obvious that this high coverage is inconsistent with the primary goal of ner models, which is expected to identify unseen mentions from new data by capturing the generalization knowledge about names and contexts. for observed mentions, other techniques, such as entity linking @xcite , would be more appropriate and effective. third, these benchmarks generally provide decent training data, and therefore the context diversity of all entity types can be sufficiently learned. in this paper, we refer to the ner tasks with strong name regularity, high mention coverage and with sufficient training instances as regular ner. and it proves that the state-of-the-art neural networks can easily exploit such name regularity, mention coverage and context diversity knowledge, and therefore achieve state-of-the-art performance in these benchmarks. unfortunately, when it comes to a more general scenario, there are significant discrepancies between regular benchmarks and ner in open settings. in this paper, we want to shed some light on the impact of the discrepancies between regular and open ner, and provides some valuable insights into the construction of general ner models in a more effective and efficient way. specifically, we want to answer the following question: can pretrained supervised neural networks still generalize well on ner when either weaker name regularity, lower mention coverage or inadequate context diversity exists? it is non-trivial to answer this question because currently no well-established benchmark concentrates on these issues. to this end, this paper exploits the efficacy of the above three kinds of information by conducting a series of experiments based on randomization test @xcite . specifically, we design several mention replacing mechanisms, which can erase specific kinds of information on-demand from current ner benchmarks. by applying the same supervised models on both vanilla and information-erased data, we can investigate how much the models will rely on particular erased information to identify entity mentions. generally, we propose to erase name regularity, mention coverage and context diversity respectively using the following kinds of randomization test, whose examples are shown in to verify the above findings, we further conduct a verification experiment by constructing a new dataset derived from wikipedia, which focuses on entity types with weak name regularity. to the best of our knowledge, this is the first work that tries to investigate such critical differences between regular and open ner. from both the randomization test and the verification experiment, we reach the following main conclusions: ## randomization test on ner to investigate the discrepancies between regular and open ner, this paper controllably erases target information from the vanilla data via a variant of randomization test @xcite in non-parametric statistics. concretely, to probe the effect of a specific kind of information, we erase it in vanilla data by randomly replacing entity mentions with particular irregular utterances. after that, we learn and compare ner models on both the vanilla benchmark and the information-erased version, in order to evaluate the models' robustness and generalization ability when target information is absent. the results of our randomization test can serve as a frame of reference for open ner, where the erased information is often truly absent. specifically, three kinds of information are particularly considered, and four kinds of strategies are used in our randomization test. ## experiments on open ner 4.1 data preparation to further verify the conclusions from our randomization test, we propose to conduct experiments on a real-world open ner dataset, which focuses on real world entity types with weaker name regularity than previously-used benchmarks. because currently no suitable dataset is available for verifying our conclusions, this paper constructs a new dataset from wikipedia. specifically, we consider four entity types in our experiments, including movie, song, book and tv series. we extract all sentences in wikipedia which contain mentions linking to entities of these types as our experiment dataset. from them, we randomly sample 10,000 sentences as the test set and 2,000 sentences as the development set, and part of the remaining data will be used as the training data according to the following different settings. finally, there are 2875, 2791, 598 and 580 mentions for movie, tv series, song and book in the test set respectively. note that different from real scenarios, this dataset only keeps sentences containing at least one mention, due to the partial labeling nature of wikipedia, this dataset only keeps sentences containing at least one mention, and the performance on this dataset may over-estimate the precision than in real applications. although this may different from real scenarios, we believe it can still lead to reasonable conclusions. ## related work by contrast, to the best of our knowledge, this is the first work which investigates the essential difference between regular and open ner. by conducting both randomization test @xcite and verification experiments, we analyze the impact of name regularity, mention coverage and context pattern sufficiency and shed light on future open ner studies. ## conclusion and future work this paper investigates whether current state-ofthe-art models on regular ner can still work well on open ner. from the perspective of name regularity, mention coverage and context diversity, we conducted both randomization test and verification experiments to evaluate the generalization ability of models. our investigation leads to three valuable conclusions, which shows the necessity of de-cent name regularity to identify unseen mentions, the hazard of high mention coverage to model generalization, and the redundancy of enormous data to capture context patterns. the above findings shed light on the promising directions for open ner, including 1) exploiting name regularity more efficiently with easilyobtainable resources such as gazetteers; 2) preventing the overfit on popular in-dictionary mentions with constraints or regularizers; and 3) reducing the need of training data by decoupling the acquisition of context knowledge and name knowledge.