Towards Building More Robust NER datasets: An Empirical Study on NER Dataset Bias from a Dataset Difficulty View (2023)

Abstract: Recently, many studies have illustrated the robustness problem of Named Entity Recognition (NER) systems: the NER models often rely on superficial entity patterns for predictions, without considering evidence from the context. Consequently, even state-of-the-art NER models generalize poorly to out-of-domain scenarios when out-of-distribution (OOD) entity patterns are introduced. Previous research attributes the robustness problem to the existence of NER dataset bias, where simpler and regular entity patterns induce shortcut learning. In this work, we bring new insights into this problem by comprehensively investigating the NER dataset bias from a dataset difficulty view. We quantify the entity-context difficulty distribution in existing datasets and explain their relationship with model robustness. Based on our findings, we explore three potential ways to de-bias the NER datasets by altering entity-context distribution, and we validate the feasibility with intensive experiments. Finally, we show that the de-biased datasets can transfer to different models and even benefit existing model-based robustness-improving methods, indicating that building more robust datasets is fundamental for building more robust NER systems.

## introduction named entity recognition (ner), aiming to recognize named entities from unstructured data, is widely studied by researchers as a fundamental task in natural language processing (nlp) and a crucial task in practical applications @xcite @xcite . recently, the advances in pre-trained language models @xcite @xcite have contributed to promising performance on standard ner benchmarks, such as conll03 @xcite and ontonotes 5.0 @xcite . despite the success, recent research has demonstrated the robustness problem in ner: the stateof-the-art ner systems often rely on superficial entity patterns for predictions, while disregarding evidence from the context. consequently, the models show poor generalization ability in outof-domain (ood) scenarios where out-of-domain entity patterns are introduced @xcite @xcite . for instance, figure 1 shows two typical ood failures of sota ner models: (1) instances with out-ofvocabulary entities (upper). the entity "cathay" is unseen in the training data, however, the model fails to deduce correctly given obvious evidence "[]'s profit to grow" in the context. (2) instances with ambiguous entities (bottom). @xcite shows when an entity is labeled differently across domains, the model fails to recognize well even with supportive context, such as mistaking the person name "apple" in the example. previous works have delved into the robustness problem of ner from different perspectives. @xcite @xcite @xcite examine the ner models by constructing challenging test sets. their experiments reveal that ner models fail to "learn from context information" during training. @xcite and @xcite evaluate models on controlled entity sets and consider the poor robustness is due to the model's tendency to memorize entity patterns. @xcite further designs a series of randomization tests and demonstrates that the strong name regularity and high mention coverage rate in existing ner datasets might hinder model robustness. all these studies indicate that the poor robustness of ner models might be due to a hidden dataset bias of existing ner datasets: the entity pattern is "easier" for the model to learn, so the models are biased to learn the shortcut in entity names while paying less attention to the context. in this work, we systematically locate the origin of ner dataset bias and investigate its effect on model robustness from a dataset difficulty view. we try to answer two questions: q1: how does the entity-context distribution in the training data induce dataset bias? to answer this question, we borrow a recent concept "v-information" @xcite to measure the difficulty of entity and context for the model to learn in each dataset. we find that (1) in all ner datasets we examine, the v-information of the entity is obviously larger than that of the context, indicating the dataset distribution induces models to learn more entity than context. (2) we further design an instancelevel metric to measure the difficulty of entity and context in every single instance. we find that the largest population of instances-with equalityinformative entity and context-does not lead or even harm the models to learn context. ## can we build more robust datasets by altering entity-context distribution? based on the above analysis, in this section, we explore three potential ways to de-bias ner datasets by altering the entity-context distribution. base model and baselines in this section, we conduct all experiments based on two pre-trained lm, bert-base-cased @xcite and roberta-large @xcite . to better verify the robustness improvement of the three de-bias methods, we include several robustness-improving baselines: (1) base @xcite the base token classification model based on bert-base-cased and roberta-large, trained on the original training sets; (2) dataaug (dai and adel, 2020), which augments the training set by replacing entities with similar entities or typos entities; ## conclusion in this work, we conduct an interesting study on the model robustness problem of ner. we quantify the difficulty distribution of context and entity in existing ner datasets and reveal how the entity-context distribution affects model robustness. based on our analysis, we further explore three potential ways to de-bias the existing ner datasets by reconstructing the existing datasets and altering the entity-context distribution. with extensive experiments, we validate that the reconstructed datasets can largely improve model robustness. as these datasets can also benefit model-based robustness-improving methods, we argue that building more robust datasets is fundamental for building more robust ner systems.