Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition (2021)

Abstract: In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity. We introduce NRB, a new testbed carefully designed to diagnose Name Regularity Bias of NER models. Our results indicate that all state-of-the-art models we tested show such a bias; BERT fine-tuned models significantly outperforming feature-based (LSTM-CRF) ones on NRB, despite having comparable (sometimes lower) performance on standard benchmarks. To mitigate this bias, we propose a novel model-agnostic training method that adds learnable adversarial noise to some entity mentions, thus enforcing models to focus more strongly on the contextual signal, leading to significant gains on NRB. Combining it with two other training strategies, data augmentation and parameter freezing, leads to further gains.

## introduction recent advances in language model pre-training @xcite @xcite have greatly improved the performance of many natural language understanding (nlu) tasks. yet, several studies @xcite @xcite revealed that state-of-the-art nlu models often make use of surface patterns in the data that do not generalize well. named-entity recognition (ner), a downstream task that consists in identifying textual mentions and classifying them into a predefined set of types, is no exception. the robustness of modern ner models has received considerable attention recently @xcite @xcite @xcite . name regularity bias @xcite @xcite in ner occurs when a model relies on a signal coming from the entity name, and disregards evidence within the local context. figure 1 shows examples where state-of-the-art models @xcite @xcite fail to exploit contextual information. for instance, the entity gonzales in the first sentence of the figure is wrongly recognized as a person, while the context clearly signals that it is a location (city). to better highlight this issue, we propose nrb, a testbed designed to accurately diagnose name regularity bias of ner models by harvesting natural sentences from wikipedia that contain challenging entities, such as those in figure 1 . this is different from previous work that evaluated models on artificial data obtained by either randomizing @xcite or substituting entities by ones from a pre-defined list @xcite . nrb is compatible with any annotation scheme, and is intended to be used as an auxiliary validation set. we conduct experiments with the featurebased lstm-crf architecture @xcite and the bert @xcite fine-tuning approach trained on standard benchmarks. the best lstm-based model we tested is able to correctly predict 38% of the entities in nrb. bert-based models are performing much better (+37%), even if they (slightly) underperform on in-domain development and test sets. this mismatch in performance between nrb and standard benchmarks indicates that context awareness of models is not rewarded by existing benchmarks, thus justifying nrb as an additional validation set. entities of interest are underlined, gold types are in blue superscript, model predictions are in red subscript, and context information is highlighted in purple. models used in this study disregard contextual information and rely instead on some signal from the named-entity itself. @xcite in which learnable noise vectors are added to named-entity words, weakening their signal, thus encouraging the model to pay more attention to contextual information. applying it to both feature-based lstm-crf and fine-tuned bert models leads to consistent gains on nrb (+13 points) while maintaining the same level of performance on standard benchmarks. the remainder of the paper is organized as follows. we discuss related works in section 2. we describe how we built nrb in section 3, and its use in diagnosing named-entity bias of state-of-the-art models in section 4. in section 5, we present a novel adversarial training method that we compare and combine with two simpler ones. we further analyze these training methods in section 6, and conclude in section 7. ## related work robustness and out-of-distribution generalization has always been a persistent concern in deep learning applications such as computer vision @xcite , speech processing @xcite , and nlu @xcite @xcite @xcite . one key challenge behind this issue in nlu is the tendency of models to quickly leverage surface form features and annotation artifacts @xcite , which is often referred to as dataset biases @xcite . we discuss related works along two axes: diagnosis and mitigation. ## mitigating bias in this section, we investigate training procedures that are designed to enhance the contextual awareness of a model, leading to better performance on nrb without impacting in-domain performance. these training procedures are not supposed to use any external data. in fact, nrb is only used as a diagnosing corpus, once the model is trained. we propose 3 training procedures that can be combined, two of them are architecture-agnostic, and one is specific to fine-tuning bert. ## analysis so far, we have shown that state-of-the-art models do suffer from name regularity bias, and we proposed model-agnostic training methods that are able to mitigate this bias to some extent. in section 6.1, we provide further evidence that our training methods force the bert-large model to better concentrate on contextual cues. in section 6.2, we replicate the evaluation protocol of @xcite in order to clear out the possibility that our training methods are only valid on nrb. last, we perform extensive experiments on name regularity bias under low resource (section 6.3) and multilingual (section 6.4) settings. ## conclusion in this work, we focused on the name regularity bias of ner models, a problem first discussed in @xcite . we propose nrb, a benchmark we specifically designed to diagnose such a bias. as opposed to existing strategies devised to measure it, nrb is composed of real sentences with easy to identify mentions. we show that current state-of-the-art models, perform from poorly (feature-based) to decently (fined-tuned bert) on nrb. in order to mitigate this bias, we propose a novel adversarial training method based on adding some learnable noise vectors to entity words. these learnable vectors encourage the model to better incorporate contextual information. we demonstrate that this approach greatly improves the contextual ability of existing models, and that it can be combined with other training methods we proposed. significant gains are observed in both low-resource and multilingual settings. to foster research on ner robustness, we encourage others to report results on nrb and wts. 13 this study opens up new avenues of investigation. conducting a large-scaled multilingual experiment, characterizing the name regularity bias of more diversified morphological language families is one of them, possibly leveraging massively multilingual resources such as wikiann @xcite , polyglot-ner (al-rfou et al., 13 english and multilingual nrb and wts are available at http://rali.iro.umontreal.ca/rali/?@xmath0. also, non-sequential labeling approaches for ner like the ones of @xcite and @xcite have reported impressive results on both flat and nested ner. we plan to measure their bias on nrb and study the benefits of applying our training methods to those approaches. finally, we want to investigate whether our adversarial training method can be successfully applied to other nlp tasks.