Leveraging Expert Guided Adversarial Augmentation For Improving Generalization in Named Entity Recognition (2022)

Abstract: Named Entity Recognition (NER) systems often demonstrate great performance on in-distribution data, but perform poorly on examples drawn from a shifted distribution. One way to evaluate the generalization ability of NER models is to use adversarial examples, on which the specific variations associated with named entities are rarely considered. To this end, we propose leveraging expert-guided heuristics to change the entity tokens and their surrounding contexts thereby altering their entity types as adversarial attacks. Using expert-guided heuristics, we augmented the CoNLL 2003 test set and manually annotated it to construct a high-quality challenging set. We found that state-of-the-art NER systems trained on CoNLL 2003 training data drop performance dramatically on our challenging set. By training on adversarial augmented training examples and using mixup for regularization, we were able to significantly improve the performance on the challenging set as well as improve out-of-domain generalization which we evaluated by using OntoNotes data. We have publicly released our dataset and code at https://github.com/GT-SALT/Guided-Adversarial-Augmentation .

## introduction deep learning models have achieved great performance on many natural language processing (nlp) problems @xcite . however, many recent works have shown that these models often rely on spurious correlations which are not necessarily the causal artifacts. thus, these models perform well on the in-distribution test set but are likely to exhibit a huge performance decline on out-of-distribution data (e.g. real world data) @xcite @xcite @xcite . prior works have constructed adversarial examples for benchmarking the generalization ability of state-of-the-art nlp models on out-ofdistribution examples @xcite @xcite . proposed approaches such as random word swapping @xcite and the appending of a sentence to the end of text @xcite do not take into consideration the unique linguistic properties and variations associated with named entities. as a key problem setting involving the classification of semantic categories of entities (e.g., organizations, locations) @xcite , ner is still in need of improved benchmarks of true generalization. previous works (bernier-colborne and langlais, 2020; @xcite have shown that words which have different entity labels in different scenarios often lead to frequently occurring errors of ner models. this can be especially problematic in specific domain applications where this challenging case is common. for example, when training an ner model for political text mining, it would be of great importance to differentiate between the categories of clinton (person) and the clinton foundation (organization). we make use of this as the inspiration for designing expert-guided heuristic linguistic patterns for creating a high quality adversarial dataset for ner. leveraging such expert-guided heuristics, we propose an automated procedure for adversarial augmentation. we use this automated procedure to first generate adversarial examples from the test data. since some of these automatically generated adversarial examples may lack quality in terms of syntax or semantics, we manually select only the examples that are of high quality for the construction of the challenging test set. the performance of state-of-the-art ner systems drops severely on this challenging test set. to alleviate this degradation, we first use the proposed heuristics to augment the training examples (without manually filtering the data for quality), which proves to be effective. we further utilize mixup @xcite as a regularization technique to interpolate the representations of the original examples and the augmented examples, leading to a smoother decision boundary and improved generalization ability @xcite . ## related work generating adversarial examples adversarial data augmentation @xcite severely influences a model's predictions without changing human judgements. it is widely leveraged to test the generalization ability of models @xcite . for example, jia and liang (2017) fools a reading comprehension system by inserting distracting sentences. @xcite leverages synthesized or natural typos to attack character-based translation models. however, few prior works have explored the generation of adversarial examples specifically for ner. @xcite performed augmentations by concatenating sentences, swapping/inserting/deleting a random character in an entity, entity swapping with outof-vocabulary entities, and cross category swapping. @xcite also took a random entity swapping approach but only selected entities of the same label to preserve linguistic correctness. in this work, we purposely alter the entity type by adding/deleting tokens in predefined word phrase sets and alter the surrounding context. ## expert-guided adversary generation current ner models often deal with unambiguous cases where one entity often gets assigned to the same label. by inducing challenging cases using the overlapping categories @xcite ) that alter the entity and its label, models can then be tested to see whether they are only learning spurious correlations between the token and the label. for the construction of adversarial examples by the altering of entity types, we define three components: (i) eligibility check: we only augment entities that are eligible to change their entity types. (ii) entity token change: by adding or deleting certain predefined tokens, we change the entity type of the original tokens to a target type. (iii) entity context change: to deal with ambiguous tokens, we further add some predefined contexts that correspond to the target entity type. note that predefined words/phrases/contexts used in different scenarios form different predefined word phrase sets, into which embed expert knowledge. during the automatic generation process, we randomly sample from the corresponding word phrase sets. organization for transitioning to organiza-tion, an example is considered eligible if an entity only contains one token (e.g. "brazil"). entity token change in this case refers to inserting words and phrases which are often used behind or after some tokens to form an organization (e.g. add "university" after "brazil"). such words and phrases form a set of size 44, including "university of " (inserted before) and "department" (inserted after). entity context change for organization involves inserting a suitable context after the newly formed organization entity, such as "and its team" and "'s office". such phrases form a set of size 42. location different from transitioning to orga-nization, we want to instead ensure the augmented entity of type location is a real world location. to achieve this, we combine the eligibility check and entity token change: we first define a word phrase set containing words and phrases that are likely to form an organization when concatenated to a location, such as "bank of " (before america). such phrases form a set of size 82. we then perform eligibility check by locating those organization entities containing one of such phrases and change their entity type by deleting those phrases (e.g. delete "re" from "munich re" person similar to transitioning to organiza-tion, an example is considered eligible for transitioning to person if an entity only contains one token (e.g. "colts"). entity token change in this situation refers to the insertion of a token representing a person's last name after the original token to change the entity type to person (e.g. add "zardari" after "colts"). such predefined tokens for insertion form a set of size 152, including examples such as "dutra" and "martin". entity context change for a person then involves inserting a suitable context after the newly formed entity, such as "and her team" and "and his company". such phrases form a set of size 49. we include more examples of word phrases in the appendix ( ## mixup with adversarial examples adversarial training improves a model's robustness to adversarial examples by directly training on ad-versarial examples, however, such training might hurt generalization @xcite or cause overfitting on adversarial features @xcite (predefined word phrases in our case). to this end, we leverage mixup @xcite to mitigate these issues and further improve generalization on the basis of adversarial training @xcite . given a pair of data points (x, y) and (x , y ), where x denotes a data point and y denotes its label in a one-hot representation, mixup @xcite creates a new data point by the interpolation of the data and their labels as shown below with λ being drawn from a beta distribution: in this work, (x, y) is a training example that is eligible for heuristic augmentation and is paired with its heuristically modified version (x , y ). since textual data is discrete and cannot be mixed in the input space, the interpolation of the two examples is computed in the hidden space. following @xcite , let h @xmath0..h n } be the hidden representations after the m-th layer where they are the concatenation of the token representations. the hidden representation for each token in the original example at the mth layer h m is linearly interpolated with h m , the representation for each token in the augmented example, by a ratio λ: then ĥm is passed to the (m + 1)-th layer, and the labels for the final output logits are mixed at the same ratio. m is randomly sampled from {8, 9, 10}. the mixing parameter λ is sampled from a beta distribution: λ ∼ b(α, β), where α and β determine the skew of the beta distribution. in this work, we use two different beta distributions from which to sample λ. for each pair of data points, two mixed data points are generated. one data point is closer to the original examples and the other is closer to the adversarial examples. see appendix b for more details. ## conclusion this work proposed an expert-guided adversarial augmentation for ner consisting of the altering of entity types by strategic selection and modification of tokens and their contexts. using this augmentation strategy on conll 2003 and manually filtering the generated examples for quality, we constructed a high-quality challenging test set for the ner task. we show that sota ner systems suffer from dramatic performance drop when evaluated on our challenging set. beyond simply using the proposed augmentation for adversarial training, we demonstrated that leveraging mixup between original examples and their augmented versions can outperform state-of-the-art baselines on in-distribution data, the challenging set, and few-shot generalization to out-of-domain data.