An Analysis of Simple Data Augmentation for Named Entity Recognition (2020)

Abstract: Simple yet effective data augmentation techniques have been proposed for sentence-level and sentence-pair natural language processing tasks. Inspired by these efforts, we design and compare data augmentation for named entity recognition, which is usually modeled as a token-level sequence labeling problem. Through experiments on two data sets from the biomedical and materials science domains (i2b2-2010 and MaSciP), we show that simple augmentation can boost performance for both recurrent and transformer-based models, especially for small training sets.

## introduction modern deep learning techniques typically require a lot of labeled data @xcite . however, in real-world applications, such large labeled data sets are not always available. this is especially true in some specific domains, such as the biomedical and materials science domain, where annotating data requires expert knowledge and is usually time-consuming @xcite . different approaches have been investigated to solve this low-resource problem. for example, transfer learning pretrains language representations on self-supervised or rich-resource source tasks and then adapts these representations to the target task @xcite . data augmentation expands the training set by applying transformations to training instances without changing their labels @xcite . recently, there is an increased interest on applying data augmentation techniques on sentence-level and sentence-pair natural language processing (nlp) tasks, such as text classification @xcite , natural language inference @xcite and machine translation @xcite . augmentation methods explored for these tasks either create augmented instances by manipulating a few words in the original instance, such as word replacement @xcite @xcite , random deletion @xcite , or word position swap @xcite ; or create entirely artificial instances via generative models, such as variational auto encoders @xcite or back-translation models @xcite . different from these sentence-level nlp tasks, named entity recognition (ner) does predictions on the token level. that is, for each token in the sentence, ner models predict a label indicating whether the token belongs to a mention and which entity type the mention has. therefore, applying transformations to tokens may also change their labels. due to this difficulty, data augmentation for ner is comparatively less studied. in this work, we fill this research gap by exploring data augmentation techniques for ner, a token-level sequence labeling problem. our contributions can be summarized as follows: ## related work in this section, we survey previously used data augmentation methods for nlp tasks, grouping them into four categories: word replacement: various word replacement variants have been explored for text classification tasks. @xcite and @xcite replace words with one of their synonyms, retrieved from an english thesaurus (e.g., wordnet). @xcite replace words with other words that are predicted by a language model at the word positions. @xcite replace uninformative words with low tf-idf scores with other uninformative words for topic classification tasks. for machine translation, word replacement has also been used to generate additional parallel sentence pairs. @xcite replace words in both the source and the target sentence by other words uniformly sampled from the source and the target vocabularies. @xcite search for contexts where a common word can be replaced by a low-frequency word, relying on recurrent language models. @xcite replace a randomly chosen word by a soft word, which is a probabilistic distribution over the vocabulary, provided by a language model. in addition, there are two special word replacement cases, inspired by dropout and masked language modeling: replacing a word by a zero word (i.e., dropping entire word embeddings) @xcite , or by a [mask] token @xcite . mention replacement: @xcite augment a question answering training set using an external knowledge base. in particular, they extract nominal groups in the training set, perform string matching with entities in wikidata, and then randomly replace them with other entities of the same type. in order to remove gender bias from coreference resolution systems, @xcite propose to generate an auxiliary dataset where all male entities are replaced by female entities, and vice versa, using a rule-based approach. swap words: @xcite randomly choose two words in the sentence and swap their positions to augment text classification training sets. min et al. (2020) explore syntactic transformations (e.g., subject/object inversion) to augment the training data for natural language inference. s ¸ahin and steedman (2018) rotate tree fragments around the root of the dependency tree to form a synthetic sentence and augment low-resource language part-of-speech tagging training sets. generative models: yu et al. (2018) train a question answering model with data generated by backtranslation from a neural machine translation model. @xcite and @xcite use a sequence-to-sequence model to generate diversely augmented utterances to improve the dialogue language understanding module. @xcite convert data from a high-resource language to a lowresource language, using a bilingual dictionary and an unsupervised machine translation model in order to expand the machine translation training set for the low-resource language. label-wise token replacement (lwtr): for each token, we use a binomial distribution to randomly decide whether it should be replaced. if yes, we then use a label-wise token distribution, built from the original training set, to randomly select another token with the same label. thus, we keep the original label sequence unchanged. taking the instance in synonym replacement (sr): our second approach is similar to lwtr, except that we replace the token with one of its synonyms retrieved from wordnet. note that the retrieved synonym may consist of more than one token. however, its bio-labels can be derived using a simple rule: if the replaced token is the first token within a mention (i.e., the corresponding label is 'b-entitytype'), we assign the same label to the first token of the retrieved multi-word synonym, and 'i-entitytype' to the other tokens. if the replaced token is inside a mention (i.e., the corresponding label is 'i-entitytype'), we assign its label to all tokens of the multi-word synonym. mention replacement (mr): for each mention in the instance, we use a binomial distribution to randomly decide whether it should be replaced. if yes, we randomly select another mention from the original training set which has the same entity type as the replacement. the corresponding bio-label sequence can be changed accordingly. for example, in ## conclusion we survey previously used data augmentation methods for sentence-level and sentence-pair nlp tasks and adapt them to ner, a token-level task. through experiments on two domain-specific data sets, we show that simple data augmentation can improve performance even over strong baselines.