Local Additivity Based Data Augmentation for Semi-supervised NER (2020)

Abstract: Named Entity Recognition (NER) is one of the first stages in deep language understanding yet current NER models heavily rely on human-annotated data. In this work, to alleviate the dependence on labeled data, we propose a Local Additivity based Data Augmentation (LADA) method for semi-supervised NER, in which we create virtual samples by interpolating sequences close to each other. Our approach has two variations: Intra-LADA and Inter-LADA, where Intra-LADA performs interpolations among tokens within one sentence, and Inter-LADA samples different sentences to interpolate. Through linear additions between sampled training data, LADA creates an infinite amount of labeled data and improves both entity and context learning. We further extend LADA to the semi-supervised setting by designing a novel consistency loss for unlabeled data. Experiments conducted on two NER benchmarks demonstrate the effectiveness of our methods over several strong baselines. We have publicly released our code at https://github.com/GT-SALT/LADA

## introduction named entity recognition (ner) that aims to detect the semantic category of entities (e.g., persons, locations, organizations) in unstructured text @xcite , is an essential prerequisite for many nlp applications. being one of the most fundamental and classic sequence labeling tasks in nlp, there have been extensive research from traditional statistical models like hidden markov models @xcite and conditional random fields @xcite , to neural network based models such as lstm-crf @xcite and blstm-cnn-crf @xcite , and to recent pre- different kinds of data augmentation approaches have been designed to alleviate the dependency on labeled data for many nlp tasks, and can be categorized into two broad classes: (1) adversarial attacks at token-levels such as word substitutions @xcite or adding noise @xcite , (2) paraphrasing at sentence-levels such as back translations @xcite or submodular optimized models @xcite . the former has already been used for ner but struggles to create diverse augmented samples with very few word replacements. despite being widely utilized in many nlp tasks like text classification, the latter often fails to maintain the labels at the token-level in those paraphrased sentences, thus making it difficult to be applied to ner. we focus on another type of data augmentations called mixup @xcite , which was originally proposed in computer vision and performed linear interpolations between randomly sampled image pairs to create virtual training data. @xcite ; @xcite adapted the idea to textual domains and have applied it to the preliminary task of text classification. however, unlike classifications where each sentence only has one label, sequence labeling tasks such as ner usually involve multiple interrelated labels in a single sentence. as we found in empirical experiments, it is challenging to directly apply such mixup technique to sequence labeling, and improper interpolations may mislead the model. for instance, random sam-pling in mixup may inject too much noise by interpolating data points far away from each other, hence making it fail on sequence labeling. to fill this gap, we propose a novel method called local additivity based data augmentation (lada), in which we constrain the samples to mixup to be close to each other. our method has two variations: intra-lada and inter-lada. intra-lada interpolates each token's hidden representation with other tokens from the same sentence, which could increase the robustness towards word orderings. inter-lada interpolates each token's hidden representation in a sentence with each token from other sentences sampled from a weighted combination of k-nearest neighbors sampling and random sampling, the weight of which controls the delicate trade-off between noise and regularization. to further enhance the performance of learning with limited labeled data, we extend lada to the semi-supervised setting, i.e., semi-lada, by designing a novel consistency loss between unlabeled data and its local augmentations. we conduct experiments on two ner datasets to demonstrate the effectiveness of our lada based models over state-of-the-art baselines. prior work like snippext @xcite , mixtext @xcite and advaug @xcite generalized the idea to the textual domain by proposing to interpolate in output space @xcite , embedding space @xcite , or general hidden space @xcite of textual data and applied the technique to nlp tasks such as text classifications and machine translations and achieved significant improvements. ## method based on the above interpolation based data augmentation techniques, in section 3.1, we introduced a local additivity based data augmentation (lada) for sequence labeling, where creating augmented samples is much more challenging. we continue to describe how to utilize unlabeled data with lada for semi-supervised ner in section 3.4.