Syntax-driven Data Augmentation for Named Entity Recognition (2022)

Abstract: In low resource settings, data augmentation strategies are commonly leveraged to improve performance. Numerous approaches have attempted document-level augmentation (e.g., text classification), but few studies have explored token-level augmentation. Performed naively, data augmentation can produce semantically incongruent and ungrammatical examples. In this work, we compare simple masked language model replacement and an augmentation method using constituency tree mutations to improve the performance of named entity recognition in low-resource settings with the aim of preserving linguistic cohesion of the augmented sentences.

## introduction deep neural networks have proven effective for a wide variety of tasks in natural language processing; however, these networks often require large annotated datasets before they begin to outperform simpler models. such data is not always available or sufficiently diverse, and its collection and annotation can be an expensive and slow process. the trend of fine-tuning large-scale language models originally trained using self-supervision has helped to alleviate the need for large annotated datasets, but this approach relies on the dataset for fine-tuning being diverse enough to train a model that generalizes well. careful data augmentation can help to improve dataset diversity and ultimately the model's generalizability. data augmentation, a technique to generate data given training set characteristics, continues to play a critical role in low-resource settings; however, the majority of work on data augmentation focuses on improving document-level tasks such as text classification. far less attention has been paid to token-level tasks @xcite . prevailing approaches to sequence tagging tasks such as named entity recognition (ner) require token-level ground truth. naive replacement-based methods for augmentation may introduce noise in the form of sentences that are ungrammatical, semantically vacuous, or semantically incongruent. whether in the form of insertions, deletions, or substitutions, care must be taken with token-level augmentation to preserve linguistic cohesion. there is evidence that large language models may possess some syntactic knowledge @xcite . work by @xcite suggests that incorporating syntactic tasks into pre-training improves the performance of large language models. inspired by these findings, we investigate how constituency trees might be used to guide data augmentation. we use treebased transformations to mutate sentences while minimizing undesired side effects of syntactic manipulation (i.e., preserving linguistic cohesion). related work by @xcite explores a similar approach in different settings. they focus on the effect of constituency based replacement in single classification and pair sentence classification tasks, while this work examines token-level classification. we compare our syntax-driven method with no augmentation (our baseline), augmented data generated through cloze-style @xcite masked language modeling using a bert-based classifier, successful approaches introduced by @xcite , and two of the top-performing augmentation strategies according to past work: synonymbased replacement and mention-based replacement. following prior work, we we use the i2b2-2010 english language dataset @xcite for ner. ## related work a variety of approaches have been explored for document-level data augmentation. the usage of backtranslation to generate augmented samples was introduced by @xcite . @xcite explored synonym replacement, random in-sertion, random swap, and random deletion for text classification. @xcite introduced an approach using monte carlo tree search (mcts) to guide generation of synthetic data. augmentation for token-level sequence tagging, however, is understudied. simple approaches for token-level classification (e.g., synonym replacement, mention replacement, shuffling, etc.) was investigated by @xcite . they used a sample of 50, 150, and 500 sentences to simulate a low-resource setting. token linearization (tl) was introduced by @xcite . the main idea of tl is to incorporate ner tags inside the training sentences themselves. ding et al. (2020) experimented with various sizes of training data across six languages. a similar idea to that presented here has been used in parallel research by @xcite . their approach, treemix, works similarly to our approach in that it replaces a phrase with another phrase from another another training instance by swapping phrases with the same constituent labels. @xcite demonstrated that treemix outperforms a method that selects a random span of text as the replacement for a target phrase, suggesting that @xmath0. ## conclusion and future work in this work, we examined data augmentation with a large-scale language model (lm) and constituency tree mutation (st). we compared these augmentation methods with a baseline and previously proposed strategies for data augmentation: synonym replacement (sr) and mention replacement (mr). we found that sr performance is still most effective, by a small margin, but the performance degrades quickly as the data size increased. we have also observed that both lm and cr retained their performance throughout larger dataset sizes. we also showed that cr performance seems to be consistent in its improvement as the augmented dataset size increases, while the lm showed degrading performance with more augmented data. future work should include improvements that hybridize the syntactic transformations with a largescale language model. one possibility to increase the performance of the baseline language model is to train it to recognize phrase-level constituents and functional categories to understand more about constituency tags by first randomly swapping a few tokens with part of speech tags. for example, the original sentence is "i take my medicine.", then the pre-training sentence is "i vb my medicine." and "i take my nn.". we hypothesize that this pretraining will improve the prediction performance of the baseline language model that we used for cr augmentation by attending to functional categories. another possibility is to assign different weights to datapoints that inform the model how much to "trust" augmented data compared to gold data. this weight could be in the form of different learning rate.