Data Augmentation for Low-Resource Named Entity Recognition Using Backtranslation (2021)

Abstract: The state of art natural language processing systems relies on sizable training datasets to achieve high performance. Lack of such datasets in the specialized low resource domains lead to suboptimal performance. In this work, we adapt backtranslation to generate high quality and linguistically diverse synthetic data for low-resource named entity recognition. We perform experiments on two datasets from the materials science (MaSciP) and biomedical (S800) domains. The empirical results demonstrate the effectiveness of our proposed augmentation strategy, particularly in the low-resource scenario.

## introduction most recently, various deep learning methods have demonstrated state of art performance for many natural language processing tasks such as text classification, sentiment analysis and named entity recognition. the availability of large training datasets is crucial to achieve this improved performance and avoid overfitting. however, in many real-world applications collecting such large training data is not possible. this is especially true for specialized domains, such as the material science or biomedical domain, where annotating data requires expert knowledge and is usually time-consuming and expensive. data augmentation (da) @xcite ) has been investigated to overcome this low resource problem. label preserving synthetic data generation is widely used in computer vision @xcite @xcite and speech domains @xcite . the discrete nature of language makes it difficult to adapt data augmentation strategies from computer vision and speech to natural language processing. unlike computer vision, where hardcoded transformations (such as rotation, masking, cropping etc.) can be easily applied without changing the label semantics, the manipulation of a single word in a sentence could change its meaning. recently, there is an increased interest in applying data augmentation to natural language processing tasks. most augmentation methods explore sentence-level tasks such as sentiment analysis @xcite ), text classification @xcite and sentence-pair tasks such as natural language inference (min et al., 2020) and machine translation @xcite . the augmentation methods either employ heuristics such as word replacement @xcite @xcite , word swap @xcite or random deletion @xcite to generate augmented instances by manipulating a few words in the original sentence; or generate completely artificial instances via sampling from generative models such as variational autoencoders @xcite or backtranslation models @xcite . the sequence labelling tasks such as named entity recognition (ner) and part-of-speech tagging (pos) involves prediction at the token level. this makes applying token-level transformation difficult as such manipulations may change the corresponding token level label. the existing da methods for sequence labelling uses dependency tree morphing (sahin and steedman, 2018), mixup @xcite to generate queried samples in the active learning scenario @xcite , sample novel sequences from a trained language model @xcite and apply pre-defined heuristics such as label-wise token and synonym replacement @xcite . the existing sequence labelling da methods are limiting as they: a). rely on linguistics resources like dependency parser or wordnet b). involves training a language model c). generate grammatically incoherent sequences d). cannot generate linguistically diverse sequences. motivated by the advancements in machine translation and the availability of high-quality machine translation systems @xcite @xcite , in this work we adapt backtranslation to the task of ner. backtranslation (bt) can automatically generate diverse paraphrases of a sentence or a phrase by naturally injecting linguistic variations. the injected linguistic variations can be further diversified by introducing layers of intermediate language translations. in this work, we generate paraphrases of one or several phrases in a sentence. we empirically demonstrate the effectiveness of our proposed method on two domain-specific ner datasets. ## related work there is an abundance of recent work on da methods for nlp tasks, we refer the readers to feng et al. for an extensive survey. in this section we narrow our focus to existing da methods for sequence labelling tasks like ner and pos. we categorize existing da methods for sequence labelling into two categories: rule-based: da primitives, which use predefined easy-to-compute transformations. we briefly describe six of such transformations proposed in the existing work: replace a token with another token of the same entity type at random @xcite . ## conclusion in this paper, we adapt backtranslation to the tokenlevel sequence tagging task of ner. we show that backtranslation can generate high-quality coherent and linguistically diverse synthetic data for ner. the experiments on two domain-specific datasets demonstrate the effectiveness of backtranslation as a competitive data augmentation strategy for ner.