DAGA : Data Augmentation with a Generation Approach for Low-resource Tagging Tasks (2020)

Abstract: Data augmentation techniques have been widely used to improve machine learning performance as they facilitate generalization. In this work, we propose a novel augmentation method to generate high quality synthetic data for low-resource tagging tasks with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings. For the supervised settings, we conduct extensive experiments on named entity recognition (NER), part of speech (POS) tagging and end-to-end target based sentiment analysis (E2E-TBSA) tasks. For the semi-supervised settings, we evaluate our method on the NER task under the conditions of given unlabeled data only and unlabeled data plus a knowledge base. The results show that our method can consistently outperform the baselines, particularly when the given gold training data are less.

## introduction a large amount of training data is often essential for neural model performance, especially, for large networks. having more training data can help reduce overfitting and improve model robustness. however, preparing a large amount of annotated data is usually costly, labor intensive and time-consuming. data augmentation @xcite is a useful technique for synthetic data generation, which is widely used in computer vision @xcite ; all words and their tags are paired up by inserting tags before (or after) the words (o tags removed). @xcite and speech @xcite . however, due to the complexity of language, it is more challenging to apply data augmentation techniques to natural language processing (nlp). unlike computer vision and speech, where handcrafted rules (such as rotation, cropping, masking, etc.) can be easily applied to transform original data, it is difficult to generalize such rules for languages. although simple distortion usually does not change the semantics of visual information, deleting or replacing a single word could completely change the meaning of the sentence. one successful method for data augmentation in nlp is back translation @xcite @xcite , where a translation model is used to translate monolingual sentences from target language to source language to generate synthetic parallel sentences. other successful methods include: systematically reordering the dependents of some nodes in gold data to generate synthetic data for dependency parsing @xcite , leveraging knowledge base for question generation @xcite and using simulation-based approach to generate a set of prerequisite toy tasks for qa @xcite . besides, synonym replace-ment, random deletion/swap/insertion, generation with vae or pre-trained language models are also used in some nlp tasks @xcite @xcite @xcite , but mainly for translation and classification tasks. compared with the above-mentioned downstream tasks like translation and classification, sequence tagging is more fragile when it is confronted with data augmentation noises due to the finer granularity of the (token-level) task. annotating unlabeled data with a weak tagger, leveraging aligned bilingual corpora to induce annotation and synonym replacement are three attempted data augmentation methods for sequence tagging @xcite @xcite . weakly labeled data will inevitably introduce more noise. note that annotating unlabeled data with a weak tagger requires in-domain data and in-domain knowledge, otherwise it may suffer from domain-shift problem @xcite . leveraging aligned bilingual corpora requires additional resources, which may not be available for low resource languages. synonym replacement often relies on additional knowledge, e.g., wordnet @xcite , which is a manually designed dictionary that may have low coverage (or not available) for low-resource languages. in this work, we investigate data augmentation with a generation approach for sequence tagging tasks. we first linearize the labeled sentences as shown with an example in figure 1 . then a language model (lm) is trained on the linearized data and used to generate synthetic labeled data. unlike employing weak taggers to label unseen data, our method unifies the processes of sentence generation and labeling using a lm. concretely, a word and its tag in a pair (e.g., "b-per jose") are trained to be generated together, in which the tag-word pairs with high probability will be chosen by the lm during generation ( §3.2). our method does not require additional resources like wordnet. nevertheless, if unlabeled data or knowledge bases are available, our method is also flexible to utilize these resources with a simple but effective conditional generation technique ( §3.4). although some recent work @xcite @xcite also leverages lm for data augmentation, their methods are conditioned on sentence-level tags to generate or modify training data, hence applicable to classification tasks exclusively. to the best of our knowledge, we are the first to utilize generative language models to generate fine-grained synthetic data from scratch for sequence tagging tasks, which introduces a new paradigm for data augmentation in nlp. furthermore, our method does not rely on large pre-trained models and in fact, it employs a simple one-layer recurrent language model @xcite , which is more convenient to train. our method demonstrates encouraging performance when trained on just a few thousand sentences ( §4). to verify the effectiveness of our method, we conduct extensive experiments on different sequence tagging tasks, including named entity recognition (ner), part-of-speech (pos) and end-toend target based sentiment analysis (e2e-tbsa). our method consistently outperforms the baseline methods in both supervised and semi-supervised settings. different from the baseline methods, our method generates novel synthetic data from scratch, and thus introduces more diversity to reduce overfitting. for the semi-supervised settings, our method demonstrates strong ability to exploit useful information from unlabeled data and knowledge base. part-of-speech (pos) tagging part-of-speech tagging consists of assigning a tag that represents a grammatical class to each word in a given sentence. it is a critical component of most nlp systems and is fundamental to facilitate downstream tasks such as syntactic parsing @xcite and opinion analysis @xcite . the current state-ofthe-art pos taggers can achieve over 97.80% accuracy on ptb-wsj @xcite and yield over 96.50% average test accuracy across 21 high-resource languages in ud 1.2 (heinzerling and strube, 2019). however, one of the problems with current pos taggers is that their accuracy can decrease significantly on lowresource languages and rare words @xcite . ## proposed method we propose a novel data augmentation method for sequence tagging tasks. we first linearize labeled sentences, and then train a language model to learn the distribution of words and tags from the linearized sequences for generating synthetic training data. a conditional generation technique is also proposed to exploit unlabeled data and knowledge bases when they are available. ## experiments in this section, we present our experiments in both supervised and semi-supervised settings. in the supervised settings, only gold data are used for augmentation. in the semi-supervised settings, we also leverage unlabeled data and knowledge bases. ## a closer look at synthetic data in this section, we explore in more details why the synthetic data generated by our method can help improve sequence tagging performance. through a closer look at the generated data, we have several interesting findings. ## conclusion in this paper, we show that language models can be used to generate high quality synthetic data for sequence tagging tasks. the generated data introduce more diversity to reduce overfitting, since they are generated from scratch instead of modifying the gold training data. our proposed method demonstrates promising performance improvements on various tagging tasks, especially in the low-resource settings. besides, experiments demonstrate that our method can also effectively utilize unlabeled data and knowledge base for semisupervised training.