Style Transfer as Data Augmentation: A Case Study on Named Entity Recognition (2022)

Abstract: In this work, we take the named entity recognition task in the English language as a case study and explore style transfer as a data augmentation method to increase the size and diversity of training data in low-resource scenarios. We propose a new method to effectively transform the text from a high-resource domain to a low-resource domain by changing its style-related attributes to generate synthetic data for training. Moreover, we design a constrained decoding algorithm along with a set of key ingredients for data selection to guarantee the generation of valid and coherent data. Experiments and analysis on five different domain pairs under different data regimes demonstrate that our approach can significantly improve results compared to current state-of-the-art data augmentation methods. Our approach is a practical solution to data scarcity, and we expect it to be applicable to other NLP tasks.

## introduction large-scale pre-trained language models (plms) such as bert @xcite and t5 @xcite have shown impressive performances on a wide variety of nlp tasks. following the paradigm of self-supervised pre-training and finetuning, these models have achieved state-of-theart performance in many nlp benchmarks such as question answering @xcite , machine translation @xcite , and text summarization @xcite . however, due to the discrepancy of training objectives between language modeling and downstream tasks, the performance of such models may be limited by data scarcity in low-resource domains @xcite . data augmentation is effective in addressing data scarcity. previous work @xcite has mainly focused on using in-domain data to synthesize new datasets for training. when applied to low-resource domains, however, these approaches may not lead to significant improvement gains as the data in lowresource domains is not comparable to that in high-resource domains in terms of size and diversity. recently, many studies @xcite have revealed the potential of leveraging data from high-resource domains to improve low-resource tasks. despite their impressive results, directly using abundant data from high-resource domains can be problematic due to the difference in data distribution (e.g., language shift) and feature misalignment (e.g., class mismatch) between domains @xcite . in this work, we explore the potential of employing style transfer as a way of data augmentation in cross-domain settings. style transfer on natural language aims to change the style-related attributes of text while preserving its semantics, which makes it a reasonable alternative for the purpose of data augmentation. here, we take the named entity recognition (ner) task as a case study to investigate its effectiveness. the general pipeline is shown in figure 1 . compared to the text classification task, the ner task is more difficult as it requires to jointly modify the tokens and their corresponding labels. one of the critical challenges here is the lack of parallel style transfer data annotated with ner labels. our workaround to the lack of data is to figure out how to leverage a non-ner parallel style transfer dataset and a nonparallel ner dataset. both datasets contain pairs of sentences in source and target styles, respectively. this scenario is much more realistic since resources for style transfer tend to be taskagnostic. at the same time, it is easier to come figure 1 : the general pipeline of employing style transfer as data augmentation. for each sample from the source domain, we transfer it to the target domain by changing its style-related attributes as synthetic data, and then use it to train ner models. in the figure, the source and target domains are newswire and social media, respectively, and the underline words are entity instances. by nonparallel task-specific datasets with different styles. moreover, a solution that can successfully exploit data in this manner will be relevant in general data augmentation scenarios beyond ner. we formulate style transfer as a paraphrase generation problem following previous work @xcite and propose a novel neural architecture that uses plms as the generator and discriminator. the generator aims to transform text guided by task prefixes while the discriminator aims to differentiate text between different styles. we jointly train the generator and discriminator on both parallel and nonparallel data to learn transformations in a semi-supervised manner. we apply paraphrase generation directly on the parallel data to bridge the gap between source and target styles. for nonparallel data, we use a cycle-consistent reconstruction to re-paraphrase back the paraphrased sentences to its original style. additionally, to guarantee the generation of valid and coherent data, we present a constrained decoding algorithm based on prefix trees along with a set of key ingredients for data selection to reduce the noise in synthetic data. we systematically evaluate our proposed methods on 5 different domain pairs under different data regimes. experiments and analysis show that our proposed method can effectively generate synthetic data by imitating the style of a target low-resource domain and significantly outperform previous state-of-theart methods. in summary, our contributions are as follows: style transfer style transfer aims to adjust the stylistic characteristics of a sentence while preserving its original meaning. it has been widely studied in both supervised @xcite @xcite @xcite and unsupervised manners @xcite @xcite @xcite @xcite . although style transfer is frequently utilized in many nlp tasks such as dialogue systems @xcite , sentiment transfer @xcite , and text debiasing @xcite , its application on data augmentation still remains understudied by the nlp community. to facilitate research in this direction, we study style transfer as data augmentation and propose a novel approach to explore transferring the stylerelated attributes of text to add more variations in the training data. data augmentation data augmentation has been recently receiving increasing and widespread attention in the field of ner. the mainstream methods can be group into two categories: rulebased approaches @xcite @xcite , and model-based approaches @xcite @xcite @xcite @xcite . although previous methods have shown promising results, they may not perform well in low-resource domains as the size and diversity of the training data are limited. recent work have studied data augmentation by leveraging the data from high-resource domains. @xcite investigated data transformation with implicit textual patterns while @xcite explored replacing entities between domains with cross-domain anchor pairs. motivated by their impressive results, we further study how to bridge the gap of data difference between domains and explore a better to leverage the data in high-resource domains for data augmentation. ## problem formulation and preliminaries considering a nonparallel ner dataset d consisting of source data d src from a high-resource domain and target data d tgt from a low-resource domain, training a model directly on d src and evaluating on d tgt is expected to give low prediction performance due to the stylistic differences (e.g., lexicons and syntax) between domains. in this work, we transform the data from a source domain to a target domain by changing text style and use the resulting transformed data to improve ner performance on d tgt . to this end, we assume access to a dataset p that contains pairs of parallel source and target sentences, respectively, to provide supervision signals for style transfer and a pre-trained generative model g θ based on an encoder-decoder @xcite architecture. given an input sequence @xmath0, x 2 , ..., x n } of length n , the generator g θ is pre-trained to maximize the conditional probability in an autoregressive manner: data preparation applying pre-trained generative models requires converting ner tokens and labels into a linearized format. in this work, we assume that the ner dataset follows the standard bio labeling schema @xcite . previous work @xcite has explored pre-pending each label to its corresponding token so that the model can capture the dependency and relationship between tokens and labels. however, we argue that this approach requires prohibitively large amounts of training data to adequately model the labeling schema. recent work has also found that this format introduces too many hallucinated tokens and thus makes the learning problem significantly harder as the model needs to track token indices implicitly @xcite . in the scenario where the number of annotated data is limited, the model may be confused with the labeling schema and tend to generate noisy samples. to mitigate this issue, we propose to linearize the data by only @xmath1. for instance, the sample "a rainy day in [new] b-loc [york] i-loc " will be converted to "a rainy day @xmath2. furthermore, we also prepend task prefixes to the beginning of the sentences to guide the direction of style transfer, where a prefix is a sequence of words for task description specifying the source and target styles of transfer (e.g., "transfer source to target: "). ## method in this work, we explore style transfer as a data augmentation method to improve the performance of ner systems on low-resource domains. we propose an adversarial learning framework to generate a paraphrase of the source data in a highresource domain whose style conforms to the target data in a low-resource domain. the proposed framework comprises two main components: (1) paraphrase generation, which aims to rephrase the sentence to a different style with supervision, and (2) cycle-consistent reconstruction, which aims to transfer the sentence to a different style and then back to its original style with no supervision. the overall architecture is shown in figure 2 . besides, we design a constrained decoding algorithm to guarantee the generation of valid samples and present a set of key ingredients to select high-quality generated sentences. ## experiments in this section, we present the experimental setup and results. we extensively evaluate our proposed method on five different domain pairs and compare our proposed method with state-of-the-art systems on data augmentation for ner in crossdomain scenarios. ## conclusion in this paper, we propose an effective approach to employ style transfer as a data augmentation method. specifically, we present an adversarial learning framework to bridge the gap between different text styles and transfer the entity knowledge from high-resource domains to low-resource domains. additionally, to guarantee the generation of valid and coherent data, we design a constrained decoding algorithm along with a set of key ingredients for data generation and selection. we undertake experiments on five different domain pairs. the experimental results show that our proposed method can effectively transfer the data across domains to increase the size and diversity of training data for low-resource ner tasks. for the future work, we plan to explore style transfer as data augmentation in cross-lingual settings (i.e., transfer entity knowledge across languages instead of just domains). additionally, since our approach is based on pre-trained language models, it would be interesting to explore leveraging pre-trained knowledge for data augmentation.