Data Augmentation for Cross-Domain Named Entity Recognition (2021)

Abstract: Current work in named entity recognition (NER) shows that data augmentation techniques can produce more robust models. However, most existing techniques focus on augmenting in-domain data in low-resource scenarios where annotated data is quite limited. In this work, we take this research direction to the opposite and study cross-domain data augmentation for the NER task. We investigate the possibility of leveraging data from high-resource domains by projecting it into the low-resource domains. Specifically, we propose a novel neural architecture to transform the data representation from a high-resource to a low-resource domain by learning the patterns (e.g. style, noise, abbreviations, etc.) in the text that differentiate them and a shared feature space where both domains are aligned. We experiment with diverse datasets and show that transforming the data to the low-resource domain representation achieves significant improvements over only using data from high-resource domains.

## introduction named entity recognition (ner) has seen significant performance improvements with the recent advances of pre-trained language models @xcite . however, the high performance of such models usually relies on the size and quality of training data. when used under low-resource or even zero-resource scenarios, those models struggle to generalize over diverse domains @xcite , and the performance drops dramatically due to the lack of annotated data. unfortunately, annotating more data is often expensive and time-consuming, and it requires expert domain knowledge. moreover, annotated data can quickly become outdated in domains where language changes rapidly (e.g, social media), lead-figure 1 : examples from ontonotes 5.0 dataset and temporal twitter dataset. the language variations and abbreviations in the text from social media domain make it clearly different from the formal text in newswire domain. a common approach to alleviate the limitations mentioned above is data augmentation, where automatically generated data can increase the size and diversity in the training set, while resulting in model performance gains. but data augmentation in the context of ner is still understudied. approaches that directly modify words in the training set (e.g, synonym replacement @xcite and word swap @xcite ) can inadvertently result in incorrectly labeled entities after modification. recent work in ner for low resource scenarios is promising @xcite but it is limited to same domain settings and improvements decrease drastically with smaller sizes of training data. to facilitate research in this direction, we investigate leveraging data from high-resource domains by projecting it into low-resource domains. based on our observations, the text in different domains usually presents unique patterns (e.g. style, noise, abbreviations, etc.). as shown in figure 1 , the text in the newswire domain is long and formal, while the text in the social media domain is short and noisy, often presenting many grammar errors, spelling mistakes, and language variations. in this work, we hypothesize that even though the textual patterns are different across domains, the semantics of text are still transferable. additionally, there are some invariables in the way the named entities appear and we assume that the model can learn from them. in this work, we introduce a cross-domain autoencoder model capable of extracting the textual patterns in different domains and learning a shared feature space where domains are aligned. we evaluate our data augmentation method by conducting experiments on two datasets, including six different domains and ten domain pairs, showing that transforming the data from high-resource to lowresource domains is a more powerful method than simply using the data from high-resource domains. we also explore our data augmentation approach in the context of the ner task in low-resource scenarios for both in-domain and out-of-domain data. to summarize, we make the following contributions: ## related work data augmentation aims to increase the size of training data by slightly modifying the copies of already existing data or adding newly generated synthetic data from existing data @xcite . it has become more practical for nlp tasks in recent years, especially in lowresource scenarios where annotated data is limited @xcite . without collecting new data, this technique reduces the cost of annotation and boosts the model performance. previous work has studied the data augmentation for both token-level tasks @xcite and sequence-level tasks @xcite . related to data augmentation on ner, @xcite conducted a study that primarily focuses on the simple data augmentation methods such as synonym replacement (i.e., replace the token with one of its synonyms) and mention replacement (i.e., randomly replace the mention with another one that has the same entity type with the replacement). @xcite studied sequence mixup (i.e., mix eligible sequences in the feature space and the label space) to improve the data diversity and enhance sequence labeling for active learning. @xcite presented a novel approach using adversarial learning to generate high-quality synthetic data, which is applicable to both supervised and semi-supervised settings. in cross-domain settings, ner models struggle to generalize over diverse genres @xcite . most existing work mainly studies domain adaptation @xcite @xcite which aims to adapt a neural model from a source domain to achieve better performance on the data from the target domain. @xcite proposed a zero-resource cross-domain framework to learn the general representations of named entities. @xcite studied the knowledge of domain difference and presented a novel parameter generation network. other efforts include the different domain adaptation settings @xcite and effective cross-domain evaluation @xcite . in our work, we focus on cross-domain data augmentation. the proposed method aims to map data from a high-resource domain to a low-resource domain. by learning the textual patterns of the data from different domains, our proposed method transform the data from one domain to another and boosts the model performance with the generated data for ner in low-resource settings. ## proposed method in this work, we propose a novel neural architecture to augment the data by transforming the text from a high-resource domain to a lower-resource domain for the ner task. the overall neural architecture is shown in figure 2 . we consider two unparalleled datasets: one from the source domain d src and one from the target domain d tgt . we first linearize all sentences by inserting every entity label before the corresponding word. at each iteration, we randomly pair a sentence from d src and a sentence from d tgt as the input to the model. the model starts with word-by-word denoising reconstruction and then detransforming reconstruction. in denoising reconstruction, we aim to train the model to learn a compressed representation of an input based on the domain it comes from in an unsupervised way. we inject noise into each input sentence by shuffling, dropping, or masking some words. the encoder is trained to capture the textual semantics and learn the pattern that makes each sentence different from sentences in other domains. then we train the decoder by minimizing a training objective that measures its ability to reconstruct each sentence from its noisy version in its corresponding domain. in detransforming reconstruction, the goal is to transform sentences from one domain to another domain based on their textual semantics. we first transform each sentence from the source/target domain to the target/source domain with the model from the previous training step as the input. the encoder then generates latent representations for transformed sentences. after that, different from denoising reconstruction, the decoder here is trained to reconstruct each sentence from its transformed version in its corresponding domain. besides denoising and detransforming reconstruction, we also train a discriminator to distinguish whether the latent vector generated by the encoder is from the source domain or target domain. in this case, the encoder can generate a meaningful intermediate represen-tation. otherwise, the model would bypass the intermediate mapping step between domains and replace it by memorizing rather than generalizing. in the following sections, we will introduce the details of our model architecture and the training algorithm. ## experiments in this section, we will introduce the cross-domain mapping experiment and the ner experiment. in the cross-domain mapping experiment, we analyze the reconstruction and generation capability of the proposed model. we then tested our proposed method and evaluated the data generated from our model on the ner task. details of the data set, experimental setup, and results are described below. ## analysis in this section, we take a step further towards exploring our approach in the context of the ner task in low-resource scenarios. specifically, we investigate two crucial questions on data augmentation for both in-domain and out-of-domain datasets. q1: does the model require a large number of training data from the target domain? to answer this question, we randomly select 5%, 10%, 20%, 40%, 80% of samples from the target domain as the train data to train our cross-domain autoencoder model. we consider nw as the source domain and sm as the target domain. after training the model, we generate samples from nw to sm and merge them into the training set from nw as the new training set. then we do evaluation on the test set from the sm. we establish a baseline that only uses the data from nw for training the model. figure 3 shows the results of model performance on the ner task. from this figure, we can see that our method can consistently achieve higher f1 scores than the baseline. even if there is only 5% training samples from the target domain, our model can still achieve significant improvements, averaging 4.81% over the baseline. q2: can we generate enough data to reach competitive results in the target domain? we train our cross-domain autoencoder model with all the data from nw and only 5% data (totally 500 samples) from sm. then we generate synthetic data from nw to sm. after that, we do evaluation on the ner task by combining 5% data from sm and different numbers of training samples as training data. figure 4 shows the f1 scores achieved by sequence labeling model. with 5% data from sm, the model can only reach 65.25% f1 score. when we add more generated samples, the model can reach up to 77.59% f1 score, pushing the model performance by 12.34%. ## discussion and limitation in this work, we explore how to leverage existing data from high-resource domains to augment the data in low-resource domains. we introduce a cross-domain autoencoder model that captures the textual patterns in each domain and transforms the data between different domains. however, there are several limitations: 1) the maximum lengths of the generated sequences: at each iteration, the maximum lengths of the generated sequences xtgt and xsrc are set to the same as original input sequences x src and x tgt , respectively. this is not an ideal case because, intuitively, a short sentence in the source domain may correspond to a long sentence in the target domain. fixing the maximum lengths of the generated sequences may hurt the model on capturing the semantics of original input sequences and result in a lower quality of reconstruction. 2) unparalleled datasets: in our experiments, the datasets d src and d tgt are unparalleled, which means the sentences x src and x tgt do not correspond to each other. when we generate sequences from one domain to another, there is no guidance and thus we cannot control the quality of the generated sequences xtgt and xsrc . although our proposed method can not outperform the upper bound (i.e, training the model on the data from the target domain), it can achieve a significant improvement than only using the data from the source domain, providing a strong lower bound baseline for semi-supervised learning. ## conclusion in this work, we present a novel neural architecture for data augmentation where the model learns to transform data from a high resource scenario to data that resembles that of a low resource domain. by training the model on reconstruction loss, it can extract the textual patterns in each domain and learn a feature space where both domains are aligned. we show the effectiveness of our proposed method by evaluating a model trained on the augmented data for ner, concluding that transforming text to low-resource domains is more powerful than only using the data from high-resource domains. our future work includes three directions: i) explore how to embed more supervision about forcing a better alignment in the latent space, ii) design a strategy to control the quality of the generated sequences, and iii) generalize our method to other nlp tasks such as text classification.