Controlled Data Generation via Insertion Operations for NLU (2022)

Abstract: Use of synthetic data is rapidly emerging as a realistic alternative to manually annotating live traffic for industry-scale model building. Manual data annotation is slow, expensive and not preferred for meeting customer privacy expectations. Further, commercial natural language applications are required to support continuously evolving features as well as newly added experiences. To address these requirements, we propose a targeted synthetic data generation technique by inserting tokens into a given semantic signature. The generated data are used as additional training samples in the tasks of intent classification and named entity recognition. We evaluate on a real-world voice assistant dataset, and using only 33% of the available training set, we achieve the same accuracy as training with all available data. Further, we analyze the effects of data generation across varied real-world applications and propose heuristics that improve the task performance further.

## introduction one of the common challenges to deploying natural language understanding (nlu) techniques at scale in commercial applications is the necessity for continuous annotation of user data. models can then be re-trained and updated to capture new usage patterns. this process is expensive, labor intensive, and time-consuming. at an age when user privacy is becoming the focus of increased concern in all ai applications, manual review of user data normally required for such annotation becomes highly undesirable. consequently, multiple initiatives are undertaken towards minimizing the amount of human annotations needed for training nlu models. data augmentation (da) refers to strategies that aim at increasing the diversity of training samples without explicitly collecting new data. in this work, we present a generative model that is used to generate labeled synthetic data. given a set of utterance templates foot_0 that we construct from a limited amount of labeled data, our goal is to generate synthetic utterances and augment the original (reduced) training data, with the objective of improving the model robustness and performance. we focus on the special case where the synthetic data must retain a specific fine-grained interpretation of the original utterance, such as token-level annotation. for example, we would like to control the composition of entities @xcite in the training data when expanding to new features while maintaining nlu model performance. in our proposed approach, we control the desired annotation by re-framing the generation process as insertion rather than left-to-right generation. we preserve the desired entities in the synthetic example by including them in the model's input during generation and introduce methods to explicitly prevent entity corruption during the generation process. our contributions are as follows: (i) we propose a novel synthetic data generation technique using insertion transformers that allows for token-level control over the generated synthetic utterance. (ii) we demonstrate the usefulness of the proposed approach for nlu model building. our model which is trained using a limited fraction of user data combined with synthetic data matches the performance of a model trained with the entire real data. (iii) we apply domain-specific heuristics to improve the quality of synthetic data, which would further improving task performance. ## background our nlu models are responsible for interpreting the corresponding domain, intent and actionable slots of customer utterances. these categories are modularized, i.e., utterances belonging to a partic- playmusicintent albumname(shake it off) artistname(taylor swift) playmusicintent can you play albumname(shake it off) by artistname(taylor swift) now input template generated output ular domain (e.g., books) are supported by a specific set of intents (e.g., playbook) and actionable slots (e.g., bookname), and served by the domainspecific intent classification (ic) and named entity recognition (ner) models. in this work we experiment and evaluate ic and ner tasks across multiple domains. we explore the use of synthetic data as an additional source for training the models of these domains. while a number of data augmentation techniques for natural language have been proposed, ranging from token-level perturbations @xcite to paraphrase generation @xcite and auto-regressive models @xcite @xcite , these techniques can not be directly applied to token labeling tasks such as ner. specifically, synthetic data generation for ner involves two additional challenges: ## experimental setup in order to study the effectiveness of synthetically generated data, we evaluate nlu model performance in reduced data regime. for each domain, we build multiple ic-ner models using all real data, a reduced set of real data and combination of real and synthetic data. all models within a domain share the same training hyper-parameters, including architecture and encoder. they differ only in training data composition. similar to @xcite , we limit the focus of this work to synthetic data generation and defer hyper-parameter optimization to future work. we use apache mxnet @xcite to build both git and ic-ner models in this work. full: this baseline is trained using all real data and default training hyper-parameters for each domain. this setup reflects the current performance of nlu models in production and serves as an estimate for lower bound in error metric for all other models. reduced: we train another baseline using onethird of real data. our reduction of two-thirds of the data is motivated by a privacy control feature allowing customers to delete their data. given the trends, we estimated a worst case drop of 67% in our annotated data before it can be refilled with more human annotations. to simulate this worst case scenario, we randomly downsample across all utterances. duplicate: to reduce the impact of hyperparameters we also train a model with the reduced set duplicated to reach the full training size. we refer to it as duplicate. we note that duplication has been used as a baseline for data augmentation in @xcite @xcite eda: easy data augmentation (eda) consists of four simple operations: synonym replacement, random insertion, random swap, and random deletion. eda has shown to be a strong baseline, outperforming complex model-based baselines particularly for small datasets @xcite . similar to git, eda can provide control and flexibility over the type of data generated, which is a key requirement from our users. git: (ours). we use the reduced set to finetune domain-specific git models and also as input templates during inference, with fixed hyperparameters. during inference, we control the number of generated synthetic utterances which is augmented with reduced set. we test two configurations: in git_50, the fraction of synthetic to real data is set to 50% while with git_200, the fraction of synthetic to real data is set to 200%. in the former, synthetic data size is kept smaller than real data while in latter, we add enough synthetic data to compensate for removed data. ## discussion while the overall regression appears modest, there exists significant variation among domains (figure 3 ). we can see that git improves semer only among certain domains when compared to reduced (e.g., music but not sports). in general, domains with relatively higher traffic exhibit moderate regression (<5%). recall that for simplicity we use the same hyper-parameters across all domains. while we observe that git_50 (wiki) and git_50 (wiki+in-domain) configurations provide overall improvements over reduced, we investigate whether data reduction effects are related to improvements with synthetic data addition. specifically, using the null hypothesis that the relative semer (%) between data reduction (@xmath0, we estimate the pearson correlation between them using two-tailed t-distribution. in ## limitations since our primary focus in this work was developing insertion transformers for da, we did not explore extensive hyper-parameter optimization while building ic-ner models using combination of real and synthetic data. for example, we observed that adding the same fraction of synthetic data results in significant performance variations across domains, suggesting that per-domain parameter optimization may be yield improved performance. ## conclusion we demonstrated da using git as a feasible data generation technique to mitigate reduced annotation volumes for ic and ner tasks. we showed that nlu models trained on 33% real data and synthetic data perform similar to models trained on full real data. further, on domains with highest semer regressions we improved the quality of synthetic data by filtering them with model confidence scores. among domains which benefit from synthetic data, we showed that appropriate carrier token insertion enhances utterances' semantics and their value as training samples. in the future, we would like to explore data generation with entities replaced through knowledge base sampling. such finer control over entities better supports new feature expansion and enhances customer privacy.