Data Augmentation for Voice-Assistant NLU using BERT -based Interchangeable Rephrase (2021)

Abstract: We introduce a data augmentation technique based on byte pair encoding and a BERT-like self-attention model to boost performance on spoken language understanding tasks. We compare and evaluate this method with a range of augmentation techniques encompassing generative models such as VAEs and performance-boosting techniques such as synonym replacement and back-translation. We show our method performs strongly on domain and intent classification tasks for a voice assistant and in a user-study focused on utterance naturalness and semantic similarity.

## introduction with conversational assistants becoming more and more pervasive in everyday life, task-oriented dialogue systems are rapidly evolving. these systems typically consist of spoken language understanding (slu) models, tasked with determining the domain category and the intent of user utterances at each turn of the conversation. the ability to quickly train such models to meet changing and evolving user needs is necessary. however, developers often find themselves in situations with access to very little labeled training data. this is especially true when new functions are deployed, and a large user base has not had a chance to utilize the function, thus, limiting the number of available utterances that can be labeled for training. furthermore, the process of labeling large amounts of data can be time consuming and expensive. more recently, these challenges have been both enhanced and complicated by privacy concerns and legislation that may prevent the use of user utterances for training. much of the recent research addressing data paucity has focused on pre-training using selfsupervision and vast amounts of unlabeled data @xcite @xcite . pre-trained models can later be fine-tuned with a much smaller amount of labeled data for specific tasks. in this work, instead of pre-training, we explore methods that enhance and expand the taskspecific training set by using data augmentation. while models such as bert prove to be both useful and relevant, we show that data augmentation during the fine-tuning stage can boost performance even on these large pre-trained models. we implement and compare several pre-existing techniques for data augmentation on natural language understanding (nlu) tasks such as domain classification (dc) and intent classification (ic) for a voice assistant. we also introduce a new method of data augmentation called interchangeable rephrase (ir) with the goal of "rephrasing" an existing utterance using new language while maintaining the original intent or goal (see ## related work recurrent neural network (rnn)-based vae generative models @xcite explicitly model properties of utterances like topic, style, and other higher-level syntactic features. the variational component helps in generating diverse text, thus, we use vaes as a candidate for transforming and augmenting text data in our experiments. building upon unconditioned text generation, conditional vae (cvae) @xcite generates more relevant and diverse text conditioned on certain control attributes, e.g. tense, sentiment @xcite , style @xcite . in this work, our goal is to maintain semantic similarity, therefore, we generate text by conditioning on the original intent or goal. guu et al. generate novel sentences from prototypes by exploiting analogical relationships of sentences. they have shown that generated sentences have a varied style. we extend this idea by using prototype utterances and then editing it into a new utterance or rephrase, using both the vae and cvae architecture, to generate augmented rephrase data. we call these models vae-edit and cvae-edit. back translation (bt) is the process of translating an utterance in a certain language to another language and then translating it back to the original language. certain question answering (qa) models @xcite have observed that backtranslation generates diverse paraphrases, while preserving the semantics of the original sentences. in our case, we use back-translations as rephrases to augment data. another form of simple augmentation techniques is eda: easy data augmentation @xcite . they consist of four operations: synonym replacement, random insertion, random swap and random deletion. they show a boost in text classification tasks with these operations on smaller datasets. since the number of labeled of spoken utterances are limited, we compare to this approach for our nlu tasks. automatic speech recognition (asr) module, which converts audio input into text, introduces some errors in the process before feeding it into downstream nlu tasks. without modifying asr or nlu components, an utterance correction module can be used to help with denoising data @xcite . the reconstructed utterances can be further used as data augmentation. ppdb is a well known paraphrase database consisting of automatically generated paraphrases @xcite . we use this database to rephrase utterances by identifying short phrases within the utterance and replacing them with a related phrase according to the database and pos (e.g. "there is a lot of" → "there are plenty of"). ## interchangeable rephrase bert is pre-trained using two tasks: masked lm (mlm) and next sentence prediction (nsp). in our rephrase task, the end-objective is almost identical to the mlm training, and only this procedure is used to train our self-attention model. mlm training allows the model to predict appropriate word(s) to replace the masked token depending on the context of the rest of the phrase. each input token corresponds to a final hidden vector that is fed into an output softmax over the vocabulary. thus, to rephrase an utterance that has more or fewer tokens than the original, the desired number of tokens for the rephrase must be known a priori. to allow rephrases with unknown lengths, we use byte pair encoding (bpe) to group word ngrams into single tokens @xcite . by using bpe, an individual token may represent a sequence of several words, but the model can still be trained to predict only a single token. we perform bpe on a set of training relevant to the end tasks to get the most frequent n-gram sequences and include these in the model's vocabulary. similarly to ppdb, we assume that many of these n-grams are synonyms and are interchangeable. for example, in the context of a virtual assistant skill that enables finding and reciting recipes, n-gram short phrases such as a "how to make", "tell me how to cook" and "teach me to make" can all be used in the place of the n-gram "how do i make" in the utterance "how do i make a margherita pizza." this interchangeable property is the foundation of our rephrase and data augmentation system. the bert-like self-attention model allows for the predictions to be made based on the context of the input utterance and bpe tokenization allows for variable-length outputs while still only having to predict a single token. though the new bpebased vocabulary requires re-training of the model (pre-trained bert cannot be used), it remains structurally the same as the original bert model. we refer to our bert based interchangeable rephrase model as bert-ir. figure 1 provides an overview of the rephrase model. in order to maintain intent such that the bigram "turn on" in the utterance "turn on the lights" is not inadvertently replaced with "turn off", negative examples and an intent feature are included in a fine-tuning step. the fine-tuning process minimizes a loss function based on the cosine similarity function we emphasize that it is not necessary to use a bert model for this rephrase method. it is technically possible to use a model architecture identical to that of bert. however, given that this is not a fine-tuning task and the model needs to be fully re-trained to support the new n-gram tokens a less resource intensive and data hungry model is preferred. in our experiments we use a bert-like model that leverages self-attention, but has only a fraction of the total parameters of bert. ## experiments in the following experiments, we examine and compare our proposed method with various data aug-mentation techniques in the context of utterance generation (rephrase) for voice assistants. first, we study the properties of the utterances the systems are capable of generating. an ideal data augmentation method should create data that either expands upon or fills in missing gaps of the original training distribution, while still being inherently natural and meaningful. in this context, given an input utterance (from the original distribution), the goal is to generate rephrases that are semantically similar, yet, different enough to positively alter the original training distribution. this is measured in our next two experiments in which we compare the performance of augmented datasets by training an utterance domain classifier (dc) and an intent classifier (ic). finally, we perform a user study to examine the quality of utterance generation in terms of naturalness and semantic similarity. data. we use an original dataset comprised of utterances for a set of 63 skills (domains) for a voice assistant. the skills range from playing a song on spotify to turning on/off in-home appliances to providing the weather. the developer of each skill provides a set of training utterances that users can say as an entry point to the function. each utterance is annotated with an intent, and on average there are 530 utterances per skill (maximum 2000 and minimum 9). for the dc task, of the 43540 utterances in the dataset, 6590 are held out for validation and the remaining utterances are used for training and processed for augmentation. additionally, we use a separate test set that was collected and labeled through user trials and crowd sourcing. each domain in this test set has roughly 900 test utterances. for the ic task, we consider 5 different skills (domains) from the above dataset: weather, sys-temapp, smartthings, tvcontrol and tvsettings. we allocate 30% of the data in each skill for testing and use the remaining for training including augmentation. (see appendix a.3 for more details). ## discussion in our experiments, we show that vae models are capable of generating diverse rephrases. however, these rephrases do not preserve the original meaning. this likely contributed to poorer performance on dc and ic. the discriminator of cvae models trained to condition on a domain (dc) or intent (ic) helps improve semantic similarity resulting in slightly improved performance of nlu tasks. the vae-edit and cvae-edit models perform quite poorly on all comparisons as they don't necessarily preserve the meaning of the utterance when transforming them into an altered style. back-translation yields relatively diverse rephrases; however, it poorly conserves the original meaning. dae just tends to copy a higher fraction of n-grams, while not changing the meaning of the utterance. eda has boosted the dc and ic performance, compared to other methods. however, it merely changes a small percent of words in utterances by replacement operations, which are not always grammatically sound. our bert-ir yields the best performance on the nlu tasks, with a relative error reduction of 46.26% on dc and 43.4% on ic as compared to no additional augmentation. similar to ppdb, our approach generates rephrases with a lower wordoverlap and a significantly higher semantic similarity with the original utterance. the user study reveals that bert-rephrase is an improvement over the ppdb baseline, but does not perform at the level of human generated utterances in terms of naturalness and intelligibility. for examples of generated rephrases see ## conclusion we introduced bert-ir, a simple augmentation strategy based on byte pair encoding and a bertlike self-attention model to generate diverse, natural and meaningful rephrases in the context of utterance generation for voice assistants. we demonstrate that bert-ir performs strongly on spoken understanding tasks like domain classification and intent classification and in a user-study focused on evaluating quality of rephrases based on naturalness and interpretability (intent preservation). intent classification: