Data-Efficient Paraphrase Generation to Bootstrap Intent Classification and Slot Labeling for New Features in Task-Oriented Dialog Systems (2020)

Abstract: Recent progress through advanced neural models pushed the performance of task-oriented dialog systems to almost perfect accuracy on existing benchmark datasets for intent classification and slot labeling. However, in evolving real-world dialog systems, where new functionality is regularly added, a major additional challenge is the lack of annotated training data for such new functionality, as the necessary data collection efforts are laborious and time-consuming. A potential solution to reduce the effort is to augment initial seed data by paraphrasing existing utterances automatically. In this paper, we propose a new, data-efficient approach following this idea. Using an interpretation-to-text model for paraphrase generation, we are able to rely on existing dialog system training data, and, in combination with shuffling-based sampling techniques, we can obtain diverse and novel paraphrases from small amounts of seed data. In experiments on a public dataset and with a real-world dialog system, we observe improvements for both intent classification and slot labeling, demonstrating the usefulness of our approach.

## introduction intent classification and slot labeling are two fundamental components in task-oriented dialog systems, producing a formal meaning representation for an utterance that the system can act upon to fulfill the user's request. as shown in figure 1 , it is typically modeled by classifying the utterance into a set of supported intents and labeling its sequence of tokens with expressed slots. we refer to the combined output of these two steps as the interpretation of the utterance. while the performance on these tasks has by now reached high accuracies on benchmarks like snips @xcite , the widespread use of real-world systems like apple's siri, amazon's alexa and google's assistant leads to new research challenges. as such systems are constantly expanding their functionality, bootstrapping new features is a common task that we focus on in this work. in this paper, we define a new feature as a set of one or more intents and related slots that were not known to the system before. the main challenge when introducing a new feature is that typically, only very little seed training data is available, which makes it difficult to train good intent and slot models for the feature. however, manually collecting annotated data is expensive and time-consuming, slowing down the expansion of the system's functionality. we therefore aim to reduce that time and effort by automatically augmenting the seed training data and follow the idea of recent work to leverage paraphrase generation @xcite . through paraphrasing, we can automatically increase the amount of training data, but for the data to be useful, we have to ensure that the new utterances are diverse and different from what we already have. if not, the data augmentation would degenerate to simply upsampling the seed data. that can on its own already be helpful, but it does not expose any new input examples to the model during training which can help the model learn and generalize even better. on the other hand, we also have to ensure that paraphrases remain realistic and natural utterances that are representative of the real-world data distribution. in contrast to previous work, we propose a data-efficient augmentation approach that can work with very small amounts of seed examples. first, we propose using an interpretation-to-text model that can leverage similarities to existing intents and shared slots when generating paraphrases for new features. in addition, instead of requiring pairs of paraphrases, it can be trained on single utterances annotated with intents and slots as they already exist for the downstream tasks. second, we further propose new paraphrase sampling strategies that increase the amount and diversity of obtained paraphrases by using random sampling and shuffling the input representation. and third, we obtain token-level slot labels for the paraphrases via alignment-based label projection, instead of relying on self-labeling with a baseline model as in prior work. these differences make our approach very data-efficient, such that it can produce diverse and novel training examples for a new feature from just 100 seed utterances. we evaluate our technique by simulating the introduction of new features on snips, a common benchmark for intent classification and slot labeling in english, and on german data from a real-world dialog system. in both settings, we see substantial improvements for new features without negative effects on existing features, demonstrating the usefulness of our approach despite little seed data being available. ## related work intent classification and slot labeling have been studied for several decades as fundamental building blocks of task-oriented dialog systems, dating back at least to the introduction of the atis corpus @xcite and subsequent work, e.g. @xcite . in recent years, much progress has been made by applying deep learning techniques, such as recurrent networks @xcite , and modeling both tasks jointly with a single model @xcite . approaches that further improve knowledge sharing between the tasks, such as slot-gates @xcite or bidirectional task connections @xcite , continued this line of work. yet, sufficiently large training datasets are required for such advanced machine learning approaches to be effective. to overcome the lack of training data for new features and avoid costly data collections, several proposals have been made in the recent past. machine translation can be used to obtain training examples if the feature already exists for other languages @xcite . cross-lingual transfer learning is another technique to effectively use such existing data @xcite . if a feature is already being actively used, feedback signals from users, such as paraphrases or interruptions, can be identified in user interactions to obtain additional training data @xcite . if unlabeled utterances exist for the feature, pairs of paraphrases between labeled and unlabeled data might be found, which allows deducing labels for the unlabeled utterance from the paraphrase relationship @xcite . in our work, we make no assumption about the availability of labeled data in other languages, user feedback or unlabeled data, as none of them is necessarily available when bootstrapping a new feature. therefore, the closest existing work to ours is @xcite and @xcite , who both deal with the setup where only seed examples are available. @xcite propose using conditional variational auto-encoders to generate paraphrases for the seed data and show that the paraphrases increase intent classification performance in their experiment. in contrast to our work, they do not evaluate on slot labeling and do not suggest a technique to add slot labels to the paraphrases. @xcite generate paraphrases for seed examples with a transformer network and self-label them with a baseline intent and slot model. when added as training data, this combination of paraphrasing with self-supervised learning improves both tasks in their experiment. our approach differs mainly in terms of data efficiency, as we do not rely on a baseline model for self-labeling -which in turn re-quires sufficient seed data for training -and in addition, we leverage data from existing features and use more advanced sampling strategies to obtain sufficient amounts of high-quality paraphrases from little seed data. we therefore demonstrate that our approach is applicable to earlier stages of bootstrapping by running our experiments on orders of magnitude less seed data than @xcite . furthermore, our paraphrase generation approach differs by training an interpretation-to-text model instead of a text-totext model, such that no corpus of paraphrase pairs is needed. we rely exclusively on the already existing data for the downstream tasks of intent classification and slot labeling. beyond our specific use case, the task of paraphrase generation in general has gained much attention in recent years, driven partly by large-scale datasets such as quora questions pairs foot_0 and the wikianswers foot_1 corpus, which enabled training complex models. @xcite proposed one of the first neural models, relying on residual lstms. further work explored auto-encoders @xcite , generator-discriminator models trained with reinforcement learning @xcite or decompositions into multiple recurrent and transformer models that paraphrase at different levels @xcite . all approaches in this line of work follow the text-to-text approach and assume training data in the form of paraphrase pairs. because existing large datasets of this type focus on questions, we found direct application of models trained on them to the dialog domain to yield very unnatural paraphrases. more closely related to the interpretation-to-text approach we use in this paper is work on generating natural language from structured data. various versions of this task, differing mainly by the type of input, have been studied, e.g. generation from abstract meaning representations @xcite or from tabular data @xcite , as well as generation in unsupervised settings with denoising autoencoders @xcite . the e2e nlg challenge @xcite , a recent competition carried out on a dataset from the restaurant domain with slot-based input representations, resembles our paraphrase generation scenario very closely. among 62 competition entries, the organizers found that sequence-to-sequence models, as also used in our work, were most popular and also very competitive. however, a difference of that line of work is that paraphrases are evaluated only intrinsically, which does not necessarily reflect their usefulness for intent classification and slot labeling. ## data augmentation approach we formalize the introduction of a new feature as having two sets of training data d n and d o , where the former contains the seed examples for the new feature and the latter has all available examples for existing features. an example (u, i, s) ∈ d o ∪ d n consists of a tokenized utterance u, the true intent label i and true slot labels s in bio-format. we refer to the label of a slot as the slot name and the covered tokens as its slot value. let ŝ be the mapping of slot names to their values, e.g. ŝ = {movie-type : movies, spatial-rel : closest, loc-type : movie theatre} for the example in figure 1 . in our scenario, d n contains one or a few new intents not yet in d o foot_2 and is very small. we propose a paraphrase generation model pg that produces new examples d @xmath0. in this section, we will describe how pg learns to paraphrase utterances, how we sample diverse paraphrases from it and how we project labels onto them to obtain examples d p . ## experimental setup we experiment with our data augmentation approach by simulating the introduction of a new feature on a public dataset with english utterances and on internal data from a real-world dialog system in german. ## conclusion we proposed a data augmentation approach for seed data of new features in dialog systems that relies on interpretation-to-text paraphrase models, shuffling and random sampling to generate paraphrases and alignment-based label projection. we demonstrated that using the resulting new training examples improves performance for intents and slots on an english benchmark and german dialog system data.