How to Tame Your Data: Data Augmentation for Dialog State Tracking (2020)

Abstract: Dialog State Tracking (DST) is a problem space in which the effective vocabulary is practically limitless. For example, the domain of possible movie titles or restaurant names is bound only by the limits of language. As such, DST systems often encounter out-of-vocabulary words at inference time that were never encountered during training. To combat this issue, we present a targeted data augmentation process, by which a practitioner observes the types of errors made on held-out evaluation data, and then modifies the training data with additional corpora to increase the vocabulary size at training time. Using this with a RoBERTa-based Transformer architecture, we achieve state-of-the-art results in comparison to systems that only mask trouble slots with special tokens. Additionally, we present a data-representation scheme for seamlessly retargeting DST architectures to new domains.

## introduction dialog state tracking (dst) is a common problem for modern task-oriented dialog systems that need to be capable of tracking user requests. commonly, there is an ontology that defines slots that must be filled according to a user's utterances -e.g., a restaurant slot that is filled in with a restaurant name given by the user. a key problem for dsts is that the values that fill a slot at inference may have never been encountered at training time (consider that the set of all possible restaurant names is bound only by the limits of language). in this work, we address the problems of training on a domain with effectively limitless possible vocabulary, and aim to create a dst system capable of scaling to unseen vocabulary at inference. we do this by first utilizing a language model (lm) based transformer that is capable of handling any possible input and output in a textual manner, letting the same exact architecture scale to new intents, slots, and slot values, with no modifications needed. additionally, we present a practical data augmentation procedure for analyzing and addressing issues in the development of a dst system, leading to state-of-the-art performance. ## related work work in dst has taken a number of different approaches. the annual dst challenge (dstc) has undergone eight iterations (although from the sixth competition on, it has been the more broad dialog system technology challenge) @xcite . the m2m:simulated dialogue @xcite dataset for dialog state tracking has been addressed by a number of different approaches. @xcite used a bi-directional gru @xcite along with an oracle delexicalizer to generate a candidate list for slot filling. @xcite later used a bi-directional lstm @xcite without the oracle delexicalization to generate candidate lists for slot filling. @xcite use two bi-directional lstms -one at the utterance level, the other at the dialog level -to perform the dialog state tracking. however, this work is only tested on the simulated dataset sim-gen, meaning there is no comparison with the more challenging human crafted utterances contained in sim-r and sim-m. the closest approach to the one detailed in this paper is that of @xcite . they used a system based off of bert @xcite , but removed the language-model head and instead used two specialized heads: one that does per-slot utterance level classification to determine whether a given slot is active in the utterance or is the special dontcare token, and another per-slot head that predicts whether a token represents the beginning or end of the span for that type of slot. our model differs in that we do not need to alter the architecture of the model with specialized heads, and instead fine-tune the existing language model head. in their experimentation, they adjusted the level of slot-specific dropout using targeted feature dropout, first used by xu and sarikaya (2014), where slots are replaced with a special [unk] token. our approach also differs in that instead of simply dropping out slots, we use the more nuanced method of targeted data augmentation. finally, data augmentation has been widely used for improving the robustness of dialog systems. hou et al. (2018) used a lstm-based sequence-to-sequence network to map from generic utterances (e.g., "show me @xmath0.g., "where is @xmath1. this approach requires delexicalization and only alters grammatical structure, which is quite different from our approach which leaves grammatical structure alone, instead altering the non-delexicalized slot values. @xcite perform data augmentation via four different approaches: (1) replace words (excluding proper nouns, qualifiers, personal pronouns, and modal verbs) with their synonyms, (2) remove all stop words, (3) use existing neural machine-translation technology to translate from the source language to another and back again (similar to that of @xcite , except they do not train their own seq2seq network), and (4) use an existing paraphraser to paraphrase the utterance. ## method our goal in this work is to to create a robust, readily extensible dialog state tracking system that requires minimal to no alteration of network architecture if the schema and/or domain of the dia-log task changes. for instance, imagine a system that is being developed for the restaurant domain under a schema in which a set of slots are specified: cuisine, price, location. now imagine that later it becomes necessary to add a new slot: kidfriendliness. instead of changing the architecture and retraining from scratch, we would prefer to be able to fine-tune the existing model with the new slot now present. additionally, we incorporate targeted data augmentation to combat over-fitting when a domain has limited vocabulary. ## data augmentation there are a number of common issues in the datasets for these dialog tasks, including: to counteract these issues, researchers have proposed a number of different data augmentation schemes (see section 2). at the outset of our study, we tried the 10% slot-specific dropout used by @xcite , but our model still overfit to the training set. to combat this, we devised the following procedure: in our work, we were using the sim-r and sim-m datasets @xcite , which are concerned with restaurant reservations and movie tickets respectively. we noticed that our system was nearly perfectly able to handle requests related to time, date, and number of people -slots whose values come from small structured sets -but was having difficulty with movie titles, restaurant names, and locations, even with the targeted 10% dropout. we found corpora for movie names (42,306 movie titles found on wikipedia as of 2013 @xcite ), restaurant names (1445 humorous restaurant names @xcite ), and locations (2067 us settlement names from 1880 to 2010 @xcite ) which we then used to randomly replace the respective slots at training time at a rate of 50%. we note that our replacement has two major effects. ## experiments as previously mentioned, we used the sim-r and sim-m datasets @xcite . this is because we found them to be of high quality (but with room for improvement), and there was a recent state-ofthe-art approach that used a similar transformerbased architecture to compare against @xcite . to assess the performance of the models, we use joint goal accuracy @xcite , the standard metric for assessing dst systems. at each turn of dialog, the ground truth must be perfectly matched. for this specific work, we fine-tuned the roberta masked language model of @xcite ; specifically, we used the huggingface transformers library @xcite . all models were trained with the adam optimizer with an initial learning rate of 5e -5, epsilon of 1e -8, a linear learning rate schedule over 20 epochs, and an attention mask rate of 15%. we compare three approaches in the experiment. (1) roberta-lm, the roberta lm architecture with 10% slot-specific dropout; (2) roberta-separate, the roberta lm architecture with 50% slot-specific replacement, with separate models trained on the sim-m and sim-r datasets; and (3) roberta-combined, the roberta lm architecture with 50% slot-specific replacement, with a single model trained on the combined sim-m and sim-r datasets. ## conclusions and future work in this paper, we make two contributions. first, we introduce a process for a) examining the source of errors in dialog state tracking on held-out evaluation data, and b) correspondingly augmenting the dataset with corpora to vastly increase the vocabulary at training time. like earlier work that selectively masked slot values, this prevents the system from overfitting to specific values found in the training data. furthermore, however, it forces the system to learn a wider range of values, rather than syntactic features only, vastly improving the performance. second, we do this in the context of a language model based transformer, that due to the language-based nature of its representation -slots are simply represented as tokens concatenated to user utterances -is capable of transferring seamlessly between and working jointly on different datasets without the need to change the underlying architecture. in the future, we would like to address other forms of targeted data augmentation, addressing grammatical differences in addition to vocabulary modifications. puyang xu and ruhi sarikaya. 2014. targeted feature dropout for robust slot filling in natural language understanding. in fifteenth annual conference of the international speech communication association.