Comparing Data Augmentation Methods for End-to-End Task-Oriented Dialog Systems (2024)

Abstract: Creating effective and reliable task-oriented dialog systems (ToDSs) is challenging, not only because of the complex structure of these systems, but also due to the scarcity of training data, especially when several modules need to be trained separately, each one with its own input/output training examples. Data augmentation (DA), whereby synthetic training examples are added to the training data, has been successful in other NLP systems, but has not been explored as extensively in ToDSs. We empirically evaluate the effectiveness of DA methods in an end-to-end ToDS setting, where a single system is trained to handle all processing stages, from user inputs to system outputs. We experiment with two ToDSs (UBAR, GALAXY) on two datasets (MultiWOZ, KVRET). We consider three types of DA methods (word-level, sentence-level, dialog-level), comparing eight DA methods that have shown promising results in ToDSs and other NLP systems. We show that all DA methods considered are beneficial, and we highlight the best ones, also providing advice to practitioners. We also introduce a more challenging few-shot cross-domain ToDS setting, reaching similar conclusions.

## introduction task-oriented dialog systems (todss) aim to understand and fulfil a user's goal in a particular application domain (e.g., booking tickets, restaurant reservations) through natural language conversation @xcite . they traditionally include several modules to handle a variety of processing stages. a natural language understanding (nlu) module typically extracts structured information from each user's utterance, usually in the form of slot-value pairs (e.g., for destination, travel date). a dialog state tracker (dst) keeps track of the information that has been revealed about the user's goal (dialog state) throughout the conversation. a policy optimizer (po) de-cides the system's next action (e.g., request information for a particular unfilled slot of the dialog state, or recommend a restaurant). a natural language generation (nlg) module realises communication actions of the system in natural language. in spoken conversations, automatic speech recognition (asr) and text-to-speech (tts) modules are also included @xcite , though here we consider only written conversations. when the modules of a tods are trained separately, large volumes of annotated training data are needed, since each module requires its own input/output training examples. in end-to-end todss @xcite , a single system is jointly trained to handle all (or most) of the processing stages, reducing (or at least harmonising) the kinds of annotated training data required, since each training example can be used by all (or most) of the modules @xcite . recent transformer-based end-to-end todss @xcite @xcite aim to reduce the necessary training data further, by leveraging pre-training on unlabelled data. however, the best performance is still obtained by fine-tuning on domain-specific annotated training datasets @xcite , which are costly to obtain, especially when moving to a new domain (e.g., from booking airline tickets to reserving restaurants). llms that promise adequate results in few-or even zero-shot settings @xcite have been underexplored in todss, and work along this direction has focused mostly on the dst module, with contradicting results @xcite . thus cost-effective methods to obtain or augment training sets for todss are needed. we experiment with data augmentation (da), which adds synthetic training examples to an existing annotated (possibly small) training set. da methods have shown promising results in various nlp tasks, e.g., text classification, text generation, question answering @xcite @xcite , but have not been systematically compared in end-to-end todss (section 6). we simulate the case where the available annotated training set is limited, which is often the case with real-life tods applications, especially when moving to new application domains. we experiment with two end-to-end pre-trained transformer-based todss, namely ubar @xcite and galaxy @xcite , and two english textual conversational datasets, multi-woz 2.0 @xcite and kvret @xcite . we vary both the size of the initial annotated training set (e.g., 2%, 10%, 25% of multiwoz's training set) and the expansion factor (generating one, two, or four synthetic examples from each original one, i.e., x2, x3, x5 expansion). we also introduce a more challenging few-shot cross-domain setting: in each iteration, we select a single domain as the target one, we remove from the training set all but 20 (few-shot) dialogs of the target domain, we generate synthetic training dialogs from the 20 dialogs, and evaluate only on test dialogs from the left-out target domain. 1 we compare eight da methods that have shown promising results in todss (mostly when training individual modules) and other nlp systems. the da methods we consider are of three types. word-level augmentation methods replace words of the original training examples by similar words, e.g., words that have similar word embeddings or words suggested by models that predict masked words. sentence-level augmentation includes paraphrasing sentences of the original training data via back-translation (translating to another language and back), invoking systems trained specifically to perform paraphrasing, or by prompting large language models (llm); we also include in this type methods that rotate (swap) parts of the dependency tree of the original sentence. finally, dialog-level augmentation methods exploit dialog-specific annotations of the original training examples (e.g., dialog states), which are available in tods datasets. our contributions are: (1) we conduct a systematic comparison of da methods in an end-to-end tods setting, the largest (in terms of da methods) comparison to date in this setting. (2) we show that substantial performance gains can be obtained with da methods, even when using pre-trained models, 1 for example, the domains of multiwoz are: train, taxi, hotel, restaurant, hospital, police and attraction. and we offer concrete advice to tods practitioners (section 5). (3) we introduce and utilise a new few-shot cross-domain evaluation setting for da approaches in tods, where we show that da methods again boost performance. ## experimental setup in all experiments, we use three different randomly sampled training subsets (e.g., 3 different 10% subsets) and report averages. ## advice to tods practitioners based on our experiments, we provide the following advice to tods practitioners. (1) although da methods are currently not widely used in tods (section 6), they can lead to very substantial gains when annotated training examples are scarce, even when using pre-trained end-to-end models. however, data scarcity depends on the difficulty of the task (e.g., whether the dialogues are single-or multi-domain), it cannot be judged using a universal threshold of initial annotated training instances, and is best assessed using performance scores. (2) although all da methods considered led to substantial improvements, dialog-state da methods (at least the ones we considered) should be the last choices, as they led to a smaller performance boost and they also require dialog-specific annotations (e.g., dialog states) that are difficult to obtain in practice. (3) among the other da methods consid-ered, the best overall performers were word2vecbased word substitution (section 2.1.1), rotating parts of the dependency tree (section 2.2.3), and paraphrasing the utterances by invoking paraphrasing models or prompting llms (sections 2.2.2, 2.2.4). we recommend using any of these methods, provided that the resources they require are available; e.g., rotation requires a dependency parser for the target language, prompting requires an llm supporting the target language and adds the cost of invoking an llm. (4) generating more than two synthetic instances per original one is worth considering only when the original annotated training instances are very scarce (e.g., when using only 2% of multiwoz's multi-domain training set). paraphrasing as a form of da was also used by @xcite and @xcite for slot-tagging (subpart of dst) and rg; they implemented their own paraphrasing model based on rnns. similarly, @xcite train a bart model @xcite for paraphrasing in an end-to-end tod setting. @xcite , also employ da in a tods setting, by substituting masked tokens of the input using an lm (much as in section 2.1.1). @xcite augment their dataset by replacing slot-values with randomly generated strings to simulate references to out-ofvocabulary (oov) words, a simple yet effective approach for models that learn to copy words from input to output by incorporating the copy-mechanism @xcite . @xcite randomly change the letter order of slot-values to improve the robustness of their tods. improving robustness is also the main goal of @xcite , who generate turns with replaced slot-values not encountered in the training set. the closest previous work is that of @xcite . they experiment with four da methods in an end-to-end tods setting, including stopword removal, paraphrasing, back-translation, synonym substitution (and their combination). we compare more (eight) da methods, of three different types (word-level, sentence-level, dialoglevel), while quan and xiong consider fewer (four) methods of the word-level and sentence-level types only. furthermore, we also experiment with a more challenging dataset containing dialogs that may belong (the same dialog) in multiple domains (mul-tiwoz), whereas quan and xiong use the single-domain camrest676 @xcite , and (like us) kvret, which covers three domains, but each dialog pertains to a single domain. contrary to the work of quan and xiong, we also experiment with several sizes of initial annotated training set (e.g., 2%, 10%, 25% in multiwoz) and different expansions (x2, x3, x5), and we also introduce a new few-shot cross domain setting (section 3.1). ## conclusions and future work we performed the largest to date (in terms of methods and types) empirical comparison of da methods in the context of end-to-end todss, using two pre-trained transformer todss (ubar, galaxy) and two datasets (multiwoz, kvret). we varied both the size of the initial annotated training set (e.g., 2%, 10%, 25% of mul-tiwoz's training set) and the expansion factor (generating one, two, or four synthetic instances from each original one). we also introduced a new, more challenging few-shot cross-domain evaluation setting. we showed that substantial performance gains can be obtained with da methods, even when using pre-trained end-to-end models, and we offered concrete advice to tods practitioners regarding when to use da, which da methods to prefer, and with which expansion factors (section 5). in future work, we plan to explore combinations of da methods, to check if the synthetic training instances they produce are complementary. we also plan to investigate how well da methods perform in low-resource languages (e.g., languages for which llms or reliable dependency parsers may not be available). finally, we aim to measure the costs (computational, monetary) of da methods.