Controlled Data Augmentation for Training Task-Oriented Dialog Systems with Low Resource Data (2023)

Abstract: Modern dialog systems rely on Deep Learning to train transformer-based model architectures. These notoriously rely on large amounts of training data. However, the collection of conversational data is often a tedious and costly process. This is especially true for Task-Oriented Dialogs, where the system ought to help the user achieve specific tasks, such as making reservations. We investigate a controlled strategy for dialog synthesis. Our method generates utterances based on dialog annotations in a sequence-to-sequence manner. Besides exploring the viability of the approach itself, we also explore the effect of constrained beam search on the generation capabilities. Moreover, we analyze the effectiveness of the proposed method as a data augmentation by studying the impact the synthetic dialogs have on training dialog systems. We perform the experiments in multiple settings, simulating various amounts of ground-truth data. Our work shows that a controlled generation approach is a viable method to synthesize Task-Oriented Dialogs, that can in turn be used to train dialog systems. We were able to improve this process by utilizing constrained beam search.

## introduction the current success of large language models (llms) is the result of multiple factors. one of them is the availability of high-quality data in large amounts. specifically, the prospering adoption of these llms as chatbots, as pioneered by chat-gpt @xcite , was made possible due to the usage of human feedback during training in the rlhf framework @xcite . the category of task-oriented dialog (tod) systems describes a specific kind of chatbot that aims to help users achieve tasks, such as booking hotels or making reservations, using external services. it can therefore be seen as a language-based interface to these services. figure 1 : the general setup of the study. we use n gold standard dialogs to train a generator, which synthesizes dialogs that are in turn used to train a dialog system (steps 1.2 to 4.). to evaluate the improvement (step 5), we compare the performance of the dialog system to the baseline, where we only use the n dialogs for training (step 1.1). in the recent past, different model architectures have been proposed to solve this problem. to evaluate them, the original multiwoz @xcite dataset and its successors are widely used as benchmarks. it contains more than ten thousand dialogs spanning multiple domains (e.g., attraction, hotel and restaurant), paving the way for the adoption of large end-to-end models. while this benchmark allows for useful comparisons of different architectures and approaches, we deem this scenario of having multiple thousand dialogs to be infeasible for most real-life tod use cases, where usually not nearly as many dialogs are available. this is due to the fact that the types of dialog needed to train a tod system are different from those used to train, e.g., a social chatbot, since the tod conversations need to include the usage of the external systems. therefore, it is typically not only necessary to have two users (while using the wizard-of-oz technique; @xcite , but to have at least one of them interact with the external service to provide the necessary information to achieve the task. we therefore investigate the synthesis of suitable utterances from given annotations to train tod systems. we call this method "controlled generation for training" (cg4t). for this, we simulate lower resource scenarios based on the multiwoz dataset, allowing us to both evaluate the models on this common benchmark and also to test the method in different, more realistic settings. with the objective of ensuring that the generated utterances contain the entities defined in the annotation, we additionally investigate the effect of constrained beam search during the generation phase. to sum up, in our work, we aim to answer four research questions (rq): tod systems and the multiwoz dataset. since the publication of the original multiwoz @xcite dataset, there have been multiple updated versions, mostly to fix label noise @xcite @xcite . the setting introduced by the dataset allows for two types of evaluation: dialog state tracking (dst) and response evaluation. since recent publications manage to get near-perfect results for the response evaluation as measured by the inform and success metric @xcite , we decide to focus on the dst task. there has been varied research (e.g., @xcite @xcite on specific model architectures, using distinct approaches to solve the dst task. @xcite propose som-dst, which uses an explicit state memory and predicts the operation to perform for each slot at every turn, e.g., carryover or update. on the other hand, star @xcite tries to leverage the correlations between the slots with a slot selfattention mechanism. constrained beam search. when generating outputs with an nlg model, the goal of exhaustively finding the sequence with the highest probability is infeasible in most cases. since for a sequence of length m over a vocabulary with length v, the computational cost would be o(v m ). accordingly, beam search is commonly adopted as an approximation and used to create output sequences with a trained llm. performing a beam search with a beam size of b consists of keeping b candidates, for which a greedy search is continued, and finally the candidate sequence with the highest probability is picked as the prediction. the constrained beam search (cbs; @xcite is a variant of this method, which tries to enforce the existence of specific words, so-called constraints, in the output. since for the dst task it is important to have utterances containing the entities exactly as given in the annotation, we hope that adopting the cbs scheme in our method will increase the amount of annotations appearing in the generated utterances. ## proposed method and materials we use the multiwoz 2.4 @xcite dataset as the basis for all of our experiments. this dataset is the result of multiple corrective iterations of the original multiwoz data @xcite . it contains roughly ten thousand dialogs, of which two thousand in total are reserved as validation and test data. we make use of the official split between training, validation and test data. the dialogs were created with the wizard-of-oz (woz) the goal of our method is to synthesize a large amount of dialogs based on a small amount of collected data. instead of generating the whole dialog in one pass, we predict each utterance one after another based on the turn annotations with a sequence-to-sequence generator model. this can be described as using the annotations as a blueprint for the dialog. the main steps of the method are given as 1.2, 2, 3, and 4 in fig. 1 . they first include the preprocessing of the data, so they fit a specified input/output format (x, y) for the sequence-to-sequence model (1.2). this format is depicted in fig. 3 . with this, we train a sequence-to-sequence model (2.) as the generator ŷ = g cbs (x). we write a generator model that uses the constrained beam search as its decoding strategy as g cbs (x). using the trained g cbs (x) we perform the data augmentation by synthesizing dialogs using cbs (3.), and finally train the tod system on this augmented data (4.). these steps are described in more detail below. ## experiments our experiments can be divided into two stages. the first stage consists of studying the generator and relates to rq 1 (feasibility) and rq 2 (constrained beam search). the second focuses on the dialog system and aims to answer rq 3 (effectiveness). to evaluate the proposed method, we look at four different, simulated low(er) resource settings to answer rq 4 (magnitude of data). in each of those, we assume that a certain amount n of annotated dialogs have been collected, which will be used for the cg4t approach. concretely, we use @xmath0, 750, 1000, 2500} in an attempt to understand the effect different magnitudes of data have. ## qualitative error analysis to better understand the weaknesses of the generator, we performed a qualitative error analysis by comparing generated dialogs to their ground-truth. in the following, we will reference the errors with regard to the generated dialog in fig. 5 and its ground-truth in fig. 6 in the appendix a. first, errors in the annotation will naturally be replicated in the generated utterance. as an example, while in the ground-truth the user requests the reference number for the ticket and the system delivers it, the generated dialog does not mention the reference number at all. however, since the annotation does not contain the reference number either, we cannot expect it to be generated. second, we can see that sometimes the speaker role seems to not be completely taken into account, leading to formulations that are unexpected for the dialog system, such as the "yes, please" response the system gave to the user's request to book a train ticket. lastly, while the concrete text of the slot values will in general be correct due to the constrained beam search, the embedding of them into context still contains errors, both semantically and syntactically. an example we can see in the generated dialog is first that the chatbot produces a faulty utterance "there are 5 trains peterborough on wednesday". thus, at the same time, prematurely giving the number of trains for the destination but also not specifying it is the destination. the premature destination mention is due to it already being part of the belief state annotation for this turn. to sum up, while the model in general achieves decent results, there are still multiple caveats, and the error analysis emphasizes the importance of the annotations. ## results and discussion our experiments show that the model is able to generate utterances based on the provided annotations. these utterances resemble the language defined by the training data, as is shown by the bertscore (cf. tab. 3 ). the improvements with increased n are negligible for this metric. this can be attributed to the fact that, due to fine-tuning a pretrained llm, the model already had good general language generation capabilities to begin with. however, with regard to the average utterance length (cf. tab. 1) and lexical richness (cf. tab. 2), the generated text does not perfectly align with the training data. nevertheless, we consider the model to have a good, but not perfect, language generation capability for this specific task. besides language generation, rq 1 also tends to the task-completion, which in this case is adhering to the annotations. we report the results for the annot r metric in tab. 4. these show that the model is also capable of generating utterances that adhere to the annotations. the performances for the different n-scenarios are comparable. it is an important insight that the annot r metric is more expressive. this makes up for a weakness of the metric, which stems from the different kind of annotations that exist. the dialog in fig. 5 shows that when few annotations are provided for an utterance, the generation can lead to an unfitting utterance. for example, when the user asks to book the train ticket for one person, the system answers with "yes, please". from the usual dialog flow the multiwoz conversations have, it is clear that this utterance usually belongs to a user and not the system. the generator mistakenly used this as a confirmation, even though it is unfitting for this speaker in this scenario. since both the language generation and the taskcompletion capabilities are sufficient, we answer rq 1 positively: we successfully used a pretrained sequence-to-sequence model to generate synthetic dialogs based on annotations. the effect of the constrained beam search is also shown in tables tab. 3 and tab. 4. as is to be expected from the method, constraining the generation led to slightly worse results in the bertscore metrics. this is consistent over all n. however, regarding the annot metric, the constrained beam search improved the outcome substantially. with it, the annot r relative to the test data was nearperfect over all n. a special case is @xmath1, where annot @xmath2. this means that on average, the utterances predicted by the generator model contained more annotations than the actual test data. this effect can be explained by the small sample size, as well as the fact that the ground-truth annot of the test data over all n is roughly 0.73. the large increase in annot score with a small decline in bertscore metrics suggests that cbs is useful for annotation-based generation, which leads us to answer rq 2 positively. as a result, we utilized constrained beam search for the second stage of experiments. tab. 5 shows the results for the selected tod system architectures for the n-scenarios. we report both the results of using only these n dialogs, and additionally using the synthetic dialogs, i.e., with cg4t. the first finding is that for all n and both architectures, cg4t did improve the jga, therefore showing that the synthetic dialogs generated from annotations can indeed increase the performance of both open-vocabulary and ontology-based tod systems. consequently, we answer rq 3 positively as well. still, the results show that with larger n, the benefit gained through the usage of cg4t decreases. this is to be expected given the nature of the method, which led us to pose rq 4. finally, tab. 5 demonstrates the significant differences in jga between the two approaches. regarding rq 4 we evaluate the combined results from experiments in stages one and two. stage one showed that only a few gold standard dialogs are needed to get good language generation capability. moreover, thanks to cbs, we can also attain satisfying results for task-completion. the second stage showed that the more dialogs available, the lesser the improvement. from this we conclude that a) the less data one has, the more sense it makes to use this method and b) for this concrete scenario, the method is nonessential past a magnitude of 2500 dialogs. ## conclusion in this work, we studied whether it is practicable to synthesize dialogs based on annotations to augment the collected ground-truth data for training a tod system. to this end, we focused on four research questions regarding the feasibility (rq 1), the effect of cbs during decoding (rq 2), the performance improvement when using the synthetic dialogues to train a dialog system (rq 3), and the relation between available data and the effect of the method (rq 4). we saw that even with small amounts of dialog, we can train a generator that creates utterances from annotations using a sequence-to-sequence strategy. while the constrained beam search had slightly adverse effects on the language generation capabil-ities, it provided significant improvements to the task-completion, i.e., adhering to the annotation. augmenting the training data with the synthetic dialogues relevantly increased the performance of the dialog system as measured by the jga metric. the smaller the number of available conversations, the greater the effect of the cg4t approach. past a scale of roughly 2500 collected ground-truth dialogs, we would discourage augmenting the training data in this way since the improvements are too small for the increased effort. our research showed at multiple stages the importance of having annotations of the highest quality when using the proposed approach.