Velocidapter: Task-oriented Dialogue Comprehension Modeling Pairing Synthetic Text Generation with Domain Adaptation (2021)

Abstract: We introduce a synthetic dialogue generation framework, Velocidapter, which addresses the corpus availability problem for dialogue comprehension. Velocidapter augments datasets by simulating synthetic conversations for a task-oriented dialogue domain, requiring a small amount of bootstrapping work for each new domain. We evaluate the efficacy of our framework on a task-oriented dialogue comprehension dataset, MRCWOZ, which we curate by annotating questions for slots in the restaurant, taxi, and hotel domains of the MultiWOZ 2.2 dataset (Zang et al., 2020). We run experiments within a low-resource setting, where we pretrain a model on SQuAD, fine-tuning it on either a small original data or on the synthetic data generated by our framework. Velocidapter shows significant improvements using both the transformer-based BERTBase and BiDAF as base models. We further show that the framework is easy to use by novice users and conclude that Velocidapter can greatly help training over task-oriented dialogues, especially for low-resourced emerging domains.

## introduction humans perform dialogue interactions to accomplish common tasks: work email threads, nursepatient conversations, customer service conversations, etc. (cf. a task-oriented dialogue is a form of information exchange where the system obtains user preferences (i.e. slot values for attributes) by conversation. the dynamic flow between speakers in these dialogues introduces additional challenges such as: among recent data augmentation studies, @xcite contribute the sole prior work explicitly on task-oriented dialogue comprehension. however, their synthetic data generation is scoped within a clinical scenario, with templates of inquiryresponse pairs between nurses and patients. this limits dialogue-specific traits, such as mind change and co-reference, to consecutive turns only. inspired by this prior work, we introduce velocidapter, which can augment a handful of taskoriented dialogues to a synthetic dataset that is larger by several orders of magnitudes. figure 1 shows a simple, intuitive example of velocidapter's synthetic generation in the restaurant booking domain. different from prior work, we define templates as dialogue chunks (i.e. several contiguous turns), which we call discourse templates. this lets us design dialogue-specific challenges that span over multiple dialogue turns (e.g. mind change, zero anaphora, etc.). we further aim to expand prior work by addressing scalability issues for task-based dialogue comprehension by leveraging synthetic generation with a mutual concept: domain adaptation (da). this pairing is synergistic: da gives the model the necessary pretraining to generalize well, and the synthetic generation process yields sufficient data in the target domain to effectively fine-tune the model. to use velocidapter, a user extracts pairs of discourse templates from a few development dialogues in the target domain (cf. colored dialogue chunks within dialogues 1 and 2 in figure 1 ), a value list for each slot (cf. slot values in figure 1 ), and a question list for each slot. with these inputs, velocidapter simulates a synthetic corpus of taskoriented dialogues by mixing turn templates from several dialogues and filling templates with values from the slot value list. finally, it matches each dialogue to a set of questions according to the slots they contain. this synthetic dataset is then used to train or fine-tune a dialogue comprehension model in the target domain. we contribute a new dataset, mrcwoz, to evaluate our framework foot_0 . this dataset is generated from the existing large dialogue corpus, multiwoz 2.2 @xcite , which is used for dst (dialogue state tracking) task. we form training and test sets of mrcwoz from the respective sets in multiwoz by annotating questions for each unique slot type in the restaurant, hotel, and taxi domains. note that the formation of mrcwoz is completely separate from our augmentation framework. we show that within a low resource setting, models using our framework significantly outperform models using original target data (raw data). specifically, velocidapter outperforms the raw training by 0.26, 3.82, and 13.23 f1 scores in the restaurant, hotel, and taxi domains, respectively. these gains are obtained at little human time cost and are robust: through a user study, we show that templates extracted by a novice human in under an hour, still lead to significant improvements over raw training. to the best of our knowledge, this is the first study to make use of the inherent clustered structure of task-oriented conversations to augment a large set of instantiated dialogue datasets. our framework is also the first to address dialoguespecific challenges that span over several turns within a machine comprehension perspective. we thus conclude that this approach potentially can greatly facilitate the rapid advancement of understudied task-oriented dialogue areas, which lack sufficient corpora. a few incorporate a conversational component to the mrc task. dream @xcite , friend-sqa @xcite and a study by @xcite are all dialogue comprehension datasets. although a valuable source, these do not apply to taskoriented dialogue comprehension, as all three are open-domain and multi-party. in contrast, coqa and quac do employ two-party dialogue; however, their task is to conversationally answer questions about a passage, diverging from our task definition @xcite . ## velocidapter: data generation framework let us first formalize our task. our goal is to create a task-oriented dialogue-augmentation framework f , that given a list of dialogue turn templates t , a slot label-value list s v , and finally a slot labelquestion list s q , can generate a large dialogue comprehension dataset d. f creates individual synthetic dialogues in d by composing turn templates from t , filling these turn templates with values from s v , and finally matching these to questions from s q . d then can be used to train or fine-tune a task-oriented dialogue comprehension model. task-oriented dialogues can be deconstructed as having dialogue units that convey slot values for particular attributes. we name these atomic units that are composed to creat dialogues in our framework as discourse templates. velocidapter takes as input a set of manually-extracted discourse templates and outputs instantiated dialogues that are of orders of magnitudes larger in scale. this facilitates the robust training of large models from just a few dialogue instances. figure 2 shows the end-toend pipeline of our framework. to use velocidapter a user extracts the turn templates from a small, taskoriented dialogue development set (e.g. in figure 2 turn templates in 2a are extracted from dialogues in 1a), a list of values for each slot (2b), and a list of questions for each slot (3b). velocidapter then generates individual dialogues (3a) and their associated inquiry-response pairs (4a), by executing three steps: (1) structured corpus construction, (2) dialogue template generation, and (3) dialogue corpus generation. ## experiments to evaluate we need a dataset for dialogue comprehension. unfortunately, no suitable dataset exists for this purpose, so we pick an existing dialogue dataset and retrofit it for our evaluation purposes. we start with the multiwoz dataset, commonly used for dst. we choose a range of domains from multiwoz to work with to showcase domain agnostic feature of our framework. we leave hospital and police domains out, following past work @xcite since they lack correct annotations and validation and test sets. from the remaining five we choose restaurant, hotel, and taxi domains as their pools of slot labels show very few overlaps thus resulting in a diverse dataset. the resulting corpus contains 2,409 dialogues, averaging 8.92 turns per dialogue, and 12.2 tokens per turn. but since multi-woz does not come with dialogue comprehension questions natively, we supply our our own hand annotated questions as detailed next. speaker turn user hello, i would like to find a place to dine in, there will be restaurant bookpeople of us. system what cuisine would like to try? user let's try food type, please. system okay sounds good. user sorry, i want to have food type instead. system which part of the city would you favor? user the arbitrary-city area is too far from my place, i think city area would work the best. system okay, does restaurant bookday sound good? user yes, that should work. system great, your booking is successful. anything else i can help you with? user this is all i wanted for today, thank you. system thanks, good bye. speaker turn user hello, i would like to find a place to dine in, there will be 4 of us. system what cuisine would like to try? user let's try british, please. system okay sounds good. user sorry, i want to have italian instead. system which part of the city would you favor? user the center is too far from my place, i think south would work the best. system okay, does friday sound good? user yes, that should work. system great, your booking is successful. anything else i can help you with? user this is all i wanted for today, thank you. system thanks, goodbye. train test domain # dial # s-q # dial # s-q hotel 650 2859 71 318 restaurant 1250 4495 65 316 taxi 321 965 52 157 for each slot type in multiwoz, we manually create a list with a few questions. we then match each dialogue to a set of questions according to the slots present in the dialogue to create our multi-woz dialogue comprehension dataset, which we term mrcwoz. as a result of this process, mr-cwoz pairs each dialogue with an average of 4.2 questions. the domain-specific statistics of mrc-woz data can be seen in we also randomly sample a small development set, vel dev containing few dialogue (e.g. 2-10 dialogues) from the training set of each domain to extract turn templates for velocidapter. during sampling, we ensure that the final set of dialogues cover all possible slots encountered in the test set so that the trained model will be exposed to each slot at least once (e.g. food type, booking day, etc.). we fine-tune the bert-base @xcite and bidaf models @xcite in experiments representing three different scenarios/datasets: (1) in a high-resource scenario on mrcwoz, which serves as an upper bound for our experimental setup. we term the models that are fine-tuned with this dataset woz large; (2) in a low resource setting on the small vel dev set, which uses only a handful of dialogues. we term models fine-tuned with this other training set woz small. (3) in our proposed setting on our framework's synthetic dataset. we term models trained with this set as velocidapter. considering that our synthetic data is generated by templates extracted from the vel dev set, this is a low resource scenario. moreover, we also train a model version that also has its respective pre-trained versions on squad, we add an "sq" prefix to the name of each model to denote them: (1 )-sq+woz large (2 )-sq+velocidapter (3 )-sq+woz small. the careful reader will note that the second and third settings are directly comparable, as they both utilise the same vel dev dataset, but our framework multiply augments this initial dataset to a large volume of synthetic data. we evaluate the performance of models on the mrcwoz test set using the proposed f 1 and exact match (em) metrics as in squad @xcite , using the official evaluation scripts provided. ## conclusion in this work, we introduce a template-based augmentation framework for the task-oriented dialogue comprehension task. our framework, velocidapter, combines the two mutually beneficial concepts of synthetic data generation and domain adaptation to strategically utilize limited human input to greatly enrich sparse dialogue training data. velocidapter leverages the turn-based nature of dialogue to strategically involve humans-in-the-loop to greatly reduce error in a robust fashion. it can be used to augment task-specific domain dialogues in the low-resource, few-shot setting by generating several orders of magnitude larger datasets, substantially decreasing dialogue-specific errors of a model (e.g. partial value match, value mismatch, etc.). this process only requires a little manual intervention: under an hour's time of a novice human creator for each new domain. our experiments indicate that velocidapter is a viable approach in addressing the data gap in comprehension of task-oriented dialogue systems. in the future, we look forward to using our framework on other task-oriented dialogue tasks. we further want to discover the automated extraction of dialogue chunks and generation of templates which can also benefit from controlled text generation techniques.