DG 2: Data Augmentation Through Document Grounded Dialogue Generation (2022)

Abstract: Collecting data for training dialog systems can be extremely expensive due to the involvement of human participants and the need for extensive annotation. Especially in document-grounded dialog systems, human experts need to carefully read the unstructured documents to answer the users’ questions. As a result, existing document-grounded dialog datasets are relatively small-scale and obstruct the effective training of dialogue systems. In this paper, we propose an automatic data augmentation technique grounded on documents through a generative dialogue model. The dialogue model consists of a user bot and agent bot that can synthesize diverse dialogues given an input document, which is then used to train a downstream model. When supplementing the original dataset, our method achieves significant improvement over traditional data augmentation methods. We also achieve great performance in the low-resource setting.

## introduction most of human knowledge is stored in the form of documents, ranging from answering factoid questions @xcite to providing how-tos on millions of tasks @xcite . how to comprehend and retrieve relevant knowledge from documents given a user query is a challenging research problem. inspired by real-world applications, there have been more works @xcite @xcite that aims to tackle this challenge. in this work, we focus on the task of conversational information seeking based on the associated documents, which are often referred to as document-grounded dialogue systems @xcite . recent works have introduced various datasets for building document-grounded conversational question answering and dialogue systems. some work such as quac @xcite and coqa @xcite first explored the direction of figure 1 : an example from doc2dial of dialogue conversation produced from grounding to an associated document. the agent must select the correct spans and engage in a fluent manner to generate a proper response. conversational question answering. then, sharc @xcite added follow-up questions by agents. later, doc2dial @xcite ) further included the dialogue actions and domains, which aims to simulate more kinds of real-life scenarios. however, such dataset is typically hard to scale up to new domains, as it requires carefully crafted dialogue flows and expensive human annotations. however, as the relations between conversations and documents become more complex, the cost of collecting large-scale datasets also becomes more expensive. as a consequence, one main obstacle for developing scalable and effective document grounded dialog systems is the lack of sufficient data. in chit-chat scenarios, recent works such as dialogpt @xcite , meena @xcite , and blender @xcite have achieved high performance by taking the advantage of training on a large-scale corpus. similarly, task-oriented dialog systems such as ardm @xcite and simpletod @xcite have also utilized largescale corpora or pre-trained models to achieve good performance. the aforementioned models were trained with millions of samples, while the current document-grounded dialogue datasets like doc2dial @xcite only contain thousands of conversations. training on such a smallscale dataset constrains the performance of neural network models. therefore, augmenting existing datasets can help build a more effective documentgrounded dialogue system. one popular approach to augmenting datasets is to paraphrase existing seed data. the most straightforward form of paraphrasing is to directly use a model trained to generate paraphrase pairs @xcite . back-translation serves as another type of paraphrasing, which first translates a sentence into another language and then back again @xcite . back-translation ensures the quality and correctness of the augmented data and often shows improvement in downstream models. both methods aim to provide variety to the training data without greatly altering the semantics of the original sentences. however, these methods only operate on the existing dialogue data and fail to take advantage of the available document for augmentation. another direction for data augmentation is to generate examples from scratch by grounding to auxiliary documentation. @xcite generate question-answer pairs with a model pre-trained on available training data. this often requires additional filtering or denoising measures to ensure correctness of generated data. also, these models are built for the purposes of single-turn question answering, rather than multi-turn dialogues. inspired by @xcite , we propose an automatic document-grounded dialogue generation (dg 2 ) method that augments the amount of data available for training a dialogue system. the model consists of a user bot and an agent bot that alternately generates utterances to complete a conversation. the user bot includes a span extraction model that can first select a passage and then predict the rationale start and end positions inside a passage. the agent bot has a denoising mechanism to filter out generated rationales irrelevant to the conversation. the user bot begins by selecting a passage from the document that is most relevant to the current context. it then selects a rationale span from this passage and generates the user utterance. the agent bot takes the selected span from the user bot, and then checks if it can find the correct ratio-nale span, and finally generates the agent response. this process repeats until an entire dialogue is generated. we evaluate our model on a representative document-grounded dialog dataset doc2dial @xcite . we test and generate additional dialogs with both the seen documents and unseen documents. we augment the original dataset and train it on a downstream model. the results show that our method improves the performance of the downstream model after augmentation. we also test scenarios of low-resource settings. we train and evaluate the generative models with only 25%, 50%, 75% data. experimental results show that our method perform well even when training data is scarce. ## document-grounded dialogue setup a dialogue can be thought of as a series of turns between two interlocutors. within goal-oriented dialogues, we refer to the first speaker as the user, and the second speaker as the agent, whom we model as @xmath0, a 1 ), (u 2 , a 2 ), ...(u t , a t )]. in a document-grounded setting, the conversation revolves around the topics and entities mentioned in the associated document. a document is composed of a series of text passages, which are themselves broken down further into spans. dialogue success is determined by following the typical success metrics for any given task, where the only difference is that the outcome of the conversation is likely to depend on the ability to reason about the contents of the document. while sophisticated architectures are certainly capable of improving document-grounding, we take a data-centric approach instead by generating new dialogues from the documents to serve as additional training data for the downstream model. ## data augmentation via dg 2 we propose document-grounded dialogue generation (dg 2 ) as a method of data augmentation. we aim to generate a complete and coherent dialogue given a document by building two bots talking to each other. given a document c, we can model a dialog d between the user and the agent with: we further decompose the model into three parts: passage selection, rationale extraction, and utterance generation. we also apply a filtering model to ensure the quality of generated utterances. ## experiments we first introduce the datasets evaluated with our method, then the baselines for comparisons, and in the end our method's implementation details. doc2dial consists of two subtasks around identifying relevant spans based on dialogue context and producing cohesive responses based on extracted rationales @xcite . formulated as a span selection task, user utterance understanding requires an agent to interpret user queries in the context of the dialogue history and then select the relevant span from the associated document. predicted spans are graded based on exact match (em) and f1-score. exact match is when the predicted span exactly lines up with the actual span. f1-score balances the recall and precision of the predicted uni-grams compared to the gold span. ## results and analysis this section shows the results for the full dataset and low-resource settings. we also conduct human evaluation on the generated dialogues. afterwards, we discuss the results by analyzing generated examples. ## ethical consideration the models and approaches introduced in our work involve using synthetic data as an enhancement to existing datasets for modeling document-grounded dialogue. for the existing datasets, they are often dialogue simulation data generated by human workers based on their understanding of the associated document content and dialogue context. there are potential biases or toxic content introduced in the existing simulation during data collection. we can address such concerns by making efforts to improve the quality of the generated data that has shown its effectiveness in the downstream task. therefore, our method can add an extra layer of safety and privacy if we only use generated data for training downstream models. future work can explore how data augmentation can help to build a more private and safe dataset. ## conclusion to address the problem of limited data in documentgrounded dialogue systems, we propose dg 2 to perform data augmentation via dialogue generation. our technique generates diverse utterances grounded on the given document while filtering the utterances to ensure quality and correctness when training on the downstream model. we demonstrated the effectiveness of our pipeline by showing the improvement over the previous data augmentation methods. we additionally show competitive results in the low-resource setting when a limited amount of human annotated data is available for training. future work will explore more techniques of filtering to improve data quality. we hope this spurs further research into document-grounded augmentation techniques for dialogue systems.