Causal Document-Grounded Dialogue Pre-training (2023)

Abstract: The goal of document-grounded dialogue (DocGD) is to generate a response by anchoring the evidence in a supporting document in accordance with the dialogue context. This entails four causally interconnected variables. While task-specific pre-training has significantly enhanced performances on numerous downstream tasks, existing DocGD methods still rely on general pre-trained language models without a specifically tailored pre-training approach that explicitly captures the causal relationships. To address this, we present the first causally-complete dataset construction strategy for developing million-scale DocGD pre-training corpora. Additionally, we propose a causally-perturbed pre-training strategy to better capture causality by introducing perturbations on the variables and optimizing the overall causal effect. Experiments conducted on three benchmark datasets demonstrate that our causal pre-training yields substantial and consistent improvements in fully-supervised, low-resource, few-shot, and zero-shot settings.

## introduction goal-oriented dialogue, focusing on assisting users in achieving their goals through natural language interactions, has made significant progress in recent years @xcite . nonetheless, these systems frequently encounter constraints in providing information that extends beyond what can be obtained from particular databases or domains. to address this issue, researchers have proposed the goal-oriented document-grounded dialogue (docgd) task @xcite , which leverages documents as the external knowledge source to support dialogue systems in meeting the diverse information needs of users. recently, task-specific pre-training has shown an extraordinary ability to boost performances 𝑑 𝑑 𝑒 𝑟 𝑐 on downstream tasks, by mitigating the gaps between pre-training and fine-tuning due to different data distributions and training objectives @xcite @xcite . nonetheless, the study of pre-training for docgd is hindered due to the difficulty of modeling the causal effects, which is a prominent characteristic of docgd. as shown in figure 1 , the task requires the model to identify evidence in a document based on the dialogue context, and then utilize the grounding evidence to generate a corresponding response. this process involves the interplay of four variables that are causally connected. to attain precise modeling of causal effects during pre-training, two challenges must be overcome: (1) the scarcity of large-scale and causally-complete docgd datasets for pre-training, as opposed to dialogue generation tasks that have the advantage of utilizing conversational data from various social media sources @xcite , (2) the traditional likelihood objective (e.g., @xcite ) being insufficient to capture the causal relationships among variables. for the first challenge, we propose a novel strategy for building a docgd pre-training corpus. we define a dataset as causally-complete if it includes all the variables related to a task and encompasses all reasonable causal relationships among these variables. our strategy involves two steps. firstly, we transform wikipedia documents into dialogues by generating pseudo-user utterances and modifying evidence in the documents to serve as agent responses. secondly, we extract grounding documents embedded in urls and insert virtual evidence to supplement dialogues from reddit. both steps guarantee that the datasets are causallycomplete, and they complement one another, as the former has authentic evidence with synthetic dialogues, while the latter possesses authentic dialogues with synthetic evidence. to tackle the second challenge, we propose a causally-perturbed pre-training strategy that enhances the modeling of causality in our pre-training datasets. our approach entails introducing causal interventions to both the document and evidence variables while optimizing the total effect of responses for different causes. the total effect comprises the natural direct effect (nde) and total indirect effect (tie) @xcite . in essence, the nde quantifies the impact of irrelevant sentences in the supporting document, while the tie captures the influence of evidence (detailed explanations in §3.3). our objective is twofold: to enhance the model's resilience to perturbations in irrelevant sentences by minimizing the nde, and to promote reliance on evidence in generating dialogue responses by maximizing the tie. to achieve this, we retain relevant evidence while perturbing the remaining parts of the document to improve response consistency when using two versions, thus reducing the nde. additionally, we eliminate evidence from the document while preserving other information, subsequently decreasing the likelihood of generating original responses, thus maximizing the tie. overall, we refer to the two aforementioned strategies jointly as causal document-grounded dialogue (causaldd). we thoroughly conduct experiments and analyses on three docgd benchmark datasets. our results, obtained through fullysupervised, few-shot, low-resource, and zero-shot scenarios, and evaluated by both automatic and human assessment, convincingly demonstrate the effectiveness of our pre-training corpus construction and causally-perturbed pre-training strategy. especially, causaldd even outperforms gpt-3.5. ## methodology in this section, we first take a casual-effect perspective on the docgd task in §3.1. we then propose two strategies to overcome the challenges discussed in §1: (1) a dataset construction strategy in §3.2 for building a causally-complete pre-training corpus; ## conclusion in this paper, we demonstrate that modeling complete causal relationships among variables (i.e., documents, dialogue contexts, evidence, and responses) is necessary for pre-training for documentgrounded dialogue task (docgd). we propose a strategy for creating causally-complete pretraining datasets and design a causally-perturbed pre-training strategy to model the causality of docgd. to the best of our knowledge, this is the first work that analyzes docgd from a causal perspective. extensive experiments and analyses verify that our causal pre-training method causaldd significantly improves performance under fullysupervised, few-shot, and low-resource settings, while also accelerating convergence and enhancing the ability to handle complex cases.