M ulti D oc2 D ial: Modeling Dialogues Grounded in Multiple Documents (2021)

Abstract: We propose MultiDoc2Dial, a new task and dataset on modeling goal-oriented dialogues grounded in multiple documents. Most previous works treat document-grounded dialogue modeling as machine reading comprehension task based on a single given document or passage. In this work, we aim to address more realistic scenarios where a goal-oriented information-seeking conversation involves multiple topics, and hence is grounded on different documents. To facilitate such task, we introduce a new dataset that contains dialogues grounded in multiple documents from four different domains. We also explore modeling the dialogue-based and document-based contexts in the dataset. We present strong baseline approaches and various experimental results, aiming to support further research efforts on such a task.

## introduction with the recent advancements in nlp, there has been a surge of research interests and efforts in developing conversational systems for various domains. an important task in the field is conversational question answering and document-grounded dialogue modeling. prior work typically formulates the task as a machine reading comprehension task assuming the associated document or text snippet is given, such as quac @xcite , sharc @xcite , coqa @xcite , or-quac @xcite and doc2dial @xcite . however, such task setup neglects the common real-life scenarios where a goaloriented conversation could correspond to several sub-goals that are addressed in different documents. in this work, we propose a new task and dataset, multidoc2dial, on modeling goal-oriented dialogues that are grounded in multiple documents. we illustrate the proposed task in figure 1 . it includes a goal-oriented dialogue with four segments on the left and three relevant documents on the right. each dialogue segment indicates that all turns within it are grounded in a same document, e.g., turns from a3 to a7 in seg-2 are all grounded in doc-2. the blue dashed lines connect a dialogue turn with its corresponding relevant span in a document. the red dotted lines with arrows indicate that the dialogue flow shifts among the grounding documents through the conversation, i.e., doc-1 → doc-2 → doc-1 → doc-3. this example highlights certain challenges for identifying the relevant grounding content among different documents dynamically in a conversation. for instance, agent response a2 mentions 'insured' as an important condition based on a span in doc-1. however, there are no more details about 'insured' in doc-1. to further discuss about 'insured', the conversation naturally switches to refer to another document doc-2. another challenge is to handle deep dialogue context. for instance, to provide response to u4 or u6 in seg-2, the agent needs to understand the context of 'disability benefit' mentioned in seg-1. there are also cases such as seg-4 where its turn u10 starts a new question related to doc-3 but seems independent of the previous segments. the task goal in the example is naturally simple, but it still reveals the realistic expectation of document-grounded dialogue modeling that is still yet to be met. to the best of our knowledge, there is no existing task or dataset that addresses the scenarios where the grounding documents of goal-oriented dialogues are unknown and dynamic. to facilitate the study in this direction, we introduce a new dataset that contains conversations that are grounded in multiple documents. to construct the dialogue flows that involve multiple documents, we derive a new approach from the data collection pipeline proposed in @xcite . the newly composed dialogue flows have multiple dialogue segments where two adjacent segments are u10: hmm, i haven't checked my ssa online account for long time could you remind me where to start? a11: do you plan to apply online? if so, you can follow the link to fill out an application form the basics about disability benefits the social security disability insurance (ssdi) program pays benefits to you and certain family members if you are "insured". when you start receiving disability benefits, certain members of your family may qualify for benefits based on your work. ## model we propose to formulate the tasks on predicting agent turn as an end-to-end generation task inspired by the recent development of retriever-reader architecture @xcite . we consider retrieval-augmented generation (rag) @xcite as the base model. it includes a retriever component and a generative reader component. the retriever aims to retrieve the most relevant document passages given dialogue query; the generator, a pre-trained seq2seq model in our case, takes the combined dialogue query and top-n document passages as input and generates the target output. since we need to deal with the contextual information of dialogue turns and document passages, we further investigate how to utilize dialogue-based and document-based structure information for the retriever component. ## experiments we evaluate the proposed approaches on two tasks for predicting next agent turn, i.e., grounding generation and response generation. given current turn, dialogue history and all available documents in the dataset, we aim to evaluate generated text along with intermediate retrieval results. we split the data into train/validation/test sets as shown in ## conclusion and future work we introduced multidoc2dial, a new task and dataset that deals with goal-oriented dialogues that have multiple sub-goals corresponding to different documents. we proposed two tasks for predicting next agent turn that we formulate as generation tasks. we presented strong baseline approaches based on retriever-reader architecture and experimented with different variances of neural retrievers. for future work, we aim to address the ambiguity in the open-book dialogue modeling.