doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset (2020)

Abstract: We introduce doc2dial, a new dataset of goal-oriented dialogues that are grounded in the associated documents. Inspired by how the authors compose documents for guiding end users, we first construct dialogue flows based on the content elements that corresponds to higher-level relations across text sections as well as lower-level relations between discourse units within a section. Then we present these dialogue flows to crowd contributors to create conversational utterances. The dataset includes over 4500 annotated conversations with an average of 14 turns that are grounded in over 450 documents from four domains. Compared to the prior document-grounded dialogue datasets, this dataset covers a variety of dialogue scenes in information-seeking conversations. For evaluating the versatility of the dataset, we introduce multiple dialogue modeling tasks and present baseline approaches.

## introduction the task of reading documents and responding to queries has been the trigger of many recent research advances. on top of the development of contextual question answering quac @xcite and coqa @xcite , more recent work mantis @xcite and @xcite included more kinds of user intents for querying over documents; while sharc @xcite added follow-up questions from agents and binary answers from users for the inference over documents. these exciting works confirm the importance of modeling document-grounded dialogue. yet, it involves more complex scenes in practice, which requires better understanding of the inter-relations between conversations and documents. thus, we aim to investigate how to create the training instances to further approach real-world applications of document-grounded dialogue for information seeking tasks. in this work, we propose a new dataset of goaloriented document-grounded dialogue. figure 1 shows sample utterances from dialogues d1, d2 and d3 between an assisting agent and a user, and an example document in the middle. d1 and d2 are grounded in the given document, while d3 is irrelevant to the document. it illustrates two different types of contexts that we aim to capture: (1) dialogue-based context, where a query could be formed by a single or multiple turns, and (2) document-based context, which corresponds to varied forms of knowledge represented in the document. more specifically, dialogue-based context of a query could be initiated by a user (e.g., u1 in d1) or an agent (e.g., a3 in d1), and carried out through multiple turns by both roles (e.g., all turns in d2). document-based context could involve structural elements in documents, such as the headers t1 and t2 or list items of m1 and m2, as well as textual discourse units, such as clauses (e.g, "if your clothing has been damaged"). for creating such dataset, we consider the document contents for social welfare websites,such as ssa.gov and va.gov, which guide users to access various forms of information. we develop a pipeline approach for dialogue data construction. inspired by how human authors compose user-facing web content, we utilize both the highlevel hierarchical relations between document components, as well as the low-level semantic relations between discourse units @xcite to dynamically create outlines of dialogues, or we call dialogue flows. a dialogue flow is a sequence of interactions between an assisting agent and a user. each turn contains a dialogue scene that is defined by a dialogue act, a role (user or agent) and a piece of grounding content from a document. then we present these dialogue flows to crowd contributors to create conversational utterances. such approach helps to avoid additional noise from the post-hoc ## doc2dial we introduce doc2dial, a new dataset that includes (1) a set of documents; and (2) conversations between an assisting agent and an end user, which are grounded in the associated documents. figure 1 presents sample utterances from different dialogues along with a sample document from va.gov in the middle. it illustrates some prominent features in our dataset, such as the cases where a conversation involves multiple interconnected sub-tasks under a general inquiry (e.g., d1); or the cases where a conversation involves multiple interactions to verify the conditional contexts for one query (e.g., d2). recent work, such as @xcite , has started to address the challenge of modeling complex contexts by allowing follow-up questions from agents based on natural language inference rules extracted from the relevant documents. however, it also simplified the task by using only restricted forms of questions and binary answers. in our work, we not only encourage free-form utterances, but also aim to include various dialogue scenes that provoke inquires with different document-based and dialog-based contexts. a user query can be formed in single-turn or multiple-turn manners: (1) the user explicitly states a context that is associated with a text-span that contains a solution to the query, e.g., u5 on t4; ## related work our work is mainly focused on modeling dialogues that are grounded in documents. it is generally inspired by the recent substantial interests on the challenges of machine reading comprehension and conversational qa, such as coqa @xcite , quac @xcite and doqa @xcite . those tasks aim to support conversational question answering, which involves understanding a text passage and answering a series of interconnected questions that appear in a conversation. these tasks add the complexity of coreference resolution and contextual reasoning to the reading comprehension challenges such as squad @xcite , yet aim at identifying a solution from a given list of candidates by reasoning over spans from a document. our task shares those challenges and additionally introduces the dialogue scenes where the agent asks questions when the user query is identified as under-specified or additional verification required for a resolute solution. another recent work @xcite extends multiwoz @xcite by adding turns that are grounded in the faq knowledge for certain entity and domain. the documentbased knowledge used in our work is beyond faqs with entity as context but whole documents with more complex contexts. in addition, ours is also largely related to conversational search tasks, such as mantis @xcite . similarly, it also provides multi-turn conversations with varied user intents that are grounded in documents from stack exchange website. in addition to the domain difference, one major distinction is that the grounding in mantis is determined by the hyperlinks to a document. our grounding is defined at at a much finer level in addition to the link to a document. to the best of our knowledge, the closest related work to ours is sharc @xcite with dialogues that are grounded to a span of a given text snippet. it also proposes to address under-specified questions by requiring follow-up questions that are answerable with yes/no answers in similar domains. our dataset goes beyond sharc in several aspects nonetheless: we exploit not only paragraph-level structure but also higher-level document structure, we create conversations over much longer span of document content, where utterances are freeformed, as opposed to yes/no answers. ## conclusion we have introduced doc2dial, a new dialogue dataset for goal-oriented tasks that are grounded in documents from multiple domains. compared to previous work, our dialogues cover a greater variety of dialogue scenes that correspond to a much wider span of document content. for evaluation, we investigated three types of dialogue tasks and proposed baseline approaches. we hope this work will inspire and assist both dialogue and document modeling for tackling more real-life dialogue tasks.