D oc2 B ot: Accessing Heterogeneous Documents via Conversational Bots (2022)

Abstract: This paper introduces Doc2Bot, a novel dataset for building machines that help users seek information via conversations. This is of particular interest for companies and organizations that own a large number of manuals or instruction books. Despite its potential, the nature of our task poses several challenges: (1) documents contain various structures that hinder the ability of machines to comprehend, and (2) user information needs are often underspecified. Compared to prior datasets that either focus on a single structural type or overlook the role of questioning to uncover user needs, the Doc2Bot dataset is developed to target such challenges systematically. Our dataset contains over 100,000 turns based on Chinese documents from five domains, larger than any prior document-grounded dialog dataset for information seeking. We propose three tasks in Doc2Bot: (1) dialog state tracking to track user intentions, (2) dialog policy learning to plan system actions and contents, and (3) response generation which generates responses based on the outputs of the dialog policy. Baseline methods based on the latest deep learning models are presented, indicating that our proposed tasks are challenging and worthy of further research.

## introduction the last decade has witnessed a dramatic change in how humans interact with information retrieval systems. although traditional search engines still play an important role in our daily life, the wide adoption of smart devices with small screens requires systems to answer user requests more concisely. early attempts focus on answering independent questions @xcite , whereas recent studies pay attention to handling interconnected questions via conversations around a single passage @xcite or documents @xcite . yet, the nature of heterogeneous documents and our conversational setting pose challenges that require further attention. we, therefore, develop doc2bot foot_0 with these considerations in mind. the first concerns the nature of heterogeneous documents, which often contain different types of structures such as tables and sequences. to answer questions regarding such structural types, systems need to acquire various skills. figure 1 shows a conversation between a user and an agent, where the agent has access to a collection of documents. in this conversation, every utterance except the first one depends on the conversation history and the grounded documents. the document contains diverse structures including conditions (n2), sequences (n3), and tables (n4). each type of structure poses different challenges for the agent. for example, the agent needs to understand the rule text in n2 to ask questions such as a3 and a5. the agent also needs to be able to summarize the procedure in n3 to produce the answer (a13) or jump to a specific step of the procedure if being asked (a15). in addition, the agent needs to be capable of summarizing the tables (a19) or answering questions with row and column constraints (a21). note that there have been previous attempts @xcite @xcite to address different structures in isolation. however, little research has been done on handling multiple types of structures in the same document context. the second concerns the nature of our conversational setting, which is to help users seek information from documents. since user information needs are often under-specified, it is desirable for systems to ask clarifying questions. this resembles the task of relevance feedback and query refinement in traditional information retrieval. however, in doc2bot, system feedback is in the form of natural u0 a1 hello, i'd like to know about the social security and medical insurance of hz city. which one do you want to know? industrial injury insurance, unemployment insurance or medical insurance? u2 a3 u4 a5 ## dataset collection this section details the process of collecting doc2bot, including 4 stages: 1) document collection which selects targeted domains and documents; 2) document graph construction which unifies heterogeneous structures from multiple domains to build document graphs; 3) dialog flow generation that simulates the agenda of a user seeking information from a document graph; and 4) dialog collection where crowd-collaborators write dialogs based on the generated dialog flows. ## data analysis document data dialog data doc2bot contains 6,619 dialogues with 101,994 turns (see for evaluation, we define several tasks (see section 5) and split the dialog dataset into a training set, consisting of 70% of dialogues, as well as a validation set of 10% and a test set of 20%. ## conclusion this paper presented doc2bot, a novel dataset for dgds for information seeking. unlike prior datasets, doc2bot contains examples that simultaneously test the ability of machines to comprehend heterogeneous documents and clarify user information needs. we proposed three main tasks associated with doc2bot: (1) dialog state tracking, which tracks user intentions during the conversation, (2) dialog policy learning, which plans the next system action and contents, and (3) response generation, which is to generate system responses based on the outputs of the dialog policy. we then presented baselines for our tasks using several contemporary models. our experimental results showed that: 1) both dialog states information and document structure information are important for the task of dialog policy learning; 2) planning system actions helps improve response generation; and 3) the tasks of dialog state tracking and dialog policy learning, which are essential for response generation, are still very challenging with joint accuracy of only 34.55% and 43.13% respectively. we hope that our dataset and such observations will be helpful for future research in this direction.