D ial D oc 2022 Shared Task: Open-Book Document-grounded Dialogue Modeling (2022)

Abstract: The paper presents the results of the Shared Task hosted by the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering co-located at ACL 2022. The primary goal of this Shared Task is to build goal-oriented information-seeking conversation systems that are grounded in the domain documents, where each dialogue could correspond to multiple subtasks that are based on different documents. The task is to generate agent responses in natural language given the dialogue and document contexts. There are two task settings and leaderboards based on (1) the same sets of domains (SEEN) and (2) one unseen domain (UNSEEN). There are over 20 teams participating in Dev Phase and 8 teams participating in both Dev and Test Phases. Multiple submissions significantly outperform the baseline. The best-performing system achieves 52.06 F1 and the total of 191.30 on the SEEN task; and 34.65 F1 and the total of 130.79 on the UNSEEN task.

## introduction goal-oriented document-grounded dialogue systems enable end users to interactively query about domain-specific information based on the given documents. the tasks of querying document knowledge via conversational systems continue to attract a lot of attention from both research and industrial communities for various applications such as or-convqa @xcite , multidoc2dial @xcite , qrecc @xcite , topi-ocqa @xcite and abg-coqa @xcite . the previous shared task @xcite by the first dialdoc workshop addressed the task of goal-oriented information-seeking dialogue systems in the machine reading comprehension setting, where the dialogue is aiming at querying about the information provided in a given * work done while at ibm research document @xcite . however, in reallife scenarios, for conversation in a given domain, the grounding document is often unknown, a dialogue turn could arbitrarily correspond to any document, hence each dialogue could be grounded in multiple documents. thus, we propose to explore the open-book closed-domain setting for goaloriented information-seeking dialogue systems that are grounded in the given domain documents. we introduce the shared task at the second dialdoc workshop on document-grounded dialogue and conversational question answering (di-aldoc 2022 shared task). the shared task aims to deal with the information-seeking goal-oriented dialogues that have multiple sub-goals corresponding to different documents. the input includes the dialogue history, the current user turn, and a set of domain documents, the output is the agent's utterance in natural language. it comprises two tasks that address two different evaluation settings: (1) the seen task where the test data shares the same sets of domains as the training data; and (2) the unseen task where the test data is all in one unseen domain different from the training data. we host the leaderboards for dev and test phases for the seen and unseen tasks respectively on eval.ai foot_0 . there are over 20 teams participating in dev phase and 8 teams participating in both dev and test phases. multiple submissions significantly outperform the baseline. the best-performing system achieves 52.06 f1 and the total of 191.30 on the seen task comparing to 35.85 and 126.21 by the baseline; and 34.65 f1 and the total of 130.79 on the unseen task comparing to 19.26 and 59.52 by the baseline. in this report, we first describe the dataset and the two task settings. then, we summarize the approaches and evaluation results of several top participating teams. ## dataset in this shared task, the dataset is based on mul-tidoc2dial introduced by @xcite . it contains 4796 conversations with an average of 14 turns grounded in 488 documents from four domains including va.org and studentaid. org. for document data, each document includes a title, the body content with the span/section information as well as the html mark-ups such as list and title. for dialogue data, each turn in a dialogue contains: (1) the speaker role, (2) the dialogue act, (3) the grounding text span along with the title of the document, and (4) human generated utterance in natural language. each dialogue contains one or multiple segments where each indicates that all turns within one segment are grounded in the same document. ## task description our shares task centers on building open-book goal-oriented dialogue systems, where an agent could provide an answer or ask follow-up questions for clarification or verification. the main goal is to generate grounded agent responses in natural ## evaluation the evaluation is focused on the groundedness and naturalness of the generated agent response. we consider the automatic metrics as intrinsic evaluation metrics, and human annotations for extrinsic evaluations. ## conclusion we present the results of dialdoc 2022 shared task. world-wide researchers and practitioners brought their individual perspectives on the task through this data competition. we received over 500 submissions during the dev phase by over 20 teams for both seen and unseen leaderboards. for the final test phase, there were officially 8 teams submitted to the seen leaderboard and 6 teams submitted to the unseen leaderboard. most of the submissions during test phase beat the baseline performance by large margins.