D ial D oc 2021 Shared Task: Goal-Oriented Document-grounded Dialogue Modeling (2021)

Abstract: We present the results of Shared Task at Workshop DialDoc 2021 that is focused on document-grounded dialogue and conversational question answering. The primary goal of this Shared Task is to build goal-oriented information-seeking conversation systems that can identify the most relevant knowledge in the associated document for generating agent responses in natural language. It includes two subtasks on predicting agent responses: the first subtask is to predict the grounding text span in the given document for next agent response; the second subtask is to generate agent response in natural language given the context. Many submissions outperform baseline significantly. For the first task, the best-performing system achieved 67.1 Exact Match and 76.3 F1. For the second subtask, the best system achieved 41.1 SacreBLEU and highest rank by human evaluation.

## introduction goal-oriented conversational systems could assist end users to query information in documents dynamically via natural language interactions. meanwhile, there is a vast number of documents in which individuals and organizations choose to present their interests and knowledge to the world for broad applications. thus, it attracts a lot of attentions from researchers and practitioners from different fields. there have been significant individual research threads that show promises in handling heterogeneous knowledge embedded in the documents @xcite , including (1) unstructured content such as text passages (coqa @xcite , quac @xcite , sharc @xcite , doqa @xcite , doc2dial @xcite ); (2) semi-structured content such as tables or lists (sqa @xcite , hybridqa @xcite ); (3) mul-timedia such as images and videos with associated textual descriptions (recipeqa @xcite , pstuts-vqa @xcite , mi-moqa @xcite ) despite these recent advances, the challenge remains for handling multiturn queries of complex dialogue scenarios @xcite and then respond based on the most relevant content in documents of various types from wide domains. as a step forward, we propose a shared task and competition to invite researchers to bring their individual perspectives and advance the field in joint effort. we introduce dialdoc 2021 shared task, which focuses on building goal-oriented informationseeking dialogue that are grounded in textual content. in particular, the goal is to develop a dialogue system to comprehend multi-turn queries and identify the most relevant knowledge in the associated document for generating agent responses in natural language. it includes two subtasks for predicting agent response. the first subtask (subtask 1) is to predict the grounding text span in the given document for next agent response; the second subtask (subtask 2) is to generate agent response in natural language given the contexts. the dataset used for the task is a goal-oriented document-grounded dialogue dataset doc2dial @xcite . we hosted the leaderboards for dev-test and test phase on eval.ai for two subtasks respectively. there are a total of 23 teams that participated dev-test phase. for final test phrase, 11 teams submitted to the leaderboard of subtask 1, and 9 teams submitted to the leaderboard of subtask 2. for the first task, the best system achieved 67.09 exact match and 76.34 f1. for the second subtask, the best system achieved 41.06 sacrebleu and rank the best by human evaluation. in this work, we first describe the dataset and the two subtasks. then, we provide a summary of the evaluation results from participating systems. ## dataset we use doc2dial dataset foot_0 introduced in @xcite , which contains 4793 goal-oriented dialogues and a total of 488 associated grounding documents from four domains for social welfare: dmv, va, ssa, and studentaid. in this dataset, dialogues contain the scenarios when agent ask follow-up questions for clarification or verification based on dialogue-based and document-based context. each turn is annotated with (1) grounding span from the associated document, (2) dialogue act, e.g., query, respond and (3) speaker role, either agent or user. for developing models, we divide the data into training, validation and test split based on the number of dialogues. for evaluating the models, we provide a dev-test set which contains about 30% test dataset. the final test set also includes dialogue and document data from an unseen domain cdccovid that is not in the training, validation or dev-test set. the dialogues of unseen domain were collected in the same data collection process as published doc2dial dataset. ## task description this shared task focuses on building goal-oriented information-seeking dialogue systems. the goal is to teach a dialogue system to identify the most relevant knowledge in the associated document for generating agent responses in natural language. it includes two subtasks on predicting agent response. the agent can either provide an answer or ask follow-up question. here we only consider the cases that use queries are answerable. ## baseline subtask 1 the baseline model for subtask 1 is based on bert-qa @xcite . for each token, it computes the probabilities of start and end positions by a linear projection from the last hidden layers of the bert model. then it multiplies the scores of the start and end positions for estimating the probability of the corresponding span. as a baseline, we fine-tune bert-base on doc2dial dataset where the input is dialogue query and the associated document context. the dialogue query is the concatenation of dialogue turns in reverse order. ## conclusion we presented the results of 1st dialdoc 2021 shared task, which included two subtasks on document-grounded goal-oriented dialogue modeling. we received submissions from a total of 17 teams during entire phase for subtask 1, and 9 teams for subtask 2. all submissions during final test phase outperformed baselines by a large margin for both subtasks. by organizing this shared task, we hope to invite researchers and practitioners to bring their individual perspectives on the subject, and to jointly advance the techniques toward building assistive agents to access document content for end users by conversing.