U G ent- T2K at the 2nd D ial D oc Shared Task: A Retrieval-Focused Dialog System Grounded in Multiple Documents (2022)

Abstract: This work presents the contribution from the Text-to-Knowledge team of Ghent University (UGent-T2K) to the MultiDoc2Dial shared task on modeling dialogs grounded in multiple documents. We propose a pipeline system, comprising (1) document retrieval, (2) passage retrieval, and (3) response generation. We engineered these individual components mainly by, for (1)-(2), combining multiple ranking models and adding a final LambdaMART reranker, and, for (3), by adopting a Fusion-in-Decoder (FiD) model. We thus significantly boost the baseline system’s performance (over +10 points for both F1 and SacreBLEU). Further, error analysis reveals two major failure cases, to be addressed in future work: (i) in case of topic shift within the dialog, retrieval often fails to select the correct grounding document(s), and (ii) generation sometimes fails to use the correctly retrieved grounding passage. Our code is released at this link.

## introduction most prior research on document-grounded dialog systems assumes a single document for each dialog @xcite @xcite . there are relatively few works on multi-document grounded (mdg) dialog modeling, which requires a dialog system to (i) retrieve grounded passages (or documents) given the user question, and then (ii) generate responses based on the retrieval results and dialog context. real-world applications (e.g., administration question answering, travel booking assistance and procedural task guidance) for mdg are challenging because of more complex user behavior in such dialogs on diverse information sources. in particular, for (i) retrieval of grounding text passage(s) the challenges pertain to keeping track of dialog state, topic shift (e.g., switching from driving license requirements to car insurance), vocabulary mismatch, vague question formulation, etc. furthermore, (ii) response generation needs to appropriately phrase the answer to fit in a human(-like) dialog rather than simply copying a source document snippet. we leverage the recently released dialog dataset, multidoc2dial @xcite , to tackle aforementioned challenges. we build a pipeline system (fig. 1 ) comprising (1) a document retriever, (2) a passage retriever, and (3) an answer generator fusing multiple grounding input passages. given the dialog context (i.e., the dialog history and user question), a document retriever searches given supporting documents to select the top-m related ones. subsequently, these full documents are segmented into shorter passages ranked by a passage retriever. for these retrieval components (1)-(2), we use an ensemble approach -combining bm25, cosine similarity, etc.; for passage retrieval, we included dense passage retrieval (dpr; @xcite -followed by a reranking step using lam-bamart @xcite . the top-k passages are fused with the dialog context by a response generator to produce knowledge-grounded responses, based on fusion in decoder (fid; @xcite . we contribute with: (i) a multi-stage pipeline system comprising first the grounding text retrieval stages, split further into document and subsequent passage retrieval components (both using a multi-feature ensemble system), and second an answer generation model fusing information from multiple passages; (ii) experiments demonstrating that our pipeline system outperforms the baseline method by a large margin (over +10 points for both f1 and sacrebleu); (iii) insightful error analysis, suggesting that the main shortcomings of the current system include failures of (a) the retrieval stages in case of topic shifts by the user, and (b) the answer generation stage to identify the correct grounding passage among its inputs. our codes are released at https://github. com/yiweijiang2015/ugent-t2k-dialdoc 2 task definition the multidoc2dial shared task comprises two subtasks: in the seen-domain (referenced by subscript s), the system can rely on training data comprising both exemplary dialogs as well as the corresponding document set from the domains it will be tested on, whereas in the unseen-domain (referenced by u ) no related dialogs nor documents have been seen by the system before. in general, for both subtasks, a system first retrieves relevant documents from a document pool (d s or d u ) given the dialog context, i.e., a user's question q i (i is the turn number) and the full conversation history @xmath0. current state-of-the-art solutions split long documents into passages (p s or p u ) to facilitate more fine-grained location of the grounding information. second, the grounding information g (span(s) or passage(s)) for q i has to be identified within the passages of retrieved documents. the multidoc2dial dataset was curated such that g for each question can be exactly found within one document, while the full dialog's answers jointly may span multiple documents, thus requiring a model to decide when to switch to a different document. (note that, depending on how exactly a document is split into shorter passages, g may extend over more than one passage.) third, a generation model takes as input g and @xmath1.e., f1_u, sacrebleu, rouge-l and meteor). ## model the next subsections detail the aforementioned components (1)-(3) of our pipeline system. ## conclusion we propose a pipeline system for dialogs grounded in multiple documents. our system consists of a document retriever, a passage retriever and a multipassage-fusing generator. the retriever is designed to limit the passage search space by first ranking documents, which proves to enhance the passage retrieval performance considerably for the multi-doc2dial shared task. compared to the baseline rag model, our multi-passage-fusing generator achieves better knowledge-grounded answer generation. based on error analysis of our current system, future work will focus on the topic shift issue for conversational retrieval and investigate the knowledge misrecognition problem for dialog generation.