Docalog: Multi-document Dialogue System using Transformer-based Span Retrieval (2022)

Abstract: Information-seeking dialogue systems, including knowledge identification and response generation, aim to respond to users with fluent, coherent, and informative answers based on users’ needs. This paper discusses our proposed approach, Docalog , for the DialDoc-22 (MultiDoc2Dial) shared task. Docalog identifies the most relevant knowledge in the associated document, in a multi-document setting. Docalog , is a three-stage pipeline consisting of (1) a document retriever model (DR. TEIT) , (2) an answer span prediction model , and (3) an ultimate span picker deciding on the most likely answer span, out of all predicted spans. In the test phase of MultiDoc2Dial 2022, Docalog achieved f1-scores of 36.07% and 28.44% and SacreBLEU scores of 23.70% and 20.52%, respectively on the MDD-SEEN and MDD-UNSEEN folds.

## introduction introducing a machine-generated dialogue with a human level of intelligence has been consistently among dreams of artificial intelligence with a vast number of applications in different domains, ranging from entertainment @xcite to healthcare systems @xcite . in such a system, the machine has to (i) understand the flow of conversation, (ii) raise informative questions, and (iii) answer problems in different domains of interest, and in some cases it has to act as an all-knowing agent @xcite . recent advances in nlp have made this dream closer to reality. in the last decade, the success of the neural language model in language understanding and generation has encouraged more and more contributions from both academia and industry in the area of conversational artificial intelligence @xcite . the major efforts in conversational artificial intelligence can be categorized into three subareas @xcite : in this system paper, we present our work on the dialdoc shared task 2022 centered on developing a qa dialogue system. a common approach to this problem comprises two subtasks of (i) knowledge identification (ki) to retrieve the knowledge from the documents and (ii) response generation @xcite to generate an answer based on the retrieved knowledge @xcite . the multi-document scenario, meaning that the related documents have to be retrieved before the answer generation, is the main distinction between the dialdoc shared tasks in 2021 and 2022. to tackle this problem, we propose a three-stage pipeline, called docalog, consisting of (1) document retriever model (dr. teit), (2) an answer span prediction model, a state-of-the-art transformer-based model taking single documents (dr. teit results) as input and outputting the answer span for every input document, and (3) an ultimate span picker deciding on the most likely answer span, out of all predicted spans in the step (2). in multidoc2dial 2022 challenge, during the test phase, docalog achieved an f1-score of 36.07% and a sacrebleu of 23.70% on the mdd-seen, and an f1-score of 28.44% and a sacrebleu of 20.52% on the mdd-unseen. ## related work the main focus of dialdoc shared tasks has been on developing task-oriented information-seeking dialogue systems, an important setting in the domain of conversational ai @xcite . some of the performing models in this domain have been caire @xcite , scirdt @xcite , and rwth @xcite . the proposed approaches of caire and scirdt utilize additional data for the augmentation of pre-trained language models in span detection, and rwth @xcite model uses neural retrievers for obtaining the most relevant document passages. in a broader context, the major work in document-grounded dialogue modeling can be divided into the following categories: (i) qa in an unstructured content, e.g., coqa @xcite , quac @xcite , sharc @xcite , doqa @xcite , and doc2dial @xcite (ii) qa in a semi-structured content, such as tables or lists, e.g., sqa @xcite , and hybridqa @xcite and thirdly (iii) qa in a multimedia content (images and videos with associated textual descriptions), e.g., recipeqa @xcite , pstuts-vqa @xcite , and mimoqa @xcite . ## results document retriever: in our experiments, dr. teit achieved a precision@5 of 86% and a mean reciprocal rank (mrr) of 0.72 indicating that on average, the hit is among the first two retrieved documents and it would be more than sufficient to take top-5 documents to the next step, i.e., span detection. ## conclusions we proposed docalog, a solution for the dialdoc-22 challenge. docalog is a three-stage pipeline consisting of (1) a document retriever model (dr. teit), (2) an answer span prediction model, and