Building Goal-oriented Document-grounded Dialogue Systems (2021)

Abstract: In this paper, we describe our systems for solving the two Doc2Dial shared task: knowledge identification and response generation. We proposed several pre-processing and post-processing methods, and we experimented with data augmentation by pre-training the models on other relevant datasets. Our best model for knowledge identification outperformed the baseline by 10.5+ f1-score on the test-dev split, and our best model for response generation outperformed the baseline by 11+ Sacrebleu score on the test-dev split.

## introduction there has been a recent surge of interest in building domain-specific question answering (qa) systems, in both academia and industry. compelling realworld applications include customer services and decision-support, wherein there is strong reliance on such qa systems to be of high quality. a significant challenges for building qa systems is that domain-specific data is relatively sparse and much noisier, compared to samples from well-studied public benchmark datasets. also, when the answer is not explicitly present in the context, models must generate new answers instead of extracting from document, adding complexity to the problem. in this paper, we make efforts toward building domain-oriented question answering systems, by tackling the two doc2dial shared-tasks 1 : knowledge identification and response generation. for knowledge identification (subtask1), the main goal is to identify the grounding knowledge, in form of a document span, for the next-agent conversational turn. for response generation (subtask2), the main objective is to generate the next-agent response, in natural language. we experiment with various baseline models, and we developed and evaluated our proposed solutions. some improvement strategies we tried include post-processing, hyper-parameter tuning, and pre-training on other well-known datasets such as squad @xcite . we found that with carefully-selected hyperparameters, and with pre-processing and postprocessing heuristics, the baseline model's performance can be significantly improved: our best model is able to out-perform the provided baseline by 10.5+ f1-score on the test-dev split for subtask1, and our best model for subtask2 out-performed the baseline by 11+ sacrebleu score on test-dev. ## related work there are many previous works that study the problem of dialogue-based question answering. some of them only focused on answering the questions based on dialogue history alone @xcite , while, for others, the dialogue and question-answer pairs are based on a document @xcite . most of these tasks are extractive in nature, meaning that the exact answer can be located in the document or dialogue. among them, coqa @xcite is the most similar task to doc2dial dataset. the main objective of the coqa challenge is to measure machine learning models' ability to comprehend text and answer related questions that appear in a conversation; also, because some answers may not appear explicitly in the document, the model may be required to synthesize the answer based on evidence. the two sub-tasks we study in this paper differ from those described above-mainly in terms of dataset attributes. the doc2dial dataset mostly contains long documents and dialogues that inter-connect with each other. moreover, the ground-truth answers in doc2dial are usually long as well, which makes the associated prediction tasks harder to tackle. thus, the models and heuristics we have developed are mainly targeted towards handling these specific scenarios and problems. ## result and analysis for subtask1, we report f1-score and exact-match score on the dev set for our proposed method. for subtask2, we report sacrebleu @xcite on the dev set. finally, we report the test set results achieved with our best model, for both tasks.