Improving Multiple Documents Grounded Goal-Oriented Dialog Systems via Diverse Knowledge Enhanced Pretrained Language Model (2022)

Abstract: In this paper, we mainly discuss about our submission to MultiDoc2Dial task, which aims to model the goal-oriented dialogues grounded in multiple documents. The proposed task is split into grounding span prediction and agent response generation. The baseline for the task is the retrieval augmented generation model, which consists of a dense passage retrieval model for the retrieval part and the BART model for the generation part. The main challenge of this task is that the system requires a great amount of pre-trained knowledge to generate answers grounded in multiple documents. To overcome this challenge, we adopt model pretraining, fine-tuning, and multi-task learning to enhance our model’s coverage of pretrained knowledge. We experimented with various settings of our method to show the effectiveness of our approaches.

## introduction recently, deep learning-based dialog systems have attracted much attention from academia and the industry. the main challenge of dialog systems is to generate fluent responses consistent with the users' text input. as pre-trained language models (plms) (e.g., bart @xcite and gpt2 @xcite ) have emerged, dialog systems have taken advantage of plms @xcite @xcite , which can enhance the quality of dialog response by applying implicit language knowledge. however, these systems lack knowledge of specific topics and thus show weakness in conducting an in-depth conversation with humans. there have been various works for knowledge-grounded dialogue systems to address this problem. @xcite knowledge grounded dialogue models are capable of generating precise responses based on both the dialogue context and external sources. therefore, researchers have usually constructed dialogue flows grounded in related documents @xcite or knowledge graphs @xcite @xcite . in particular, @xcite have introduced the doc2dial tasks for goal-oriented document-grounded dialog systems. compared to previous works, doc2dial has introduced a more challenging setting with multiturn queries and aims to generate natural language responses from relevant grounding document. on top of that, they also propose the multidoc2dial dataset @xcite ,which is built upon the doc2dial dataset. multidoc2dial dataset is more closely related to real-life scenarios than the prior work since the agent generates responses based on multiple documents as grounding knowledge. due to its multi-document setting, utilizing knowledge has become more complex. to utilize external knowledge in dialogue, knowledge grounded models generally consist of a retrieval model and a generative model. recently, the retrieval augmented generation (rag) model @xcite has been proposed to leverage both parametric @xcite and non-parametric memory @xcite ) methods by combining pre-trained seq2seq models and the dense vector index of grounding documents. however, the rag model lacks knowledge related to question answering and dialogue generation. in this paper, our team jpl proposes four approaches to enhance rag's diverse knowledge: multi-task learning, data augmentation, pretraining and contrastive learning. multi-task learning, extra pretraining on conversational question answering datasets, and data augmentation enhance the model's task-oriented knowledge. contrastive learning for the dpr retriever module strengthen task-agnostic knowledge. we participate in the second dialdoc shared task held by acl, multi-doc2dial: modeling dialogues grounded in multiple documents @xcite . these methods cultivate the dialogue model's capability to use complex external knowledge on top of plm's inherent power. ## methodology we use four methods to enhance the model's ability to efficiently utilize external grounding knowledge especially on dialogue modeling. ## conclusion in this paper, we explain our submissions to the multidoc2dial shared task. we utilize various conversational qa datasets and methods to improve the given baseline model. our rag model is composed of dpr for the retriever and bart for the generator. we train dpr with contrastive learning with an extra hard negative sample. bart is pretrained on conversational qa datasets, coqa and doc2dial. on the end-to-end level, we implement multi-task learning to utilize model knowledge obtained from the previous grounding task that is trained on augmented data. all of the mentioned techniques enhance the model performance compared to the suggested baseline model.