Document-Grounded Goal-Oriented Dialogue Systems on Pre-Trained Language Model with Diverse Input Representation (2021)

Abstract: Document-grounded goal-oriented dialog system understands users’ utterances, and generates proper responses by using information obtained from documents. The Dialdoc21 shared task consists of two subtasks; subtask1, finding text spans associated with users’ utterances from documents, and subtask2, generating responses based on information obtained from subtask1. In this paper, we propose two models (i.e., a knowledge span prediction model and a response generation model) for the subtask1 and the subtask2. In the subtask1, dialogue act losses are used with RoBERTa, and title embeddings are added to input representation of RoBERTa. In the subtask2, various special tokens and embeddings are added to input representation of BART’s encoder. Then, we propose a method to assign different difficulty scores to leverage curriculum learning. In the subtask1, our span prediction model achieved F1-scores of 74.81 (ranked at top 7) and 73.41 (ranked at top 5) in test-dev phase and test phase, respectively. In the subtask2, our response generation model achieved sacreBLEUs of 37.50 (ranked at top 3) and 41.06 (ranked at top 1) in in test-dev phase and test phase, respectively.

## introduction the dialdoc21 shared task is a task that generates a proper response by finding a knowledge span from a document associated with a dialogue history. it consists of two subtasks; subtask1 for finding useful knowledge spans from a document and subtask2 for generating proper responses based on the knowledge spans. the doc2dial dataset, the dataset for the dialdoc21 shared task, contains conversations * equal contribution between users and agents in real-world situations. the user and the agent engage in a conversation associated with a document, and the agent should provide the user with document-grounded information in order to guide the user. in this paper, we propose two models to perform the dialdoc21 shared task using a pre-trained language model. in particular, we show that in the process of finetuning the pre-trained model, the proposed input representations significantly contribute to improving performances. ## related work the baseline models for the subtask1 and the sub-task2 were proposed by @xcite , the composers of doc2dial datasets. they formulated the subtask1 as a span selection, inspired by extractive question answering tasks such as squad task @xcite . @xcite proposed a method to reflect the differences between knowledge spans used for each turn and current knowledge span candidates. the differential information is fused with or disentangled from the contextual information to facilitate final knowledge selection. @xcite constructed input presentation using word, dialog state and positional embedding. ## task description in the subtask1, our goal is to find a relevant knowledge span required for agent's response in a conversation composed of multi-turns from a given document. inspired by @xcite , we propose a joint model to perform both dialogue act prediction and knowledge span prediction. in the subtask2, our goal is to generate agent's response if you have a surviving divorced spouse they could get the same benefits as your widow or widower provided that your marriage 9 33 lasted 10 years or more. benefits paid to a surviving divorced spouse won't affect the benefit amounts your other 10 34 survivors will receive based on your earnings record. if your former spouse is caring for your child who is under age 16 or disabled and gets 35 benefits on your record , 36 they will not have to meet the length -of -marriage rule. the child must be your natural or legally adopted child. in natural language based on a dialogue history and a document associated with the dialogue history. the dialogue history consists of speakers and utterances. then, the document consists of sentences, tags, titles, and so on. based on these structural information of the dialogue history and the document, we define special tokens and embeddings. then, we propose a method to reflect these special tokens and embeddings to the well-known bart model @xcite . the doc2dial dataset contains goal-oriented dialogues and knowledge documents. for developing models, three sub-datasets in four domains (dmv, va, ssa, and studentaid) were deployed; a train dataset, a validation dataset, and a test-dev dataset. for evaluating the models, a test dataset in five domains (i.e., four seen domains (dmv, va, ssa, studentaid) and an unseen domain (covid-19)) was used. the test-dev dataset embodied 30% of the test dataset except for the unseen domain. ## curriculum learning to improve performances, we train the proposed models through curriculum learning @xcite . figure 2 illustrates the training process by curriculum learning. we first divide the training dataset into n buckets and train n teacher model (i.e., a teacher model per bucket). in this paper, n is set to four. then, we measure performances of each teacher model by using n-1 dataset except for those used for training each teacher model. based on the performances of the teacher models, we assign each data to difficulty levels. ## conclusion we proposed a document-grounded goal-oriented dialogue system for the dialdoc21 shared task. the proposed model used various special tags and embeddings for enriching input representation of pre-trained language models, roberta-large for knowledge span prediction and bart for response generation. in addition, curriculum learning was adopted to achieve performance improvements. in the subtask1, our span prediction model achieved f1-scores of 74.81 (ranked at top 7) and 73.41 (ranked at top 5) in test-dev phase and test phase, respectively. in the subtask2, our response generation model achieved sacrebleus of 37.50 (ranked at top 3) and 41.06 (ranked at top 1) in test-dev phase and test phase, respectively.