Technical Report on Shared Task in D ial D oc21 (2021)

Abstract: We participate in the DialDoc Shared Task sub-task 1 (Knowledge Identification). The task requires identifying the grounding knowledge in form of a document span for the next dialogue turn. We employ two well-known pre-trained language models (RoBERTa and ELECTRA) to identify candidate document spans and propose a metric-based ensemble method for span selection. Our methods include data augmentation, model pre-training/fine-tuning, post-processing, and ensemble. On the submission page, we rank 2nd based on the average of normalized F1 and EM scores used for the final evaluation. Specifically, we rank 2nd on EM and 3rd on F1.

## introduction our team scir-dt participates in the dialdoc shared task in the document-grounded dialogue and conversational qa workshop at the acl-ijcnlp 2021. there are two sub-tasks based on the doc2dial dataset @xcite . the dataset contains goal-oriented conversations between a user and an assistive agent. each dialogue turn is annotated with a dialogue scene, which includes role, dialogue act, and grounding in a document (or irrelevant to domain documents). the documents are from different domains, such as social security and veterans affairs. sub-task1 is knowledge identification which requires identifying the grounding knowledge in form of document span for the next agent turn. the input is dialogue history, current user utterance, and associated document. the output should be a text span. the evaluation metrics are exact match (em) and f1 @xcite . sub-task2 is text generation which requires generating the next agent response in natural language. the input is dialogue history and associated document. the output is agent utterance. the evaluation metrics are sacrebleu @xcite and human evaluations. we only participate in sub-task 1. ## our method we first use two data augmentation methods to obtain a 5-times larger augmented dataset. we use the augmented data to re-train bert and roberta with the whole word masking technique and finetune bert, roberta, and electra models. we test several span post-processing methods and then propose an ensemble method with trainable parameters for final text span selection. the pipeline we used in this competition is illustrated in figure 1 . ## conclusion we introduced our submission for doc2dial shared task. in sub-task 1, our model is based on roberta and electra. we propose a simple but efficient ensemble method for knowledge selection in multi-turn dialogue. our team scir-dt ranks 2nd on the final submission page. apart from the methods we introduced, there are other methods that could further improve the performance of our model. for example, @xcite proved the dialogue act information was useful for subtask 1; there are some noisy data such as empty responses in the dialogue data could be filtered out during training; employing machine reading comprehension dataset such as squad @xcite or cqa dataset such as coqa @xcite for pre-training and fine-tuning may also be helpful. however, due to the time limitation, we did not try all these methods during the competition. we hope these methods and experiences would be helpful for future contestants.