Team JARS : D ial D oc Subtask 1 - Improved Knowledge Identification with Supervised Out-of-Domain Pretraining (2021)

Abstract: In this paper, we discuss our submission for DialDoc subtask 1. The subtask requires systems to extract knowledge from FAQ-type documents vital to reply to a user’s query in a conversational setting. We experiment with pretraining a BERT-based question-answering model on different QA datasets from MRQA, as well as conversational QA datasets like CoQA and QuAC. Our results show that models pretrained on CoQA and QuAC perform better than their counterparts that are pretrained on MRQA datasets. Our results also indicate that adding more pretraining data does not necessarily result in improved performance. Our final model, which is an ensemble of AlBERT-XL pretrained on CoQA and QuAC independently, with the chosen answer having the highest average probability score, achieves an F1-Score of 70.9% on the official test-set.

## introduction question answering (qa) involves constructing an answer for a given question in either an extractive or an abstractive manner. qa systems are central to other natural language processing (nlp) applications like search engines, and dialogue. recently, qa based solutions have also been proposed to evaluate factuality @xcite and faithfulness @xcite of abstractive summarization systems. in addition to popular qa benchmarks like squad @xcite , and mrqa-2019 @xcite , we have seen qa challenges that require reasoning over human dialogue. some notable examples being quac @xcite and coqa @xcite . these datasets require the model to attend to the entire dialogue context in the process of retrieving an answer. in this work, we are interesting in building a qa system to help with human dialogue. @xcite introduced a new dataset of goal-oriented dialogues (doc2dial) that are grounded in the associated documents. each sample in the dataset consists of an information-seeking conversation between a user and an agent where agent's responses are grounded in faq-like webpages. dialdoc shared task derives its training data from the doc2dial dataset and proposes two subtasks which require the participants to (1) identify the grounding knowledge in form of document span for the next agent turn; and (2) generate the next agent response in natural language. in this paper, we describe our solution to the subtask 1. this subtask is formulated as a span selection problem. therefore, we leverage a transformerbased extractive question-answering model @xcite to extract the relevant spans from the document. we pretrain our model on different qa datasets like squad, different subsets of mrqa-2019 training set, and conversational qa datasets like coqa and quac. we find that models pretrained on out-of-domain qa datasets substantially outperform the baseline. our experiments suggest that conversational qa datasets are more useful than mrqa-2019 data or its subsets. in the following sections, we first present an overview of the dialdoc shared task ( §2), followed by our system description ( §3) and a detailed account of our experimental results, and ablation studies ( §4, §5). the dataset shared during the shared task was divided into train/validation/testdev/test splits. train and validation splits were provided to the participants to facilitate model development. during phase 1, the models were evaluated on testdev whereas, the final ranking was done on the performance on the test set. pre-processing using the pre-processing scripts provided by the task organizers, we converted the doc2dial dataset into squad v2.0 format with questions containing the latest user utterance as well as all previous turns in the conversation. this is in line with previous work from @xcite which showed that including the entire conversational history performs better than just considering the current user utterance. dialogue context is concatenated with the latest user utterance in the reverse time order. the output of this pre-processing step consisted of 20431 training, 3972 validation, 727 testdev, and 2824 test instances. ## system description as discussed earlier, subtask 1 of dialdoc shared task is formulated as a span selection problem. therefore, in order to learn to predict the correct span, we use an extractive question-answering setup. ## experimental setup in this section, we discuss our experimental setup in detail. ## results we now present the results for different experimental setups we tried for dialdoc subtask 1. ## conclusion our submission to the dialdoc subtask 1 performs continual pretraining of a transformer-based encoder on out-of-domain qa datasets. experiments with different qa datasets suggest that conversational qa datasets like coqa and quac are highly beneficial as their setup is substantially similar to doc2dial, the downstream dataset of interest. our final submission ensembles two albert-xl models independently pretrained on coqa and quac and achieves an f1-score of 70.9% and em-score of 53.5% on the competition test-set.