Momentum Contrastive Pre-training for Question Answering (2022)

Abstract: Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching. In order to address this problem, we propose a novel Momentum Contrastive pRe-training fOr queStion anSwering (MCROSS) method for extractive QA. Specifically, MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs. Hence, the pre-trained models can better transfer the knowledge learned in cloze-like samples to answering natural questions. Experimental results on three benchmarking QA datasets show that our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios.

## introduction the task of extractive question answering (qa), which aims to select an answer span from a passage given a query, is a major focus of nlp research. currently, deep learning systems @xcite @xcite have achieved competitive results with humans on large-scale qa datasets @xcite @xcite . however, the collection of high-quality natural qa pairs is still a labor-intensive task, especially for the construction of domain-specific qa systems. to alleviate such data availability restrictions, pretraining methods have been drawing increasing attention @xcite . typically, pre-training methods for extractive qa generate cloze-like query-passage pairs with text-matching techniques. for instance, the span selection pre-training (sspt) method @xcite generates these pairs with the wikipedia corpus using bm25. nevertheless, these methods have two major issues. firstly, the format of in jordan's second season of the playoffs against the [blank], he even scored 63 points ## method the structure of our proposed mcross method is shown in fig. 2 . along with the cloze-like samples s @xmath0, p, a) generated by sspt @xcite , we create positive natural samples s natural+ = (q n , p, a) containing natural queries q n given passages and answers (p, a) from s cloze with t5 answer-aware question generator @xcite which is fine-tuned with squadv1.1 training set. the details of pre-training datasets are illustrated in appendix a.1. though different in format, the cloze-like queries q c and positive natural queries q n are semantically similar since they have the same answers a in given supporting passages p. therefore, we expect that the predicted probability distribution of answer span in positive pairs (s cloze , s natural+ ) should be closer. on the other hand, we generate negative query pairs (s cloze , s natural-) with different answers or supporting passages. correspondingly, the predicted answer span distribution of s natural-is expected to be far from that of s cloze . mcross achieves the goal with two pre-training tasks as follows. ## conclusion this paper presents a novel pre-training method mcross for extractive qa which contains two tasks: 1) contrastive learning and 2) answer term prediction. specifically, mcross adapts moco frameworks to maintain consistency in answering cloze-like and natural questions, enabling pretrained models to have a more comprehensive understanding of supporting passages. the empirical experiments on three public datasets demonstrate that our approach can obtain noticeable improvements in extractive qa tasks in supervised and zero-shot scenarios.