x M o C o: Cross Momentum Contrastive Learning for Open-Domain Question Answering (2021)

Abstract: Dense passage retrieval has been shown to be an effective approach for information retrieval tasks such as open domain question answering. Under this paradigm, a dual-encoder model is learned to encode questions and passages separately into vector representations, and all the passage vectors are then pre-computed and indexed, which can be efficiently retrieved by vector space search during inference time. In this paper, we propose a new contrastive learning method called Cross Momentum Contrastive learning (xMoCo), for learning a dual-encoder model for question-passage matching. Our method efficiently maintains a large pool of negative samples like the original MoCo, and by jointly optimizing question-to-passage and passage-to-question matching tasks, enables using separate encoders for questions and passages. We evaluate our method on various open-domain question answering dataset, and the experimental results show the effectiveness of the proposed method.

## introduction retrieving relevant passages given certain query from a large collection of documents is a crucial component in many information retrieval systems such as web search and open domain question answering (qa). current qa systems often employ a two-stage pipeline: a retriever is firstly used to find relevant passages, and then a fine-grained reader tries to locate the answer in the retrieved passages. as recent advancement in machine reading comprehension (mrc) has demonstrated excellent results of finding answers given the correct passages @xcite , the performance of open-domain qa systems now relies heavily on the relevance of the selected passages of the retriever. traditionally the retrievers usually utilize sparse keywords matching such as tf-idf or bm25 @xcite , which can be efficiently implemented with an inverted index. with the popularization of neural network in nlp, the dense passage retrieval approach has gained traction @xcite . in this approach, a dual-encoder model is learned to encode questions and passages into a dense, low-dimensional vector space, where the relevance between questions and passages can be calculated by the inner product of their respective vectors. as the vectors of all passages can be pre-computed and indexed, dense passage retrieval can also be done efficiently with vector space search methods during inference time @xcite . dense retrieval models are usually trained with contrastive objectives between positive and negative question-passage pairs. as the positive pairs are often given by the training data, one challenge in contrastive learning is how to select negative examples to avoid mismatch between training and inference. during inference time, the model needs to find the correct passages from a very large set of pre-computed candidate vectors, but during training, both positive and negative examples need to be encoded from scratch, thus severely limiting the number of negative examples due to computational cost. one promising way to reduce the discrepancy is momentum constrastive learning (moco) proposed by @xcite . in this method, a pair of fast/slow encoders are used to encode questions and passages, respectively. the slow encoder is updated as a slow moving average of the fast encoder, which reduces the inconsistency of encoded passage vectors between subsequent training steps, enabling the encoded passages to be stored in a large queue and reused in later steps as negative examples. unfortunately, directly applying moco in question-passage matching is problematic. unlike the image matching tasks in original moco paper, the questions and passages are distinct from each other and not interchangeable. furthermore, the passages are only encoded by the slow encoder, but the slow encoder is only updated with momentum from the fast encoder, not directly affected by the gradients. as the fast encoder only sees the questions, the training becomes insensitive to the passage representations and fails to learn properly. to solve this problem, we propose a new contrastive learning method called cross momentum contrastive learning (xmoco). xmoco employs two sets of fast/slow encoders and jointly optimizes the question-passage and passage-question matching tasks. it can be applied to scenarios where the questions and passages require different encoders, while retaining the advantage of efficiently maintaining a large number of negative examples. we test our method on several open-domain qa tasks, and the experimental results show the effectiveness of the proposed approach. to summarize, the main contributions of this work are as follows: ## related work there are mainly two threads of research work related to this paper. ## discussion how to select/create negative examples is an essential aspect of passage retrieval model training. xmoco improves passage retrieval model by efficiently maintaining a large set of negative examples, while previous works mainly focus on finding a few hard examples. it is desirable to design a method to take the best from both worlds. as described in section 4.5, we can combine the two approaches under a simple multitask framework. but this multitask framework also has its drawbacks. firstly, it loses the computational efficiency of xmoco, especially if the method of generating the hard examples is expensive. secondly, the large set of negative examples in xmoco and the set of hard examples are two separate sets, while ideally, we want to maintain a large set of hard negative examples. to this end, one possible direction is to employ curriculum learning @xcite . assuming the corresponding passages for similar questions can serve as hard examples for each other, we can schedule the order of training examples so that similar questions are trained in adjacent steps, resulting more hard examples to be kept in the queue. we plan to explore this possibility in future work. ## conclusion in this paper, we propose cross momentum contrastive learning (xmoco), for passage retrieval task in open domain qa. xmoco jointly optimizes question-to-passage and passage-to-question matching, enabling using separate encoders for questions and passages, while efficiently maintains a large pool of negative samples like the original moco. we verify the effectiveness of the proposed method on various open domain qa datasets. for future work, we plan to investigate how to better integrate hard negative examples into xmoco.