Recovering Gold from Black Sand: Multilingual Dense Passage Retrieval with Hard and False Negative Samples (2022)

Abstract: Negative samples have not been efficiently explored in multilingual dense passage retrieval. In this paper, we propose a novel multilingual dense passage retrieval framework, mHFN, to recover and utilize hard and false negative samples. mHFN consists of three key components: 1) a multilingual hard negative sample augmentation module that allows knowledge of indistinguishable passages to be shared across multiple languages and synthesizes new hard negative samples by interpolating representations of queries and existing hard negative samples, 2) a multilingual negative sample cache queue that stores negative samples from previous batches in each language to increase the number of multilingual negative samples used in training beyond the batch size limit, and 3) a lightweight adaptive false negative sample filter that uses generated pseudo labels to separate unlabeled false negative samples and converts them into positive passages in training. We evaluate mHFN on Mr. TyDi, a high-quality multilingual dense passage retrieval dataset covering eleven typologically diverse languages, and experimental results show that mHFN outperforms strong sparse, dense and hybrid baselines and achieves new state-of-the-art performance on all languages. Our source code is available at https://github.com/Magnetic2014/mHFN.

## introduction passage retrieval, which matches relevant passages to queries, is an important task in information retrieval (ir). it can be also integrated as a core component to solve many natural language processing (nlp) problems, e.g., open domain question answering @xcite , fact checking @xcite , etc. powered by large scale pretrained language models (e.g. bert @xcite , roberta @xcite , and t5 @xcite )), dense passage retrieval, which explores dense vector representations to match relevant passages, has attracted growing interest @xcite @xcite @xcite @xcite . dense passage retrieval models usually adopt a bi-encoder (also known as dual-encoder) architecture, one encoder for encoding queries, the other for learning passage representations, which can be computed offline. with dense representations of a query and document, passage ranking is recast as a nearest neighbor search problem that can be efficiently solved by similarity search toolkits tailored for dense vectors, such as faiss @xcite . dense passage retrieval models are usually trained via contrastive learning (cl), which encourages query representations to be close to positive (i.e., relevant) passages and away from negative (i.e., irrelevant) passages in the learned semantic space. many previous studies demonstrate the effectiveness of cl in dense passage retrieval @xcite @xcite . under this cl-based dense passage retrieval setting, using more negative samples has proved beneficial to models @xcite @xcite @xcite . specifically, hard negative samples (i.e., negative samples which are similar to positive samples) are more desirable than ordinary negative samples for tuning the dense representations of queries and passages @xcite . as multilingual pretrained language models (e.g., mbert @xcite ) have exhibited cross-lingual generalization and knowledge transfer from high-resource to low-resource languages, it is natural to extend the monolingual dpr model @xcite to multilingual dpr (e.g., mdpr @xcite ) by replacing the monolingual backbone model with a multilingual pretrained language model. such a multilingual setting is promising for languages without sufficient training data. unfortunately, negative samples are not efficiently explored in existing multilingual dense passage retrieval models. first, current multilingual dense passage retrieval models develop hard negative samples for each language in a separate and independent way, which is unable to share the common features of indistinguishable passages across languages. second, in order to increase the number of negative samples, the in-batch and cross-batch negative technique are widely used in dense passage retrieval @xcite , where negative samples are sampled from the same batch as positive samples and shared across all gpus. however, this will quickly exhaust gpu memory with growing negative samples, hence making it difficult to increase the number of negative samples further. third, as the number of negative samples increases, false negative samples (i.e., unlabeled relevant passages) will also grow in number, which would make it harder for training to converge and mislead the direction of optimization since false negative samples are actually positive. as shown in the figure 1 , it is desirable to make representations of unlabeled false negative samples close to positive samples and keep hard negative samples as distant as possible. in this paper, our key interest lies in a multilingual dense passage retrieval task, where a unified retrieval model is used to retrieve in-language (i.e., in the same language as the given query) relevant passages for multiple languages. to efficiently utilize negative samples (especially hard and false negative samples) and solve the aforementioned issues, we propose mhfn, a new multilingual dense passage retrieval framework with hard and false negative samples. to model hard negative samples, we propose a multilingual hard negative sample augmentation module that shares hard negative samples across languages and generates synthetic negative sample representations. in this way, we force the model to learn efficient features to distinguish between positive and hard negative samples that are similar to each other. to handle false negative samples, we introduce a multilingual negative sample cache queue that stores negative samples from previous batches in each language as candi- query positive passages hard negative passages false negative passages other random negative passages (a) mdpr query positive passages hard negative passages false negative passages other random negative passages (b) mhfn @xcite , mhfn learns better passage representations, especially for hard and false negative samples. our contributions can be summarized as follows: multilingual and cross-lingual dense passage retrieval researchers have been utilizing crosslingual knowledge transfer to enhance monolingual retrieval for low-resource languages since the advent of multilingual pretrained language models. both @xcite and @xcite investigate zero-shot transfer using a cross-encoder architecture, in which they first fine-tune mbert on the source language, then apply the model to the target language directly. however, the crossencoder architecture they use is slow in practice. in contrast, bi-encoders equipped with nearest neighbor vector search tools such as faiss @xcite run much faster than cross-encoders since the dense representations can be computed and indexed in advance. @xcite also utilize mbert to perform retrieval in the many-to-many scenario. by retrieving in a multilingual pool, they train a model to answer a query in any specific languages. however, this diverges from our setting, where we solely focus on using a unified model to conduct in-language retrieval for multiple languages. for datasets, mr. tydi foot_0 @xcite collects queries and passages in eleven typologically diverse languages from wikipedia. to the best of our knowledge, it is the only publicly available dataset that can be used for our multilingual passage retrieval experiments. other datasets, such as xor-tydi @xcite and clirmatrix (sun and duh, 2020), focus on cross-lingual retrieval instead. based on the mr. tydi dataset, @xcite empirically investigate the best practice of training multilingual dense passage retrieval models. however, they have not studied the impact of the number of negative samples and the side effect of false negative samples. ## methodology in this section, we elaborate the proposed mhfn, as illustrated in figure 2 , which aims to utilize hard and false negative samples in multilingual dense passage retrieval. it is comprised of three essential components: a multilingual hard negative sample augmentation module that shares hard negative knowledge across languages, a multilingual negative sample cache queue that stores extra negative samples in each language beyond the limit of in-batch negative samples, and an adaptive false negative sample filter that filters out unlabeled false negative samples from candidate negative samples during training. ## conclusion in this paper, we have presented a novel multilingual dense passage retrieval model, mhfn, which efficiently explores hard and false negative samples. it can 1) efficiently share hard negative samples across languages and generate augmented highquality hard negative samples, 2) increase the number of multilingual negative samples, and 3) adaptively filter out unlabeled false negative samples from all candidate negative samples for effective training. experiments and in-depth analysis validate the effectiveness of mhfn and demonstrate that it outperforms the strong sparse, dense, and hybrid baselines, setting new state-of-the-art results on the mr. tydi dataset.