Lexicon-Enhanced Self-Supervised Training for Multilingual Dense Retrieval (2022)

Abstract: Recent multilingual pre-trained models have shown better performance in various multilingual tasks. However, these models perform poorly on multilingual retrieval tasks due to lacking multilingual training data. In this paper, we propose to mine and generate self-supervised training data based on a large-scale unlabeled corpus. We carefully design a mining method which combines the sparse and dense models to mine the relevance of unlabeled queries and passages. And we introduce a query generator to generate more queries in target languages for unlabeled passages. Through extensive experiments on Mr. TYDI dataset and an industrial dataset from a commercial search engine, we demonstrate that our method performs better than baselines based on various pre-trained multilingual models. Our method even achieves on-par performance with the supervised method on the latter dataset.

## introduction information retrieval (ir) aims to retrieve relevant passages for a given query, which plays a critical role in many industry scenarios such as open-domain question answering (qa) @xcite and web search @xcite . traditionally, bag-of-words (bow) retrieval systems such as tf-idf and bm25 @xcite were widely used, which mainly depend on keyword matching between queries and passages. with the development of large-scale pretrained language models (plms) @xcite such as bert, dense retrieval methods @xcite show quite effective performance. these methods usually employed a dual-encoder architecture to encode both queries and passages into dense embeddings and then perform approximate nearest neighbor searching @xcite . recently, some works found that dense retrievers perform poorly in the zero-shot multilingual settings @xcite due to the distributional shift. to boost the performance of dense retrievers, some previous methods for cross-domain retrieval can be directly adopted to unsupervised multilingual dense retrieval. there are two important kinds: 1) generating training data in target languages. for example, kulshreshtha et al. applied self-training to generate labeled data and further proposed back-training @xcite to obtain more high-quality data. qgen @xcite proposed to use a query generator to generate in-domain queries. 2) leveraging sparse retrievers, which is more effective in the unsupervised setting, to enhance dense retrievers. for example, spar @xcite proposed to distill knowledge from bm25 to the dense model and laprador @xcite proposed to enhance the dense model by multiplying the similarity with the bm25 score. however, there are three major problems when directly adopting these methods to multilingual dense retrieval. first, zero-shot multilingual query generators suffer from grammatical adjustment and accidental translation problems @xcite . as a result, zero-shot query generators only provide little help in bridging the gap among different languages. second, hybrid dense and sparse models such as laprador and spar get high latency in the inference stage foot_0 . finally, dense retrieval is different from other tasks, it not only needs positive query-passage pairs but also needs negative query-passage pairs @xcite . however, previous methods such as the back-training focus on positive pairs and simply take the top passages of bm25 as negative passages. although training data in target languages is very expensive, unlabeled queries and passages can be easily obtained from search engines such as google and bing. in this paper, we propose a novel method that augments data in target languages by combining sparse and dense models, namely lestm, which stands lexicon-enhanced self-supervised training for multilingual dense retrieval. first, as we mentioned above, sparse retrievers mainly depend on keyword matching between queries and passages and dense retrievers mainly depend on the language modeling ability of pre-trained models, which indicates the sparse and dense models perform retrieval in different aspects @xcite . in addition, the sparse-dense hybrid retriever is significantly better than both sparse and dense models @xcite . both can demonstrate that sparse and dense models notice different characteristics and are complementary. therefore, we craft a lexiconenhanced retrieval module to mine positive and negative passages for each unlabeled query in target languages, which leverages the retrieval results of both sparse and dense models. we treat passages that both sparse and dense models regard are relevant as positive passages, and passages that one model regards are relevant but the other regards are irrelevant as negative passages. furthermore, we employ a query generator to generate queries for passages in target languages due to the limited number of unlabeled queries. the query generation methods have been shown to significantly improve the performance of retrieval models in the monolingual setting @xcite . considering the grammatical adjustment and accidental translation problems, we first use the mined positive querypassage pairs to train a query generator. then, we use the trained model to generate more queries in target languages. considering that there may exist more relevant passages to the generated queries, we use both sparse and dense retrievers to filter the generated samples. finally, using only unlabeled data from target languages, lestm iteratively mines query passage pairs by the lexicon-enhanced retriever and generator, trains a new better retriever and query generator using these mined pairs, mines again for better query passage pairs, and repeats. in summary, our contributions are as follows. cross-lingual (domain) retrieval. these tasks aims to investigate the retrieval capabilities under cross-lingual @xcite or cross-domain @xcite setting. the methods for these tasks can be divided into two main categories: model transfer methods and data transfer methods. the model transfer methods for cross-domain focus on pre-training sentence representation. for example, gtr @xcite and cpt @xcite propose that scaling up the model size can significantly improve the performance of dense models. contriever @xcite and laprador @xcite propose to use contrastive learning to learn sentence aware representation. for cross-lingual, they focus on learning multilingual representations by pre-training (conneau and lample, 2019; @xcite such as mbert @xcite and xlmr @xcite . the data transfer methods mainly focus on obtaining noisy training data in the target domain or target languages. for example, back-training @xcite and qgen @xcite propose to use a query generator to generate indomain queries. cora @xcite leverages a generator to help mine retrieval training data and dr.decr @xcite mines lots of parallel data to perform cross-lingual distillation. ## preliminaries in this section, we give a brief review of dense retrieval and then present how to apply models to multilingual dense retrieval. overview. given a query q and a corpus c, the retrieval task aims to find the relevant passages for the query from a large corpus. usually, a dense retrieval model employs two dense encoders (i.e., bert) e q (•) and e p (•). they encode queries and passages into dense embeddings, respectively. then, the model uses a similarity function, often dot-product, to perform retrieval: training. the training of retrieval is metric learning essentially. the goal is to narrow the distance between the query and the relevant passages (a.k.a., positive passages) and widen the distance between the query and the irrelevant passages (a.k.a., negative passages). let {q i , p + i , p - i,0 , p - i,1 , . . . , p - i,n } be the i-th training sample. it consists of one query, one positive passage, and n negative passages. then we can employ a contrastive loss function, called @xcite , to optimize the model: in practice, we cannot use all passages in the corpus c as negative passages due to the limitation of resources. therefore, a common practice is sampling a subset from the corpus c as negative samples, and many studies focus on which distribution the negative passages sampled from is better @xcite . multilingual setting. this setting aims to transfer knowledge from the source language to the target languages. in this setting, only labeled data from the source language is available. and the trained model will be directly evaluated on target languages. note that the setting is different to crosslingual retrieval whose queries and passages are in different languages. in this setting, the queries and the passages are in the same language and just training data from the source language (e.g., english) is available. ## methodology in this section, we present the proposed lestm. the overview is presented in figure 1 . we first present the augmentation method which combines sparse and dense retrievers. then, we present how to use the mined data to train the query generator, generate new data, filter the generated samples, and fine-tune the dense retriever. finally, we summarize the full training process. ## experiments in this section, we construct experiments to demonstrate the effectiveness of the proposed method. ## conclusion in this paper, we propose a novel augmentation method that combines sparse and dense retrievers for multilingual retrieval. we firstly designed a passage mining method based on the results of both sparse and dense retrievers. after that, we utilized the mined data to train a query generation model and generate more training data. extensive experimental results show that the proposed method outperforms the baselines, and can significantly improve the state-of-the-art performance. currently, we directly utilize a large number of unlabeled queries in target languages. as future work, we will investigate how to augment training data without any unlabeled queries in target languages.