Empowering Dual-Encoder with Query Generator for Cross-Lingual Dense Retrieval (2022)

Abstract: In monolingual dense retrieval, lots of works focus on how to distill knowledge from cross-encoder re-ranker to dual-encoder retriever and these methods achieve better performance due to the effectiveness of cross-encoder re-ranker. However, we find that the performance of the cross-encoder re-ranker is heavily influenced by the number of training samples and the quality of negative samples, which is hard to obtain in the cross-lingual setting. In this paper, we propose to use a query generator as the teacher in the cross-lingual setting, which is less dependent on enough training samples and high-quality negative samples. In addition to traditional knowledge distillation, we further propose a novel enhancement method, which uses the query generator to help the dual-encoder align queries from different languages, but does not need any additional parallel sentences. The experimental results show that our method outperforms the state-of-the-art methods on two benchmark datasets.

## introduction information retrieval (ir) aims to retrieve pieces of evidence for a given query. traditional methods mainly use sparse retrieval systems such as bm25 @xcite , which depend on keyword matching between queries and passages. with the development of large-scale pretrained language models (plms) @xcite such as bert, dense retrieval methods @xcite show quite effective performance. these methods usually employed a dual-encoder architecture to encode both queries and passages into dense embeddings and then perform approximate nearest neighbor searching @xcite . recently, leveraging a cross-encoder re-ranker as the teacher model to distill knowledge to a dual- figure 1 : the performance of cross-encoder and query generator when varying the number of training samples and retrievers. we use bm25 and dpr as retrievers, respectively. for the cross-encoder (bert-large), we use retrieved top-100 passages which do not contain the answer as negative and contrastive loss for training. for the query generator (t5-base), we firstly train it with the query generation task and then fine-tune the model with the same setting as bert-large. the reported performance is the top-5 score of re-ranked top 500 passages on the nq test set. however, there are two major problems when scaling the method to the cross-lingual dense retrieval setting. firstly, the cross-encoder typically requires large amounts of training data and high-quality negative samples due to the gap between pre-training (token-level task) and finetuning (sentence-level task), which are usually not satisfied in the cross-lingual setting @xcite . due to expensive labeling and lack of annotators in global languages, especially low-resource languages, the training data in cross-lingual are quite limited. then with the limited training data, the dual-encoder is not good enough to provide high-quality negative samples to facilitate the crossencoder. secondly, the cross-lingual gaps between different languages have a detrimental effect on the performance of cross-lingual models. although some cross-lingual pre-training methods such as in-foxlm @xcite and labse @xcite have put lots of effort into this aspect by leveraging parallel corpus for better alignment between different languages, these parallel data are usually expensive to obtain and the language alignment could be damaged in the fine-tuning stage if without any constraint. to solve these problems, we propose to employ a query generator in the cross-lingual setting, which uses the likelihood of a query against a passage to measure the relevance. on the one hand, the query generator can utilize pre-training knowledge with small training data in fine-tuning stage, because both of its pre-training and fine-tuning have a consistent generative objective. on the other hand, the query generation task is defined over all tokens from the query rather than just the [cls] token in the cross-encoder, which has been demonstrated to be a more efficient training paradigm @xcite . as shown in figure 1 , with the number of training samples dropping, the performance of bert-large drops more sharply than t5-base. besides, the query generator is less sensitive to highquality negative samples. as we can see, using bm25 as the retriever to mine negative samples for re-ranker training, the gap between cross-encoder and query generator is smaller than the gap using dpr as the retriever. finally, the query generator can provide more training data by generation, which is precious in the cross-lingual setting. to sum up, the query generator is more effective than the cross-encoder in the cross-lingual setting. based on these findings, we propose a novel method, namely quick, which stands query generator improved dual-encoder by cross-lingual knowledge distillation. firstly, at the passage level, we employ a query generator as the teacher to distill the relevant score between a query and a passage into the dual-encoder. secondly, at the language level, we use the query generator to generate synonymous queries in other languages for each training sample and align their retrieved results by kl-divergence. considering the noise in the generated queries, we further propose a scheduled sampling method to achieve better performance. the contributions of this paper are as follows: ## preliminaries in this section, we give a brief review of dense retrieval and re-ranking. the overviews of all methods are presented in figure 2 . ## methodology in this section, we present the proposed quick. the overview of the proposed method is presented in figure 3 . we start with the training of the query generator, then present how to perform distillation and alignment training for the dual-encoder, and we finally discuss the entire training process. retrieved passages passage encoder 𝑃𝑃(𝑞𝑞|𝑝𝑝) query decoder query generator cross-lingual dual-encoder cross-lingual dual-encoder cross-lingual query passage answer cross-lingual alignment query-likelihood distillation query scheduled sampling query-likelihood distillation ## experiments in this section, we construct experiments to demonstrate the effectiveness of our method. ## conclusion in this paper, we showed that the cross-encoder performs poorly when there are not sufficient training samples which are hard to obtain in the crosslingual setting. then we proposed a novel method that utilizes the query generator to improve the dual-encoder. we firstly proposed to use a query generator as the teacher. after that, we proposed a novel alignment method for cross-lingual retrieval which does not need any parallel corpus. extensive experimental results show that the proposed method outperforms the baselines and significantly improves the state-of-the-art performance. currently, our method depends on training data in all target languages. as future work, we will investigate how to perform the proposed method for zero-shot cross-lingual dense retrieval.