Cross-Lingual Training of Dense Retrievers for Document Retrieval (2021)

Abstract: Dense retrieval has shown great success for passage ranking in English. However, its effectiveness for non-English languages remains unexplored due to limitation in training resources. In this work, we explore different transfer techniques for document ranking from English annotations to non-English languages. Our experiments reveal that zero-shot model-based transfer using mBERT improves search quality. We find that weakly-supervised target language transfer is competitive compared to generation-based target language transfer, which requires translation models.

## introduction dense retrieval uses dense vector representations for semantic encoding and matching. however, most existing work focuses on high-resource languages such as english, where large-scale annotations are readily accessible. in this work, we focus on improving term-matching retrieval for low(er)-resource languages. we explore techniques for leveraging relevance judgments in english to train dense retrievers for document retrieval in non-english languages. our experimental results show that combining dense retrieval and term-matching retrieval can obtain effectiveness improvements. also, weakly-supervised target language transfer yields effectiveness competitive to generationbased target language transfer. this extended abstracted is an abridged version of @xcite . ## cross-lingual relevance transfer we leverage the dpr model of @xcite , but using mbert as the backbone model. during inference, we apply both bag-of-words exact term matching and dense retrieval. the relevance score of each document combines termmatching scores with dense retrieval similarity via s doc " α ¨sterm `p1 ´αq ¨sdense , where α is tuned via cross-validation. model-based transfer. by exploiting the zeroshot cross-lingual transfer ability of pretrained transformers such as mbert @xcite , we train the dense retriever in the source language and apply inference directly to the target languages. target language transfer. to bridge the language gap between training and inference, we explore two techniques for creating a target language transfer set. (1) generation-based query synthesis, where the goal is to leverage powerful generation models to predict reasonable queries given documents in the target language. we choose mbart @xcite as our query generation model. the input of the model is the passage and its learning target is the corresponding query. we use the translate-train technique to obtain the generation models. more specifically, we leverage google translate to translate english querydocument pairs into the target languages. then, we use passages in the target language collections as input and generate corresponding queries in the same language. (2) weakly-supervised query synthesis. we can automatically build the target language transfer set without manual annotation effort by treating the titles of wikipedia articles as queries and the corresponding documents as positive candidates. we also retrieve top 1000 documents with bm25 for each query; documents except for the positive candidate are labeled as negative. ## results and discussions finding #1: zero-shot model-based transfer improves term-based matching. the results of zero-shot model-based transfer are shown in model (2) and model (3). comparing with the corresponding baselines, we observe that model-based transfer, either nq zero-shot or ms zero-shot, can improve retrieval effectiveness on p@20 for all collections, except nq zero-shot on the trec3-es dataset. we do not observe a clear winner between nq and ms, though. these results indicate that mbert-based dpr effectively transfers relevance matching across languages. finding #2: target language transfer benefits certain collections, and wiki query synthesis is better than query generation. target language transfer results are shown in model (4) and model (5). ms ñ qgen and ms ñ wiki denote the two-stage training strategy with differ-ent transfer sets, where qgen denotes generationbased query synthesis and wiki denotes weaklysupervised query synthesis from wikipedia. by comparing model (4) with model (3), we observe that second-stage training with generation-based query-document pairs can improve the effectiveness of p@20 over zero-shot model-based transfer on the chinese, french, hindi, bengali, and spanish collections. however, we see little improvement in terms of ap for all collections. by comparing model (5) with model ( 3 ), we find that secondstage training with weakly-supervised training data can improve p@20 over the zero-shot baselines on french, hindi, bengali, and spanish. furthermore, by comparing these two transfer sets, we observe that, except for chinese, wiki obtains better retrieval effectiveness than qgen, which requires translation models for the target languages (which are expensive to build and not available for all languages). ## conclusion we investigate the effectiveness of three transfer techniques for document ranking from english training data to non-english target languages. our experiments in six languages demonstrate that zeroshot transfer using mbert-based dense retrieval models improves term-based matching methods, and fine-tuning on augmented data in target languages can further benefit certain collections.