Learning Cross-Lingual IR from an E nglish Retriever (2022)

Abstract: We present DR.DECR (Dense Retrieval with Distillation-Enhanced Cross-Lingual Representation), a new cross-lingual information retrieval (CLIR) system trained using multi-stage knowledge distillation (KD). The teacher of DR.DECR relies on a highly effective but computationally expensive two-stage inference process consisting of query translation and monolingual IR, while the student, DR.DECR, executes a single CLIR step. We teach DR.DECR powerful multilingual representations as well as CLIR by optimizing two corresponding KD objectives. Learning useful representations of non-English text from an English-only retriever is accomplished through a cross-lingual token alignment algorithm that relies on the representation capabilities of the underlying multilingual encoders. In both in-domain and zero-shot out-of-domain evaluation, DR.DECR demonstrates far superior accuracy over direct fine-tuning with labeled CLIR data. It is also the best single-model retriever on the XOR-TyDi benchmark at the time of this writing.

## introduction multilingual models are critical for the democratization of ai. cross-lingual information retrieval (clir) @xcite @xcite , for example, can find relevant text in a high-resource language such as english even when the query is posed in a different, possibly low-resource, language. in this work, we develop useful clir models for this constrained, yet important, setting where a retrieval corpus is available only in a single high-resource language (english in our experiments). a straightforward solution to this problem could use machine translation (mt) to translate the query into english, and then perform english ir @xcite . while such a two-stage process can produce reasonably accurate predictions, an alternative end-to-end approach that can tackle the problem purely cross-lingually, i.e., without involving mt for inference, would clearly be more efficient and cost-effective. pre-trained multilingual masked language models (plms) such as multilingual bert @xcite or xlm-roberta (xlm-r) @xcite can provide the foundation for such a one-step solution, as simply fine-tuning a plm with labeled clir data would yield a crosslingual retriever @xcite . here we first run an empirical evaluation of these two approaches on a public clir benchmark @xcite , which involves both in-domain and zero-shot out-of-domain tests. we use colbert @xcite )a state-of-the-art (sota) neural ir model that has been shown to outperform other recent methods such as dpr @xcite )-as our ir architecture and xlm-r as the underlying plm for both methods @xcite . results indicate that the mtbased solution can be vastly more effective than direct en + clir fine-tuning, with observed differences of 22.2-28.6 recall@5k-tokens ( §3). crucially, the modular design of the former allows it to leverage additional english-only training data for its ir component, providing significant boosts to its performance. the above findings lead naturally to the central research question of this paper: can a highperformance clir model be trained that can operate without having to rely on mt? to answer the question, instead of viewing the mt-based approach as a competing one, we propose to leverage its strength via knowledge distillation (kd) into an end-to-end clir model, which we call dr.decr (dense retrieval with distillation-enhanced cross-lingual representation). kd @xcite is a powerful supervision technique typically used to distill the knowledge of a large teacher model about some task into a smaller student model @xcite . here we propose to use it in a slightly different context, where the teacher and the student retriever are identical in size, but the former has superior performance simply due to utilizing mt output and consequently operating in a high-resource and low-difficulty monolingual environment. we run two independent kd operations ( §2.2). one directly optimizes an ir objective by utilizing labeled clir data: parallel questions (english and non-english) and corresponding relevant and non-relevant english passages. the teacher and the student, dr.decr, are shown the english and non-english versions of the questions, respectively; the training objective is for dr.decr to match the soft query-passage relevance predictions of the teacher. the second kd task is representation learning from parallel text, where dr.decr learns to encode a non-english text in a way that matches the teacher's encoding of the aligned english text, at the token level. the cross-lingual token alignments needed to create the training data for this task are generated using a greedy alignment process, which exploits the multilingual representation capabilities of the underlying plm encoders. in our evaluation on the xor-tydi benchmark @xcite , dr.decr outperforms the fine-tuned colbert baseline by 25.4 (in-domain) and 14.9 (zero-shot) recall@5k-tokens, recovering much of the performance loss from the mt-based solution. it is also the best single-model ir system on the xor-tydi leaderboard 1 at the time of this writing. ablation studies show that each of our two kd processes contribute significantly towards the final performance of dr.decr. our contributions can be summarized as follows: ## conclusion we train highly effective end-to-end cross-lingual ir models by distilling the knowledge of an english retriever. we propose separate processes to teach ir and multilingual text representations, and present for the latter a cross-lingual alignment algorithm that only relies on the underlying masked language model's multilingual representation capabilities. supervised and zero-shot evaluations show that our model recovers much of the performance lost due to operating in an efficient crosslingual mode. our kd-based method also yields new single-model sota results on the xor-tydi benchmark. future work will explore ir on unseen languages and evaluation on additional datasets.