Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval (2022)

Abstract: State-of-the-art neural (re)rankers are notoriously data-hungry which – given the lack of large-scale training data in languages other than English – makes them rarely used in multilingual and cross-lingual retrieval settings. Current approaches therefore commonly transfer rankers trained on English data to other languages and cross-lingual setups by means of multilingual encoders: they fine-tune all parameters of pretrained massively multilingual Transformers (MMTs, e.g., multilingual BERT) on English relevance judgments, and then deploy them in the target language(s). In this work, we show that two parameter-efficient approaches to cross-lingual transfer, namely Sparse Fine-Tuning Masks (SFTMs) and Adapters, allow for a more lightweight and more effective zero-shot transfer to multilingual and cross-lingual retrieval tasks. We first train language adapters (or SFTMs) via Masked Language Modelling and then train retrieval (i.e., reranking) adapters (SFTMs) on top, while keeping all other parameters fixed. At inference, this modular design allows us to compose the ranker by applying the (re)ranking adapter (or SFTM) trained with source language data together with the language adapter (or SFTM) of a target language. We carry out a large scale evaluation on the CLEF-2003 and HC4 benchmarks and additionally, as another contribution, extend the former with queries in three new languages: Kyrgyz, Uyghur and Turkish. The proposed parameter-efficient methods outperform standard zero-shot transfer with full MMT fine-tuning, while being more modular and reducing training times. The gains are particularly pronounced for low-resource languages, where our approaches also substantially outperform the competitive machine translation-based rankers.

## introduction in recent years, neural rankers @xcite @xcite , trained on large-scale datasets @xcite @xcite , have substantially pushed the performance on various retrieval benchmarks. since such models are generally too computationally involved (i.e., too slow) for ad-hoc retrieval on large document collections, they are commonly leveraged as rerankers, i.e., they rerank the output of some fast model (e.g., bm25) that produces the initial ranking. large-scale datasets for training neural rerankers, however, exist only in english, which impedes their adoption in retrieval scenarios that involve other languages: (a) monolingual retrieval in other languages and (b) cross-lingual information retrieval (clir) in which, for a given query in one language, one needs to determine relevance of documents written in one or more other languages. while clir is often instantiated in the form of standalone tasks (e.g., to allow users from different countries to search over the aggregated global collection of covid-19 news and findings in their native language @xcite ), it also supports a range of ir-backed nlp tasks such as cross-lingual question answering @xcite , entity linking @xcite , and crosslingual summarization @xcite . a truly multilingual search engine requires reliable estimation of both monolingual (for a wide range of languages) as well as cross-lingual query-document relevance, which both crucially rely on the alignment of text representations across different languages @xcite . the lack of large-scale retrieval datasets in languages other than english means that monolingual reranking for those languages has to be achieved by means of cross-lingual transfer of a reranking model trained on english relevance judgments. pretrained massively multilingual transformers (mmts) like multilingual bert (mbert) @xcite or xlm-r @xcite have been leveraged to this effect, but were shown to require substantial task-specific (i.e., ranking-oriented) fine-tuning for reliable prediction of semantic similarity and relevance scores @xcite . mmts offer zero-shot cross-lingual transfer of neural (re)ranking models out of the box -an mmt is finetuned on english relevance judgments and then employed in (monolingual or cross-lingual) retrieval tasks that involve other languages. conceptually, via such transfer, no fine-tuning data (i.e., relevance judgments) is required for the target language(s). this procedure, in principle, enables downstream zero-shot transfer to any language seen by the mmt in pretraining (e.g., for mbert, 104 languages). however, in language understanding tasks @xcite , massive performance drops have been observed when transferring between distant languages, and especially in transfer to low-resource languages, underrepresented in mmt pretraining @xcite . our results ( §4) confirm these findings for ad-hoc ir. this is the consequence of the effect known as the curse of multilinguality @xcite : sharing mmt parameters (i.e., its fixed parameter budget/capacity) across more and more languages makes text representations for individual languages less accurate. this effect is especially detrimental to low-resource languages, those least represented in multilingual pretraining corpora. what is more, large-scale full fine-tuning on the source language data (e.g., english) is likely to lead to catastrophic forgetting and interference effects @xcite that further bias the multilingual representation space towards the source language, at the expense of representation quality for low-resource languages. besides the standard zero-shot cross-lingual transfer @xcite , other cross-lingual transfer approaches, commonly applied in other nlp tasks, such as training data translation @xcite , or leveraging external word-level alignments @xcite , as well as distant supervision @xcite have been explored as means to improve the crosslingual transfer of neural rankers in ir. while translation-based approaches are competitive for high-resource languages, they may not be as effective for low-resource languages for which reliable mt models are missing; also, translation-based cross-lingual transfer has been shown to suffer from unwanted artifacts, such as "translationese" @xcite . contributions. even if one would have sufficient amounts of labelled data in target languages, training language-or language-pair specific neural rerankers for all languages and language pairs would be prohibitively computationally expensive and unsustainable @xcite . in this work we additionally remedy for this by composing (re)rankers in a modular way that enables more sustainable cross-lingual transfer. concretely, we introduce neural (re)ranking models for crosslingual and multilingual document retrieval based on mmts that enable much more parameter efficient fine-tuning and more effective cross-lingual transfer for relevance prediction. our (re)rankers are based on two styles of modular components: 1) adapters @xcite @xcite and 2) sparse fine-tuning masks (sftms) @xcite . when integrated into the architecture of a pretrained mmt, both allow for (1) the pretrained multilingual knowledge to be fully preserved, alleviating the negative interference and forgetting effects, and (2) offer additional language-specific model capacity which is used to improve the mmts' representations for target languages, thus remedying for the curse of multilinguality. we provide an extensive evaluation of both approaches in (i) zero-shot transfer for monolingual retrieval and (ii) clir, on two established benchmarks @xcite . as an additional contribution, we expand the clef dataset @xcite with three query languages from the turkic family (turkish, kyrgyz, and uyghur, the latter two being low-resource languages), typologically and etymologically distant from the indo-european languages. 1 our results show that our modular neural (re)rankers are not only faster to train, but also outperform standard zero-shot transfer based on full mmt fine-tuning, and especially so in retrieval tasks that involve linguistically distant and low-resource languages. moreover, our adapter-and sftm-based rerankers generally outperform a strong preranker that utilizes state-of-the-art machine translation. ## methodology we first introduce the general multi-stage ranking (i.e., preranking-reranking) framework, commonly used in information retrieval tasks, within which our work is embedded. we then introduce adapters and sparse fine-tuning masks (sftms), and present how to leverage them as crucial vehicles of the parameter-efficient cross-lingual transfer of the reranking component. ## experimental setup adapter and sftm training. we train adapters following the recommendations from pfeiffer et al. @xcite . unless noted otherwise, we train las with the reduction factor of 2 (i.e., h/@xmath0, for 250k steps with batch size 64 and learning rate of 1e-4. for ras we experimented with the different reduction factors: 1, 2, 4, 8, 16, 32 (cf. §4). following @xcite , for fair comparisons between adapters and sftms, we set the mask size k for sftms to the same number of parameters that adapters with a certain reduction factor have. 5 reranking training. we train mbert-based foot_5 rerankers on ms-marco @xcite , with a linear warm-up over the first 5k updates, in batches of 32 instances with a maximum sequence length of 512, and using a learning rate of 2e-5. we evaluate the model on the validation data every 25k updates and choose the checkpoint with the best validation performance. evaluation data. we evaluate the models on the standard clef-2003 benchmark @xcite foot_6 as well as on the recently introduced hc4 benchmark @xcite . with clef, we use monolingual test collections in en, de, it, ru, and fi for moir, and experiment with the following cross-lingual directions: en-{fi, de, it, ru}, de-{fi, it, ru}, fi-{it, ru}. each experimental run covers 60 queries, whereas the document collection sizes are as follows: ru -17k, fi -55k, it -158k, and de -295k. we additionally evaluate the models in clir tasks with clef queries posed in lower-resource languages. to this end, (i) we leverage swahili (sw) and somali (so) queries @xcite , where the queries were obtained via manual translation of english queries; (ii) we create another set of translated clef queries in three languages: turkish (tr), kyrgyz (kg), and uyghur (ug). the new set covers one high-resource and two lowresource languages and is intended to facilitate and diversify evaluation of clir with low-resource languages in future work. the queries were constructed via the standard post-editing procedure borrowed from other data collection tasks @xcite : we obtained initial query translations via google translate, which were then post-edited by native speakers. hc4 comprises queries and document collections in three languages: persian (fa), russian (ru) and chinese (zh). compared to clef, hc4 collections are considerably larger, spanning 646k, 486k and 4.72m documents per each respective language, associated with 50 test queries in each language. we use title and description fields as queries following @xcite . hc4 is used in moir experiments. baseline models. the primary baseline for our adapter-and sftm-based transfer is the standard and well established method for zero-shot transfer of english-trained rerankers (macavaney et al., 2020), termed monobert. this is the reranking cross-encoder where we allow for full-tuning of the underlying monolingual or multilingual bert ## results and discussion cross-lingual retrieval (clir). tables 1 and 2 show the clir results, for fourteen language pairs 8 https://turkic-interlingua.org from the augmented clef 2003 benchmark foot_7 using distil dmbert and nmt+bm25 as stage 1 prerankers, respectively. with distil dmbert as the preranker ( the language pairs in tables 1 and 2 consist of high-resource languages for which large parallel corpora and, consequently, reliable nmt models exist. however, even when starting from a more competitive mt-based preranker (nmt+bm25; cross-lingual transfer for moir. it is worth noting that all moir scores are substantially higher than clir results from tables 1 and 2 . this is expected and reflects the fact that matching representations within a languagewhere models can still rely on exact lexical matches between queries and documents -is easier than aligning text representations across languages. ## related work next to adapters and sftms there exist other parameter efficient transfer (pet) methods. for example, bitfit trains only bias vectors @xcite , lora trains low-rank decompositions of weight matrices in dense layers @xcite and methods that learn continuous prompts @xcite @xcite . in the context of retrieval for 12 this phenomenon has been reported to occur in lowresource and out-of-domain settings @xcite . we confirm this finding as we find hallucinations appearing more often in en so than in en sw query translations. english, concurrent work focuses on the learningefficiency @xcite and out-of-domain generalization @xcite of pet methods, whereas we investigate pet both on task-and language-level adaption for clir. ## conclusion in this work, we introduced modular and parameterefficient neural rerankers for effective cross-lingual retrieval transfer. our models, based on adapters and sparse fine-tuning masks, allow for decoupling of language-specific and task-specific (i.e., ranking) knowledge. we demonstrate that this leads to more effective transfer to cross-lingual ir setups as well as to better cross-lingual transfer for monolingual retrieval in target languages with no relevance judgment improving over strong prerankers based on state-of-the-art nmt. encouragingly, we observe particularly pronounced gains for low-resource languages included in our evaluation. we hope that our results will encourage a broader investigation of parameter-efficient neural retrieval in monolingual and cross-lingual setups. we make our code and resources available at: https:// github.com/rlitschk/modularclir .