Boosting Zero-shot Cross-lingual Retrieval by Training on Artificially Code-Switched Data (2023)

Abstract: Transferring information retrieval (IR) models from a high-resource language (typically English) to other languages in a zero-shot fashion has become a widely adopted approach. In this work, we show that the effectiveness of zero-shot rankers diminishes when queries and documents are present in different languages. Motivated by this, we propose to train ranking models on artificially code-switched data instead, which we generate by utilizing bilingual lexicons. To this end, we experiment with lexicons induced from (1) cross-lingual word embeddings and (2) parallel Wikipedia page titles. We use the mMARCO dataset to extensively evaluate reranking models on 36 language pairs spanning Monolingual IR (MoIR), Cross-lingual IR (CLIR), and Multilingual IR (MLIR). Our results show that code-switching can yield consistent and substantial gains of 5.1 MRR@10 in CLIR and 3.9 MRR@10 in MLIR, while maintaining stable performance in MoIR. Encouragingly, the gains are especially pronounced for distant languages (up to 2x absolute gain). We further show that our approach is robust towards the ratio of code-switched tokens and also extends to unseen languages. Our results demonstrate that training on code-switched data is a cheap and effective way of generalizing zero-shot rankers for cross-lingual and multilingual retrieval.

## introduction cross-lingual information retrieval (clir) is the task of retrieving relevant documents written in a language different from a query language. the large number of languages and limited amounts of training data pose a serious challenge for training ranking models. previous work address this issue by using machine translation (mt), effectively casting clir into a noisy variant of monolingual retrieval @xcite @xcite . mt systems are used to either train ranking models on translated train-ing data (translate train), or by translating queries into the document language at retrieval time (translate test). however, clir approaches relying on mt systems are limited by their language coverage. because training mt models is bounded by the availability of parallel data, it does not scale well to a large number of languages. furthermore, using mt for ir has been shown to be prone to propagation of unwanted translation artifacts such as topic shifts, repetition, hallucinations and lexical ambiguity @xcite @xcite . in this work, we propose a resourcelean mt alternative to bridge the language gap and propose to use artificially code-switched data. we focus on zero-shot cross-encoder (ce) models for reranking @xcite . our study is motivated by the observation that the performance of ces diminishes when they are transferred into clir and mlir as opposed to moir. we hypothesize that training on queries and documents from the same language leads to monolingual overfitting where the ranker learns features, such as exact keyword matches, which are useful in moir but do not transfer well to clir and mlir setups due to the lack of lexical overlap @xcite . in fact, as shown by @xcite on bi-encoders, representations from zero-shot models are weakly aligned between languages, where models prefer non-relevant documents in the same language over relevant documents in a different language. to address this problem, we propose to use code-switching as an inductive bias to regularize monolingual overfitting in ces. generation of synthetic code-switched data has served as a way to augment data in cross-lingual setups in a number of nlp tasks @xcite @xcite . they utilize substitution techniques ranging from simplistic re-writing in the target script @xcite , looking up bilingual lexicons @xcite to mt @xcite . previous work on improving zero-shot transfer for ir includes weak supervision @xcite , tuning the pivot language @xcite , multilingual query expansion @xcite and crosslingual pre-training @xcite @xcite . to this end, code-switching is complementary to existing approaches. our work is most similar to @xcite , who use bilingual lexicons for full termby-term translation to improve moir. concurrent to our work, @xcite show that codeswitching improves the retrieval performance on low-resource languages, however, their focus lies on clir with english documents. to the best of our knowledge, we are the first to systematically investigate (1) artificial code-switching to train ces and (2) the interaction between moir, clir and mlir. our contributions are as follows: (i) we show that training on artificially code-switched data improves zero-shot cross-lingual and multilingual rankers. (ii) we demonstrate its robustness towards the ratio of code-switched tokens and effectiveness in generalizing to unseen languages. (iii) we release our code and resources. foot_0 2 methodology reranking with cross-encoders. we follow the standard cross-encoder reranking approach (ce) proposed by @xcite , which formulates relevance prediction as a sequence pair (query-document pair) classification task. ces are composed of an encoder model and a relevance prediction model. the encoder is a pre-trained language model @xcite that transforms the concatenated input [cls] q [sep] d [sep] into a joint query-document feature representation, from which the classification head predicts relevance. finally, documents are reranked according to their predicted relevance. we argue that fine-tuning ces on monolingual data biases the encoder towards encoding features that are only useful when the target setup is moir. to mitigate this bias, we propose to perturb the training data with code-switching, as described next. artificial code-switching. while previous work has studied code-switching (cs) as a natural phenomenon where speakers borrow words from other languages (e.g. anglicism) @xcite , we here refer to codeswitching as a method to artificially modify monolingual training data. in the following we assume availability of english (en-en) training data. the goal is to improve the zero-shot transfer of ranking models into cross-lingual language pairs x-y by training on code-switched data en x -en y instead, which we obtain by exploiting bilingual lexicons similar to @xcite . we now describe two cs approaches based on lexicons: one derived from word embeddings and one from wikipedia page titles (cf. appendix a for examples). code-switching with word embeddings. we rely on bilingual dictionaries d induced from crosslingual word embeddings @xcite and compute for each en term its nearest (cosine) cross-lingual neighbor. in order to generate en x -en y we then use d en )x and d en )y to code-switch query and document terms from en into the languages x and y, each with probability p. this approach, dubbed bilingual cs (bl-cs), allows a ranker to learn interlingual semantics between en, x and y. in our second approach, multilingual cs (ml-cs), we additionally sample for each term a different target language into which it gets translated; we refer to the pool of available languages as seen languages. code-switching with wikipedia titles. our third approach, wiki-cs, follows @xcite and uses bilingual lexicons derived from parallel wikipedia page titles obtained from inter-language links. we first extract word n-grams from queries and documents with different sliding window of sizes n p t1, 2, 3u. longer n-gram are favored over shorter ones in order to account for multi-term expressions, which are commonly observed in named entities. in wiki cs we create a single multilingual dataset where queries and documents from different training instances are code-switched into different languages. ## experimental setup models and dictionaries. we follow @xcite and initialize rankers with the multilingual encoder mminilm provided by @xcite . we report hyperparameters in appendix c. for bl-cs and ml-cs we use multilingual muse embeddings foot_1 to induce bilingual results significantly different from zero-shot are marked with * (paired t-test, bonferroni correction, p ă 0.05). ## results and discussion we observe that code-switching improves crosslingual and multilingual re-ranking, while not impeding monolingual setups, as shown next. transfer into moir vs. clir. we first quantify the performance drop when transferring models trained on en-en to moir as opposed to clir and mlir. comparing zero-shot results between different settings we find that the average moir performance of 25.5 mrr@10 ( the transfer performance greatly varies with the language proximity, in clir the drop is larger for setups involving typologically distant languages (ar-it, ar-ru), to a lesser extent the same observation holds for moir (ar-ar, ru-ru). this is consistent with previous findings made in other syntactic and semantic nlp tasks @xcite . the performance gap to fine-tuning on translated data is much smaller in moir (+4 mrr@10) than in clir (+11.1 mrr@10) and mlir (+8.3 mrr@10). our aim to is close this gap between zero-shot and full finetuning in a resource-lean way by training on codeswitched queries and documents. code-switching results. training on codeswitched data consistently outperforms zero-shot models in clir and mlir ( ## conclusion we propose a simple and effective method to improve zero-shot rankers: training on artificially code-switched data. we empirically test our approach on 36 language pairs, spanning monolingual, cross-lingual, and multilingual setups. our method outperforms zero-shot models trained only monolingually and provides a resource-lean alternative to mt for clir. in mlir our approach can match mt performance while relying only on bilingual dictionaries. to the best of our knowledge, this work is the first to propose artificial code-switched training data for cross-lingual and multilingual ir.