C ontrastive M ix: Overcoming Code-Mixing Dilemma in Cross-Lingual Transfer for Information Retrieval (2024)

Abstract: Multilingual pretrained language models (mPLMs) have been widely adopted in cross-lingual transfer, and code-mixing has demonstrated effectiveness across various tasks in the absence of target language data. Our contribution involves an in-depth investigation into the counterproductive nature of training mPLMs on code-mixed data for information retrieval (IR). Our finding is that while code-mixing demonstrates a positive effect in aligning representations across languages, it hampers the IR-specific objective of matching representations between queries and relevant passages. To balance between positive and negative effects, we introduce ContrastiveMix, which disentangles contrastive loss between these conflicting objectives, thereby enhancing zero-shot IR performance. Specifically, we leverage both English and code-mixed data and employ two contrastive loss functions, by adding an additional contrastive loss that aligns embeddings of English data with their code-mixed counterparts in the query encoder. Our proposed ContrastiveMix exhibits statistically significant outperformance compared to mDPR, particularly in scenarios involving lower linguistic similarity, where the conflict between goals is more pronounced.

## introduction multilingual pretrained language models (mplms) have been a key ingredient in cross-lingual transfer. several studies @xcite @xcite have highlighted that the inherent multilinguality of these models facilitates knowledge transfer from high-resource to low-resource languages, even in data-scarce scenarios. in cross-lingual transfer, code-mixing @xcite has been effective in both sentence-level and token-level zero-shot crosslingual tasks. code-mixing, employed in the absence of target language data, involves sentences containing words from multiple languages, aligns representations across different languages by training mbert @xcite on code-mixed data. however, code-mixing is less effective in the case of cross-lingual transfer for information retrieval (ir), compared to training solely on english data. in contrast to cross-lingual transfer for classification tasks, where the primary challenge is representation alignment, ir introduces the additional hurdle of representation matching between relevant query-passage pairs, where dpr @xcite is a de facto standard architecture. our hypothesis is that code-mixing, while contributing to the shared goal of representation alignment, may adversely affect the ir-specific challenge of querypassage matching, by introducing noise into text embeddings. our contribution aims to strike a balance between the positive and negative effects of codemixing on zero-shot ir performance, introducing contrastivemix as our proposed solution. our distinction is disentangling the two objectives of representation alignment and relevance matching into two contrastive loss functions, involving both english and code-mixed data, to facilitate relevance matching. specifically, we incorporate an additional contrastive loss, aligning embeddings of english data with their code-mixed counterparts, solely in the query encoder. with these two loss functions, our method aligns cross-lingual representations while mitigating interference caused by embedding noise during ir learning. as a result, contrastivemix effectively achieves both language alignment and relevance matching. experimental results demonstrate that con-trastivemix outperforms mdpr, with statistically significant distinctions observed across eight di-verse languages. in contrast, a naive code-mixing approach consistently falls short when compared to mdpr. these eight languages can be categorically divided into two groups based on linguistic similarities, one belonging to the indo-european language family, the same as english (or the high similarity group), and the other constituting a low similarity group. as the challenge of representation alignment is lower in the former group, consistently highlighted by the effectiveness of mplms in such group evidenced by @xcite al. (2020); krishnan et al. (2021); xu et al. (2022) , mitigating the adverse effects on alignment becomes more apparent in the low similarity group. our observations confirm that the impact of contrastivemix is notably more pronounced in the low similarity group, resulting in a widened performance gap of 1.29 mrr@100 when compared to mdpr. our contributions can be summarized as follows: (1) we demonstrate that training ir models on code-mixed data is not effective for cross-lingual transfer. (2) we propose a method to enhance zeroshot ir performance through code-mixing. (3) we analyze the effectiveness of code-mixing in ir through representation alignment. ## conclusion this paper identifies and overcomes a dilemma in using code-mixing for cross-lingual transfer in ir. specifically, we add contrastive loss, designed to align the embeddings of english sentences with their code-mixed counterparts, as a key component of the training objective. our approach experimentally demonstrated better performance than mdpr, with statistical significance.