Language Agnostic Multilingual Information Retrieval with Contrastive Learning (2023)

Abstract: Multilingual information retrieval (IR) is challenging since annotated training data is costly to obtain in many languages. We present an effective method to train multilingual IR systems when only English IR training data and some parallel corpora between English and other languages are available. We leverage parallel and non-parallel corpora to improve the pretrained multilingual language models’ cross-lingual transfer ability. We design a semantic contrastive loss to align representations of parallel sentences that share the same semantics in different languages, and a new language contrastive loss to leverage parallel sentence pairs to remove language-specific information in sentence representations from non-parallel corpora. When trained on English IR data with these losses and evaluated zero-shot on non-English data, our model demonstrates significant improvement to prior work on retrieval performance, while it requires much less computational effort. We also demonstrate the value of our model for a practical setting when a parallel corpus is only available for a few languages, but a lack of parallel corpora resources persists for many other low-resource languages. Our model can work well even with a small number of parallel sentences, and be used as an add-on module to any backbones and other tasks.

## introduction information retrieval (ir) is an important natural language processing task that helps users efficiently gather information from a large corpus (some representative downstream tasks include question answering, summarization, search, recommendation, etc.), but developing effective ir systems for all languages is challenging due to the cost of, and therefore lack of, annotated training data in many languages. while this problem is not unique to ir the language contrastive loss incorporates the non-parallel corpora in addition to the parallel ones. it encourages the distances from a sentence representation, which can be a sample from both the parallel corpora and the nonparallel corpora, to the two embeddings of a paralleled pair to be the same. in this work, we propose to use the semantic contrastive loss and the language contrastive loss to jointly train with the information retrieval objective, for learning cross-lingual representations that encourage efficient lingual transfer ability on retrieval tasks. our semantic contrastive loss aims to align the embeddings of sentences that have the same semantics. it is similar to the regular infonce @xcite loss, which forces the representations of parallel sentence pairs in two languages to be close to each other, and away from other negative samples. our language contrastive loss aims to leverage the non-parallel corpora for languages without any parallel data, which are ignored by the semantic contrastive loss. it addresses the practical scenario wherein parallel corpora are easily accessible for a few languages, but the lack of such resources persists for many low-resource languages. the language contrastive loss encourages the distances from a sentence representation to the two embeddings of a paralleled pair to be the same. figure 1 illustrates how the two losses improve language alignment. in experiments, we evaluate the zero-shot cross-lingual transfer ability of our model on monolingual information retrieval tasks for 10 different languages. experimental results show that our proposed method obtains significant gains, and it can be used as an add-on module to any backbones. we also demonstrate that our method is much more computationally efficient than prior work. our method works well with only a small number of parallel sentence pairs and works well on languages without any parallel corpora. ## background: multilingual dpr dense passage retriever (dpr) @xcite uses a dual-encoder structure to encode the queries and passages separately for information retrieval. to generalize to multilingual scenarios, we replace dpr's original bert encoders with a multilingual language model xlm-r @xcite to transfer english training knowledge to other languages. concretely, given a batch of n query-passage pairs (p i , q i ), we consider all other passages p j , j ̸ = i in the batch irrelevant (negative) passages, and optimize the retrieval loss function as the negative log-likelihood of the gold passage: ## contrastive learning for cross-lingual generalization the multilingual dense passage retriever only uses english corpora for training. to improve the model's generalization ability to other languages, we leverage two contrastive losses, semantic contrastive loss and language contrastive loss. figure 2 shows our model framework. specifically, the semantic contrastive loss @xcite pushes the embedding vectors of a pair of parallel sentences close to each other, and at the same time away from other in-batch samples that have different semantics. the language contrastive loss focuses on the scenario when there is no parallel corpora for some languages, which encourages the distance from a sentence embedding to paralleled embedding pairs to be the same. ## training the two contrastive losses are applied to the passage encoder only. experiments show that applying them to both the passage encoder and the query encoder would result in unstable optimization, where we see weird jumps in the training loss curves. the joint loss with the information retrieval loss, the semantic contrastive loss, and the language contrastive loss is ## related work dense monolingual / multilingual information retrieval study recently attracts great attention, which mainly benefits from (1) supervised finetuning based on large pre-trained language models and (2) self-supervised contrastive learning. dense passage retrieval @xcite ) is the framework first proposed for monolingual superivsed finetuning on information retrieval. it uses a bert-based dual-encoder structure to encode the query and the candidate passages into embeddings. similar to monolingual ir, supervised finetuning can also be applied to multilingual pretrained language models (lms) for multilingual ir. commonly uesd multilingual pretrained lms include multilingual bert (mbert, @xcite and xlm-r @xcite , both of which are trained on large corpora representing about 100 languages primarily with the masked language modeling task. these models do not use any explicit objective to improve the alignment between language sentences. recent efforts in nlp field have provided easy access to parallel corpora, e.g. @xcite . many multilingual language models use additional parallel data to improve lingual transfer ability. infoxlm @xcite uses parallel corpora to pre-train xlm-r by maximizing mutual information between multilingual multi-granularity texts. labse @xcite pre-trains bert with masked language model and translation language model on the monolin-gual data and bilingual translation pairs. self-supervised contrastive learning is another way used to improve the cross-lingual alignment. contrastive learning maximizes the agreement between positive samples, and minimizes the similarity of positive and negative ones @xcite . for language representation learning, @xcite apply contrastive learning to train a discriminative model to learn language representations. for multilingual representation learning, contrastive learning has been used to improve cross-lingual transfer ability by using additional parallel data @xcite or by leveraging other self-supervision signals @xcite . ## conclusion in this paper, we present a model framework for multilingual information retrieval by improving lingual adaptation through contrastive learning. our experiments demonstrate the effectiveness of our methods in learning better cross-lingual representations for information retrieval tasks. the two contrastive losses can be used as an add-on module to any backbones and many other tasks besides information retrieval.