Language Concept Erasure for Language-invariant Dense Retrieval (2024)

Abstract: Multilingual models aim for language-invariant representations but still prominently encode language identity. This, along with the scarcity of high-quality parallel retrieval data, limits their performance in retrieval. We introduce LANCER, a multi-task learning framework that improves language-invariant dense retrieval by reducing language-specific signals in the embedding space. Leveraging the notion of linear concept erasure, we design a loss function that penalizes cross-correlation between representations and their language labels. LANCER leverages only English retrieval data and general multilingual corpora, training models to focus on language-invariant retrieval by semantic similarity without necessitating a vast parallel corpus. Experimental results on various datasets show our method consistently improves over baselines, with extensive analyses demonstrating greater language agnosticism.

## introduction multilingual text encoders @xcite @xcite aim to provide representations that capture the essential meaning of texts, irrespective of language identity. these models show promise in information retrieval (ir) tasks, outperforming traditional lexicalbased methods like bm25 @xcite @xcite . despite their potential, these models often underperform with unseen or underrepresented languages @xcite . this paper explores strategies to enhance language invariance in multilingual retrieval settings. we introduce language concept erasure for language-invariant dense retrieval (lancer), a multi-task learning framework designed to induce language invariance in the representation space of dense retrieval models. we focus * work completed while at umass amherst. particularly on the challenges posed by englishonly fine-tuning, where a multilingual model is fine-tuned using only english queries and documents. while the ideal is for document relevance to transcend language barriers, training regimes typically lack comprehensive multilingual coverage. this often leads to a performance gap between languages well-represented in the training data and those that are not, exacerbated by the scarcity of reliable relevance judgments in many languages. language bias @xcite and data scarcity @xcite are known as limiting factors of the use of dense retrieval model performance on monolingual (non-english), crosslingual, and multilingual foot_0 retrieval tasks. this degrading performance is illustrated in figure 1 , which presents retrieval results on the lareqa dataset @xcite , a retrieval dataset with parallel queries and documents in 11 languages. when the number of languages on which models are evaluated is increased, performance degrades across all models. our approach, lancer, regularizes the fine-tuning process of the retrieval model by an auxiliary language invariance objective. building upon the concept of guardedness @xcite , the inability to linearly predict a given concept (in our case, language identity) from a representation, our regularization scheme encourages guarded representations with respect to language identity. specifically, we penalize the correlation between the representations and the language of the text that is represented. this auxiliary objective can be computed using any texts that are annotated by their language, i.e., it can make use of any multilingual non-parallel corpus, and does not require any specialized parallel data or human-annotation of semantic similarity. as such, it is particularly useful for languages that are not well-represented in commonly used retrieval datasets. the dense retrieval models developed using our framework exhibit reduced language bias, improving all three backbone encoders on two multilingual retrieval datasets. benefiting from language-invariant representations, these models also show substantial improvements in monolingual and cross-lingual retrieval tasks. notably, on miracl @xcite , lancer improves mdpr by 13.6% and mcontriever by 32.5% at ndcg@10, also outperforming an llm-based data augmentation method foot_1 . ## methodology our objective is to diminish the languagedependent features within the embedding space of a dense retrieval model, enabling it to produce language-invariant representations across various linguistic contexts while preserving effective retrieval results. the guardedness objective (ravfogel et al., 2023)-the inability to linearly predict the concept from the representation-has been shown to be equivalent to having a zero crosscorrelation matrix between the representations and the concept labels @xcite . we take inspiration from this condition, and propose a correlation-based loss function that penalizes any linear correlation between the language labels and the representations, while at the same time, we aim to preserve effective retrieval results in the training language (that is, english). ## conclusion in this work, we introduce lancer, a multi-task training framework designed to improve languageinvariant dense retrieval. the core of lancer is the language concept erasure task, which reduces the language-specific signals present in the multilingual dense vectors by preventing linear classifiers from detecting the language labels. paired with the retrieval task, lancer enables the model to prioritize learning language-invariant knowledge for query-document matching. we conduct experiments across all possible linguistic settings of an ir task (e.g., monolingual, cross-lingual, and multilingual). the extensive results from these experiments demonstrate the effectiveness of lancer in building languageinvariant dense retrieval models. in multilingual contexts, lancer outperforms knowledge transfer using parallel data. furthermore, in monolingual tasks across 18 languages, lancer, as a zero-shot approach, surpasses an in-domain data augmentation method based on llms. for future work, we are interested in extending the language concept to more general concept(s) to improve domain adaptation and convert lancer to a general framework for model debiasing for downstream applications.