Language Bias in Multilingual Information Retrieval: The Nature of the Beast and Mitigation Methods (2024)

Abstract: Language fairness in multilingual information retrieval (MLIR) systems is crucial for ensuring equitable access to information across diverse languages. This paper sheds light on the issue, based on the assumption that queries in different languages, but with identical semantics, should yield equivalent ranking lists when retrieving on the same multilingual documents. We evaluate the degree of fairness using both traditional retrieval methods, and a DPR neural ranker based on mBERT and XLM-R. Additionally, we introduce ‘LaKDA’, a novel loss designed to mitigate language biases in neural MLIR approaches. Our analysis exposes intrinsic language biases in current MLIR technologies, with notable disparities across the retrieval methods, and the effectiveness of LaKDA in enhancing language fairness.

## introduction information retrieval (ir) is the process of obtaining relevant information from a large collection of data according to a user's information needs. this information may exist in various formats, including text documents, images, or videos. conventionally, the collection is a corpus of text documents, and user information needs are expressed in plain text queries. ir serves as a foundational technology in numerous nlp applications including questionanswering systems @xcite , and is also assuming an increasingly pivotal role in supporting the advancement of large language models (llms) for text understanding and knowledge inference @xcite . multilingual information retrieval (mlir) entails queries being in different languages, with the results for a query in a given language being across multiple languages (including the source language of the query). mlir has particular importance as it enables (multilingual) users to access information that may be unavailable or limited in their native language, thereby fostering cultural and linguistic diversity. research has shown that mlir systems often exhibit biases towards certain languages due to factors like morphological complexity @xcite and resource availability @xcite . for instance, @xcite found that documents in higher-resource languages tend to be ranked higher in mlir. this phenomenon is particular notable when the models are built upon multilingual pretrained models @xcite . another case of language bias in mlir is shown in figure 1 . given semantically equivalent queries in different languages and the same documents, we are interested in determining the consistency of the obtained ranking lists. this forms the basis for our investigation of language fairness in mlir from a query-level perspective. our study compares mlir methods using semantically equivalent queries in 24 european languages, which we use to search a fixed multilingual document collection. these parallel query sets are from the original dataset, not machine-translated, and are based on human-annotated document tags. in repurposing them as queries and using the tags as relevance judgements, we fashion a multilingual ir collection with massively-multilingual parallel query sets. our work makes four main contributions: foot_0 1. novel evaluation metric for fairness under ranking: we propose the mean rank correlation (mrc) score to evaluate language fairness under mlir, based on the premise that semantically-equivalent queries in different languages should yield consistent document rankings. 2. novel dataset: we develop the multieup-v2 dataset, consisting of semantically parallel queries and multilingual documents, along with demographic attributes. this dataset serves as a benchmark for future fairness research in mlir. 3. quantification of language (un)fairness: we analyze language fairness in mlir across different languages and language families, and find that bm25 exhibits larger language bias than neural retrieval frameworks like mdpr. additionally, higher-resource languages tend to be associated with higher degrees of language fairness. 4. proposal of a new ranking bias mitigation method: we propose the language kldivergence alignment (lakda) loss to mitigate language bias in mlir, demonstrating its effectiveness within the mdpr neural retrieval framework with multilingual text encoders mbert and xlm-r. ## language bias in mlir in this section, we examine language bias in mlir. first, we introduce a novel metric for quantifying language fairness, our evaluation benchmark, and introduce a method for mitigating language bias. ## experiments and findings our language fairness experiment consists of two main parts: the detection and comparison of language bias among different ranking methods (section 3.1) and the mitigation of fairness bias (section 3.2). ## related work fairness in information retrieval (ir) has been extensively studied through two primary dimensions: individual fairness and group fairness. these frameworks are crucial in ensuring equitable access to information, addressing concerns related to biases in ranking systems. individual fairness refers to the principle that similar items (in this case, documents) should be treated similarly @xcite . in ir, this means that if two documents are equally relevant to a query, they should receive similar rankings. a violation of individual fairness occurs when two comparable documents are ranked differently due to irrelevant factors, such as their format or metadata. this concept is rooted in the idea of consistency and uniform treatment, ensuring that the system does not unfairly prioritize or penalize specific documents that are otherwise similar in content and relevance. group fairness, on the other hand, ensures that predefined groups (such as demographic groups or, in our case, languages) are treated equitably in the ranking process @xcite @xcite . the goal is to prevent bias against any group by ensuring that the system does not favor one group over another. in ir, this often translates to ensuring that documents associated with a protected group (e.g., underrepresented languages or communities) are not system-atically ranked lower than those associated with unprotected groups. group fairness frameworks attempt to mitigate historical and societal biases that might seep into the retrieval process, making sure that members of different groups have equitable access to information. in our work, we extend this concept to multilingual ir, treating each language as a group and ensuring that rankings are fair and consistent across languages. two key fairness metrics in group fairness that align with our work are probability of equal expected rank (peer) and attention weighted ranked fairness (awrf). while these metrics primarily emphasize document-level fairness, our approach uniquely focuses on query-level fairness. in our context, we argue that the retrieval system should provide consistent performance across languages, ensuring that the language of the query does not affect the user's ability to access relevant information. this promotes inclusivity, ensuring that users from different linguistic backgrounds experience similar outcomes when interacting with the system, ultimately fostering equal access to information. ## background knowledge multieup the european parliament (ep) serves as a crucial forum for political debate and decisionmaking in the european union. during debates, members of the european parliament (meps) discuss topics in their own languages, and debates are then transcribed in the original languages, and indexed with multilingual topics. as such, the data is naturally occurring in the 24 official languages of the eu, and expertly transcribed and multilingually annotated. additionally, we have access to basic demographic details of each of the meps, making it the perfect target for the study of fairness in an ir context, in terms of both language and protected attributes was crafted. the eu has published different language versions of all titles, providing semantically identical queries for investigating language fairness in mlir. an earlier version of the multieup dataset was published in 2023 covering debates up to october 2022 @xcite . in this work, we have expanded the dataset using the same data collection and preprocessing procedures, to include debates up to 2024. this doubles the total data volume, and provides a sufficient sample size to research neural ranking methods. we additionally augment each document with comprehensive metadata of the author, including gender, nationality, political affiliation, and age, for use in exploring fairness with respect to protected attributes. unlike mlir datasets such as mmarco @xcite , a multilingual version of the ms marco @xcite , that relies on machine translation, our benchmark queries and documents are original rather than translated versions. this reduces noise and ensures the linguistic authenticity of the corpus. another commonly used mlir datasets mr. tydi @xcite and miracl @xcite , are actually mixed monolingual ir dataset, since they were structured such that queries in different languages are matched only with documents in the same language. this limits the comparability of results across different languages. our benchmark addresses this limitation by introducing semantically parallel queries across multiple languages, enabling comprehensive analysis of language fairness in mlir. dpr dense passage retrieval (dpr: karpukhin et al. ( 2020 )) is a neural retrieval framework initially proposed for monolingual supervised finetuning. dpr uses dual encoders: one for encoding queries and another for encoding passages (documents), both based on the bert architecture @xcite . the primary advantage of dpr over traditional retrieval models like bm25 is its ability to embed both queries and documents into a shared dense vector space, enabling efficient nearest-neighbor search for retrieval. the relevance of a document to a query is determined by the similarity between their embeddings, typically using the dot product as a similarity measure. in our work, we employ mdpr using mbert and xlm-r to handle multilingual queries and documents. these models are fine-tuned on parallel query-document pairs from multiple languages, allowing the system to generalize across different languages. the use of mdpr allows us to explore how multilingual language models handle language biases, which often favor high-resource languages over low-resource ones. furthermore, we investigate the performance of these models on the mul-tieup dataset, assessing their ability to ensure fair and equitable retrieval across 24 languages, thus promoting fairness in multilingual ir. ## conclusion we introduced a novel benchmark, multieup-v2, for investigating language fairness in multilingual information retrieval (mlir) systems. additionally, we proposed the mean rank correlation (mrc) score to assess language fairness in mlir systems, which ensures that queries in different languages but with the same semantic meaning retrieve similar documents. our findings indicate that the traditional ir method bm25 exhibits larger language biases than dpr with multilingual pretrained language models. furthermore, we designed the language kl-divergence alignment (lakda) loss to mitigate language bias, and found that incorporating lakda loss into dpr improves language fairness substantially without sacrificing retrieval performance.