Improving BGE - M 3 Multilingual Dense Embeddings for N igerian Low Resource Languages (2025)

Abstract: Multilingual dense embedding models such as Multilingual E5, LaBSE, and BGE-M3 have shown promising results on diverse benchmarks for information retrieval in low-resource languages. But their result on low resource languages is not up to par with other high resource languages. This work improves the performance of BGE-M3 through contrastive fine-tuning; the model was selected because of its superior performance over other multilingual embedding models across MIRACL, MTEB, and SEB benchmarks. To fine-tune this model, we curated a comprehensive dataset comprising Yorùbá (32.9k rows), Igbo (18k rows) and Hausa (85k rows) from mainly news sources. We further augmented our multilingual dataset with English queries and mapped it to each of the Yoruba, Igbo, and Hausa documents, enabling cross-lingual semantic training. We evaluate on two settings: the Wura test set and the MIRACL benchmark. On Wura, the fine-tuned BGE-M3 raises mean reciprocal rank (MRR) to 0.9201 for Yorùbá, 0.8638 for Igbo, 0.9230 for Hausa, and 0.8617 for English queries matched to local documents, surpassing the BGE-M3 baselines of 0.7846, 0.7566, 0.8575, and 0.7377, respectively. On MIRACL (Yorùbá subset), the fine-tuned model attains 0.5996 MRR, slightly surpassing base BGE-M3 (0.5952) and outperforming ML-E5-large (0.5632) and LaBSE (0.4468).

## introduction nigeria is home to hundreds of languages, yet its three major tongues: hausa, yorùbá, and igbo-are still considered low-resource for information retrieval (ir) tasks. these languages are morphologically rich and linguistically complex, featuring phenomena such as agglutinative affixes and, in the case of yorùbá and igbo, tonal diacritics that alter word meaning. a key challenge is that text in * equal contribution. traditional lexical retrieval methods (e.g., bm25 or tf-idf ranking) are insufficient for these lowresource, morphologically rich languages. lexical ir relies on exact or near-exact token overlap between query and document, an assumption that breaks down when words have many surface forms or when spelling variations (such as omitted diacritics) are common. consequently, purely lexical approaches struggle to retrieve semantically relevant content if there is no literal token match. this limitation is well-documented as the semantic and vocabulary mismatch problem. for example, a yorùbá user might search for "àwòrán" (meaning "picture"), but a document containing the synonym "fotò" (a borrowed word for "photo") would be missed by lexical matching. recent advancements in neural ir show promising solutions by introducing dense multilingual embedding models, such as labse @xcite , me5 @xcite , and bge-m3 @xcite . these models encode queries and documents into a shared vector space, enabling semantic matching beyond lexical similarity @xcite . despite their effectiveness, general multilingual models do not obtain a very high performance for low-resource languages such as yorùbá, igbo, and hausa, as opposed to english. @xcite . fine-tuning multilingual models on targeted datasets has emerged as a promising strategy for improving retrieval performance on low-resource languages. more recently, the miracl dataset (18 languages) @xcite was used to finetune retrieval models, and a single model trained on all languages achieved robust performance, even outperforming some monolingual-tuned models on their own language @xcite . recognized for its state-of-the-art performance across multilingual retrieval benchmarks such as miracl and seb, we decided to fine-tune the bge-m3 model @xcite , as it offers substantial potential for improvement through contrastive fine-tuning. a technique that encourages the embedding model to minimize distances between semantically similar document-query pairs and maximize distances for dissimilar pairs @xcite @xcite . our contributions are as follows: queries were taken from headlines or sub-topics, with the matching article content as the positive document. we added english-translated queries using the gemma3-27b model to support multilingual retrieval. about 15k yoruba, 15k igbo, and 15k hausa queries ( 45k total) were translated and paired with their original-language documents, creating english-yoruba, english-igbo, and english-hausa pairs for alignment. ## results for evaluation, we utilized the held-out portions of the wura dataset as our primary benchmark for all three languages. the wura dataset contains annotated query-document pairs in yoruba, igbo, and hausa, making it well-suited for evaluating our multilingual retriever in each language. we partitioned wura's data into validation and test splits to tune the model and assess final performance. approximately 60% of the wura queries (up to a maximum of 2,000 per language) were set aside as a validation set for development and hyperparameter tuning. the remaining 40% of the queries (again up to 2,000 per language) was reserved as the final test set on which we report results. importantly, these evaluation queries were never seen during training (as ensured by the preprocessing step that removed wura validation examples from the training data). in addition to wura, we also evaluated on the yoruba subset of miracl @xcite a widely used multilingual retrieval benchmark that provides monolingual ad-hoc retrieval tasks over wikipedia across 18 languages with hundreds of thousands of high-quality relevance judgments-following its standard development/test protocol to cross-check robustness. we evaluate retrieval performance primarily using mean reciprocal rank (mrr) (2), which measure the model's ability to successfully retrieve the correct document for each query in the test set. ## conclusion this study has shown that fine-tuning multilingual embedding models, particularly bge-m3, can significantly improve information retrieval performance for low-resource nigerian languages such as yoruba, igbo, and hausa. through contrastive learning and cross-lingual alignment using english translated queries mapped to one of yoruba, igbo and hausa documents, the fine-tuned models achieved a results and outperformed established baselines. our findings emphasize that lowresource languages can benefit greatly from recent advances in large-scale multilingual embeddings when appropriately adapted. the outcomes also reinforce the potential for building inclusive, language aware ir systems that serve diverse linguistic communities.