Optimized Text Embedding Models and Benchmarks for A mharic Passage Retrieval (2025)

Abstract: Neural retrieval methods using transformer-based pre-trained language models have advanced multilingual and cross-lingual retrieval. However, their effectiveness for low-resource, morphologically rich languages such as Amharic remains underexplored due to data scarcity and suboptimal tokenization. We address this gap by introducing Amharic-specific dense retrieval models based on pre-trained Amharic BERT and RoBERTa backbones. Our proposed RoBERTa-Base-Amharic-Embed model (110M parameters) achieves a 17.6% relative improvement in MRR@10 and a 9.86% gain in Recall@10 over the strongest multilingual baseline, Arctic Embed 2.0 (568M parameters). More compact variants, such as RoBERTa-Medium-Amharic-Embed (42M), remain competitive while being over 13 \times smaller. Additionally, we train a ColBERT-based late interaction retrieval model that achieves the highest MRR@10 score (0.843) among all evaluated models. We benchmark our proposed models against both sparse and dense retrieval baselines to systematically assess retrieval effectiveness in Amharic. Our analysis highlights key challenges in low-resource settings and underscores the importance of language-specific adaptation. To foster future research in low-resource IR, we publicly release our dataset, codebase, and trained models at https://github.com/kidist-amde/amharic-ir-benchmarks.

## introduction as a foundational task in natural language processing (nlp), document retrieval plays a crucial role in applications such as open-domain question answering @xcite and factchecking @xcite . traditional retrieval systems such as tf-idf and bm25 @xcite match queries to documents based on lexical overlap. while efficient, they struggle with vocabulary mismatch and semantic ambiguity, limiting their generalizability to synonyms and paraphrases. these challenges are particularly pronounced in morphologically rich languages, where high inflectional variability and complex morphology complicate exact-match retrieval. suboptimal tokenization in multilingual models further exacerbates these issues, leading to over-segmentation and inefficient subword representations @xcite . as a result, word-based indexing methods fail to capture non-concatenative morphology, affixation, and orthographic variations, degrading retrieval effectiveness. to address these limitations, retrieval models must move beyond lexical overlap and incorporate robust semantic representations. neural retrieval models. recent work has introduced several families of neural retrieval methods that leverage transformer-based pre-trained language models to improve retrieval effectiveness, particularly in monolingual english settings. these methods have significantly advanced document ranking, achieving state-of-the-art performance in benchmarks such as ms @xcite and natural questions @xcite . broadly, they fall into three main categories @xcite : (i) learned sparse retrieval (e.g., @xcite , which enhances queries and documents with context-aware term expansions; (ii) dense retrieval (e.g., dpr, @xcite , which maps text into dense vector spaces for efficient retrieval, employing a dual-encoder architecture that encodes queries and documents separately, a design that limits their effectiveness for fine-grained relevance modeling; and (iii) cross-encoders (e.g., @xcite , which address this limitation by jointly encoding query-document pairs, capturing richer contextual interactions, with a computational overhead that restricts their use to re-ranking candidate documents @xcite . as an alternative, late-interaction models (e.g., colbert, @xcite , introduce token-level interactions and strike a balance between the efficiency of dense retrieval and the expressiveness of cross-encoders. a newer paradigm, generative information retrieval @xcite @xcite , uses pre-trained encoder-decoder models to consolidate indexing, retrieval, and ranking into a single generative framework. while promising, genir lags behind dense retrieval in handling large-scale datasets and accommodating dynamic corpora, requiring further study of its scalability and adaptability @xcite . research gap. despite these advances, neural retrieval remains understudied for morphologically complex, low-resource languages like amharic. most retrieval models are optimized for highresource languages, and prior work has largely focused on cross-lingual transfer from these languages @xcite . despite advancements in multilingual embedding models @xcite , these approaches remain inadequate for morphologically rich languages due to suboptimal tokenization, poor subword segmentation, and weak cross-lingual transfer @xcite . section 2 further explores the importance of addressing this gap in information retrieval research. our contribution. to address the gap identified above, we focus on amharic and introduce optimized retrieval models and benchmarks, making the following key contributions: (i) amharic text embeddings: we develop dense retrieval models for amharic, leveraging amharic bert and roberta as base models, improving passage ranking accuracy for morphologically complex text. (ii) the first systematic benchmark for amharic: we evaluate both sparse and dense retrieval models on amharic, establishing strong baselines for future research. (iii) a language-specific vs. multilingual analysis: we show that amharic-optimized models consistently outperform multilingual embeddings, underscoring the value of language-specific adaptation. (iv) a public benchmark dataset: we repurpose the amharic news text classification dataset (am-news) by treating headlines as queries and corresponding articles as passages, creating ms mar-co-style query-passage pairs with heuristic relevance labels. this enables reproducible evaluation of passage ranking models for amharic. we refer to this processed version as the amharic passage retrieval dataset. the dataset is publicly available on hugging face, foot_0 and all code and preprocessing scripts are released on github. 3 ## motivation recent studies highlight systemic shortcomings in low-resource language technologies, leading to retrieval failures, biased outputs, and exposure to harmful or policy-violating content @xcite . for example, nigatu and raji (2024) find that amharic-speaking youtube users frequently encounter such content due to retrieval systems misinterpreting user intent behind benign queries. these errors stem from foundational limitations in information retrieval (ir) systems, which are optimized for highresource languages like english and struggle with morphologically complex languages like amharic. the consequences extend beyond search engines: @xcite demonstrate that retrieval failures in machine translation propagate gender bias, defaulting amharic occupational terms to male forms even when the context is gender-neutral. such errors reflect broader research gaps in nlp, where systems disproportionately prioritize highresource languages, thereby exacerbating inequities faced by underrepresented linguistic communities @xcite . amharic, the working language of ethiopia's federal government and one of the most widely spoken semitic languages @xcite , presents unique challenges for ir. its root-based templatic morphology allows a single root to generate numerous derived forms through affixation and vowel pattern changes. these morphological variations, combined with the ge'ez script, an abugida writing system with 33 base characters and over 230 syllabic forms, make amharic structurally and morphologically distinct from indo-european and other high-resource languages. as a result, conventional retrieval models tend to underperform without language-specific adaptation. addressing these challenges requires amharic-specific embedding models tailored for passage retrieval. while recent efforts @xcite have advanced amharic nlp, their primary focus has not been on optimizing retrieval performance. our work fills this gap by developing and benchmarking retrieval methods specifically adapted to amharic's linguistic characteristics, laying a foundation for more equitable and semantically accurate information access in low-resource language settings. ## related work retrieval systems commonly adopt a two-stage pipeline to optimize efficiency and effectiveness: (i) first-stage retrieval efficiently retrieves candidate documents using lightweight methods such as sparse or dense retrieval. (ii) re-ranking refines the results using computationally more intensive models, such as cross-encoders. sparse retrieval. sparse retrieval is fundamental in ir, with bm25 known for its efficiency, interpretability, and cross-domain robustness @xcite . however, it struggles with vocabulary mismatch and morphological variability, challenges that are particularly acute in morphologically rich languages like amharic. learned sparse retrieval (lsr) methods @xcite attempt to mitigate these issues by dynamically weighting and expanding terms, thereby enhancing relevance while maintaining interpretability @xcite . however, lsr faces limitations in low-resource settings due to the scarcity of annotated data, dialectal diversity, and morphological complexity (e.g., amharic's templatic morphology), which necessitate subwordaware tokenization or morphological analyzers that are often unavailable. dense retrieval. dense retrieval encodes queries and documents into a shared semantic space using neural network encoders, enabling efficient retrieval via approximate nearest neighbor (ann) search based on embedding similarity @xcite @xcite . while it helps mitigate lexical mismatch, its effectiveness in low-resource languages is hindered by the need for large-scale labeled training data. multilingual models such as mbert @xcite , xlm-r @xcite , and african language-specific models like serengeti @xcite and afriberta @xcite partially address data scarcity through crosslingual pretraining. however, their effectiveness in morphologically complex languages like amharic has not been thoroughly investigated. recent advances in unsupervised contrastive learning, such as contriever @xcite , have demonstrated strong zero-shot and multilingual retrieval performance, especially in crosslingual transfer scenarios. nonetheless, their effectiveness in morphologically complex languages like amharic remains unexplored, as current evaluations do not account for challenges arising from root-based and templatic morphologies. beyond data scarcity, retrieval performance is further constrained by morphological complexity and tokenization challenges. amharic's templatic morphology often causes standard subword tokenizers to over-segment words into non-morphemic units, leading to fragmented representations that obscure semantic relationships. broader research on multilingual tokenization quality @xcite shows that excessive segmentation in morphologically rich languages introduces noise into subword representations, degrading performance in downstream tasks. despite recent advances in multilingual dense retrieval, state-of-the-art models such as arctic embed 2.0 @xcite and multilingual e5 @xcite , which topped the mteb embedding leaderboard foot_2 at the time of our study, continue to struggle with highly inflected languages. these models often produce suboptimal tokenizations, fragmented subword representations, and inefficient embeddings, ultimately limiting their retrieval effectiveness. our empirical findings in section 6.3 illustrate the extent to which tokenization errors impair retrieval performance in amharic. bridging the gap in amharic ir. retrieval systems are primarily optimized for high-resource languages, exacerbating performance disparities in low-resource settings like amharic @xcite . prior research in amharic ir has explored pre-trained embeddings (word2vec, fast-text, amroberta, belay et al., 2021), morphological tools (e.g., annotation frameworks, wordnetbased query expansion, @xcite , and cross-lingual transfer via multilingual models @xcite . however, systematic evaluations of sparse and dense retrieval architectures remain absent, making principled comparisons difficult and leaving the effectiveness of different paradigms in amharic ir largely unexamined. @xcite introduce 2airtc, a trec-style test collection for standardized amharic ir evaluation, but it lacks baseline retrieval benchmarks and complete relevance judgments, making recall-based assessments unreliable. to ensure robust evaluation, we conduct our main experiments on the amharic passage retrieval dataset, which we derive by preprocessing the amharic news text classification dataset (am-news) (azime and mohammed, 2021) into ms marco-style query-passage pairs (see section 5). a detailed analysis of 2airtc, its limitations, and our supplementary evaluations on this dataset is provided in appendix a. to address these gaps, our work introduces amharic-specific retrieval models that incorporate both strong and compact encoder backbones (section 4.2), optimized using contrastive training to better handle amharic's morphological complexity. we also develop and evaluate a late-interaction col-bert model tailored for amharic, and benchmark both sparse and dense retrieval architectures. this enables rigorous, reproducible comparisons across retrieval paradigms. ## methodology in this section, we outline our approach to amharic dense retrieval. we begin by reviewing dense retrieval and colbert architectures, which underpin our framework. we then introduce our amharic embedding models, describing their architecture, training setup, and optimization strategy. ## experimental evaluation and results in this section we present our empirical evaluation, which is structured around the following research questions: rq1 how well do amharic-optimized embeddings improve ranking accuracy compared to general-purpose multilingual embedding models? (section 6.1) recall model params mrr@10 ndcg@10 @10 @50 @100 rq2 how do different retrieval paradigms compare in effectiveness, establishing a benchmark for amharic passage retrieval? (section 6.2) rq3 how does tokenization quality, particularly subword segmentation, impact retrieval effectiveness in morphologically rich, lowresource languages like amharic? (section 6.3) rq4 to what extent does the base model size influence retrieval performance in late interaction models for low-resource settings like amharic? (section 6.4) ## conclusion we introduced dense retrieval models and established the first systematic benchmark for amharic passage retrieval. our models consistently outperform multilingual baselines, underscoring the importance of linguistic adaptation for morphologically rich, low-resource languages. we also show that tokenization quality, especially subword fertility, significantly impacts retrieval performance: compact segmentations improve ranking accuracy, while over-segmentation harms semantic alignment. our main experiments use the amharic passage retrieval dataset (derived from amnews using heuristic labels), and we include supplementary results on 2airtc in the appendix. however, both datasets present evaluation challenges: the former lacks gold-standard relevance judgments, and the latter has incomplete labeling. these limitations underscore the need for more robust evaluation resources and motivate future research directions. to address these gaps, future work should focus on: (i) designing morphology-aware or byte-level tokenizers tailored to amharic's templatic structure, (ii) improving training with hard negative mining and curriculum-based strategies, and (iii) extending evaluation to document-level and multi-hop retrieval. creating a high-quality, human-annotated benchmark with expert-labeled relevance, dialect variation, and morphological features, through collaboration with local institutions will be critical for aligning ir systems with real-world amharic information needs.