N eigh XLM : Enhancing Cross-Lingual Transfer in Low-Resource Languages via Neighbor-Augmented Contrastive Pretraining (2025)

Abstract: Recent progress in multilingual pretraining has yielded strong performance on high-resource languages, albeit with limited generalization to genuinely low-resource settings. While prior approaches have attempted to enhance cross-lingual transfer through representation alignment or contrastive learning, they remain constrained by the extremely limited availability of parallel data to provide positive supervision in target languages. In this work, we introduce NeighXLM, a neighbor-augmented contrastive pretraining framework that enriches target-language supervision by mining semantic neighbors from unlabeled corpora. Without relying on human annotations or translation systems, NeighXLM exploits intra-language semantic relationships captured during pretraining to construct high-quality positive pairs. The approach is model-agnostic and can be seamlessly integrated into existing multilingual pipelines. Experiments on Swahili demonstrate the effectiveness of NeighXLM in improving cross-lingual retrieval and zero-shot transfer performance.

## introduction recent progress in natural language processing (nlp) has brought impressive performance to english and other high-resource languages across a wide range of tasks. however, for genuinely lowresource languages, models still struggle due to the lack of labeled data and effective transfer. early multilingual models such as mbert @xcite , xlm @xcite , and xlm-r @xcite exhibit some crosslingual transfer capabilities, but their alignment remains limited, especially for typologically distant languages. to mitigate this limitation, recent research has focused on incorporating explicit cross-lingual signals into pretraining at multiple levels of granularity, including token-level @xcite , word-level @xcite @xcite , sentence-level @xcite , and syntax-level @xcite @xcite . these methods enhance alignment by modeling cross-lingual consistency at their respective levels, providing stronger supervision across languages. in addition, contrastive learning techniques have shown strong potential in improving sentence representations, both in monolingual (e.g., sim-cse; @xcite and multilingual (e.g., con-sert; @xcite contexts. despite recent advances such as alignment-based techniques and contrastive pretraining that better exploit existing corpora, a fundamental bottleneck persists: the scarcity of labeled data for low-resource languages. to address this, recent work explores pseudo-supervision strategies that simulate labeled pairs from monolingual corpora. ernie-m @xcite , for example, constructs pseudo-parallel sentence pairs via backtranslation, but the resulting supervision is only as reliable as the underlying translation system, which often generates noisy or semantically inaccurate outputs in low-resource settings due to the scarcity of parallel training data. alternatively, @xcite mine cross-lingual neighbors in embedding space as training pairs. however, in the absence of strong initial alignment, particularly for typologically distant and under-resourced language pairs, cross-lingual nearest neighbors in the embedding space may not be semantically aligned. training on such misleading neighbors can reinforce incorrect associations and degrade cross-lingual generalization. this highlights a core challenge: how to obtain more high-quality labeled supervision for low-resource languages. in this paper, we propose neighxlm, a neighbor-augmented contrastive pretraining framework that enriches target-language supervision without relying on translation systems. while annotated data are scarce, large unlabeled corpora are often available, even for low-resource languages. pretrained multilingual encoders, trained on language modeling objectives, implicitly capture intralanguage semantic relationships by positioning semantically similar sentences closer in the embedding space. neighxlm exploits this property by mining semantically similar neighbors from unlabeled corpora, thereby constructing high-quality positive pairs to enhance contrastive pretraining. figure 1 illustrates the neighxlm framework. we evaluate neighxlm on swahili (sw), covering diverse downstream tasks including cross-lingual sentence retrieval and zero-shot transfer tasks such as classification and question answering. results across multiple benchmarks show that neighxlm consistently outperforms the base model, demonstrating its effectiveness in enhancing cross-lingual transfer for genuinely low-resource languages. ## conclusion in this work, we propose neighxlm, a neighboraugmented contrastive pretraining framework for improving cross-lingual transfer in low-resource settings. by leveraging intra-language semantic relations to mine high-quality neighbors, our method enriches supervision beyond limited parallel data and enhances cross-lingual alignment. experiments show that neighxlm consistently improves retrieval and zero-shot transfer performance.