Mapping Cross-Lingual Sentence Representations for Low-Resource Language Pairs Using Pre-trained Language Models (2025)

Abstract: In this work, we explore different linear mapping techniques to learn cross-lingual document representations from pre-trained multilingual large language models for low-resource languages. Three different mapping techniques namely Linear Concept Approximation (LCA), Linear Concept Compression (LCC), and Neural Concept Approximation (NCA) and four multilingual language models such as mBERT, mT5, XLM-R, and ErnieM were used to extract embeddings. The inter-lingual representations were created mappings the monolingual representation extracted from multilingual language models. The experimental results showed that LCA and LCC significantly outperform NCA, with models like ErnieM achieving the highest alignment quality. Language pairs exhibit variable performance, influenced by linguistic similarity and data availability, with the Amharic-English pair yielding particularly high scores. The results showed the utility of LCA and LCC in enabling cross-lingual tasks for low-resource languages.

## introduction training such models requires extensive data and computational resources, a significant hurdle for low-resource languages where data availability is limited. to address this, we focus on leveraging existing resources and cross-lingual learning techniques to align sentences across languages, including low-resource ones. cross-lingual learning aligns text representations from one language to another, enabling effective knowledge transfer and facilitating robust multilingual systems without heavy reliance on machine translation @xcite . alignment can occur at different levels: word, sentence, or document. word-level alignment brings semantically similar words close in a shared embedding space, aiding tasks like bilingual lexicon induction @xcite . sentence-level alignment captures full context and meaning, using techniques like laser @xcite to generate language-independent sentence embeddings. document-level alignment broadens this focus, enhancing multilingual information retrieval @xcite . our study addresses a specific gap: exploring effective methods for generating cross-lingual sentence representations from pre-trained large language models. specifically, we ask: how effective are different mapping methods for learning crosslingual sentence representations in low-resource language pairs? answering this will help improve nlp inclusivity and capabilities for low-resource languages. this work builds on work by salamon et al. ( 2021 2 methodology @xcite proposed an approach using different mapping techniques for obtaining interlingual representations, which serves as an inspiration for the current work. it involves the generation of document embeddings (representations) for the source and target languages, and then finding a mapping into an inter-lingual representation space. it further allows cross-lingual transfer learning, hence avoiding the high costs of machine translation (mt) systems and challenges with lowresource languages @xcite . this study utilizes pre-trained language models to embed parallel data sets. it then employs mapping techniques to align monolingual representation spaces, creating inter-lingual document representations. this approach facilitates the effective transfer of linguistic information across different languages ## results in this section, we present the results obtained in two parts: one focused on the pre-trained models, and another one focused on the pairs of languages used. further, we provide the results across dimensions for a selection of the experiments run. ## conclusion this study evaluated the effectiveness of multiple mapping methods in aligning cross-lingual sentence representations for pairs of low-resource languages, utilizing pre-trained multilingual llms. we tested lca, lcc, and nca mapping techniques across multiple model and language pair combinations to assess their performance in lowresource settings. our findings highlight the mapping techniques' success in capturing semantic relationships across languages, with lca and lcc consistently outperforming nca. this outcome suggests that the architectural limitations of nca make it less effective in capturing the nuanced linguistic similarities required for cross-lingual alignment tasks. additionally, the variability in results across models showed erniem's superior performance overall, with xlm-r and mt5 close behind, underscoring the importance of diverse pre-training data for robust multilingual performance. mbert, by contrast, performed less effectively, highlighting the limitations posed by limited language exposure in