Cross-Lingual Document Recommendations with Transformer-Based Representations: Evaluating Multilingual Models and Mapping Techniques (2025)

Abstract: Recommendation systems, for documents, have become tools for finding relevant content on the Web. However, these systems have limitations when it comes to recommending documents in languages different from the query language, which means they might overlook resources in non-native languages. This research focuses on representing documents across languages by using Transformer Leveraged Document Representations (TLDRs) that are mapped to a cross-lingual domain. Four multilingual pre-trained transformer models (mBERT, mT5 XLM RoBERTa, ErnieM) were evaluated using three mapping methods across 20 language pairs representing combinations of five selected languages of the European Union. Metrics like Mate Retrieval Rate and Reciprocal Rank were used to measure the effectiveness of mapped TLDRs compared to non-mapped ones. The results highlight the power of cross-lingual representations achieved through pre-trained transformers and mapping approaches suggesting a promising direction for expanding beyond language connections, between two specific languages.

## introduction the rapid expansion of online information from diverse sources and the growing multilingual nature of the web underscore the escalating significance of information retrieval (ir) and recommender systems (rs). today's web is no longer limited to a single language, but is increasingly rich in multiple languages, mirroring the multilingual capacities of its global users @xcite . this diversity highlights the urgent need for cross-lingual recommender systems. traditional recommender systems often prioritize content in a single language, sidelining a wealth of multilingual documents that may hold valuable insights. this gap leads to the emergence of cross-language information access, where recommender systems suggest items in different languages based on user queries @xcite @xcite . machine learning and deep learning, which have significantly impacted language representation and processing, are pivotal to enhancing information retrieval and recommender systems, especially in the realm of document recommendation @xcite . with these advancements, documents ranging from historical texts and scientific papers to legal ones can be recommended more accurately. however, current recommender systems falter when content is available in various languages, often recommending documents in only the query language. in multinational contexts such as the european union, such limitations can hinder effective policy formation. there are two main strategies to address this gap: on the one hand, one can translate the query into multiple target languages or develop a cross-lingual representation space for documents. while this can be effective, this approach is fraught with challenges, including the need for large-scale data, the computational expense of training, and potential loss in translation, especially in domains like law that require precision. on the other hand, crosslingual representations, which focus on creating shared embedding spaces for documents across languages, are the focal point of this study @xcite . by employing mapping-aligned document embeddings and comparing their similarity with the query, it offers a computationally cheaper solution without the need for extensive fine-tuning of pre-trained large language models. the rest of the paper is organized as follows. section 2 presents the related works. the proposed methodology is presented in section 3. section 4 presents the experimental setting and the datasets used in this work. the experimental results will be presented in section 5, while the results are discussed in section 6. finally, the conclusions will be presented in section 7. ## related work the work towards generating inter-lingual and multilingual representations, which can encapsulate information across multiple languages in a unified form, has gained substantial attention in recent years. this interest spans both word-level and document-level representations. early observations, such as those introduced by @xcite , identified that word embedding spaces across languages possess structural similarities. these insights led to the development of linear mappings from one language embedding space to another, utilizing parallel vocabularies. subsequent works @xcite @xcite , have aimed to refine these cross-lingual word embeddings, mainly through modifications in space alignment methods or retrieval techniques. techniques like averaging word vectors @xcite or leveraging crosslingual knowledge bases like wikipedia @xcite or babelnet @xcite have been used to learn document-level cross-lingual representation. a notable methodology in this domain is the cross-lingual semantic indexing (cl-lsi) @xcite , which extends the well-known latent semantic indexing (lsi) to encapsulate multiple languages through the singular value decomposition of concatenated monolingual document-term matrices. an emerging strategy in both word-level, sentence-level and document-level research is the use of neural network architectures. one of the pioneer works in this direction was the work by @xcite where they used a deep neural network to directly encode long text passages in a language-independent manner. the work by @xcite used a multilingual auto-encoder to generate languageindependent sentence embeddings. recently, pretrained models such as bert @xcite have changed the landscape of cross-lingual representation research. these models have enabled the generation of sentence encoders on multilingual unlabeled corpora without the need for parallel data @xcite @xcite . concurrently, certain studies have leveraged pre-trained multilingual transformers for cross-lingual information retrieval (ir). the work by @xcite combined mbert with google translate in their information retrieval pipeline, while @xcite utilized mbert and xlm for the same purpose, emphasizing the need for fine-tuning for efficient and effective document-level results. collectively, these studies underscore the potential of transformers in cross-lingual information retrieval, paving the way for alternative methodologies such as mapping over fine-tuning, as explored in the current investigation. while these approaches have shown promise, the study herein differentiates itself by presenting a methodology that uses mapping methods to create inter-lingual representations. the novelty of this work primarily lies in the use of mapping methods to align monolingual representations obtained separately for each language from pre-trained large language models, to produce inter-lingual document-level representations. ## methods in this section, we will introduce the different large language models used in this study and the mapping approaches used to learn interlingual representation from the pre-trained large language models. ## discussion from our results, it becomes evident that transformer leveraged embeddings combined with mapping methods markedly outperform non-mapped embeddings across all models, as delineated in an examination of interestingly, both erniem and mt5, when aligned with non-mapped embeddings, showcased better performance than their transformer counterparts under identical conditions. the underlying reason might be traced back to the distinctive training data and methodologies employed by these models. in contrast to mbert and xlm-robert, which utilize mlm (and additionally nsp for mbert), erniem incorporates a broader spectrum of pre-training objectives geared towards cross-lingual alignment. this distinction could elucidate the superior performance of its non-mapped embeddings. mt5's commendable performance can be attributed to its foundational design, being inherently an encoder-decoder model, though this project exclusively utilized its encoding facet. in general, our study highlights the efficacy of transformer leveraged embeddings when synergized with mapping techniques, resulting in a noticeable performance leap over non-mapped embeddings. this aligns with the findings of @xcite , which accentuate that standalone out-of-the-box models, without refinements or supplementary techniques, are generally less efficient. however, diverging from their study, our research underscores that optimal performance doesn't solely hinge on model fine-tuning. in the realm of ir, integrating mapping techniques can be equally potent in driving commendable results. ## conclusion document recommendation stands at the forefront of information retrieval (ir) systems. within recommendation frameworks, it efficiently suggests pertinent documents in alignment with a user's query. in our research, we delved into the possibilities of crafting cross-lingual representations by harnessing embeddings from pre-existing multilingual transformers in conjunction with mapping strategies. using embeddings from these pre-trained multilingual transformers allows for document representation without requiring further training or intricate processing. nonetheless, our research illuminated that solely depending on the raw embeddings from the transformers fell short in terms of efficacy. a notable enhancement in results was witnessed when the embeddings were synergized with mapping techniques such as lca, lcc, and nca. the languages incorporated within our study hold considerable prominence across various linguistic tasks. consequently, the adopted models and mapping techniques have the potential to foster efficient representations by mapping low-resource languages onto those that are more abundantly represented. it beckons further exploration into how these mapping techniques perform when applied to low-resource languages. future research might not restrict itself to merely language pairs, as was the focus of this study, but could expand to encompass language tuples-translating from a single source language to multiple target languages. achieving this might necessitate refining the present mapping methodologies, introducing supplementary steps, or pioneering entirely novel methods. the code of this project is publicly available on github.