Adapting Multilingual Embedding Models to Historical L uxembourgish (2025)

Abstract: The growing volume of digitized historical texts requires effective semantic search using text embeddings. However, pre-trained multilingual models face challenges with historical content due to OCR noise and outdated spellings. This study examines multilingual embeddings for cross-lingual semantic search in historical Luxembourgish (LB), a low-resource language. We collect historical Luxembourgish news articles from various periods and use GPT-4o for sentence segmentation and translation, generating 20,000 parallel training sentences per language pair. Additionally, we create a semantic search (Historical LB Bitext Mining) evaluation set and find that existing models perform poorly on cross-lingual search for historical Luxembourgish. Using our historical and additional modern parallel training data, we adapt several multilingual embedding models through contrastive learning or knowledge distillation and increase accuracy significantly for all models. We release our adapted models and historical Luxembourgish-German/French/English bitexts to support further research.

## introduction exploration possibilities of historical texts, such as newspapers, have advanced rapidly due to digitization efforts by libraries and archives @xcite . traditionally, tools relied on keyword-based searches, often enhanced with semantic enrichment techniques such as named entity recognition @xcite . recent embedding benchmarks @xcite show that massively multilingual embedding models, trained on diverse multilingual corpora, perform well in both multilingual and cross-lingual semantic search. these models have also become integral in retrieval-augmented generation (rag), where they help retrieve more relevant and contextually appropriate documents, thereby improving the faithfulness of generated responses. however, for low-resource languages like luxembourgish (lb), where multilingual models have limited exposure, their performance remains uncertain. applying these models to semantic search in imperfectly digitized historical collections introduces additional challenges, as they must handle ocr errors and historical spelling variations. the disparity between these noisy, historical texts and the clean, modern digital-born data used to train multilingual models, combined with their limited support for luxembourgish, complicates the development of effective exploration tools for historical luxembourgish newspaper archives. to address this issue, we compile 2,338 historical luxembourgish news articles from different time periods and use gpt-4o to segment and translate them into modern french (fr), english (en) and german (de). the resulting parallel sentences serve as fine-tuning data to adapt existing multilingual embedding models for imperfectly digitized historical luxembourgish. our main contributions: (1) we adapt multilingual embeddings for digitized historical luxembourgish by generating training data through a prompt-based translation approach with gpt-4o. ## related work this section reviews relevant embedding models that support luxembourgish semantic search, including monolingual luxembourgish models and multilingual embeddings. reimers and gurevych (2020) use knowledge distillation through a strong paraphrase-trained english embedding model and parallel data to create cross-lingually aligned models. multiple instances of such models have been open sourced and a particularly powerful and popular one is paraphrasemultilingual-mpnet-base-v2 (m-mpnet) which was trained on over 50 languages. later within this work, we will explain how we extend this model to also support luxembourgish. the multilingual bitext mining model labse @xcite is trained with translation ranking loss and negative samples. it has been trained roughly on less than 100 luxembourgish-english sentence pairs and specializes in zero-shot bitext mining. a recent model is gte multilingual (m-gte) @xcite , a multilingual embedding model designed for long context text representation and reranking. m-gte has been trained with hard negatives and has included 50,000 luxembourgish pairs within its contrastive pre-training. specific model adaptations to luxembourgish have also been developed. one example is luxem-bert @xcite , a monolingual bert model pre-trained for luxembourgish using augmented data, partially generated by translating texts from closely related languages and incorporating relevant text sources. closely related to our work, luxembedder @xcite ) used openai's textembedding-3-small and labse to mine a set of parallel sentences for each pair of languages between luxembourgish, english, and french. these parallel sentences (up to 20,000 per pair) were then used to further fine-tune labse, improving performance on modern luxembourgish evaluation sets. however, its ability to handle luxembourgish texts from different historical periods-potentially affected by digitization errors common in large-scale historical text collections, remains unclear. our work aims to extend existing embedding models to better perform cross-lingual semantic search within a collection of historical, ocr-noisy luxembourgish texts. the conditions of these texts combined with the different spelling variations foot_1 poses an interesting generalization challenge to the models. ## method to adapt and evaluate embedding models for digitized historical luxembourgish news articles, we create parallel texts by translating them into modern german, french, and english. this allows the models to learn cross-lingual representations and improves their ability to align historical luxembourgish with contemporary languages for semantic search. ## results among the off-the-shelf(in cursive) models, labse is the strongest model in all three languages. surprisingly, luxembedder, a lb-tuned version of labse, shows an average performance drop of 9pp across language pairs in our bitext mining task, despite improved performance on the modern lb tasks. similarly, m-gte underperforms labse by 10.4pp. both openai embedding models (textembedding-3-small/large) show moderate performance. among the models contrastively adapted using the hist pairs, the performance in historical bitext mining improves significantly, reaching up to 97.8% accuracy. notably, after domain adaptation, luxembedder matches the adapted labse, reaching over 97.8% accuracy and closing the performance gap observed in the standard models. meanwhile, the customized m-gte models lag behind by about 2pp. interestingly, across all model architectures, training on any language pair improves performance similarly across all pairs, showing a positive cross-lingual transfer. these models experience significant performance drops in modern lb evaluations, particularly in paralux. however, adapting these models with mixed batches of hist and modern sentence pairs partially mitigated performance loss on the modern lb evaluation tasks. within labse and m-gte, this adaptation even improved the performance on sib-200 topic classification, while sacrificing only up to 1% of the performance on historical bitext mining. these mixed-data adapted models provide an overall stronger general backbone for cross-lingual semantic searching within collections containing both historical and modern luxembourgish. the m-mpnet model, before distilling en-lb knowledge, performs poorly in all lb evaluations, despite its proven exact matching capabilities in other languages, confirming its lack of support for the language. after distilling lb with any dataset, the model performs magnitudes better across the board. when distilled with a single dataset, the model performs best on historical semantic search tasks with the hist sentence pairs. in contrast, when distilled with modern sentence pairs, the model excels on modern lb tasks, achieving 80% accuracy on paralux foot_4 and outperforming the second-best m-gte, which achieves 70%. finally, distilling with the mixed data set yields the best results in all evaluations, demonstrating the synergy between the two sources. however, even the mixed-data distilled m-mpnet model only achieves an average accuracy across pairs of 90% in historical bitext mining, the final model: mix it all for a final allpurpose model covering both historical and modern lb, we contrastively adapt m-gte to all language pairs of hist while preserving an equal number of modern sentence pairs, regardless of language. the adaptation dataset consists of 20,000 lb-fr/en/de (hist), 20,000 lb-en (modern), and 40,000 lb-fr (modern), for a total of 120,000 sentence pairs. it is the best-performing historical semantic search m-gte model, achieving an average accuracy of 97.5% across all language pairs. this model outperforms the adapted labse and lux-embedder models on sib-200 (+6pp) and paralux (+20pp), while performing similarly to them in the historical bitext mining evaluations. ## conclusions in this work, we explore the adaptation of multilingual embedding models to digitized historical lb texts, a task where off-the-shelf models struggle due to limited exposure and their reliance on clean modern data. to address this issue, we generate parallel sentence-segmented documents by translating historical luxembourgish newspaper articles into french, english, and german using gpt-4o. to evaluate the effectiveness of adaptation, we design a historical bitext mining task with a heldout test set of 233 articles. our results show that adaptation to parallel historical data improves retrieval accuracy by up to 13pp. however, this adaptation introduces trade-offs, particularly reducing performance on modern lb tasks that require high semantic precision, such as adversarial paraphrase detection. we mitigate this problem through a balanced data mixing strategy that helps preserve modern lb performance while improving historical text semantic search capabilities. these results demonstrate the effectiveness of domain adaptation for historical text processing and suggest that such approaches could benefit lowresource languages facing digitization challenges. such improvements are particularly relevant for libraries and archives, where effective cross-lingual semantic search can improve the discoverability of historical documents and support digital exploration.