Text Generation Models for L uxembourgish with Limited Data: A Balanced Multilingual Strategy (2025)

Abstract: This paper addresses the challenges in developing language models for less-represented languages, with a focus on Luxembourgish. Despite its active development, Luxembourgish faces a digital data scarcity, exacerbated by Luxembourg’s multilingual context. We propose a novel text generation model based on the T5 architecture, combining limited Luxembourgish data with equal amounts, in terms of size and type, of German and French data. We hypothesise that a model trained on Luxembourgish, German, and French will improve the model’s cross-lingual transfer learning capabilities and outperform monolingual and large multilingual models. To verify this, the study at hand explores whether multilingual or monolingual training is more beneficial for Luxembourgish language generation. For the evaluation, we introduce LuxGen, a text generation benchmark that is the first of its kind for Luxembourgish.

## introduction recent advances in deep learning have made it extremely popular to use language models (lms) such as bert @xcite and t5 @xcite for many tasks in natural language processing (nlp) @xcite . the tasks range from text classification tasks, such as sentiment analysis @xcite and offensive language detection @xcite , to text generation tasks, such as machine translation @xcite ) and text summarisation @xcite . lms have achieved state-of-the-art results in many of these tasks @xcite . even though these lms are multilingual by design, their support and performance can suffer with languages that are not as well represented @xcite . a lot of focus of these lms tends to fall on english, as well as other high-resource languages @xcite . this fact is evidenced by the amount of data used for training certain models. for example, mt5 @xcite , which is trained using the oscar common crawl data, contains roughly 3.4tb for english, 1.4tb for chinese, 1.1tb for russian, 600gb for german and 430gb for spanish foot_0 . by contrast, luxembourgish only has 18mb, and is therefore not well-supported by many current llms. however, initial testing has shown that gpt-4o @xcite ) models can produce luxembourgish very well, demonstrating the positive effects of cross-lingual transfer-learning @xcite at an immense scale. more data would certainly improve the situation, but this may not always be possible, meaning that other options need to be considered. luxembourgish, a west germanic language, is spoken by around 400,000 people, primarily in luxembourg @xcite . in addition to the small number of speakers, luxembourg is home to a complex societal multilingualism that, historically, has favoured french and german as official languages, especially in formal and written communication. only since the advent of digital and social media, and as a result of the active language policy to support luxembourgish, have more significant amounts of text data been produced and therefore made available. as this situation is surely similar with other small varieties, means of finding and using more data are necessary. continuing with the example of luxembourgish, models such as luxembert @xcite have used data augmentation, in this case translating from german @xcite . models such as luxgpt2 @xcite rely on transfer learning. however, none of these methods have provided state-of-the-art performance like models for other languages. in fact, @xcite has shown that multilingual models outperform existing monolingual luxembourgish models in a text classification task. we, therefore, propose to fill this gap by combining the largest collection of data for the luxembourgish language available with carefully considered transfer learning. in this case, in particular, we seek to answer the question of whether similar languages included in the training can improve performance. recent research has shown that careful selection of multilingual training data improves models for 16 african languages @xcite . we therefore hypothesise that equal amounts, in terms of size and composition, of luxembourgish, german, and french will outperform monolingual and multilingual models on luxembourgish nlp tasks. in this paper, we address the gap in luxembourgish nlp and present a novel generative model that is based on the multilingual t5 architecture @xcite . moreover, we run multiple evaluations on downstream tasks to ascertain whether multilingual or monolingual pre-training is more beneficial for a luxembourgish model. to this end, we obtain training data for both german and french, the geographical and socio-cultural neighbours of luxembourg(ish), and aim to learn more about treating these languages equally in pre-training. in doing so, we also discuss the data compilation, specifically the equivalency of data. for evaluation purposes, due to the current under-represented situation of luxembourgish nlp, we re-use classification tasks @xcite and introduce new generative tasks, including news article headline generation, paraphrasing and wikipedia biography summaries. our main contributions are: the expansion of language models to encompass european languages beyond english has been a focal point in recent nlp research. the t5 (raffel et al., 2020), bert @xcite , and electra @xcite architectures have been adapted to create not just multilingual @xcite but also monolingual models that capture the linguistic nuances of individual languages. for example, chan et al. (2020) developed gbert and gelectra, and later germant5, a version of t5 pre-trained exclusively on a large german corpus. similarly, le et al. (2020) introduced flaubert, a french language model based on the bert architecture. in addition to these, carmo et al. (2020) introduced ptt5, a portuguese t5 model that outperformed previous models in portuguese text generation and comprehension tasks. araujo et al. (2024) trained a similar t5 model for spanish, and daðason and loftsson (2022) evaluated various classification tasks on monolingual icelandic models based on electra and convbert @xcite . for the russian language, @xcite developed rut5, among many others, achieving new benchmarks in russian nlp tasks. these models underline the importance of tailoring language models to specific languages to capture their unique syntactic and semantic properties. as a relatively small language, luxembourgish is less represented in nlp, particularly in comparison to french and german, its socio-cultural neighbours. luxembourgish has developed in close contact with french and german and today shares grammatical features as well as parts of the lexicon with those languages. this is especially true of german, due to luxembourgish having developed from the moselle franconian dialect @xcite , but is also true of french. while not typologically, luxembourgish has been in close contact with french, exhibiting the borrowing of grammatical structures and lexical items, as well as a lot code-switching in written texts. research in nlp for luxembourgish has started only recently, with some early exceptions: @xcite introduce various resources for nlp tasks for luxembourgish; @xcite analyse typical writing patterns (contextual n-deletion) in written transcripts of speech, and @xcite introduce a manually annotated corpus of mixed language sentences to test a word-based language identification system. recently, @xcite and @xcite introduced a state-ofthe-art pipeline for sentiment analysis, purschke (2020) published a pipeline for the automatic orthographic correction of text data and @xcite introduced a new approach to zero shot classification based on a task-specific dictionary for topic classification. for spoken data, @xcite and @xcite develop lux-asr, a performant automatic speech recognition for luxembourgish. for automatic comment moderation @xcite and orthographic normalization @xcite , models based on the t5 architecture have shown to perform well for luxembourgish. looking at language models for luxembourgish, various strategies have been tested. some have been trained using transfer-learning from german, such as luxgpt @xcite . another model for luxembourgish exists in luxembert @xcite , which used augmentation techniques to produce more luxembourgish data, and which performs on par with multilingual bert on luxembourgish language tasks such as partof-speech tagging, named entity recognition and news classification @xcite . to this end, the authors of the afore mentioned model also introduced the corresponding resources. anastasiou (2022) introduced enrich4all, another bert model for the development of a multilingual chatbot in administrative contexts, trained with a specifically annotated corpus. these models all demonstrate the various strategies used to create the models. nevertheless, it is clear that not enough resources for luxembourgish exist yet, and that research related to nlp in luxembourgish is limited. both of these reasons explain why there are only some benchmarks for classification tasks in luxembourgish, and none for generative tasks. the absence of benchmarks, small amounts of data, and lack of more generative models are areas we hope to address with this research. ## data data is essential to training a language model and plays a critical role in this research, as outlined in the introduction. because we wanted to investigate specifically the composition of the data used for training, we explain the choices made in this section. it would be reasonable to assume a typical approach of simply using all the data one can find. because luxembourgish only has a minimal amount of data available compared to german and french, this would lead to a large imbalance in the data, calling into question whether the luxembourgish part would even be worth including. as stated previously, we therefore aimed to match the german and french data largely to mirror that of the luxembourgish data in terms of size, type, and domain. we consciously set out to collect roughly equal amounts of data for each language to test specifically how much better the language model would perform in comparison to a monolingual luxembourgish model. the rtl data spans the years 2008 until 2023 and has been used for many other luxembourgish nlp research @xcite @xcite @xcite . in terms of news related text, we also added data from the leipzig (lpzg) collection @xcite which includes data from other luxembour-gish news sites. 3 for web content, we also made use of data from the leipzig collection using specifically 1 million sentences from the latest web crawl, excluding rtl. in addition, we use text from luxembourgish chat rooms. encyclopaedic text was also used in the form of wikipedia articles, which we obtained from the latest dump as of the time of training. this spans roughly 70,000 articles about various topics, but mainly biographies and information about locations. similarly, we used all example sentences from the luxembourgish online dictionary (lod). 4 finally, political speeches and debates are also represented in the corpus, which are transcribed for the chambre des députés (chamber), the national legislature chamber of luxembourg. german for german, we aimed to collect the closest corpora to the luxembourgish corpora. starting with news related text, we again made use of parts of the leipzig corpus, specifically from german news sites (in roughly equal token quantities) as well as the ten thousand german news articles dataset (10kgnad) dataset foot_4 , which consists of over 10,000 articles from an austrian newspaper across nine topics. as such, these articles are the unused part of the one million posts corpus (ompc) @xcite , which we use to replicate the situation in luxembourgish, where we have news articles and the user discussion under each article. we also use the potsdam commentary corpus (pcc) @xcite ) to add further user comments from german newspaper task train test type news title 162,882 13,852 rtl news articles positive comment 3,236 810 rtl articles & comments negative comment 3,236 810 rtl articles & comments description 11,858 2,094 wikipedia articles we used the same objective as the original t5 models @xcite . the main idea is to feed the model with corrupted (masked) versions of the original sentence and train it to reconstruct the original sequence. this denoising objective works by randomly sampling and dropping out 15% of tokens in the input sequence. all consecutive spans of dropped-out tokens are then replaced by a single sentinel token. for both of our pre-trained models, we use a learning rate of 1e-4, a batch size of 128 sequences, and a maximum sequence length of 512. we pretrain each model for 1m steps. ## luxgen: text generation benchmark for luxembourgish to evaluate the luxt5 and luxt5-grande models, we defined luxgen: a text generation benchmark for luxembourgish, consisting of four text generative tasks. due to data being limited for luxembourgish, especially in terms of benchmarks, we derive these tasks mainly from rtl data, as this already has the most metadata available. the generative tasks are all novel for luxembourgish. we believe these tasks to be the best currently available to evaluate the performance of text generation models, including recent large language models for nlp tasks in luxembourgish. an overview of the available data is presented in ## evaluation in this section, we evaluate the performance of our proposed models, luxt5 and luxt5-grande, on luxembourgish text generation tasks encompassed in the luxgen dataset, and a classification task for luxembourgish introduced by @xcite . we compare our models against several baselines, including the non fine-tuned large language models (llms) llama 3 @xcite , gpt-4o @xcite , and mis-tral, as well as fine-tuned versions of mt5 @xcite and byt5 @xcite . our evaluation comprises both automatic metrics, using bleu scores @xcite due to the lack of advanced nlg metrics for luxembourgish, standard metrics for classification, and a manual analysis to provide a comprehensive understanding of each model's capabilities in generating accurate and fluent luxembourgish text. ## conclusion this paper has demonstrated the performance of multiple language models for the luxembourgish language. the models demonstrated are all capable of both text classification and text generation tasks. while we have presented a detailed evaluation of the various models, we have also described the different datasets with which the models were trained. in doing so, we have shown that large, massively multilingual models do not necessarily perform best for small, low-resource languages. in fact, we think it is clear that smaller models, trained on the limited amount of data available for a given low-resource language, can benefit from the addition of equal amounts of linguistically related languages. our results indicate that such models can outperform larger multilingual language models consistently and can come very close to the performance of llms, like gpt, although at a considerably lower cost in terms of training data size and training time. with these findings in mind, we plan ablation studies in the future to determine the exact effects more precisely. the findings of this paper further suggest that there may be positive implications for not just lowresource languages that can benefit from sociocultural neighbours or contact languages, but also for all kinds of varieties within a given language, such as regional dialects. nonetheless, these findings require further research, which will shape our future outlook on the topic of this paper. we plan to experiment with adding and removing further languages to and from our models to assess the performance impact. we also want to look more closely at the precise quantities and composition of added data, as well as the balances in relation to other languages. furthermore, we plan to test our approach for regional varieties of german and other languages, to determine whether this approach of adding linguistically related languages or language varieties to a model can help performance.