Neural Text Normalization for L uxembourgish Using Real-Life Variation Data (2025)

Abstract: Orthographic variation is very common in Luxembourgish texts due to the absence of a fully-fledged standard variety. Additionally, developing NLP tools for Luxembourgish is a difficult task given the lack of annotated and parallel data, which is exacerbated by ongoing standardization. In this paper, we propose the first sequence-to-sequence normalization models using the ByT5 and mT5 architectures with training data obtained from word-level real-life variation data. We perform a fine-grained, linguistically-motivated evaluation to test byte-based, word-based and pipeline-based models for their strengths and weaknesses in text normalization. We show that our sequence model using real-life variation data is an effective approach for tailor-made normalization in Luxembourgish.

## introduction automatic text normalization is the task of mapping non-standard spellings to a standard @xcite . normalization thus reduces orthographic variation and noise in language data. it can serve as a pre-processing step to facilitate downstream tasks like pos-tagging and ner (e.g. @xcite . in this paper, we address orthographic normalization for luxembourgish, a germanic language currently in the process of political development, including orthographic standardization @xcite . spelling norms for luxembourgish are not a novelty, however, due to a lack of language teaching in school, written luxembourgish today is characterized by vast amounts of variation, e.g., orthographic, lexical, syntactical or regional. this has led to written luxembourgish texts adhering to the standard orthography to be rare, even in formal contexts. for this reason, we develop an automatic text normalization model for luxembourgish to reduce variation in written data as a pre-processing step for nlp tasks. luxembourgish is an under-researched language, and as such there is a lack of annotated parallel data for training and fine-tuning normalization models. to tackle this problem, our proposed solution uses word-level real-life variation data to create training data sequences and fine-tune multilingual sequenceto-sequence models. in this paper, we use byt5 @xcite and mt5 @xcite models and in addition, we benchmark the generative models gpt-4o and llama and a word-based luxembourgish correction pipeline, spellux. 1 we evaluate model performance using both quantitative metrics and a tailored qualitative evaluation of linguistic contexts. then, we compare bytebased, word-based and pipeline-based models to identify linguistic contexts in which models perform particularly well or struggle. our main contributions are therefore twofold: ## related work luxembourgish, a relatively small language, is less represented in nlp compared to its linguistic neighbors, french and german. research in nlp for luxembourgish has only recently gained momentum, with a few earlier works: adda-decker et al. @xcite introduced various resources for nlp tasks in luxembourgish; @xcite analyzed typical writing patterns (contextual n-deletion) in transcribed speech; and @xcite figure 1 : illustration of the creation of training data with the luxembourgish online dictionary (lod) sentence 'drink milk with honey, then your throat will no longer hurt.' and the variational statistical data for 'milk'. the algorithm processes every word sequentially, this illustrates only the replacement process for the word 'milk'. other developments include @xcite and @xcite , who tested various approaches for performing sentiment analysis on luxembourgish, including training bert-based models. @xcite proposed a new approach to zero-shot classification using a taskspecific dictionary for topic classification. for spoken data, @xcite and @xcite developed lux-asr, an efficient automatic speech recognition system for luxembourgish. additionally, @xcite fine-tuned language models for automatic comment moderation. some language models have been trained using transfer learning from german, such as luxgpt @xcite . other models developed for luxembourgish are luxembert @xcite and enrich4all (anastasiou, 2022). various t5 @xcite and byt5 ( xue et al., 2022 2018 ) developed an encoderdecoder architecture for text normalization in swiss german by using sequence-to-sequence models but not t5 architectures. similarly, bollmann (2018) worked on historical text normalization, comparing encoder-decoder architectures to statistical machine translations. for pipeline-based normalization, van der goot (2019) developed monoise, which has long been regarded as the state-of-the-art text normalization tool. this pipeline operates at a word level using a spelling correction module and word embeddings for various languages. furthermore, van der goot (2019) introduced the error reduction rate (err) as an evaluation metric for normalizers. monoise has also been used on the task of nested named entities in danish @xcite and for codeswitching data (van der goot and çetinoglu, 2021). for luxembourgish, purschke (2020) published spellux a pipeline for automatic orthographic correction of text data, the first text normalization tool for luxembourgish. ## methodology this section describes the methodology for finetuning and evaluating our normalization model for luxembourgish. this includes the creation of training data, the experimental setup for the model training and benchmarking, and the model evaluation process. ## results this section illustrates the results from the comprehensive quantitative evaluation and the linguistically-informed qualitative tests. the evaluated models include fine-tuned byt5 and mt5 models, generative models gpt and llama and the pipeline-based spellux. ## discussion automatic text normalization is a challenging task whose success depends on a number of factors, including a societally-anchored orthographic norm as the target of the correction task, the availability of large and standard-adherent datasets, suitable tech-nical approaches for implementing the task, and a thorough understanding of the respective strengths and weaknesses of each approach. given the current situation of written luxembourgish -with a standard under development and limited amounts of correctly spelled text -in this paper we investigate text normalization approaches and present a comprehensive evaluation suite. one of the challenges in developing text normalization tools is the use of synthetic data. while these are easy to produce based on existing corpora and orthographic rule sets, they do not represent the variation that would occur in real texts. to overcome these shortcomings, we present a new approach to generating training data with real-life variation data derived from actual texts written and corrected by writers of luxembourgish. this approach has a clear advantage over synthetic data or prompting llms, as it represents the variation space of a language realistically, according to the actual writing practices of its speakers. in particular, the combination of variants and their frequency of use allows the creation of training data that reflect the variation patterns found in real-life texts. another problem with automatic text normalization is model evaluation. given the large amount of variation found in written luxembourgish, our approach to model evaluation includes a comprehensive set of quantitative and qualitative tests. these allow for a more fine-grained and linguistically informed analysis of the model output, e.g. by comparing success rates for specific orthographic rules. in this way, our evaluation suite increases the transparency of traditional evaluation metrics. the results of the evaluation experiments show that the tested approaches not only perform differently in terms of quantitative success, e.g. err, but also show particular strengths and weaknesses for specific orthographic rules and contextual phenomena. in general, the latest version of gpt (october 2024) outperforms all other approaches, both model-based and pipeline-based. at the same time, the byt5 model presented in this paper and the spellux correction pipeline show individual strengths for certain sets of orthographic phenomena. nevertheless, we believe that working with a technical solution tailored to luxembourgish can be advantageous. first, our approach allows us to control all aspects of model training, i.e., training data, model parameters, and task implementation. second, the use of real-life variation data as a basis for model training brings our approach closer to the actual variation space found in writing practice. third, since the standard orthography is still under development, we can easily adapt and optimize our approach to future versions of the standard. fourth, by combining linguistic analysis and hyperparameter optimization, our approach offers great potential for future iterations. looking beyond the task of text normalization, our approach can also serve as a linguistic analysis tool for detecting and classifying variation patterns in written luxembourgish, for example in the context of the research project tracing attitudes and variation in online luxembourgish text archives (travolta). 13 using journalistic texts and user comments from the media platform rtl.lu, we can trace the development of individual as well as group-based writing practices outside the official spelling norm. since there is hardly any research on the development of the written domain in luxembourgish, the project can contribute to a better understanding of individual writing practices as well as the structure and dynamics of its variation space in general. ## conclusion in this paper, we present the first generative normalization model for luxembourgish by creating training data from real-life variation data. more importantly, we develop performance tests for this normalizer to achieve a comprehensive, linguisticallyinformed evaluation using both quantitative and qualitative metrics. for the creation of training data, we use a variant dictionary with frequency information to create parallel training data with incorrect and correct sentence pairs. this training data is then used to fine-tune a byt5 model and a mt5 model: the first sequence-to-sequence models fine-tuned for this task. additionally, benchmarking is performed to compare byte-based (byt5), word-based (mt5), llm-based (llama, gpt) and pipeline-based (spellux) approaches. furthermore, performance tests for luxembourgish text normalization offer a deeper insight into the strengths and weaknesses of the models, as we compare byt5, mt5 and spellux. as the performance of byt5 shows, our approach to the generation of training data is an effective method to train models while preserving a realistic variation space in the data. furthermore, the byt5 base model achieves comparable performances to other approaches with an err of 0.26. overall, this paper shows that normalization for luxembourgish is possible and achieves good results, either with prompting llms, using an already established pipeline, or with a byt5 architecture.