FST s vs ICL : Generalisation in LLM s for an under-resourced language (2025)

Abstract: LLMs have been widely adopted to tackle many traditional NLP tasks. Their effectiveness remains uncertain in scenarios where pre-trained models have limited prior knowledge of a language. In this work, we examine LLMs’ generalization in under-resourced settings through the task of orthographic normalization across Otomi language variants. We develop two approaches: a rule-based method using a finite-state transducer (FST) and an in-context learning (ICL) method that provides the model with string transduction examples. We compare the performance of FSTs and neural approaches in low-resource scenarios, providing insights into their potential and limitations. Our results show that while FSTs outperform LLMs in zero-shot settings, ICL enables LLMs to surpass FSTs, stressing the importance of combining linguistic expertise with machine learning in current approaches for low-resource scenarios.

## introduction large language models (llms) have been widely adopted to tackle many traditional nlp tasks. part of their success is attributed to their extensive pretraining, which enables them to generalize well across different domains and diverse linguistic structures. the rapid advancement of the field has led to techniques like in-context learning (icl), which have further showcased the impressive generalization capabilities of llms, allowing them to adapt to new tasks and domains with minimal training data, often requiring only a few examples. however, the effectiveness of these approaches remains uncertain in scenarios where pretrained models have limited prior knowledge of a language. this is particularly relevant for under-resourced languages, where training data is scarce or highly non-homogeneous. the performance of llms in these situations is not yet well understood, and it is unclear whether their impressive generalization capabilities can be replicated in all type of scenarios. this raises critical questions about the extent to which in-context learning and other recent innovations can bridge the gap in multilingual coverage. in this work, we examine the generalization of llms in under-resourced settings through the task of orthographic normalization across otomi language variants. we develop two approaches: a rule-based method using a finite-state transducer (fst) and an icl approach that provides the model with string transduction examples. we focus on the scenarios that are particularly difficult for a transducer-based approach. in-context learning. a paradigm where llms learn to perform tasks by recognizing patterns from a few provided examples (demonstrations). unlike supervised learning, it requires no parameter updates or separate training stage. icl enables models to make predictions by drawing analogies from the given context @xcite . it can be considered a form of generalization, but it has nuances compared to traditional machine learning @xcite . although the mechanisms that underlie and influence this type of learning are not fully understood @xcite , it has proven successful in several domains. icl can be applied in domains with minimal training data, requiring only a few examples. for example, multilingual tasks involving under-resourced languages such as machine translation and interlinear glossing @xcite @xcite . orthographic normalization this is the process of converting written text into a standardized form within a language. for many dominant languages with long-established writing conventions, this task is either well-solved or of limited concern. however, for numerous other languages, particularly those lacking a long tradition of standardization due to sociopolitical factors, normalization remains a significant challenge. these languages often exhibit high internal diversity, making the task even more complex. otomi is a group of languages spoken in central mexico (see appendix a for geographical distribution) that are part of the oto-manguean language family. there are around nine dialectal variants @xcite . otomi is an endangered language group (around 300,000 speakers) that faces a scarcity of nlp tools and digital resources, plus there is orthographic variability, with several standards in use by the speakers. automatically converting text across the different standards is a crucial upstream task for developing more advanced language technologies. in lowresource settings, common approaches for building normalizers include designing fsts based on linguistic expertise @xcite and applying neural models like seq2seq @xcite . ## findings and interpretation to compare model performance, we measure the error rate between predicted orthographic normalizations and gold standards using word error rate (wer) and character error rate (cer), the latter being particularly useful for morphologically rich languages @xcite . the overall results are shown in figure 1 , with detailed error-rate values reported in both settings, @xmath0, exhibit a similar trend: the rule-based fst approach outperforms state-of-the-art llms in orthographic normalization of otomi text when they are prompted in a zero-shot setting. this is notable since fsts are computationally lightweight compared to the extensive resources (data and infrastructure) required for training large neural models. despite their robustness across many tasks, these llms struggle to generalize to orthographic variations in an under-resourced language like otomi surprisingly, providing neural models with fewshot examples drastically improves their performance. models like gpt-4o show some of the worst performances in a zero-shot setting. still, with just 10 examples provided, the error rate decreases, outperforming fsts and becoming the best model for orthographic normalization. see for example, gpt-4o_zero (wer: 31.5% cer: 10.1%) vs. gpt-4o_few (wer: 11.2% cer: 2.6% ) in @xmath1. the plots show a clear trend: llms make more errors than fsts but surpass them in a few-shot setting, highlighting the effectiveness of icl. interestingly, the most recent models perform worst in zero-shot, contrary to expectations given their sophistication. we conjecture that they may prioritize specialized reasoning capabilities over multilingual flexibility. however, their icl capability remains remarkable. further exploration is needed. despite expectations, the ots+@xmath2, suggesting it was easier. we anticipated greater difficulty since neural models lacked source orthography information at test time. a possible reason is that otq and in-ali are similar, with few character transformations, making sentences in this norm relatively easy to normalize, even without source orthography information at test time. when inali is not the target norm, icl still helps, but llms don't consistently outperform the fst. in particular, llms struggle when the target is ots. we conjecture this is likely due to the greater availability of digital data in inali (a norm that represents an institutional effort to unify standards), whereas ots is among the least represented norms online, limiting the effectiveness of pretrained models to generate text when ots is the target. some studies suggest llms' sensitivity to input and target probabilities may affect their emergent capabilities, especially in rare languages and text sequences @xcite . ## conclusions we explored the generalization limits of llms in an orthographic normalization task, which included building a rule-based system for converting otomi text across norms and compared its performance against the neural approaches. the test set was designed to assess the models capacity to normalize cases that are difficult for a rule-based approach, i.e., code-switching and ambiguous orthographic rules. one of the main takeaways is that when working in a limited resource scenario, one can leverage knowledge of the language to build an fst, and this can be more effective than simply doing zeroshot prompting with sophisticated llms. however, once you have a fst you can use it to generate demonstrations of orthographic transductions across orthographic norms and use them to improve a neural model. our results showed that llms surpass fsts with just 10 examples in fewshot settings, they were particularly good in codeswitching cases. however, we found that when the target orthography has very limited digital representation, even with icl, llms struggle to produce normalized text with fewer errors than the fst. in summary, this highlights the potential of icl to generalize from limited data, reducing the reliance on extensive labeled datasets while reaffirming the value of linguistically informed data. this could be promising for many practical applications, including developing technologies for under-resourced languages.