Lemma Dilemma: On Lemma Generation Without Domain- or Language-Specific Training Data (2025)

Abstract: Lemmatization is the task of transforming all words in a given text to their dictionary forms. While large language models (LLMs) have demonstrated their ability to achieve competitive results across a wide range of NLP tasks, there is no prior evidence of how effective they are in the contextual lemmatization task. In this paper, we empirically investigate the capacity of the latest generation of LLMs to perform in-context lemmatization, comparing it to the traditional fully supervised approach. In particular, we consider the setting in which supervised training data is not available for a target domain or language, comparing (i) encoder-only supervised approaches, fine-tuned out-of-domain, and (ii) cross-lingual methods, against direct in-context lemma generation with LLMs. Our experimental investigation across 12 languages of different morphological complexity finds that, while encoders remain competitive in out-of-domain settings when fine-tuned on gold data, current LLMs reach state-of-the-art results for most languages by directly generating lemmas in-context without prior fine-tuning, provided just with a few examples. Data and code will be made available upon publication.

## introduction lemmatization is one of the core nlp tasks widely used during data pre-processing in such areas as information extraction, named entity recognition, and sentiment analysis, and is of particular importance for languages with complex morphology. to lemmatize a word means to transform its inflected form (e.g. chose, chosen) into its dictionary-based form, also known as lemma (e.g. choose), according to the definition of the contextual lemmatization task in @xcite . most recent approaches tend to address contextual lemmatization as a supervised classification approach, which was first proposed by chrupala et al. @xcite and became the core idea in the architecture of a variety of contextual lemmatizers @xcite @xcite . this method learns to determine the minimum number of edits necessary to convert the word into its lemma. such techniques require a large amount of annotated data, which can be especially challenging for languages with rich morphology @xcite . apart from that, the majority of lemmatization systems are evaluated mostly in-domain, while their real application is almost always out-ofdomain, namely, out of the scope of the data they have been trained on. previous work has demonstrated that lemmatizers' performance worsens substantially when deployed out-of-domain @xcite . the recent generation of large language models (llms) has exhibited strong capabilities on a wide variety of nlp tasks, such as reasoning, problemsolving, code generation, information extraction, and text composition @xcite @xcite @xcite . however, as it has also been claimed, when llms are evaluated on languages other than english or other high-resource languages (e.g., spanish), their per-formance is not as good as expected @xcite @xcite . as for the contextual lemmatization task, to the best of our knowledge, there is no empirical evidence on the capacity of such models in generating the correct lemmas, especially for high-inflected and low-resource languages. the scarcity of manually annotated data is another problem for training competitive contextual lemmatizers. the attempts to create large annotated corpora, such as universal dependencies @xcite and the unimorph project @xcite , aim to bridge this resource gap, but they still cover a very limited scope of languages and domains @xcite . in response to data scarcity, model-and data-based cross-lingual transfer for sequence labeling have been proposed @xcite @xcite . these approaches focus on overcoming the lack of data in a target language by either fine-tuning a pre-trained multilingual model on a source language (usually english) in order to make predictions for any of the target languages included in the model pre-training (model-transfer); or automatically generating labeled data for the target language (data-transfer), which is then used to train a sequence labeling model. considering the lemmatization challenges mentioned above, in this paper we raise the following research questions (rq): rq1: to what extent can the latest generation of large language models directly obtain lemmas in the target language without prior fine-tuning, especially for the languages with complex morphology? rq2: could in-context lemma generation, produced by llms, be an alternative in solving the lemmatization task out-of-domain? rq3: what is the best strategy in the scenario where the data in the target language or domain is non-existent or difficult to access? the setup to investigate these rqs is illustrated in figure 1 . hence, our contributions are the following: (i) we empirically investigate the ability of llms to perform in-context lemma generation across languages of different morphological complexity; (ii) we address the out-of-domain problem by comparing the performance of llms against encoder-only models fine-tuned on different data distribution, and (iii) we conduct a comparative analysis of the methods to overcome the data scarcity in the target language for the lemmatization task, namely, model-transfer, data-transfer and direct lemma generation, using the data in english language as a source. overall, our results suggest that, while finetuning encoders on gold data remains a competitive option for out-of-domain settings, generative llms reach state-of-the-art results in lemmatization by directly generating lemmas in-context without prior fine-tuning, provided just with a few examples. ## related work approaches to perform lemmatization evolved from being rule-based language-dependent systems @xcite @xcite @xcite to advanced multilingual architectures, trained on a large amount of annotated data @xcite @xcite . the idea of addressing lemmatization as a sequence labeling task, where the labels are induced as the minimum amount of edits necessary to convert an inflected word into its lemma, was first proposed by @xcite and has been adopted in a wide variety of systems such as the system by gesmundo and samardžić (2012), ixa pipes @xcite , lemming @xcite , the system of @xcite and morpheus @xcite , among others. the advancement of supervised techniques involving deep learning algorithms, and the development of the transformer architecture @xcite and transformer-based masked language models (mlms) such as bert @xcite and the multilingual xlm-roberta @xcite , have massively improved the performance of current contextual lemmatizers. thus, in the sigmorphon 2019 shared task @xcite on contextual lemmatization, most of the participant systems were based on mlms @xcite @xcite . the evaluation of contextual lemmatizers is almost always performed in-domain, namely, on the same data distribution used during the fine-tuning process. however, in practice, lemmatizers are usually deployed out-of-domain, which results in a significant performance degradation, especially for high-inflected languages (toporkov and @xcite . the rise and constant development of llms has demonstrated remarkable abilities in dealing with a wide variety of nlp tasks such as language understanding, reasoning, language generation, code generation, and query response, especially when using a few-shot learning approach @xcite @xcite . model families such as llama @xcite , generative pre-trained transformer (gpt) @xcite , qwen @xcite , claude @xcite , mistral @xcite or gemma @xcite , to name but a few, are experiencing continuous growth, offering llms with different parameter sizes and evaluated on various downstream tasks. nevertheless, there seems to be a noticeable difference in the performance of such models depending on the resources of the language and its complexity. @xcite demonstrate that there is a disparity in the performance of llms between agglutinative and fusional languages, giving an advantage to high-resource languages like english over more morphologically complex languages such as turkish. this is attributed to tokenization quality, stating that morphological alignment does not influence the model's performance. in one of the first systematic analyses of the morphological capabilities of llms for the tasks of inflection and reinflection, models such as chat-gpt are still far from achieving state-of-the-art results, performing on the level of some older supervised models @xcite . in their multilingual version of the wug test, @xcite indicate that llms can apply their morphological knowledge to previously unseen words and that the morphological complexity of languages is more important than their relative representation in the training data. this fact states the importance of morphology for improving low-resource language modeling. furthermore, it has also been demonstrated that llms could be used for lemma disambiguation in a dictionary-augmented approach for the endangered languages such as erzya and skolt sami (hämäläinen, 2024), reaching the level close to a human annotator. nonetheless, to the best of our knowledge, no empirical work has studied the ability of llms to generate correct lemmas. the lack of quality annotated corpora for many target languages led to the exploration of modeltransfer and data-transfer techniques. modeltransfer is based on the cross-lingual capabilities of multilingual pre-trained mlms, where the knowledge in the source data can be transferred to the target language @xcite . data-transfer aims to automatically produce labeled data for the target language, traditionally based on translation and annotation approaches @xcite . both techniques have demonstrated competitive results in the tasks of cross-lingual transfer for various sequence labeling tasks @xcite , although lemmatization has not been studied from a cross-lingual transfer perspective. ## materials and methods in this section, we describe the datasets and models we use for our experiments. we also describe the prompting method we apply to perform in-context lemma generation. ## experiments in this paper, we aim to answer the following questions: (rq1) what are the capabilities of the latest generation of llms to perform in-context learning for lemmatization? (rq2) can we use direct lemma generation to address the problem of poor out-of-domain performance, namely, when evaluated on a data distribution different from the one seen during training? (rq3) what is the optimal strategy in a scenario in which annotated training data in the target domain or language is not available? to address these questions, we conduct three sets of experiments. each experimental setup and its corresponding results are described below in a separate subsection. evaluation is performed using average word and sentence accuracy metrics. 1 we conduct 3 runs and report the average results for all experiments and models, except for claude-3.7-sonnet, where only 1 run was performed due to computational costs. to assess whether the observed differences in model performance are statistically significant, we apply mcnemar's test @xcite . ## discussion although llms show promising results in incontext lemma generation for the selected languages, their performance depends heavily on the provided examples. in prompt development across 4 languages of varied morphological complexity, the differences between the basic in a zero-shot setting and the 4-shot prompt were quite significant. we identified a number of problems when using llms for direct lemma generation, such as: (i) randomly generated output. llms such as mistral-large-instruct-2407 and claude-3.7-sonnet tend to randomly generate quotation marks, but there are also cases, exhibited for english (ewt corpus), where claude-3.7-sonnet provided the output for the same sentence more than once. in the case of llama-3.3-70b-instruct, the model gives additional explanations, for instance, in the case of ambiguous basque auxiliary verbs such as edun and izan. (ii) modified wordforms. despite the explicit instructions on not performing any changes to the initial word, even if it is misspelled, llms tend to ignore them (this was the case during the experimental prompting phase with mistral-large-instruct-2407). apart from that, it is common to lowercase the input words at the beginning of the sentence. (iii) arbitrarily skipping words. this was the most interesting observation, as there was no perceivable pattern of which words the model was skipping. for instance, for basque mistral-large-instruct-2407 ignored verbal forms such as 'egingo' ('will do' in english); (iv) struggling with certain lemmas that do not appear in the few-shot examples. this case was observed for articles in spanish for which, instead of providing the lemma for definite articles such as 'el', llms were returning the same word form of the determiner given in the input text (e.g., la, los or las). this behaviour could be changed by adding an example in the prompt to deal with these particular cases. model scale also plays a crucial role in the quality of in-context lemmatization. these are only some of the most common observations obtained during the analysis of llms' performance on contextual lemmatization. the quality of examples used in the prompt design plays an important role, and careful and elaborated example selection may be specifically beneficial for low-resource languages. a deeper analysis should be conducted regarding other potential pitfalls of llms for this task, especially for languages with more complex morphology. qwen-2.5-7b qwen-2.5-32b qwen-2.5-72b ministral-8b mistral-li-2407 language wacc sentacc wacc sentacc wacc sentacc wacc sentacc wacc sentacc in a few-shot setting. we also investigate the scenario in which no training data is available for a given language. comparing model-transfer, datatransfer, and direct lemma generation with llms, we conclude that the best lemmatization approach in such a case would also be direct in-context lemma generation, which would remain predominantly achievable with large-scale language models. finally, future work should include model comparison, broader linguistic sampling, and comprehensive prompt optimization. lemmatization is a task generally studied and evaluated in-domain, which is rather surprising as the use of lemmatizers is predominantly out-ofdomain @xcite . our work demonstrates, for the first time, the potential to perform direct lemmatization directly without any training data by applying direct in-context learning with large size llms, even for high-inflected relatively low-resource languages.