Evaluating Morphological Compositional Generalization in Large Language Models (2025)

Abstract: Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. However, their linguistic generalization capabilities remain questionable, raising doubts about whether these models learn language similarly to humans. While humans exhibit compositional generalization and linguistic creativity in language use, the extent to which LLMs replicate these abilities, particularly in morphology, is under-explored. In this work, we systematically investigate the morphological generalization abilities of LLMs through the lens of compositionality. We define morphemes as compositional primitives and design a novel suite of generative and discriminative tasks to assess morphological productivity and systematicity. Focusing on agglutinative languages such as Turkish and Finnish, we evaluate several state-of-the-art instruction-finetuned multilingual models, including GPT-4 and Gemini. Our analysis shows that LLMs struggle with morphological compositional generalization particularly when applied to novel word roots, with performance declining sharply as morphological complexity increases. While models can identify individual morphological combinations better than chance, their performance lacks systematicity, leading to significant accuracy gaps compared to humans.

## introduction large language models (llms) have recently achieved remarkable advances in the broad domain of natural language generation and understanding tasks @xcite @xcite @xcite . however, these models have also been shown to lack strong linguistic generalization capabilities @xcite ., * equal contribution † equal supervision id word ood word bound morphemes morphological productivity kişi nisi ler lik in kişiliklerin nisiliklerin model morphological systematicity kişiliklerin kişilerlikin model nisiliklerin nisilikinler ✔ ✘ @xmath0 figure 1: our morphological generalization tasks illustrated with an example in turkish. id and ood refer to in-distribution and out-of-distribution respectively. english translations are not part of the task and only shown here for illustrative purposes. recent works evaluating morphological generalization in language models have only focused on the productivity aspect with a limited coverage of inflectional forms @xcite . in this work, we address this gap by systematically investigating the morphological generalization abilities of llms through the lens of compositionality. following @xcite , we define the morphemes (smallest meaningful units in a language) foot_0 as the compositional primitives and design a novel suite of generative and discriminative language tasks based on the morphological combinations of these primitives. these tasks aim to test morphological productivity (ability to produce novel well-formed combinations of morphemes) and systematicity (ability to systematically understand novel combinations) respectively. figure 1 illustrates an example of both tasks. we evaluate several state-of-the-art instructionfinetuned large multilingual models on these tasks: gpt-4, gemini-1.5, aya-23 and qwen-2.5. to ensure our findings are not language-specific, we experiment with two morphologically rich (i.e. characterized by a large number of inflectional and derivational forms) languages: turkish and finnish. both languages share typological features (e.g. agglutination) despite being unrelated. we find that llms lack human-like morphological compositional generalization ability in agglutinative languages despite their high performance on various tasks in these languages. our analysis shows that morphological productivity, especially when applied to novel word roots is highly challenging for llms. moreover, as the morphological complexity of words increases the model performance sharply decreases (to nearly zero) while human performance is not consistently affected. on the systematicity task, while models perform much better than chance in identifying the validity of individual morphological combinations, however, this behaviour is not robust or systematic i.e. models fail to consistently determine validity of several compositions made up of the same set of morphemes. in summary, our contributions are as follows: 1) we design novel morphological generalization tasks that require compositional processing. 2) we prepare specific test suites in both turkish and finnish to measure morphological generalization and make these available for future research foot_1 . 3) using our novel tasks and test suites, we conduct a systematic analysis of morphological compositional generalization abilities of llms. 4) our findings reveal a systematic gap in llm's ability compared to humans concerning morphological generalization in agglutinative languages that also requires compositionality. ## conclusion in this paper, we proposed a novel experimental paradigm to test morphological generalization abilities of large language models through compositionality. our tasks target measuring morphological productivity and systematicity in a given language. we applied these tasks on the morphologically complex languages of turkish and finnish and evaluated morphological compositional generalization abilities of several state-of-the-art large language models. our experimental results and analysis re-veal a significant gap in the performance of llms compared to humans with respect to generalization in morphology of agglutinative languages.