Morphology Matters: Probing the Cross-linguistic Morphological Generalization Abilities of Large Language Models through a Wug Test (2024)

Abstract: We develop a multilingual version of the Wug Test, an artificial word completion experiment that is typically used to test the morphological knowledge of children, and apply it to the GPT family of large language models (LLMs). LLMs’ performance on this test was evaluated by native speakers of six different languages, who judged whether the inflected and derived forms generated by the models conform to the morphological rules of their language. Our results show that LLMs can generalize their morphological knowledge to new, unfamiliar words, but that their success in generating the “correct” generalization (as judged by native human speakers) is predicted by a language’s morphological complexity (specifically, integrative complexity). We further find that the amount of training data has surprisingly little on LLMs’ morphological generalization abilities within the scope of the analyzed languages. These findings highlight that “morphology matters”, and have important implications for improving low-resource language modeling.

## introduction large language models (llms) have been very successful in learning and generating grammaticallycorrect language as humans do @xcite . this poses the question of whether they actually have linguistic capability that would allow them to generalize beyond the training distribution @xcite . in addition, does this capability manifest differently in different languages that llms were trained on? here, we investigate whether llms' linguistic knowledge with respect to morphology differs between languages. specifically, we test the ability of multilingual llms to generalize their morphological knowledge to nonce words in six languages. testing cross-linguistic differences in the morphosyntactic abilities of llms trained on large amounts of human-generated text is particularly interesting given recent findings on the behavioral similarity between humans and language models in a variety of language learning and processing tasks @xcite @xcite and syntactic structure in the models' learned attention patterns @xcite . one of the key concerns of contemporary efforts in language modeling is to improve the ability to generalize well across the variety of human languages, especially regarding low-resource languages (e.g., @xcite @xcite @xcite . given the importance of the training data to llm's abilities @xcite , conventional wisdom would suggest that the amount of exposure to a given language would be the dominant factor in determining the models' ability to learn the language's morphological patterns. here, we argue that factors beyond the amount of training data play an important role for llms' generalization abilities, and in particular suggest that languages' morphological complexity needs to be taken into account. notably, languages vary in their degree of morphological complexity @xcite @xcite , for example in the number of morphological inflection paradigms and their degree of irregularity. a recent study has shown that llms, like humans, are particularly sensitive to the degree of compositional linguistic structure in their input when generating novel forms to new meanings in a matched experiment using a miniature artificial language, with higher degrees of compositionality leading to more systematic generalizations and to a higher agreement with humans @xcite . this finding implies that the morphological learning ability of llms across different human languages should similarly be affected by languages' degree of systematic morphological structure, as quantified by measures from typological linguistics @xcite . in the current paper, we test to what extent languages with more systematic structures are indeed learned better by llms using an established morphological knowledge test used in the field of child language acquisition: the wug-test @xcite . even though morphology is heavily studied in the field of computational linguistics (e.g. @xcite @xcite , and despite its importance to human language learning @xcite @xcite @xcite , there is little work on the cross-linguistic morphological knowledge of llms, especially with respect to the potential effect of languages' morpho-syntactic structure @xcite . rather, it has been found that llms often fail to generate the correct inflected forms of words that were not a part of their training data, regardless of the size of the training set @xcite . given that only one study to date has probed llms' morphological generalization abilities with a multilingual variant of the wug test @xcite , it is currently unclear to what extent can llms generalize their morphological knowledge to new contexts, and to what extent their generalization capabilities are affected by the morphological complexity of language compared to its representation in the training data. here, we take one step further in this line of work and test the relationship between languages' morphological structure and the generalization ability of multilingual llms. specifically, as shown in figure 1 , we develop a multilingual version of the wug test, an artificial word completion test that is typically used to probe the morphological knowledge of children with respect to inflectional and derivational morphology @xcite , and apply it to the gpt family of large language models @xcite . we consider six different languages, namely german, vietnamese, portuguese, spanish, french, and romanian, which vary in their degree of morphological complexity based on several established measures @xcite . for each language, we first employed gpt-4 to translate 23 questions with nonce words from the original wug test. the translations were then evaluated and corrected by linguistically-trained native speakers, and the nonce words were adapted to fit each language's phonotactic rules. llms were then provided with the translations as prompts (e.g., "this is a wug. now there are two of them. there are two __"), and were prompted to generate the missing inflected form (e.g,. "wugs"). since the nonce words are new, unfamiliar words, the models need to generalize their morphological knowledge beyond their training data. the model responses were then evaluated by native speakers, who judged whether the inflected and derived forms generated by the llms conform to their native language's morphological rules. we then tested llms' generalization success across languages against two measures of morphological complexity, namely, the richness of the morphological system and how irregular it is. our work complements this endeavour in several aspects: first, we opt to manually evaluate every response from the llms instead of an automated evaluation strategy (which took on the first word of the model response). second, we analyze a different set of languages, with only german overlapping across studies. and third, we test the impact of other important factors such as languages' morphological complexity scores. the effect of morphological complexity on language modeling some studies have explored the relationship between morphological complexity and the learnability of languages by llms, but show mixed results. @xcite estimated the predictability of text in a parallel corpora of 21 languages, and found that text in languages with rich inflectional morphology (and thus higher word entropy) was more difficult to predict by n-gram language models and lstm-based language models. however, when @xcite use a similar approach with three times more languages and more diverse language families, they did not find a correlation between prediction difficulty and the number of inflectional distinctions that languages have. @xcite further showed a positive correlation between multilingual language models' perplexity (how well a language model is able to predict the next work) and type/token ratios (i.e., the ratio between the number of word types and the total number of tokens in the text). more recently, park et al. ( 2021 ) used an even larger parallel corpus of 92 languages, and incorporated more measures of morphological complexity -including corpusbased measures and features from the world atlas of languages structure (wals) @xcite . using surprisal as an estimate for difficulty, they found that models' performance was correlated with several complexity measures, and that this correlation was stronger for language models whose tokenizer relied on bytepair-encoding @xcite . together, these studies imply that language learnability by llms is potentially affected by at least some of the specific morphological features of languages, though which features (and which metrics can capture them) is largely unknown -a question on which we aim to shed new light here. ## background on measuring morphological complexity languages vary in the degree of morphological complexity, which can be measured using a variety of tools (dryer and haspelmath, 2013) and dimensions @xcite . morphological complexity measures can be categorized into integrative complexity (i-complexity) and enumerative complexity (e-complexity) @xcite . e-complexity refers to the number of cases and inflectional paradigms that exist in a language's grammar. the more inflected forms a language can have (e.g., for gender, number, tense, case, mood etc.), the higher its e-complexity score is. i-complexity refers to the predictability of inflected form from its context. the more irregular a morphological paradigm is (e.g., many verbs in english show an irregular past tense inflection), i. e. how often irregular forms are used, the higher the i-complexity score. a well-known example of an e-complexity measure is lupyan and dale (2010)'s measure for morphological complexity, which was based on 28 morphological features extracted from the world atlas of language structure (wals, dryer and haspelmath, 2013), such as the number of inflectional distinctions. for i-complexity, @xcite introduced an information-theoretic measure to quantify the frequency of irregular forms, and @xcite proposed three measures to capture the the variety of word types used to encode identical information ("lexical diversity"). these measures include type-token ratio and shannon entropy (h), which measures the degree of uncertainty of words. the last measure is the zipf-mandelbrot parameter (α), based on the zipf's law of word distribution. languages with higher ttr and shannon entropy are more lexically diverse, and languages with a higher zipf parameter are less lexically diverse. for our study, we chose one representative measure for e-complexity and one for i-complexity, relying on previous comparative work that showed that different measures are highly correlated @xcite . for e-complexity, we use @xcite dale (2010 @xcite distinguished between four types of languages with respect to morphological structure. in our study, we consider two of them: inflected languages and isolating languages. while spanish, german, french, romanian, and portuguese are highly inflected languages, vietnamese is an isolating language which does not have explicit grammatical markers within word boundaries. we briefly describe the considered languages below. vietnamese is an isolating language and thus there are no bound morphemes in the form of suffixes and affixes. as such, there are no inflectional or derivational processes. instead, semantic and grammatical information is expressed using free morphemes (i.e., standalone words). for instance, vietnamese does not have plural word forms, but instead expresses plurality by adding a number word before the noun. french is an inflected language in the romance branch. verbal inflection is used to indicate tense, person, number, mood, and aspect. verbs are inflected such that they agree with the subject in terms of person and number. for example, the past tense formation process in french includes combining the correct conjugated form of the auxiliary verbs and the participle form of the main verb, which is formed by adding the correct ending morpheme to it. nouns carry number and grammatical gender, with number being governed by the endings of the nous. spanish is also inflected language belonging to the romance language family, which also includes french, portuguese and romanian. the choice of morphemes is governed by grammatical gender when inflecting nouns, pronouns, and adjectives. verbs are conjugated differently depending on whether the endings of the infinitive forms are -ar, -er, or -ir. they also include inflectional agreement with the person and number of the subject. another characteristic of spanish and other romance languages is that it has fusional morphology, such that a single word form can expresses various grammatical features. romanian, as another member of the romance family, is also highly inflected language with both nominal and verbal inflection, indicating a wide range of grammatical features. the inflected forms of nouns and adjectives are determined by the grammatical gender of the nouns as well as their endings. for verbss, there are 4 conjugation classes, depending on the endings of the infinitive forms. portuguese is an indo-european language in the romance branch, portuguese is also an inflectional language that bears similarity to spanish, although the exact number of possible distinctions/inflections and the degree of irregularity is different. for certain word endings (e.g., -s or -z), plural and singular portuguese forms are the same. german is an inflected and fusional language where affixes are added to the stem to convey grammatical information. such as number, case, aspect, and gender. there can be several affixes that encode the same grammatical information. the choice of affixes usually depends on the gender of the noun. if it is masculine, plurality is often expressed by adding -e. feminine nouns often end with -en. however, there are many additional rules in german, often involving changing the vowel to an umlaut (e.g., plural of "zug" is "züge"). ## discussion our goal was to investigate how well multilingual llms learn the underlying morphosyntactic structure of different languages and how this is influenced by languages' degree of morphosyntactic complexity. we did this by applying a wug test in 6 different languages, and evaluating the models' responses as a function of two measures of complexity, as well as the representation of the language in the training data. ## conclusion we tested whether languages' morphological complexity affected the performance of multilingual large language models on a classic language task. we ran the wug test in 6 languages and analyzed how task performance was affected by the degree of morphological complexity in each language. our results show that languages' morphological complexity (specifically, integrative complexity), is more important than its relative representation in the training data of large language models -a finding that challenges conventional wisdom and comes with important implications for low-resource language modeling. we have further identified additional error types beyond real-world bias such as english fall-back and inflection ignorance, whose cause we will explore in future work by investigating the role of tokenization.