Counting the Bugs in C hat GPT ’s Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model (2023)

Abstract: Large language models (LLMs) have recently reached an impressive level of linguistic capability, prompting comparisons with human language skills. However, there have been relatively few systematic inquiries into the linguistic capabilities of the latest generation of LLMs, and those studies that do exist (i) ignore the remarkable ability of humans to generalize, (ii) focus only on English, and (iii) investigate syntax or semantics and overlook other capabilities that lie at the heart of human language, like morphology. Here, we close these gaps by conducting the first rigorous analysis of the morphological capabilities of ChatGPT in four typologically varied languages (specifically, English, German, Tamil, and Turkish). We apply a version of Berko’s (1958) wug test to ChatGPT, using novel, uncontaminated datasets for the four examined languages. We find that ChatGPT massively underperforms purpose-built systems, particularly in English. Overall, our results—through the lens of morphology—cast a new light on the linguistic capabilities of ChatGPT, suggesting that claims of human-like language skills are premature and misleading.

## introduction do large language models (llms) possess humanlike linguistic capabilities? with the advent of the latest generation of llms such as @xcite , llama @xcite , and palm @xcite , there appears to be growing evidence for answering this question with yes @xcite : llms are capable of generating text that crowdworkers cannot distinguish from human-generated text @xcite and excel at linguistic probing tasks such as predicting grammaticality, detecting the subject and tense of clauses, and identifying the grammatical number of subjects and objects @xcite . despite these encouraging results, the existing body of work has so far examined a relatively limited part of the full spectrum of phenomena that are known to characterize human language, with a heavy focus on syntax and semantics. one area that has been neglected in particular is morphology, i.e., the capacity to create words according to systematic patterns of covariation in form and meaning @xcite . this gap in the llm literature is noteworthy given that morphology has been a hallmark of research on computational approaches to language since the very beginnings of neural language processing in the 1980s @xcite . in this study, we present the first systematic analysis of the morphological capabilities of llms, fo-cusing on chatgpt (openai, 2023a) as the most prominent and most widely-used llm. specifically, we investigate chatgpt's morphological capabilities using the wug test @xcite , an experimental paradigm in which a participant is asked to provide an inflected or derived form of a nonce word. an example for our evaluation setup is given in figure 1 . our experiments cover a broad range of morphological constructions and four typologically diverse languages: english, german, tamil, and turkish. we find that chatgpt falls short not only of human performance but also of various supervised baselines. in sum, our contributions are as follows: ## data and morphological constructions in this paper, we examine chatgpt's morphological behavior on a typologically diverse set of languages: english, german, tamil, and turkish. while english and german belong to the same language family, german has a more fusional morphological system than english. turkish is chosen since it is a non-indo-european language with a fully agglutinative morphology. tamil is chosen since it is a dravidian language exhibiting an agglutinative morphology with fusional elements. thus, in terms of the classical triangle of fusional, isolating, and agglutinative morphologies @xcite , the languages cover four different points: almost fully isolating (english), intermediate between isolating and fusional (german), intermediate between fusional and agglutinative (tamil), and fully agglutinative (turkish). furthermore, the chosen languages also cover different points in the spectrum from low-resource to high-resource, enabling us to form hypotheses about the impact of the amount of language-specific training data on the morphological capabilities of an llm. statistics for the amount of data in train, dev, and test for the baselines, as well as the number of wug test words, are given in ## methodology we compare the outputs of chatgpt under a variety of prompting regimens and a substantial set of supervised baselines (both neural and non-neural) to human annotations of the data described in appendix 3. results are evaluated using accuracy at k (acc@k), i.e., a model's response is regarded as correct if it is in line with any of the top k human responses. this evaluation method takes into account inter-speaker morphological variability, which is more wide-spread than previously thought @xcite . ## future directions morphological patterns are only one kind of generalization that can be investigated through a wuglike experimental paradigm. the form-meaning relationships encoded in language and multimodal models, including constructional and iconic pairings, can be investigated through prompting with nonce stimuli, leading to new insights regarding the generalizations they capture.