LLM s’ morphological analyses of complex FST -generated F innish words (2024)

Abstract: Rule-based language processing systems have been overshadowed by neural systems in terms of utility, but it remains unclear whether neural NLP systems, in practice, learn the grammar rules that humans use. This work aims to shed light on the issue by evaluating state-of-the-art LLMs in a task of morphological analysis of complex Finnish noun forms. We generate the forms using an FST tool, and they are unlikely to have occurred in the training sets of the LLMs, therefore requiring morphological generalisation capacity. We find that GPT-4-turbohas some difficulties in the task while GPT-3.5-turbo struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.

## data and methods previous datasets of inflected finnish words include the morphynet @xcite and unimorph @xcite corpora. we chose not to use data from these datasets for two reasons. firstly, complex words comprising unusually many morphemes make it possible to assess if the systems can generalise to many types of possible inflections instead of learning only the most common inflection types. the previous datasets do not include many extremely complex word forms, but these can be generated using a finite-state transducer (fst). secondly, since the sota llms have been trained on very large datasets harvested from the internet, it is likely that the previously published datasets are included in their training data, which would preclude fair assessment. we use the omorfi tools @xcite that are based on finite-state morphology @xcite to generate inflected forms of finnish nouns. the omorfi library includes some 500k lexemes, of which about 140k are nouns. we inflect the nouns in all possible combinations of number, grammatical case, and possessive suffix (see uniform sampling of lexemes creates a bias towards low-frequency types that are correlated with regularity of the inflection @xcite . we note that this is the case in our data, as we took a random sample of the lexemes, and this should be kept in mind when interpreting the results; there are probably not many irregularly inflected words, which makes the task easier. this is not an issue, however, given our research question of whether the llms have picked up even the most systematic inflection types from textual data. ## results the rightmost plot in figure 1 shows that besides gpt-4-turbo, the models perform poorly in the task. gpt-4-turbo is not close to perfect accuracy either, and the combined 10-shot result does not reach the result achieved by simple rnns trained with 80k words. with training set sizes of 800, 4k, 8k, 40k, and 80k words, the rnns achieved accuracies of 0.380, 0.765, 0.774, 0.821, and 0.840, respectively. the first three plots from left in figure 1 break down the classification task into the three component classification tasks: number, case, and possessive suffix. there are some differences in the strengths of the models: llama outperforms gpt-3.5 in the possessive suffix classification task, while gpt-3.5 performs better for other classification tasks. in number classification, poro outperforms llama, although llama performs better in other tasks. figure 2 shows the confusion matrices for gpt-4-turbo classifications of cases for the 0-shot and 10-shot setups. from the 0-shot confusion matrix we can see that the model does predict all classes even though we did not provide it with the names of the classes we expected it to recognise. this is not surprising, since gpt-4-turbo has no difficulties if asked to inflect a finnish word in all cases and to provide the names of the cases. it is obvious that gpt-4-turbo has a fair amount of both declarative knowledge (metalinguistic knowledge; it knows the classes) and procedural knowledge (knows how to inflect the words) of the finnish morphology. therefore, the challenge in this task comes presumably from the need to generalise to infrequently used, morphologically complex word forms. ## conclusion we conclude that even a sota llm, gpt-4-turbo, does not model finnish morphology thoroughly enough to allow it to provide morphological analyses of rare and complex word forms with a high accuracy. contrasting this with its impressive text generation capacity suggests that it utilises some other language processing heuristics, which clearly overlap somewhat with morphological rules since it rarely produces incorrect forms, but which preclude human-level systematic generalisation on our test set. gpt-4-turbo outperforms models such as gpt-3.5-turbo and llama2-70b, however, by a large margin.