A Comprehensive Comparison of Neural Networks as Cognitive Models of Inflection (2022)

Abstract: Neural networks have long been at the center of a debate around the cognitive mechanism by which humans process inflectional morphology. This debate has gravitated into NLP by way of the question: Are neural networks a feasible account for human behavior in morphological inflection?We address that question by measuring the correlation between human judgments and neural network probabilities for unknown word inflections. We test a larger range of architectures than previously studied on two important tasks for the cognitive processing debate: English past tense, and German number inflection. We find evidence that the Transformer may be a better account of human behavior than LSTMs on these datasets, and that LSTM features known to increase inflection accuracy do not always result in more human-like behavior.

## results we present experimental results in tables 1 , 2 , and 3 in terms of both inflection accuracy, and correlation with human behavior -our main focus. all correlations for neural models trained in this work are given with respect to model rating, and not the model production probability. we report results from training mgl on our data, and include the results reported k&c, c&al., and m&al. in the appropriate tables for reference. ## analysis we mainly analyze the correlation between (average) model ratings and human ratings. we find that the transformer correlates best with human ratings with few exceptions, indeed it attains a statistically significant positive correlation for all inflection classes in both languages, with the exception of /-∅/ in german. it is also highly accurate, as in previous work @xcite . regarding lstm architectural decisions, unsurprisingly, attention and bidirectionality typically increase accuracy in both languages. the positive effect of attention is similar for correlations with some exceptions. attention almost always leads to better correlations in german, with the interesting exception of /-(e)n/. given that humans rate /-(e)n/ most highly on average, the higher correlation could be because without attention, lstms are very sensitive to the high /-en/ frequency in the training set. the attentional lstms might learn the monosyllabic, neuter context that applies to the wugs, for which there are very few /-(e)n/ training examples. despite slightly higher accuracy for bidirectional lstms, unidirectional lstms tend to attain higher correlations with both human metrics for english, especially for the more frequent regular inflections. conversely in german, the bidirectional lstms correlate better for the more frequent /-(e)n/ and /-e/ classes, but unilstmattn correlates better for the rarer /-er/ and /-s/ classes. the dichotomy between just one highly productive class in english and several productive classes in german may explain the first observation: if unidirectional lstms overfit to the frequent class, then they might appear to correlate better in english, but not german. however, this would not explain the german class correlations for infrequent inflections, which could be explored in future work. ## conclusion we ask which neural architecture most resembles human behavior in a wug test. we introduce results on a wider range of architectures than previous work and find that the transformer, a state-of-theart model for morphological inflection, frequently correlates best with human wug ratings. despite this, a closer look at model ratings and productions on german plural inflection shows that neither model closely resembles human behavior. we also find that, while attention is crucial for lstm inflection accuracy, it does not always lead to higher correlations with humans. additionally, the often less accurate unidirectional model sometimes correlates better than its bidirectional counterpart, especially in the case of infrequent german plural classes. finally, while for some inflection classes more accurate models correlate better with humans, there is no clear relationship between the two metrics overall. future work might consider behavior when hyperparameters are tuned to maximize plausibility of the probability distribution rather than accuracy. additionally, these results motivate a closer look at the effect of lstm encoder directionality with respect to inflection class frequency.