Generalising to G erman Plural Noun Classes, from the Perspective of a Recurrent Neural Network (2021)

Abstract: Inflectional morphology has since long been a useful testing ground for broader questions about generalisation in language and the viability of neural network models as cognitive models of language. Here, in line with that tradition, we explore how recurrent neural networks acquire the complex German plural system and reflect upon how their strategy compares to human generalisation and rule-based models of this system. We perform analyses including behavioural experiments, diagnostic classification, representation analysis and causal interventions, suggesting that the models rely on features that are also key predictors in rule-based models of German plurals. However, the models also display shortcut learning, which is crucial to overcome in search of more cognitively plausible generalisation behaviour.

## introduction language is a complex and mysterious system, which requires that speakers systematically generalise but also that they admit exceptions @xcite @xcite . a clear illustration of this is the domain of morphology, where suffixes and affixes can be productively used to express a particular grammatical property, but where there are also several words that follow irregular patterns for the same grammatical function. for example, while the past tense for most english verbs is formed by affixing -ed (@xmath0, the past tense of break is not breaked but broke. if an english speaker encounters an unknown form such as treak, they must decide whether to attach -ed, or instead go with the irregular form troke. precisely because of such intricacies, the computational task of acquiring a equal contribution. morphological system capable of generalisation has a strong historical connection to cognitive science (e.g. @xcite . making progress on how human minds process this interesting subdomain of language, however, is a challenging enterprise: probing the internal representations of human minds is difficult and, in some cases, potentially unethical. several researchers have therefore opted to instead investigate neural models that show generalisation behaviour that is similar to humans in key aspects and use them to learn more about human language processing (for a prime example of this cycle, see @xcite @xcite . using neural models in such a fashion starts with an accurate understanding of how they approach the phenomena of interest: without that, we are constrained in our attempts to use them to devise hypotheses about human language processing and compare them to existing theories @xcite . in that vein, we present a detailed examination of how recurrent neural networks (rnns) process the complex german plural system which -contrary to the classical example of english past tense -features generalisation of multiple classes @xcite @xcite . we ask what kind of representations such models learn, whether they support human-like generalisation, and we make a start with comparing their learnt solutions with existing models of german plural inflection. we train rnns to predict the form of a german plural noun (or its plural class) from its singular form and grammatical gender and present an elaborate investigation of the resulting models. we perform behavioural analyses ( §3) of the models' predictions as well as structural analyses of their internal representations, aimed at identifying the features that the models have learnt to associate with each of the classes ( §4). for the latter, we use diagnostic classifiers @xcite that we afterwards use to intervene in the model to establish causal connections between the internal encodings and the predictions the model generates from these encodings ( §5). we find that our networks show a mixture of cognitively plausible generalisation and reliance upon 'shortcuts' or heuristics @xcite (see figure 1 ). on the negative side, the model's ability to cope with nouns in low-frequency plural classes is very brittle. our behavioural analyses reveal that the models overrely on length as a heuristic to predict the rare class -s. however, we also find that our models correctly learn that key predictors of plural class include grammatical gender and the last few letters of a word, which are the same features considered by the recent decision-tree-based cognitive model of @xcite . our diagnostic classifiers additionally show how these predictors are encoded in the model's recurrent hidden representations. this interesting overlap between neural and rule-based models raises questions as to what neural models might teach us about the cognitive implementation of rule-based domains. 1 ## methods german plural inflection comprises multiple plural classes with different frequencies @xcite @xcite @xcite . most plural nouns are inflected with one of five suffixes @xcite : -(e)n, -e, -ø, -er and -s ( class example frequency length singular plural -(e)n frau frauen 44.7% 11.7 -e hund hunde 26.3% 11.0 -ø wagen wagen 16.9% 11.3 -er wald wälder 3.5% 10.4 -s auto autos 5.4% 8.0 -? cello celli 3.2% 10.6 and -ø sometimes combine with umlaut. 2 the literature on german plural generalisation has measured success rates based on correct suffixation (as opposed to a combination of the suffix and umlaut), and for simplicity, we keep this focus in the current study @xcite . ## diagnostic classification we now look into how and where the plural classes are encoded by the model. to do so, we use diagnostic, or probing classifiers (dcs, @xcite @xcite , commonly used to estimate the extent to which hidden representations of a neural model reflect a specific linguistic property. dcs are simple classifiers that are trained to predict that property from the representation. the dc's performance on new data is assumed indicative of whether the linguistic property was, in fact, encoded. 9 our experiments with dcs target: i) how well different hidden representations encode plural classes; ii) how the dc performance evolves over time while processing an input; iii) what is special about the hidden representations' neurons that are most salient to the dc. ## interventions until now, we have only been able to relate dc features to models' behaviour by making adaptations in the inputs fed to the model. to strengthen these results, we now ask: can we also change the models' behaviour without changing the input, by changing the input's hidden representation? to do so, we use interventions @xcite that halt the model while it processes inputs and change its representations using the dc. we monitor the effect to establish a causal link between the dc's results and the model's predictions. setup following @xcite , the dc findings are linked to models' behaviour by adapting the hidden representations through the signal provided by dcs, while monitoring the impact on the models' predictions. we perform interventions on the hidden and memory cell state from the final encoder time step, by running the rnn, halting it after the encoder processed the input and intervening on the decoder's initialisation before it predicts an output. assuming h is the hidden representation, we use the dc as follows: we perform interventions with respect to the true plural classes of samples from the validation set for which the prediction is well-formed but not correct, with α empirically set to 2.0. foot_5 results figure 10a summarises the impact of interventions during the fifth epoch. for wellformed model predictions that have been assigned the wrong plural class, we can change the model's prediction to the right class in up to 43% of the samples with target -(e)n confirming that the information detected by dcs is partially also used by the model. for -e, -ø and -er, a smaller, yet still substantial percentage can be corrected (18-25%). however, this comes at a cost; some previously well-formed predictions are no longer well-formed (-? in the figure). in most of these cases, though, it is not the plural class that was corrupted, but the rest of the noun -i.e. the intervention sometimes negatively impacts the decoder's ability to recover all of the noun's characters. that the dc has picked up on class-specific features can be deduced from the fact that the interventions either change the class to the correct one, or make the predictions less well-formed, but hardly ever cause the model to emit a different incorrect plural class. in many cases, intervening does not lead to any changes in the models' predictions, as shown in figure 10b . predictions belonging to the plural class -e are typically immune to interventions, which may, again, be due to the fact that two other classes (i.e. ## related work our analysis draws upon current research investigating neural model representations. we apply these techniques to german plural generalisation, a challenging domain with an extensive cognitive and linguistic literature. ## discussion & conclusion for more than 30 years, the field of inflectional morphology has been a testing ground for broader questions about generalisation in language, centred around the extent to which explicit rules are required. in this discussion, neural networks are traditionally considered as an alternative to the explicit representation of rules. however, recent studies have shown that such models show interesting generalisation patterns -sometimes comparable to behaviour observed in humans @xcite . this raises the question of what kind of solution is implemented by neural networks to process language in seemingly rule-governed domains, how these solutions relate to rule-based models, and what it teaches us about human processing of inflectional morphology. our study takes a step in this direction by exploring how an rnn encodes generalisation behaviour. we find that an rnn shows a mixture of humanlike generalisation and reliance upon 'shortcuts'. the models correctly learn that key predictors of plural class include grammatical gender and the last few letters of a word, which are the same features used by the recent rule-based cognitive model of @xcite . our dcs show how these predictors are largely encoded in the hidden representations of the encoder. variation in the classifiers' performance may reflect that some plural classes are encoded more consistently than others; for instance, feminine gender is highly predictive of the -(e)n class. alternatively, the decoder may decide the plural class for some inputs. on the other hand, the models' ability to cope with nouns in low-frequency plural classes is very brittle. the dcs perform worst for the minority classes, it proved hard to change the model's predictions to -s in the interventions, and behavioural analyses suggested the model overly relies on length as a shortcut to predict this class. by contrast, we see model bias for the frequent class -e in overgeneralisation behaviour ( §3), in representation space dispersion ( §4), and in resistance to interventions ( §5). we speculate that the characterbased rnn may conflate the -e class with the 'e' character that appears in the -(e)n and -er classes. in summary, we contribute a detailed analysis of how an rnn processes the complex task of plural inflection in german. interestingly, we find cognitively plausible generalisation behaviour through learnt representations which echo recent rule-based models. future work could address the broader questions raised by these findings, such as what constitutes a rule given overlap in strategy between neural and rule-based models, and how a mechanistic understanding of how neural networks approach seemingly rule-governed domains might contribute to understanding how such generalisation is instantiated in the human brain.