Inflecting When There’s No Majority: Limitations of Encoder-Decoder Neural Networks as Cognitive Models for G erman Plurals (2020)

Abstract: Can artificial neural networks learn to represent inflectional morphology and generalize to new words as human speakers do? Kirov and Cotterell (2018) argue that the answer is yes: modern Encoder-Decoder (ED) architectures learn human-like behavior when inflecting English verbs, such as extending the regular past tense form /-(e)d/ to novel words. However, their work does not address the criticism raised by Marcus et al. (1995): that neural models may learn to extend not the regular, but the most frequent class — and thus fail on tasks like German number inflection, where infrequent suffixes like /-s/ can still be productively generalized. To investigate this question, we first collect a new dataset from German speakers (production and ratings of plural forms for novel nouns) that is designed to avoid sources of information unavailable to the ED model. The speaker data show high variability, and two suffixes evince ‘regular’ behavior, appearing more often with phonologically atypical inputs. Encoder-decoder models do generalize the most frequently produced plural class, but do not show human-like variability or ‘regular’ extension of these other plural markers. We conclude that modern neural models may still struggle with minority-class generalization.

## introduction morphology has historically been the site of vigorous debate on the capacity of neural models to capture human speaker behavior, and hence ground claims about speaker cognition. in 1986, rumelhart and mcclelland described a neural network model which learned to map english present tense verbs to their past tense forms. importantly, the network handled both regular verbs, whose past tense is formed systematically by adding the suffix /-(e)d/ (e.g. jumped), and irregular verbs where the present and past tenses bear no systematic relationship (e.g. ran). the authors suggested their model provided "an alternative @xcite to the implicit knowledge of rules" @xcite , a claim which sparked considerable controversy. @xcite highlighted many empirical inadequacies of the rumelhart and mcclelland model, and argued that these failures stemmed from "central features of connectionist ideology" and would persist in any neural network model lacking a symbolic processing component. recently, however, kirov and cotterell (2018, henceforth k&c) revisited the english past tense debate and showed that modern recurrent neural networks with encoder-decoder (ed) architectures overcome many of the empirical limitations of earlier neural models. their ed model successfully learns to generalize the regular past tense suffix /-(e)d/, achieving near-ceiling accuracy on held-out test data. moreover, its errors result from overapplication of the regular past tense (e.g. throwthrowed)-a type of error observed in human language learners as well-as opposed to the unattested forms produced by @xcite mcclelland's model. k&c conclude that modern neural networks can learn human-like behavior for english past tense without recourse to explicit symbolic structure, and invite researchers to move beyond the 'rules' debate, asking instead whether the learner correctly generalizes to a range of novel inputs, and whether its errors (and other behavior) are human-like. this challenge was first taken up by @xcite , who showed that, on novel english-like words designed to elicit some irregular generalizations from humans, the ed model's predictions do not closely match the human data. while these results suggest possible problems with the ed model, english may not be the best test case to fully understand these, since the sole regular inflectional class is also by far the most frequent. in contrast, many languages have multiple inflectional classes which can act 'regular' under various conditions @xcite . in this paper, we examine german number inflection, which has been identified as a crucial test case for connectionist modeling @xcite @xcite . the german plural system features eight plural markers (c.f. because the design and results of previous human studies have been somewhat inconsistent, and because we want to compare to fine-grained results from individuals (not just published averages), we first collect a new dataset of plural productions and ratings from german speakers. our speaker data show high variability: no class holds a majority overall, and two less frequent suffixes show a relative preference for phonologically atypical inputs ("non-rhymes"). we then compare our human data with the predictions of the encoderdecoder (ed) model proposed by k&c. while our human data paint a more complex picture of the german plural system than m95 claimed, nevertheless m95's central idea is borne out: when given non-rhymes, the ed model prefers the most frequent plural class, but speakers behave differently. this finding reveals that while modern neural models are far more powerful than earlier ones, they still have limitations as models of cognition in contexts like german number inflection, where no class holds a majority. the model may correctly identify the most frequent class, but fails to learn the conditions under which minority classes are productive for speakers. ## study 1: speaker plural inflection to evaluate whether neural models generalize correctly, we need to compare their behavior with that of humans on the same task. unfortunately, no existing datasets were suitable, so our first study asks how german speakers inflect novel nouns. ## study 2: encoder-decoder inflection our second study trains an encoder-decoder (ed) model on the task of german plural inflection, following the method of @xcite . we then compare its predictions on the m95 stimuli to the behavior of participants in study 1. ## discussion the current study asks whether modern encoder-decoder neural models learn the full set of correct generalizations -that is, human-like behaviorwith respect to german number inflection, which requires the learner to generalize non-majority inflectional classes. the short answer is no: our model learns part of that set. in particular, it correctly identifies /-e/ as the 'best' plural class for this context. /-e/ is the most frequent class in the training data for similar inputs (neuter gender, monosyllabic, phonologically close to m95; c.f. this outcome is not surprising when one considers that the model is trained to produce one correct form rather than a distribution over plausible forms; however, this is exactly the task faced by human language learners as well. all the models of morphology discussed here assume that exposure to correct forms alone should suffice for learning speaker-like behavior. @xcite ) note that training on single target forms produces highly skewed ed model scores, with a great deal of probability mass on the top-ranked form and instability in lower rankings, but that training on a distribution would not be a cognitively plausible alternative. however, it could be the case that german speakers do regularly encounter variable realizations of plural forms. köpcke observes that german plural inflection shows regional variation, for example northern speakers using /-s/ (die mädels 'girls') where southern dialects prefer /-(e)n/ (die mädeln). incorporating dialect-informed variability into training might be one way to encourage neural models toward speaker-like generalization. 13 parallel issues arise for model evaluation: how should we evaluate models of production when the target output is a distribution? on simplified versions of the task, such as classification @xcite , the output distribution is constrained within a space of plausible forms, but sequence-to-sequence models deal with the open-ended domain of all possible strings. for encoder-decoders, the likelihood scores produced during beam-search decoding offer an intuitive option, and k&c use these scores to evaluate their model with respect to @xcite demonstrate that these model scores are not a suitable metric for that comparison. other recent research has highlighted the limitations of both beam search and model scores globally in neural sequence-tosequence models @xcite . our results provide further evidence that lower-ranked ed predictions do not reflect cognitively plausible distributions: they contain many ill-formed outputs, and omit inflectional classes such as /-(e)n/, which is prevalent in speaker productions. an alternative to model scores is to treat each randomly initialized instance of a model as an individual, and compare aggregate productions with speaker data @xcite . for our experiments, this did not produce the distribution observed in the speaker data. the discrepancy between speaker production and rating preferences poses another challenge, as it's not clear how the ed model might represent these different task modalities. beside variability, the other key discrepancy between speaker and ed behavior is the treatment of non-rhyme words. if german has a default plural class, it should be realized more often on these phonologically atypical stimuli than the more familiar rhyme words. speakers in study 1 use /-s/ and /-(e)n/ more for non-rhymes than for rhymes. these results are consistent with earlier studies: m95 found that /-s/ was the only plural form to receive higher average ratings for non-rhymes compared to rhymes, and z&l found that speakers produced both /-(e)n/ and /-s/ more often for non-rhymes. in contrast, the ed model appears to treat /-e/ as a default, producing /-e/ inflections for under 70% of rhymes but over 90% of non-rhyme inputs. this asymmetry suggests that the model has not induced the full set of correct generalizations for german plural inflection -it has not recognized which plural classes are more productive for phonologically atypical nouns. in fact, the model's preference for /-e/, the most frequent (if non-majority) suffix, is the behavior anticipated by m95: "frequency in the input to a pattern associator causes a greater tendency to generalize" @xcite . it seems that the productivity of less frequent inflectional classes continues to challenge neural models and limit their cognitive application. ## conclusions german number inflection has been claimed to have distributional properties which make it difficult for neural networks to model. our experimental speaker data does not necessarily support all of these claims; in particular, /-s/ does not appear to be the only plural suffix which speakers treat as a 'default' for phonologically unfamiliar words, as the more frequent marker /-(e)n/ shows similar trends. nonetheless, the german plural system continues to challenge ed architectures. our neural model struggles to accurately predict the distribution of /-s/ for existing german nouns. on novel nouns, it generalizes the contextually most frequent plural marker /-e/; its predictions are less variable than speaker productions, and show different patterns of response to words which are phonologically typical (rhymes) as opposed to atypical (non-rhymes). regardless of the minority-default question, it seems that ed models do not necessarily function as good cognitive approximations for inflectional systems like german number, in which no class holds the majority. graphic filters to target native german speakers. participants were additionally asked about their age and exposure to languages other than german within the survey. participants were shown the three tasks, introduction, production, and rating, in order, meaning that participants had to produce a plural form for all 24 nonce words before performing the rating task. for the production task, participants saw the noun on its own, preceded by das, e.g. das bral. above the response box, the text die... appeared, to indicate that a plural form of the noun should be typed into the response box below the text. for the rating task, participants were prompted to rate each potential plural on a likert scale of sehr gut ('very good'; 5) to sehr schlecht ('very bad'; 1). after filtering out 42 respondents who failed a preliminary attention check, data from 150 participants was available for analysis. the cleaned, anonymized survey data will be published online along with this paper.