SIGMORPHON – U ni M orph 2022 Shared Task 0: Modeling Inflection in Language Acquisition (2022)

Abstract: This year’s iteration of the SIGMORPHONUniMorph shared task on “human-like” morphological inflection generation focuses on generalization and errors in language acquisition. Systems are trained on data sets extracted from corpora of child-directed speech in order to simulate a natural learning setting, and their predictions are evaluated against what is known about children’s developmental trajectories for three well-studied patterns: English past tense, German noun plurals, and Arabic noun plurals. Three submitted neural systems were evaluated together with two baselines. Performance was generally good, and all systems were prone to human-like over-regularization. However, all systems were also prone to non-human-like over-irregularization and nonsense productions to varying degrees. We situate this behavior in a discussion of the Past Tense Debate.

## introduction the overarching goal of this subtask of the 2022 sigmorphon-unimorph shared task on morphological inflection, in contrast with this year's and previous years' typologically informed subtasks, was to provide insight into how current stateof-the-art morphological inflection models relate to human language acquirers, to what extent they behave similarly or differently, and in what respects they perform better or worse. as such, the task was designed to be cognitively informative while still approachable for the nlp morphology community. this was achieved in two ways: first, nested training sets of increasing size were extracted from corpora of child-directed speech, following @xcite , allow us to approximate learning trajectories with batch learning models that are typical in the field today rather than incremental models which might better approximate the child lan-guage acquisition setting. second, supervision with semantic features substitutes for semantic information which children in real acquisition settings would certainly glean from their linguistic and environmental experiences. while this simplified the task considerably, it also permitted us to focus on the act of generating correct forms in the absence of other learning confounds. ## task description this task was organized very similarly to other iterations of the inflection task from the participants' perspective in order to encourage crosssubmissions with this year's large scale generalization inflection task @xcite . participants were asked to design supervised learning systems which could predict an inflected form given a lemma and a morphological feature set corresponding to an inflectional category or cell in a morphological paradigm. they were provided with several nested training sets as well as a development set and test set for each language. the train and dev sets consisted of (lemma, inflected, feature set) triples, while the inflected forms were held out from the test set. initially, only training and development sets were available to participants. they were expected to design, train, and tune their models on this data. shortly before the submission deadline, test sets with held-out inflections were released. in contrast with the large-scale subtask and previous iterations, only three languages were investigated which could be evaluated in detail: american english, standard german, and modern standard arabic. several nested training sets were released for each language in increments of 100 items. participants were asked to return predictions from models trained on each training size. ## data preparation data sets for (american) english and (standard) german were extracted from the childes collection of child-directed speech (cds) corpora @xcite . childes contains several types of corpora with various types of annotation. english was sourced from the brown @xcite foot_4 and brent @xcite corpora. these contain free dialogue between caregivers and their children alternating with lines of morphological annotation. in (1), * mot indicates that this utterance was produced by the child's mother and %mor indicates that the following line contains pos tags, lemmas, and morphological features. however, "words" in morphological annotation lines do not consistently line up one-to-one with tokens in dialogue lines, so it is not feasible to match lemma-feature pairs to inflected forms. to accomplish this, features were converted into unimorph format, and (lemma, inflected, features) triples were extracted from english unimorph @xcite . one advantage of childes is that it presents vocabulary that a typical child is likely exposed to during the acquisition process, and since it contains dialogue, it can also be used to make reasonable frequency estimates of child-directed speech. in nlp terms, it is a reasonable approximation of the training set over which children learn morphological inflection. see @xcite for more information. 2,054 nouns were sampled from childes weighted by their childes frequencies, and their plurals were extracted from unimorph. 454 of these items were sampled uniformly and reserved as the development set. 600 of the remainder were uniformly sampled from the remainder and set aside as the test set. the remaining 1,000 was used as the maximum training set. smaller nested subsets in increments of 100 were sampled from these, weighted by noun lemma frequencies in childes such that each larger subset was a superset of the smaller. training and test were sampled uniformly with respect to one another to guarantee that the test set would contain interesting test items. another reasonable approach would have been to sample the 1,000 training items by frequency from the entire data set and then sample the test items from the remainder in order to yield a training set containing more frequent items and a test set containing less frequent items. since item token frequency correlates with age of acquisition @xcite , this would correspond to a realistic scenario where systems predict later-acquired forms from their knowledge of earlier acquired forms. however, english past tense irregulars (i.e., non-ed pasts), are heavily skewed towards the high end of a zipfian frequency distribution, so such an approach would not yield many interesting test items. the german data set was created in much the same way as the english with cds frequency information sourced from the childes leo corpus @xcite and nominative plural forms matched from german unimorph. gender is known to be a predictor for plural forms @xcite , so the german unimorph features were augmented with masc, fem, or neut gender tags converted from the childes annotation lines. these were split into 600 training items, 500 development items, and 600 test items with the same frequencyweighted algorithm that was applied to english. the intersection of nouns extracted from leo and nouns present in unimorph was relatively small, so the largest training set that could be extracted only contains 600 items. ideally, the arabic data set would also be extracted from a cds corpus in order to get a reasonable estimation of a child's vocabulary. colloquial arabic varieties are unfortunately considered to be low-resourced in terms of available linguistic resources, so even though there are several dialectal cds corpora @xcite @xcite , they do not provide morphological annotations useful to the task in hand. thus, we selected modern standard arabic (msa) for the shared task. even though it has virtually no native speakers and no cds corpora, it is well-resourced and exhibits the same kinds of morphological patterns present across arabic varieties. a reasonable workaround for the lack of cds is to estimate a child size corpus from a given non-cds corpus through lemma frequencies. this will most likely contain high frequency lexemes that typically do not appear in cds corpora but will likely cover a similar distribution of morphological phenomena @xcite . for this shared task, the arabic data set was sourced from the penn arabic treebank (patb) @xcite , which is a morphologically and syntactically annotated news corpus of msa. the corpus is written using standard arabic orthography and it is fully diacritized. diacritization include short vowels, specific case and state markings, and gemination. arabic text without diacritization does not mark these critical phonological segments and thus would not be useful for the task at hand. despite including fine-grained morphological annotations, patb lacks the annotations of functional (grammatical) gender and number in addition to rationality (animacy). therefore, a version that has been enriched with additional features through the calima m sa morphological analyzer @xcite was used. plural inflections that reflect state and case were normalized to a single inflection since only pluralization was under investigation for this task. the 2,000 most frequent plural nouns were extracted according to their lemma frequencies from the train split of patb @xcite . these were then split into a training set of 1,000 items, a development set of 343 items, and a test set of 600 items using the same algorithm that split english and german. an animacy feature hum or non-hum was added was added to each noun, since it is known to impact nominal inflection patterns @xcite . ## systems the same neural and non-neural baselines were provided for this task and the 2022 typologically diverse inflection shared task. the neural system neural, @xcite , is a character-level transformer. it is identical to the system chr-trm which was used in the 2021 task with identical hyperparameters. the non-neural system, nonneur, is identical to the non-neural baseline made avail-able in 2021 and 2020. 5 three systems were submitted, the first and last of which were also submitted to the large scale generalization task: cluzh (silvan wehrli and makarov, 2022): universität zürich's system is identical to the one submitted to this year's large scale generalization subtask @xcite . their submission is a character-level transducer which operates over edit actions: insertion, deletion, substitution, and copy. they implement true mini-batch training for a substantial speed up, rendering the system more practical on larger training sets. heimorph @xcite : the team from heinrich-heine-universität düsseldorf developed a system with a self-attention transformer architecture with bigram hallucination. submitted models were trained on the enriched data setsthat include either 1,000 or 10,000 bigram-aware hallucinated word pairs, generated separately for each training set size. the system was implemented with fairseq, a pytorch-based tool. osu @xcite ): osu's system is identical to the one submitted to this year's large scale generalization subtask. this inflection system is a transformer whose input is augmented with an analogical exemplar showing how to inflect a different word into the target cell. in addition, alignment-based heuristic features indicate how well the exemplar is likely to match the output. ## evaluation whole-form accuracy was employed as the primary quantitative evaluation, though several further analyses were carried out by partitioning data over grammatical gender and other factors. performance was good overall but showed some points of divergence from human behavior. this section provides an analysis for each of the shared task's three languages. ## discussion this year's shared task investigated the performance of neural systems on an inflection task designed to mimic language acquisition. training data was mostly sourced from the childes collection of child-directed speech corpora and extracted by frequency to represent early linguistic input, and systems produced past forms and plurals for real words, simulating children producing novel (to them) forms of lemmas that they know from daily life. this was a challenging task characterized by small training data and complex patterns. nevertheless, systems performed well in terms of raw accuracy. american english past tense forms proved the easiest, followed by standard german noun plurals, then modern standard arabic noun plurals. in some ways, the submitted systems actually outperformed children -they all learned the producive -ed pattern for english past tense after only 100 training items, far earlier than what is reported for children. systems also achieved most of their performance on very small data. superhuman performance on very small data is a valuable property for real-world nlp applications. compared to early connectionist systems, modern neural morphology learners produce far fewer nonsense forms of the mail-membled type, though this still remains a problem, even in the largest training conditions evaluated here. this is consistent with the findings of @xcite , which found that what they called "silly" errors were still present in the productions of the 2017 task, but they were majorly reduced compared to early work. systems "successfully" over-regularized the english -ed past, the most frequent german noun plural types, and the arabic feminine sound plural. this is is a human-like tendency, however it cannot be said whether this indicates deep understanding of the paradigms or a simple case of frequency matching. systems under-applied rarer german noun plural types even at the largest training size, which may imply the latter, but more work would need to be done to confirm this. the most significant weakness of all three systems uncovered by this analysis is persistent inhuman over-irregularization. though rates of overirregularization varied significantly on english, all systems produced far more instances of it than child learners, and the problem was starker for arabic. all three systems dramatically overproduced sound-to-broken and broken-to-broken errors which are rare in child productions. broken plural patterns are apparently no more productive than english strong verb mutations, so their over-application has to be seen as over-irregularization. though @xcite did not categorize errors in these cognitively-minded terms, they did find evidence for over-irregularization in their analysis. they noted, for example, that one system overapplied spanish diphthongization, a pattern that applies to many verbs. the pattern is frequent but unpredictable -many verbs that could be subject to diphthongization are not. the pattern is apparently lexicalized and unproductive, and children underapply it if anything @xcite , thus the overapplication is an instance of over-irregularization. all of the systems evaluated this year happen to be neural single-route models that do not make an explicit distinction between regular and irregular items. no dual-route models were submitted for comparison. while all systems performed well, they showed the clear hallmarks of such models, in particular a tendency to over-produce overirregularization. all of the technical improvements over the decades have greatly improved overall prediction accuracy, but single-route models are still single-route models. what do these results tell us about human cognition? even if the systems had shown very humanlike performance, we could not therefore conclude that they are good models of cognition. as summarized recently in @xcite , that line of reasoning is backward. prediction is not explanation. we would need to first justify the assertion that these are theoretically plausible cognitive models. only then, if these systems were effective representations of cognition, then we should expect them behave in a human-like manner. what studies like this do provide is insight into state-of-the-art morphological learning models with ever-improving prediction capabilities. inasmuch as humans are the gold-standard in language learning and language use, one possible reason for current progress is that models are making predictions for more human-like reasons. the results here show that that intuition does not necessarily hold. the systems evaluated in this shared task were on the whole successful in their predictions but did not behave in a especially human-like manner.