Morphological Inflection: A Reality Check (2023)

Abstract: Morphological inflection is a popular task in sub-word NLP with both practical and cognitive applications. For years now, state-of-the-art systems have reported high, but also highly variable, performance across data sets and languages. We investigate the causes of this high performance and high variability; we find several aspects of data set creation and evaluation which systematically inflate performance and obfuscate differences between languages. To improve generalizability and reliability of results, we propose new data sampling and evaluation strategies that better reflect likely use-cases. Using these new strategies, we make new observations on the generalization abilities of current inflection systems.

## introduction morphological inflection is a task with widereaching applications in nlp, linguistics, and cognitive science. as the reverse of lemmatization, it is a critical part of natural language generation, particularly for languages with elaborate morphological systems @xcite . since morphological inflection is a particular type of well-defined regular string-to-string mapping problem @xcite , it is also useful for testing the properties of different neural network architectures. within cognitive science and linguistics, computational models of inflection have a long history in arbitrating between competing theories of morphological representation and acquisition (surveyed in @xcite , and inflection is often a focus of computational typology @xcite . however, despite the task's popularity, standard evaluation practices have significant weaknesses. we discuss three aspects of these practices which hamper investigators' ability to derive informative *denotes equal contribution figure 1 : the four logically possible train-eval overlap types if evaluation data consists of (lemma, feature set) pairs: both, featsonly, lemmaonly, neither, as well as @xmath0. ## defining overlap morphological inflection requires generalization over two primary dimensions: to new lemmas ("if i have witnessed the 2pl imperfective subjunctive with other verbs, how do i apply that to new verb x?") and to new inflectional categories ("if i have seen x inflected in several other categories, how do i create the 2pl imperfect subjunctive of x?"). because of the sparsity of morphological inflections in language use @xcite , both types of generalization are necessary during language acquisition as well as deployment of computational models. as with many linguistic phenomena, the attestation of inflected forms follows an extremely sparse and skewed long-tailed distribution, as do attested lemmas ranked by the proportions of their potential paradigms that are actually attested (paradigm saturation; ps), and inflectional categories ranked by the number of lemmas with which they oc-cur @xcite . for example, the median ps for spanish verbs in millions of tokens of childdirected speech is equivalent to two of its three dozen possible forms, and the 2nd person plural imperfect subjunctive only occurs with two lemmas (cf. @xcite . given the importance of both types of generalization, it is necessary to evaluate both to assess the abilities of a morphological learning model. in the evaluation made popular by the sigmorphon shared tasks, models are asked to predict inflected forms given (lemma, feature set) pairs, where feature sets can be seen as corresponding to inflectional categories or paradigm cells. generalization across lemmas is required when an evaluation pair contains a lemma that was out-of-vocabulary (oov) in training, and generalization across categories is required when an evaluation pair contains a feature set that was oov. in all, there are four logically possible licit types of evaluation pairs distinguished by their lemma and feature overlap with training. these are expressed visually in figure 1 along with two types which are unions of the other types: for a concrete illustration, consider the training and evaluation sets provided in (1)-(2). each evaluation pair exhibits a different kind of overlap. computational work in morphological inflection has generally ignored these dimensions of evaluation. in the shared task, the four overlap types were uncontrolled before 2021, which contains one partial evaluation on featsonly ∪ neither items. but, recognition of the value of these overlap types has grown recently. @xcite showed that four models consistently struggle to generalize across lemmas, concluding that test sets should avoid lemma overlap altogether. however, this proposal removes the option to contrast performance on seen and unseen lemmas. furthermore, they did not control for or evaluate feature overlap, so both vs. lemmaonly and featsonly vs. neither also cannot be distinguished. (3) summarizes their partition scheme, which distinguishes two overlap types. we call these lemmaattested (= both ∪ lemmaonly) and lemmanovel (= featsonly ∪ neither). goldman et al. (2022) partition types e0: sit v;@xmath1: see v;@xmath2: eat v;@xmath3: run v;prs;3;@xmath4 the 2022 sigmorphon-unimorph shared task was the first to report results on all four overlap types (both, featsonly, lemmaonly, neither). every system submitted to the shared task achieved much better performance on in-vocabulary feature sets (both and featsonly) than oov feature sets (lemmaonly or neither). this discrepancy even held for languages for which a model should be able to generalize: highly regular agglutinative morphology for which this type of generalization is often transparent. on the other hand, lemma attestation produced a much smaller discrepancy. following these observations, we focus our investigation on the four logical overlap types with extra emphasis on the featsattested vs. featsnovel dichotomy. we address agglutinative languages specifically in section 5.3 ## data sources and preparation we follow prior literature in providing training and evaluation data in unimorph's format. data sets were sampled from unimorph 4 @xcite and 3 (mccarthy et al., 2020) 1 aug-mented with frequencies from running text corpora. when possible, frequencies were drawn from childdirected speech (cds) corpora from the childes database @xcite , since one possible downstream application of the morphological inflection task is contribution to the computational cognitive science of language acquisition. childes lemma and morphological annotations were converted into unimorph format and intersected with unimorph to create frequency lists. foot_1 ## experimental setup one non-neural and three neural systems were evaluated. these were chosen based on their availability and performance in recent shared tasks: chr-trm @xcite is a character-level transformer that was used as a baseline in 2021 and 2022. we used the hyper-parameters suggested by the original authors for small training conditions. cluzh-gr and cluzh-b4 @xcite ) is a character-level transducer which substantially outperformed chr-trm in the 2022 shared task. the results submitted for the shared task are from an elaborate ensemble model optimized for each language. for this work, we evaluate two published variants with consistent hyper-parameters across languages, cluzh-gr with greedy decoding and cluzh-b4 with beam decoding, beam @xmath5. nonneur @xcite has been used as a baseline in sigmorphon shared tasks since 2017. it heuristically extracts transformations between lemmas and inflected forms and applies a majority classifier conditioned on the associated feature sets. nonneur was trained on combined training and fine-tuning sets so that each architecture was exposed to the same amount of data. ## results this section presents our analyses of the results. all evaluations report exact match accuracy. overall accuracy refers to average accuracy on an entire evaluation set. average overall accuracy refers to the mean of overall accuracy over all five seeds. see appendix c for full breakdowns by language and architecture. ## conclusions we investigated the roles that sampling strategy, random seeds, and overlap types play in evaluating and analyzing the results of morphological inflection tasks and conclude that common practices leave much to be desired. we argue for frequencyweighted splitting to achieve more realistic traintest distributions and feature/lemma overlap-aware sampling for directly investigating the generalization abilities of different models. the high score range observed for overlap-aware sampling relative to other strategies suggests that which feature sets happen to appear in train vs. test play a major role in the ability of a model to generalize, though future work would need to confirm this. regardless of sampling strategy, evaluation items of each overlap type should be used in addition to an overall analysis. the evaluation in this work reveals that all model types under investigation struggle to generalize to unseen feature sets, even for languages where that should be possible, a fact that has been overlooked in prior studies. finally, results drawn from one data split are unlikely to be representative, so multiple splits should be made with different random seeds and compared, particularly for shared tasks and leader boards where final model rankings matter.