(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models’ Performance (2022)

Abstract: In the domain of Morphology, Inflection is a fundamental and important task that gained a lot of traction in recent years, mostly via SIGMORPHON’s shared-tasks. With average accuracy above 0.9 over the scores of all languages, the task is considered mostly solved using relatively generic neural seq2seq models, even with little data provided. In this work, we propose to re-evaluate morphological inflection models by employing harder train-test splits that will challenge the generalization capacity of the models. In particular, as opposed to the naïve split-by-form, we propose a split-by-lemma method to challenge the performance on existing benchmarks. Our experiments with the three top-ranked systems on the SIGMORPHON’s 2020 shared-task show that the lemma-split presents an average drop of 30 percentage points in macro-average for the 90 languages included. The effect is most significant for low-resourced languages with a drop as high as 95 points, but even high-resourced languages lose about 10 points on average. Our results clearly show that generalizing inflection to unseen lemmas is far from being solved, presenting a simple yet effective means to promote more sophisticated models.

## introduction in recent years, morphological (re)inflection tasks in nlp have gained a lot of attention, most notably with the introduction of sigmorphon's shared tasks (cotterell et al., 2016, 2017, 2018; vylomova et al., 2020) in tandem with the expansion of uni-morph (mccarthy et al., 2020), a multi-lingual dataset of inflection tables. the shared-tasks sample data from unimorph includes lists of triplets in the form of (lemma, features, form) for many languages, and the shared-task organizers maintain standard splits for a fair system comparison. the best-performing systems to-date in all inflection shared-tasks are neural sequence-to-sequence models used in many nlp tasks. an lstm-based model won 2016's task (kann and schütze, 2016), and a transformer came on top in 2020 (canby et al., 2020). in 2020's task the best model achieved exact-match accuracy that transcended 0.9 macroaveraged over up to 90 languages from various language families and types. this trend of high results recurred in works done on data collected independently as well (e.g. malouf, 2017, silfverberg and hulden, 2018, inter alia). interestingly, the averaged results of 2020's shared-task include languages for which very little data was provided, sometimes as little as a couple of hundreds of examples. this has led to a view considering morphological inflection a relatively simple task that is essentially already solved, as reflected in the saturation of the results over the year and the declining submissions to the shared tasks. @xcite this also led the community to gravitate towards works attempting to solve the same (re)inflection tasks with little or no supervision (mccarthy et al., 2019; jin et al., 2020; goldman and tsarfaty, 2021). however, before moving on we should ask ourselves whether morphological inflection is indeed solved or may the good performance be attributed to some artifacts in the data. this was shown to be true for many nlp tasks in which slight modifications of the data can result in a more challenging dataset, e.g., the addition of unanswerable questions to question answering benchmarks (rajpurkar et al., 2018), or the addition of expert-annotated minimal pairs to a variety of tasks (gardner et al., 2020). a common modification is re-splitting the data such that the test set is more challenging and closer to the intended use of the models in the wild (søgaard et al., 2021). as the performance on morphological inflection models seems to have saturated on high scores, a similar rethinking of the data used is warranted. in this work we propose to construct more difficult datasets for morphological (re)inflection by splitting them such that the test set will include no forms of lemmas appearing in the train set. this splitting method will allow assessing the models in a challenging scenario closer to their desired function in practice, where training data usually includes full inflection tables and learning to inflect the uncovered lemmas is the target. we show, by re-splitting the data from task 0 of sigmorphon's 2020 shared-task, that the proposed split reveals a greater difficulty of morphological inflection. retesting 3 of the 4 top-ranked systems of the shared-task on the new splits leads to a decrease of 30 points averaged over the systems for all 90 languages included in the shared-task. we further show that the effect is more prominent for low-resourced languages, where the drop can be as large as 95 points, though high-resourced languages may suffer from up to a 10 points drop as well. we conclude that in order to properly assess the performance of (re)inflection models and to drive the field forward, the data and related splits should be carefully examined and improved to provide a more challenging evaluation, more reflective of their real-world use. ## (re)inflection and memorization inflection and reinflection are two of the most dominant tasks in computational morphology. in the inflection task, the input is a lemma and a featurebundle, and we aim to predict the respective inflected word-form. in reinflection, the input is an inflected word-form along with its features bundle, plus a feature-bundle without a form, and we aim to predict the respective inflected-form for the same lemma. the training input in sig-morphon's shared-tasks is a random split of the available (lemma,form,features) triplets such that no triplet occurring in the train-set occurs in the test-set. @xcite in such a setting, models can short-cut their way to better predictions in cases where forms from the same lemma appear in both the train and test data. this may allow models to memorize lemmaspecific alternations that make morphological inflection a challenging task to begin with. consider for example the notoriously unpredictable german plurality marking, where several allomorphs are associated with nouns with no clear rule governing the process. kind, for example, is pluralized with the suffix -er resulting in kinder tagged as nom;pl. assuming a model saw this example in the train set it is pretty easy to predict kindern for the same lemma with dat;pl features, foot_3 but without knowledge of the suffix used to pluralize kind the predictions kinden and kinds are just as likely. ## related work many subfields of nlp and machine learning in general suggested hard splits as means to improve the probing of models' ability to solve the underlying task, and to make sure models do not simply employ loopholes in the data. in the realm of sentence simplification, narayan et al. (2017) suggested the websplit dataset, where models are required to split and rephrase complex sentences associated with a meaning representation over a knowledge-base. aharoni and goldberg (2018) found that some facts appeared in both train and test sets and provided a harder split denying models the ability to use memorized facts. aharoni and goldberg (2020) also suggested a general splitting method for machine translation such that the domains are as disjoint as possible. in semantic parsing, finegan-dollak et al. (2018) suggested a better split for parsing natural language questions to sql queries by making sure that queries of the same template do not occur in both train and test, while lachmy et al. (2021) split their hexagons data such that any one visual pattern used for the task cannot appear in both train and test. furthermore, loula et al. (2018) adversarially split semantic parsing for navigation data to assess their models' capability to use compositionality. in spoken language understanding arora et al. (2021) designed a splitting method that will account for variation in both speaker identity and linguistic content. in general, concerns regarding data splits and their undesired influence on model assessments led gorman and bedrick (2019) to advocate random splitting instead of standard ones. in reaction, søgaard et al. (2021) pointed to the flaws of random splits and suggested adversarial splits to challenge models further. here we call for paying attention to the splits employed in evaluating morphological models, and improve on them. ## experiments in order to better assess the difficulty of morphological inflection, we compare the performances of 3 of the top-ranked system at task 0 (inflection) of sigmorphon's 2020 shared-task. we examined each system on both the the standard (form) split and the novel (lemma) split. when re-splitting, foot_4 we kept the same proportions of the form-split data, i.e. we split the inflection tables 70%, 10% and 20% for the train, development and test set. in terms of examples the proportions may vary as not all tables are of equal size. in practice, the averaged train set size in examples terms was only 3.5% smaller in the lemma-split data, on average. foot_5 ## discussion we proposed a method for splitting morphological datasets such that there is no lemma overlap between the splits. on the re-split of sigmor-phon's 2020 shared-task data, we showed that all top-ranked systems suffer significant drops in performance. the new split examines models' generalization abilities in conditions more similar to their desired usage in the wild and allows better discerning between the systems in order to point to more promising directions for future research -more so than the original form-split data on which all systems fared similarly. the new splitting method is likely to lead to more sophisticated modeling, for instance, in the spirit of the model proposed by liu and hulden (2021). the suggested move to a harder split is not unlike many other nlp tasks, in which challenging splits are suggested to drive the field forward. we thus call for morphological studies to carefully attend to the data used and expose the actual difficulties in modelling morphology, in future research and future shared tasks.