Lemmas Matter, But Not Like That: Predictors of Lemma-Based Generalization in Morphological Inflection (2025)

Abstract: Recent work has suggested that overlap –whether a given lemma or feature set is attested independently in train – drives model performance on morphological inflection tasks. The impact of lemma overlap, however, is debated, with recent work reporting accuracy drops from 0% to 30% between seen and unseen test lemmas. In this paper, we introduce a novel splitting algorithm designed to investigate predictors of accuracy on seen and unseen lemmas. We find only an 11% average drop from seen to unseen test lemmas, but show that the number of lemmas in train has a much stronger effect on accuracy on unseen than seen lemmas. We also show that the previously reported 30% drop is inflated due to the introduction of a near-30% drop in the number of training lemmas from the original splits to their novel splits.

## introduction morphological inflection -the task of producing an inflected form given a lemma and a set of morphosyntactic features -is a fundamental problem in nlp, with both practical and scientific implications. on the practical side, morphological inflection is a crucial tool for pipelining, particularly for lowresourced and highly-inflected languages for which end-to-end systems remain impractical or infeasible @xcite @xcite @xcite . on the scientific side, morphological inflections are often complex transformations at the word and sub-word level @xcite . investigating how neural systems handle these transformations, and the circumstances under which they are able to generalize them, provides a window into their linguistic generalization capabilities @xcite . ann kodner et al. sigmorphon 2022 train sigmorphon seen lemmas sigmorphon unseen lemmas ann goldman et al. sigmorphon 2020 train goldman 2022 train sigmorphon 2020 test goldman 2022 test figure 1 : a schematization of the train and test methods of @xcite and @xcite . previous work on lemma overlap has drawn conflicting conclusions. @xcite compare model performance on the original sig-morphon 2020 splits -in which triples were partitioned uniformly at random -to their "lemmasplits," in which entire paradigms are partitioned uniformly at random, and thus all lemmas occur only in train or only in test. @xcite a 30% average decrease in accuracy from the models trained and tested on the sigmorphon splits to those trained and tested on their lemma-splits. however, @xcite and @xcite find no performance drop on unseen lemmas. their setup involves a single training split, with a test set containing at most 50% triples with attested features; this also results in a significant portion of test triples with unseen lemmas. kodner et al. train a single model per language and find no difference in accuracy between seen and unseen lemmas in their test set. both experimental designs are conceptualized in fig. 1 . crucially, while any drop in test accuracy can be attributed directly to differences in the test items in kodner et al.'s setup, they could be attributed either to differences in the test items or the train sets in @xcite in contrast to lemma overlap, previous work on feature overlap has unequivocally shown that neural models struggle to generalize to unseen feature sets: @xcite found that all systems perform better on test triples with attested feature sets than those with unattested ones, with the gap in accuracy ranging from 32% to as high as 79%. these findings were replicated by @xcite , who performed a more in-depth analysis on 6 languages. @xcite created linguistically-informed language-specific probes, and found that 2 of the 3 models which they tested only succeeded on the simplest feature-based generalization, while the third failed on all probes. in this paper, we aim to understand the causes of the conflicting results on lemma overlap and further investigate what factors impact models' ability to generalize to novel lemmas. because of the consistent findings regarding feature overlap, we hold it constant; our goal is to understand lemma and feature overlap separately so that future work may investigate their interaction. in our first experiment ( §3), we introduce a novel splitting algorithm which fixes a single train set and ensures that 50% of test triples contain seen lemmas. we find an average accuracy drop of 11%, between those reported by @xcite . more importantly, we show that the number of training lemmas has a much stronger effect on accuracy on unseen than seen lemmas. in our second experiment ( §4), we replicate @xcite to investigate causes of our differing results, and show that the number of lemmas in train is a stronger predictor of generalization to unseen lemmas than the number of triples. since goldman et al.'s lemma-splitting held triple count roughly constant but introduced a near-30% drop in the number of lemmas from sigmorphon to their lemma-based splits, they exaggerated the true impact of lemma overlap. these results are a step towards a more complete and interpretable picture of the dimensions across which neural models succeed and fail at generalization. ## background: defining overlap in morphological inflection tasks, models are exposed to triples of (lemma, feature set, inflected form) in train. during evaluation, they are given a (lemma, feature set) pair as input and expected to correctly predict the corresponding inflected form. for example, given (walk, v;past), a model should output walked. in iterations of the sigmorphon shared task before 2022, triples are partitioned uniformly at random into train or test. while this entails that no triple occurring in train will occur in test, as both @xcite and @xcite note, it ignores the fact that lemmas or feature sets that appear in train may reappear during test separately. consider a simple example, adapted from kodner et al. (2023b): (1) example train set: t0: see seeing v;v.ptcp;prs t1: sit sat v;pst (2) example test set: e0: see v;pst e1: sit v;nfin e2: eat v;pst e3: run v;prs;3;sg though none of the triples in train (1) re-appear in test (2), the lemmas and feature sets seen in train do reappear individually. consider e0, for which both the lemma and feature set are attested separately in train, and e3, for which neither are. one might expect it to be easier for a model trained on (1) to generate the correct result for e0 than for e3. this is the insight behind studies of overlap: test pairs with novel lemmas or novel feature sets require a system to generalize along different dimensions, and evaluation measures that control specifically for these overlaps can better probe models' ability to perform these generalizations. @xcite define four types of overlap: both both the lemma and feature set are attested separately in train (e0 in example 2). lemmaonly only the lemma is attested in train, and the feature set is novel (e1 in example 2). featsonly only the feature set is attested in train, and the lemma is novel (e2 in example 2) neither neither the lemma nor the feature set is attested in train (e3 in example 2) we define lemmaseen to be any evaluation triple with a lemma attested in train (i.e., @xmath0.e., @xmath1. in this section, we test the impact of lemma overlap with a novel splitting algorithm that ensures that 50% of test triples contain seen lemmas. like kodner et al., our setup ensures a single training set per language, but like goldman et al., we focus on lemma overlap rather than feature set overlap. ## exp. 2: explaining the discrepancy the difference between our results and those of @xcite are partly explainable by the relationship between training lemmas and accuracy drop outlined above. since we fix our train size, we exclude from our analysis languages with miniscule numbers of training lemmas; goldman et al. include these languages in their analysis. indeed, the median number of training lemmas in our data is 1,015 vs. just 141 for goldman et al., despite comparable means (1,625 and 2,038, respectively). however, since goldman et al. train two separate models for each language (fig. 1 ), the resulting differences in performance could also be attributable to differences in the training data itself. even if goldman et al. had used an identical splitting algorithm across the two train-test splits, the performance of morphological anns varies across random seeds (liu and prud'hommeaux, 2022, i.a.) and initializations (e.g. @xcite . moreover, since their train sets were derived from separate splitting algorithms, they show substantial distributional differences which may be relevant to performance. firstly, though goldman et al. retain the train-test ratios from sigmor-phon 2020, the number of triples in their train splits do not match exactly since they split on entire paradigms. they report an average decrease in training triples of 3.5%, but there is a great deal of variation (σ = 6.341%), with one language (ludic) experiencing a 47.3% decrease. what's more, goldman et al.'s lemma-splitting strategy results in a 28.846% (σ = 5.221%) average drop in the number of training lemmas between the sigmor-phon 2020 splits and their splits, which they do not report. the cause of this drop is intuitive: if we assume that nearly all available lemmas will appear at least once in the sigmorphon training data, then since goldman et al. sample 70% for train, we expect about a 30% decrease in the number of training lemmas. fig. 4 shows the distributions of train triple and lemma count decreases. implicit in the conclusions drawn by @xcite is the assumption that the decrease in the number of training lemmas does not meanwe show that the number of training lemmas, both measured directly and in terms of the decrease between the two training sets, has a significant effect on accuracy and is the best predictor of accuracy drop across architectures. we further show that goldman et al.'s reported accuracy drops are substantially higher than they would have been had they held lemma counts constant. ## discussion the findings presented here suggest a more nuanced view of the role of lemma overlap than proposed by either @xcite or @xcite . with our novel splitting algorithm ( §3), we find that all neural architectures achieve substantially higher accuracy on seen than unseen lemmas, unlike kodner et al., but the difference is much lower (11%), about a third of that reported by @xcite . more importantly, however, we show that the number of lemmas in train has a much stronger effect on accuracy on test triples containing unseen lemmas than seen ones. while there are several incidental differences between our study and the two prior studies, we established one principled difference which resulted in the especially large performance drop which goldman et al. reported: the number of lemmas in the training set (absolute or relative to the original set) is the most important predictor for accuracy drop, with larger accuracy drops occurring for languages with fewer train lemmas. goldman et al. tested the effect of lemma overlap with two distinct training sets, one which allowed for lemma overlap, and one that both banned overlap and was on average 30% smaller in terms of lemma count; this was a confound which inflated accuracy drop. as we found in §4.3, goldman et al.'s data was also highly skewed towards small training sizes where the impact of this confound was at its greatest. our own study of lemma overlap was set up similarly to @xcite and @xcite in that we trained models on a single training set and tested them on distinct test sets (fig. 2 ). this eliminates the potential for confounds such as the training lemma count. unlike kodner et al., who found no effect of lemma overlap, however, we found a substantial drop. one possible reason for this discrepancy is the role of feature attestation: in our study, we ensured a 50% seen/unseen lemma split in test. doing so on at least 10,000 triple data sets caused feature set overlap to rise to 99.5%, thus controlling for this additional factor; indeed, our analysis does not significantly change when conducted only over test triples containing seen feature sets. kodner et al., on the other hand, fixed feature set overlap to a maximum of 50% and did not control for lemma overlap explicitly. it is possible that this discrepancy emerges as an artifact of the feature-based splitting algorithm or of the interaction between lemma and feature overlap. if, for example, lemma overlap has a small but significant effect for triples with seen features (what our results show) but does not play a role in triples with unseen features, this could help to explain the results of @xcite . now that we have established a better understanding of the effect of lemma overlap on triples with seen feature sets, future work should further investigate the interaction between lemma overlap and feature overlap. as discussed in §3, future work should also investigate the effect of the number of times a given lemma or feature set has been seen in train rather than making a binary attested-unattested distinction. the findings presented here have implications for our understanding of subword-level generalization in different classes of neural network and are thus an important contribution interpretability for these models. intuitively, generalizing morphological inflections to unseen lemmas is a more challenging task than generalization to seen lemmas, since it requires generalization from the lemma and feature sets only. without additional evidence from other inflected forms of the lemma, we cannot know, for example, if it follows a regular or irregular pattern. while all models performed similarly to each other on languages with the largest numbers of training lemmas (fig. 3 ), they diverged in their ability to generalize on languages with the smallest numbers of training lemmas. chr-trm, a character transformer trained on a single language, struggled the most to generalize to unseen lemmas on the smallest training sets, conforming with our expectations about the data hungry nature of transformers. on the other hand, deepspin, a multilingual model with a much simpler bilstm architecture, much more readily generalized to unseen lemmas even at the smallest training sizes and was nearly immune to performance drop. ## conclusion in this paper, we have demonstrated that the number of lemmas in the training data significantly affects the performance of neural models on unseen lemmas, but not on seen lemmas. this result entails that neural models struggle to generalize to novel lemmas when trained on small data, an important step to understanding how different classes of model generalize at a subword level. beyond just understanding how models generalize or fail to, inflection still remains an important element in nlp for morphologically rich low-resource and endangered languages, as evidenced by contributions to recent workshops on nlp for under-resourced, endangered, and indigenous language families. 8 in addition to the main results concerning generalization to seen and unseen lemmas, this paper provides a methodological contribution. we find that careful experimental design lets us not only reassess top line numbers claimed in prior work @xcite , but also provides an explanation for the causes of previous discrep-8 e.g., computel, americasnlp, rail, arabicnlp ancies and allows us to identify confounds. we evaluated the role of lemma overlap while controlling for a number of potential confounds by holding the training set constant. we augmented the reporting of top-level accuracy numbers with statistical analysis modeling the role of confounding factors on performance. it is likely that these approaches will prove useful in future work assessing patterns in morphological generalization. in particular, now that a separate understanding of the roles of feature overlap and lemma overlap has been established, future work is well-positioned to investigate their interaction. while overlap has been quantified in terms of binary attestation in train, future work may also consider the number of times a given lemma or feature set is attested in order to gain a more complete picture of factors affecting generalization.