Understanding Compositional Data Augmentation in Typologically Diverse Morphological Inflection (2023)

Abstract: Data augmentation techniques are widely used in low-resource automatic morphological inflection to address the issue of data sparsity. However, the full implications of these techniques remain poorly understood. In this study, we aim to shed light on the theoretical aspects of the data augmentation strategy StemCorrupt, a method that generates synthetic examples by randomly substituting stem characters in existing gold standard training examples. Our analysis uncovers that StemCorrupt brings about fundamental changes in the underlying data distribution, revealing inherent compositional concatenative structure. To complement our theoretical analysis, we investigate the data-efficiency of StemCorrupt. Through evaluation across a diverse set of seven typologically distinct languages, we demonstrate that selecting a subset of datapoints with both high diversity and high predictive uncertainty significantly enhances the data-efficiency of compared to competitive baselines. Furthermore, we explore the impact of typological features on the choice of augmentation strategy and find that languages incorporating non-concatenativity, such as morphonological alternations, derive less benefit from synthetic examples with high predictive uncertainty. We attribute this effect to phonotactic violations induced by StemCorrupt, emphasizing the need for further research to ensure optimal performance across the entire spectrum of natural language morphology.

## introduction compositional mechanisms are widely believed to be the basis for human language production and comprehension @xcite . these mechanisms involve the combination of simpler parts to form complex concepts, where valid combinations are licensed by a recursive grammar @xcite . however, domain general neural architectures often fail to generalize to new, unseen data in a compositional manner, revealing a failure in inferring the data-generating grammar @xcite ; lake and @xcite . this failure hinders these models from closely approximating the productivity and systematicity of human language. consider the task of automatic morphological inflection, where models must learn the underlying rules of a language's morphoysyntax to produce the inflectional variants for any lexeme from a large lexicon. the task is challenging: the models must efficiently induce the rules with only a small human-annotated dataset. indeed, a recent analysis by @xcite demonstrates that even state-of-the-art, task-specific automatic inflection models fall short of a compositional solution: they perform well in random train-test splits, but struggle in compositional ones where they must inflect lexemes that were unseen at training time. nevertheless, there is reason for optimism. several works have shown that automatic inflection models come much closer to a compositional solution when the human-annotated dataset is complimented by a synthetic data-augmentation procedure @xcite @xcite @xcite , where morphological affixes are identified and attached to synthetic lexemes distinct from those in the training dataset (fig. 2 ). however, little is understood about this prominent data augmentation method and the extent to which it can improve compositional generalization in neural word inflection. in this work, we seek to reveal the implicit assumptions about morpheme distributions made by this rule-based augmentation scheme, and analyze the figure 1 : given a small human-annotated dataset and a large pool of synthetic examples, we find that sampling a subset of data representing both diversity (a multitude of shapes) and high predictive uncertainty (shapes with a question mark) are on average more sample-efficient in improving compositional generalization in morphological inflection. to this end, our work presents the first theoretical explanation for the effectiveness of compositional data augmentation in morphological inflection. through an information-theoretic analysis (section 3), we show that this method eliminates "spurious correlations" @xcite between the word's constituent morphemes, specifically between the stem (e.g., walk) and the inflectional affix (e.g., -ed). by removing these correlations, the training data distribution becomes aligned with concatenative morphology, where a word can be broken down into two independent substructures: the stem (identifying the lexeme) and the inflectional affixes (specifying grammatical function). this finding sheds light on why the method is widely attested to improve compositional generalization @xcite , as concatenative morphological distributions are cross-linguistically prevalent @xcite . we go on to show, however, that the augmentation method tends towards removing all correlations between stems and affixes, whether spurious or not. unfortunately, this crude representation of concatenative morphology, while reasonable in broad strokes, is violated in virtually all languages to varying degrees by long-distance phonological phenomena like vowel harmony and reduplication. thus, our analysis demonstrates that while the method induces a useful approximation to concatenative morphology, there is still ample room for improvement in better handling of allomorphy and phonological alternations. building on our theoretical analysis, we investigate whether it is possible to improve the sample-efficiency with which the data augmentation method induces probabilistic independence between stems and affixes. specifically, we investigate whether we can use a small subset of the synthetic data to add to our training dataset. we find that selecting a subset that incorporates both high predictive uncertainty and high diversity (see fig. 1 ) is significantly more efficient in removing correlations between stems and affixes, providing an improvement in sample-efficiency for languages where the morphological system is largely concatenative. at the same time, in accordance with our theoretical analysis, this selection strategy impairs performance for languages where phonological alternations are common. our work contributes to a comprehensive understanding of a prominent data augmentation method from both a theoretical (section 3) and practical standpoint (section 4). through our systematic analysis, we aim to inspire further research in the analysis and evaluation of existing compositional data augmentation methods (reviewed in section 6), as well as the development of novel augmentation methods that can better capture cross-linguistic diversity in morphological patterns. ## preliminaries in automatic morphological inflection, we assume access to a gold-standard dataset d train with triples of ⟨x, y, t⟩, where x is the character sequence of the lemma, t is a morphosyntactic description (msd), and y is the character sequence of the inflected form. 2 the goal is then to learn the distribution p (y|x, t) over inflected forms conditioned on a lemma and msd . generating a synthetic training dataset with stemcorrupt. for many languages, d train is too small for models to learn and make systematic morphological generalizations. previous work has found that generating a complementary synthetic dataset d syn train using a data augmentation technique can substantially improve generalization @xcite . the technique, henceforth called stemcor-rupt, works as follows: we identify the aligned subsequences (of length 3 or greater) between a lemma x and an inflected form y, which we denote the stem. 3 we then substitute some of the characters in the stem with random ones from the language's alphabet; fig. 2 . the stemcorrupt procedure has a hyperparameter θ. it sets the probability that a character in the stem will be substituted by a random one; a higher value of θ (approaching 1) indicates a greater number of substitutions in the stem. 4 how does stemcorrupt improve compositional generalization? despite the widespread adoption of this technique for automatic morphological inflection and analysis, this fundamental question has heretofore remained unanswered. in the next section, we argue that stemcorrupt improves compositional generalization by removing correlations between inflectional affixes and the stems to which they are attached. by enforcing independence between these two substructures, stemcor-rupt facilitates productive reuse of the inflectional affixes with other lexemes. that is, our theoretical argument shows that the effectiveness of the method arises from factoring the probability distribution to approximate concatenative morphology, a cross-linguistically prevalent strategy for word formation where words can be "neatly segmented into roots and affixes" @xcite . we formalize and prove this in the following section. ## stemcorrupt induces compositional structure in this section, we analyze the ramifications of training on the synthetic training dataset generated by stemcorrupt (d syn train ) and the humanannotated training dataset (d train ). our analysis crucially, this removes the potential for any spurious correlation between these two substructures. 6 remark 2 (stem-affix dependencies). while concatenation is a cross-linguistically prevalent strategy for inflection @xcite , stems and affixes are rarely entirely independent. therefore, enforcing complete independence between these structures is an overly strong constraint that stemcorrupt places on the training data. the constraint is consistently violated in turkish, for example, where front-back vowel harmony constraints dictate that the vowels in the suffix share the same front/back feature as the initial vowel in the stem. this leads to forms like "daların" and pre-vents forms like "dalerin", "dalerın", or "dalarin" @xcite . in section 5, we show that stem-corrupt regularly generates examples violating vowel harmony, and that this can undermine its effectiveness. nevertheless, the empirical success of stemcorrupt demonstrates that the benefits of its concatenative bias outweigh its limitations. remark 3 (comparison to previous accounts). our analysis provides a simple yet the most accurate characterization of stemcorrupt. previous works have called stemcorrupt a beneficial "copying bias " @xcite or a strategy for mitigating overgeneration of common character n-grams @xcite . however, our analysis demonstrates that neither of these characterizations are entirely accurate. first, the denotation of a "copying bias" is only suggestive of the second factor in our statement p (y stem |x stem ), and does not address the impact on affix generation. in contrast, our analysis shows that both stem and affix generation are affected. furthermore, alleviating overfitting to common character sequences is also misleading, as it would suggest that stemcorrupt serves the same purpose as standard regularization techniques like label smoothing @xcite ). foot_6 ## extracting sample-efficient training sets from stemcorrupt problem setup. we use the following problem setup from @xcite to investigate whether the sample-efficiency of stemcor-rupt can be improved. recall that we have a dataset d train with gold triples of ⟨x, y, t⟩. further, we have a synthesized dataset d syn train where ## experiments and results data. we use data from the unimorph project @xcite , considering typological diversity when selecting languages to include. we aim for an evaluation similar in scope to @xcite . that is, broadly, we attempt to include types of languages that exhibit variation in inflectional characteristics such as inflectional synthesis of the verb, exponence, and morphological paradigm size @xcite . our selected languages can be seen in fig. 4 . we provide further information on the languages in appendix c. obtaining a large synthetic dataset. in order to generate the large augmentation dataset d syn train for every language, we generate 10, 000 augmented datapoints for every language by applying stem-corrupt to their respective low datasets from sigmorphon 2018 @xcite . ## related work compositional data augmentation methods. in general, such methods synthesize new data points by splicing or swapping small parts from existing training data, leveraging the fact that certain constituents can be interchanged while preserving overall meaning or syntactic structure. a common approach is to swap spans between pairs of datapoints when their surrounding contexts are identical @xcite @xcite . recently, @xcite extended this approach, eschewing rule-based splic-ing in favour of neural network-based recombination. @xcite review more compositional data augmentation techniques, situating them within the broader landscape of limited-data learning techniques for nlp. extracting high-value subsets in nlp training data. @xcite ; @xcite ; @xcite propose methods for extracting diverse sets of abstract templates to improve compositional generalization in semantic parsing. @xcite train a baseline model on a small amount of data and use entropy estimates to select new data points for annotation, reducing annotation costs for morphological inflection. @xcite identify effective data subsets for training high-quality models for question answering, finding that small subsets of ambiguous examples perform better than randomly selected ones. our work is also highly related to active-learning in nlp @xcite @xcite ; however we focus on selecting synthetic rather than unlabeled datapoints, and our experiments are geared towards compositional generalization rather than iid performance. compositional data splits in morphological inflection. assessing the generalization capacity of morphological inflections has proven a challenging and multifaceted problem. relying on standard "iid" @xcite splits obfuscated (at least) two different manners in which inflection models fail to generalize compositionally. first, @xcite uncovered that generalizing to novel lexemes was challenging for even state of the art inflection models. experiments by @xcite however showed that the stemcorrupt method could significantly improve generalization to novel lexemes. our work builds on theirs by contributing to understanding the relationship between stemcorrupt and lexical compositional generalization. specifically, we studied the structure of the probability distribution that stemcorrupt promotes (section 3), and the conditions under which it succeeds (remark 1) and fails (remark 2). second, @xcite showed that inflection models also fail to generalize compositionally to novel feature combinations, even with agglutinative languages that have typically have a strong oneto-one alignment between morphological features and affixes. discovering strategies to facilitate com-positional generalization in terms of novel feature combinations remains an open-area of research. ## conclusion this paper presents a novel theoretical explanation for the effectiveness of stemcorrupt, a widelyused data augmentation method, in enhancing compositional generalization in automatic morphological inflection. by applying information-theoretic constructs, we prove that the augmented examples work to improve compositionality by eliminating dependencies between substructures in wordsstems and affixes. building off of our theoretical analysis, we present the first exploration of whether the sample efficiency of reducing these spurious dependencies can be improved. our results show that improved sample efficiency is achievable by selecting subsets of synthetic data reflecting high structural diversity and predictive uncertainty, but there is room for improvement -both in strategic sampling strategies and more cross-linguistically effective data augmentation strategies that can represent long distance phonological alternations. overall, nlp data augmentation strategies are poorly understood @xcite and our work contributes to filling in this gap. through our theoretical and empirical analyses, we provide insights that can inform future research on the effectiveness of data augmentation methods in improving compositional generalization.