Exploring Linguistic Probes for Morphological Generalization (2023)

Abstract: Modern work on the cross-linguistic computational modeling of morphological inflection has typically employed language-independent data splitting algorithms. In this paper, we supplement that approach with language-specific probes designed to test aspects of morphological generalization. Testing these probes on three morphologically distinct languages, English, Spanish, and Swahili, we find evidence that three leading morphological inflection systems employ distinct generalization strategies over conjugational classes and feature sets on both orthographic and phonologically transcribed inputs.

## introduction the current practice in the evaluation of computational morphological inflection models, such as that employed in the sigmorphon, conll-sigmorphon and sigmorphon-unimorph shared tasks @xcite @xcite @xcite @xcite as well as in more targeted studies focused on specific languages or the generalization behavior of computational models @xcite @xcite @xcite , is to train on (lemma, inflection, features) triples and predict inflected forms from held-out (lemma, features) pairs. the algorithm for generating train-test splits is both random and language-independent, which has proven successful in distinguishing morphological inflection models at the gross quantitative level. models differ in their performance across languages and in their ability to generalize across lemmas or feature sets. in this paper, we both replicate this type of analysis and contrast it with new language-specific probes for testing models' generalization abilities *denotes equal contribution in a more controlled fashion. we examine 13 probes in three languages -english, spanish, and swahili -chosen for data availability and their distinct morphological characteristics. in addition, we investigate the effect of presentation style on performance: that is, whether the choice between phonological transcription or orthography has a substantial effect on outcomes. we report on three systems which differ widely in their behavior. they often -but not always -make reasonable linguistic generalizations, even in their incorrect predictions. in addition, we find no statistical effect for presentation style, even on english. this has implications for research that attempts to use neural networks for cognitive modeling and evaluates on orthographic data rather than more domain-appropriate phonological transcriptions. ## languages three languages were chosen whose inflectional morphologies range from entirely fusional (english), to mixed (spanish), to mostly agglutinative (swahili). in highly agglutinative languages, individual features in a set tend to correspond to distinct morphological patterns, so a model may generalize to unseen feature sets by mapping component features to their corresponding patterns. this is exemplified by the swahili example (1), in which most features correspond to individual morphemes; only the person/number prefix maps to more than one feature. on the other hand, highly fusional languages map entire feature sets to single patterns. this is shown by the spanish example (2), in which all features map together onto a single suffix. all data was adapted from kodner et al. (2023b), foot_0 which was in turn extracted from uni-morph 3 and 4 @xcite . the data was subjected to additional processing as described below. for each language, only verbs were extracted, and multi-word expressions were excluded. 2 importantly, no morpheme segmentation is provided in the unimorph data unlike our illustrative examples, so agglutinativity must be discovered by each system. english (germanic): english triples were transcribed using an ipa translation of the cmu pronouncing dictionary. 3 triples without available transliterations were discarded. when the dictionary provided multiple transcriptions, the lemma and inflection transcriptions which minimized levenshtein distance between them were chosen. spanish (romance): spanish triples were transcribed using the epitran package @xcite , which does not include stress information. unimorph treats the infinitive as the lemma and includes the 2nd person plural in its paradigms. swahili (bantu): swahili triples were transcribed using epitran. unimorph treats the bare stem as the lemma. the swahili data set is much smaller than the others and does not contain any forms with negative marking. we found many inconsistencies in the unimorph 4 feature tags which we normalized. most importantly, pfv and prf, which both indicate perfect aspect, were mapped to pfv and tag order was made consistent. triples in swc.sm with tags which could not be clearly mapped to tag sets in swc were excluded. ## data splits we created several random data splits to study dimensions of morphological generalization. as in prior work, blind splits were made without regard to specific linguistic properties of the triples. probe splits were sensitive to the properties of the triples: they were divided into relevant and irrelevant sets according to properties of the feature sets or lemmas. the irrelevant triples were split as in blind in order to pad training to the same length as blind, but irrelevant test triples were discarded. the relevant triples were split in a way specific to each probe, controlling which occurred in train+fine-tuning. for both blind and probe, each split was performed five times with unique random seeds, and each seed was used to produce parallel orthographic and transcription versions of the splits for evaluating presentation style, yielding ten samples in total. data sets contained 1600 training items + 400 for fine-tuning. blind splits contained 1000 test items. blind: following @xcite , this splitting strategy ensures that approximately 50% of test items contain oov feature sets but is otherwise blind to the identity of those feature sets. this was shown in the 2022 sigmorphon-unimorph shared task to create more opportunities for testing dimensions of generalization across feature sets than more traditional uniform random sampling @xcite . ## conclusions in this work, we present language-specific probes to evaluate the ability of computational systems to perform morphological generalizations as a complement to prior work relying on large-scale language-independent data-splitting. systems differ substantially both in their ability to perform generalizations and in their problem-solving strategies. of the two systems showing generalization ability, both perform well on the simplest probes but struggle on more complex but feasible probes. on probes designed to be impossible but insightful, both systems show some degree of reasonable generalization. in addition to this, we find no significant effect of presentation style on the behavior of the three systems. while we maintain that the choice of presentation style should be driven by application when possible (usually orthography for nlp or transcription for cognitive modeling), our results suggest that it can be reasonable to use orthography when transcription is impractical.