OSU at S ig M orphon 2022: Analogical Inflection With Rule Features (2022)

Abstract: OSU’s inflection system is a transformer whose input is augmented with an analogical exemplar showing how to inflect a different word into the target cell. In addition, alignment-based heuristic features indicate how well the exemplar is likely to match the output. OSU’s scores substantially improve over the baseline transformer for instances where an exemplar is available, though not quite matching the challenge winner. In Part 2, the system shows a tendency to over-apply the majority pattern in English, but not Arabic.

## introduction many theories of inflection production propose a central role for memorized word forms in shaping the outcomes for unknown or weakly represented words @xcite . in such memory-based models, speakers retrieve exemplar forms a from memory for which the outcomes b are known and use them to predict the outcome for a word c via a process of analogical reasoning: exemplar source a : exemplar target b :: source c : target d. this type of analogical reasoning is detectable in historical changes @xcite and in experiments with nonce-words @xcite , and underlies some influential computational models of inflection @xcite . recently, @xcite and @xcite show that transformer models for inflection prediction can also benefit from access to exemplars. osu's inflection prediction system foot_0 builds on this recent work, also using a transformer for prediction, but adds a heuristic set of "rule features" intended to make the system more flexible in its use of analogical reasoning. rule features are necessary because the source-target pair c : d may not correspond directly to the exemplar pair due to morphophonological alternations or inflection class mismatch. consider an analogy from anglo-saxon, ēþel : ēþle :: ġelīca : ġelīcan ("homeland", "equal".dat.sg), for which the target suffixes do not match. below is a prediction instance and its desired output, based on previous work: ## results osu entered systems for both part 1 (multilingual inflection; @xcite ) and part 2 (learning trajectories: @xcite ). however, we did not attempt all parts of the part 1 task. first, we ran each language from part 1 with the largest available dataset; we submitted results for the small partition only for languages which lacked a large training set. second, our system relies on being able to recall an exemplar with a known output for the target cell. thus, we did not attempt instances for which the target cell was unseen (lemma-only and neither); for such instances, we output the original lemma as a placeholder prediction. our results overall ( osu surpassed the neural baseline in the known cell, unknown lemma condition by 1% absolute or more on armenian, karelian, polish, slovak, turkish and veps (for all these except armenian, the improvement was at least 10%). it performed worse than baseline on arabic, assamese, hungarian, korean, ludic, old norse and pomak (with a 12% drop on korean) 2 . there is no obvious typological pattern in these results. two slavic languages (polish and slovak) performed excellently while a third (pomak) underperformed; similarly, one finnic language (karelian) performed well while another (ludic) did not. while several underperforming languages used non-latin scripts, which can cause trouble for inflector models @xcite , osu was the best-performing system on gothic, with some words written in gothic script and others in latin characters, and also performed well on khalkha mongolian, written in cyrillic, and on hebrew. task 2 (learning trajectories) did not involve 2 our development score for this condition in korean is 80.679%; our test score is 50.602%, suggesting there may be a dataset mismatch. held-out cells, but did vary the amount of training data. a representative set of learning curves for arabic is shown in figure 1 ; curves for english and german are qualitatively similar. our system experiences a rapid rise in performance between 100 and 300 training items, with diminishing returns around 600 items. we attribute poor early performance to under-regularization; unlike in part 1, we did not use cross-lingual training, which helps to regularize small-data inflectors @xcite . ## system design training using the osu system involves the following steps: (1) generation of training instances, (2) training a language-agnostic string edit model, (3) multilingual training, (4) language-specific finetuning. each training instance includes the input lemma, the morphosyntactic features of the output cell, the language and the language family (each encoded as a character), the exemplar lemma and form (separated by a diacritic), and the rule features. rule features for training instances are generated by aligning the lemma and output form as in @xcite , aligning the exemplar and its output, and then comparing the two. looking only at the lemma and output, we generate features to indicate whether there is a prefix, a suffix, a stem-internal edit, or no edit. based on the comparison, we indicate whether prefix/suffix/stem edits are identical between source-target pairs, contain some but not all matching characters, or are disjoint. for instance, the pair fill ∼ filled (suffix -ed) with exemplar die ∼ died (suffix -d) would be marked suffix to indicate the rule type and simi-lar.suffix to indicate that the edits match partly but not completely. the transformer is not forced to obey these features when generating outputs, but uses them to learn how to attend to the exemplar. for each training item, we generate one instance for inflection prediction (with rule features generated using the gold output) and one for feature prediction, containing the exemplar and cell label, but with the rule features as output. the transformer thus learns to predict a likely alignment configuration for each query/exemplar pair. for example, a feature prediction instance corresponding to example (2) would be: inference is a multistep process involving the following steps: (1) generation of multiple test instances with different exemplars, (2) prediction of rule features for each instance, (3) prediction of inflected forms for each instance, (4) majority voting to produce a single inflected output. in step (1), we sampled 5 random exemplars from the training set for each test item; the exemplar output was always drawn from exactly the same morphosyntactic cell as the target output. 3 we generated an instance for each test item × exemplar. we used the transformer in feature prediction mode to produce rule features for each instance (step 2), then concatenated these rule features with the inputs to produce inflection instances. by re-running the transformer on these augmented instances, we output an inflected form for each instance (step 4). finally, we chose the most likely output across the 5 exemplars as the model's final prediction, with ties broken at random. as an example of this process, suppose the instance ġelīca dat.sg occurred in the test set, and we had selected the pair ēþel : ēþle as one of our five exemplars. we would first generate a feature prediction instance (example 3) and present it to the trained transformer. suppose the transformer incorrectly assumed the suffix would be shared, and output suff same.suff (rather than replace.suff). in step (3), we create an inflection instance using these predicted features: as with any pipelined prediction system, an error cascade may occur; the transformer may not decode this instance correctly due to the incorrect features proposed in the previous step. in any case, we would collect this output, and those of the four other exemplars, and select the most frequently proposed form as the final prediction. system development was carried out before the shared task commenced, using datasets from sig-morphon 2020 @xcite ; we made no effort to tune on the 2022 datasets. ## analysis we analyze some outputs from part 2 with an alignment-based analysis tool as in @xcite ; @xcite , leveraging some of the same code as our rule feature extractor. in english, the model shows a strong preference for over-applying the regular (-ed) suffix throughout the learning process; using the 100-example (severely under-regularized) dataset, the model produces suffixes 84% of the time, but by 200 examples, this rises to 90% and continues to rise slowly thereafter. nearly all of the rise in accuracy is due to the model's gradual acquisition of orthographic allomorphs of -ed, such as drum ∼ drummed, first produced with 400 examples. no irregular allomorphs improve consistently, although some (swear ∼ swore, grind ∼ ground) are occasionally produced correctly. the zero past tense (bet ∼ bet) is produced less often as the dataset increases. in other experiments, we have observed that our model often produces zero outputs when trained with insufficient data; we believe our initial success with this class is the product of this tendency rather than learning. the lack of generalization of irregular allomorphs is generally consistent with the claim of @xcite that infants rarely produce such errors. it is not clear from results on held-out data whether a "u-shaped curve" @xcite would appear, since this phenomenon results from over-application of the regular suffix to previously memorized irregulars and would require inspection of training outputs. it is also likely that the token, as well as type, frequency distribution of the training data matters for the acquisition of irregulars @xcite . in arabic, the model is able to learn suffixing 'sound' plurals starting from the first 100 words, and performs best on these examples overall, reaching over 80% accuracy on concatenative patterns when trained on all available data. the model initially struggles with nonconcatenative 'broken' plural forms, but shows consistent improvement as the amount of training data increases. the alignment method used to generate training instances groups alternations into microclasses, taking changes in short vowel diacritics into account. one of these classes, the caccac ∼ cacaacic class, containing nouns such as maslak ∼ masaalik 'path' (35 examples), reaches 100% accuracy with 600 words. gradual improvement is also seen in nouns of the cacac ∼ 'accaac class, for example khtar ∼ akhtaar 'danger' (50 examples), which goes from 2% accuracy using 100 words to 86% accuracy on the full dataset. another interesting class is the cicaa' ∼ 'acciya class, for example binaa' ∼ 'ibniya 'building', with only 5 examples in the dataset. unlike other microclasses of similar size which the model fails to ever learn, the model is able to accurately produce 4 of the 5 examples (80% accuracy) using the 600-word and 900-word datasets (although with 1,000 words the model only produces 1 of the 5). other similar nouns, such as the caciic ∼ 'accicaa' pattern including qariib ∼ aqribaa' 'relative' (5 examples) are never learned by the model. the model's performance reflects broad generalizations found in the literature on child acquisition of dialectal arabic plural inflection. in general, while nonconcatenative 'broken' plural nouns are present in the speech of very young children, nonconcatenative inflection isn't productive until late preschool @xcite , and a study on the acquisition of plural inflection in egyptian arabic found that children as old as 15 may commonly produce errors when inflecting broken plural nouns in the language @xcite . in their study on plural acquisition of native arabic speakers across multiple age groups, @xcite found that the feminine sound plural marker is acquired earlier and faster than broken plural inflection patterns, and that differences in the production of broken plural forms are affected both by speakers' familiarity with the singular form and the type frequency of its associated plural template. both the human and machine acquisition trajectories are likely related to the sheer number of possible ways (i.e., 'templates') of nonconcatenatively relating singular and plural nouns in semitic languages. there are comparatively far fewer productive suffixes in msa (one feminine and one masculine) than there are templates (perhaps more than 70: plunkett and nakisa (1997)). ## conclusion and future work the competition alerts us to one obvious weak point: our inability to predict fillers for cells in which no training example is given. this is particularly problematic for languages with very large paradigms. such paradigms generally involve some degree of agglutination (separatist exponence) which renders low-frequency cells predictable @xcite . the relationships between cells can be modeled by using multiple input forms to predict a target @xcite . the ability to do this would be a valuable addition to our model. while our system was not the best in the competition, we are encouraged to find that analogical examples allow a transformer inflector to achieve near-state-of-the-art results. an analogical model is both cognitively plausible and easy to implement, and the resulting system is substantially more robust and generalizable than the simple transformer baseline.