SIGMORPHON – U ni M orph 2022 Shared Task 0: Generalization and Typologically Diverse Morphological Inflection (2022)

Abstract: The 2022 SIGMORPHON–UniMorph shared task on large scale morphological inflection generation included a wide range of typologically diverse languages: 33 languages from 11 top-level language families: Arabic (Modern Standard), Assamese, Braj, Chukchi, Eastern Armenian, Evenki, Georgian, Gothic, Gujarati, Hebrew, Hungarian, Itelmen, Karelian, Kazakh, Ket, Khalkha Mongolian, Kholosi, Korean, Lamahalot, Low German, Ludic, Magahi, Middle Low German, Old English, Old High German, Old Norse, Polish, Pomak, Slovak, Turkish, Upper Sorbian, Veps, and Xibe. We emphasize generalization along different dimensions this year by evaluating test items with unseen lemmas and unseen features separately under small and large training conditions. Across the five submitted systems and two baselines, the prediction of inflections with unseen features proved challenging, with average performance decreased substantially from last year. This was true even for languages for which the forms were in principle predictable, which suggests that further work is needed in designing systems that capture the various types of generalization required for the world’s languages.

## task description from the participants' perspective, this task was organized very similarly to previous iterations. participants were asked to design supervised learning systems which could predict an inflected form given a lemma and a morphological feature set corresponding to an inflectional category or cell in a morphological paradigm. they were provided with a small, and data permitting, large training set, as well as a development set and test set for each language. the train and dev sets consisted of (lemma, inflected, feature set) triples, while the inflected forms were held out from the test set. data was made available to participants in two phases. in the first phase, train and dev sets were provided, with the expectation that model development and tuning be carried out primarily on these languages. in the second phase, test sets were released for all languages during the evaluation phase. teams produced predicted inflected forms for each test set. they were given the opportunity to submit two sets of predictions from two separate models, one trained on the small training sets and one trained on the large training sets, with the latter being a super set of the former. ## description of languages this section provides brief descriptions of each language that was newly included or newly updated for this year's task. further information about returning languages can be found in previous years' papers @xcite . ## data preparation all data for this task is provided in standard unimorph format, with training items consisting of (lemma, inflected form, morphosyntactic features) triples. since the goal of the task is to predict inflected forms, the test set was presented as (lemma, features) pairs. data was canonicalized as in previous years using https://github.com/ unimorph/um-canonicalize, which ensures consistent ordering of the features in the feature sets. ## baseline systems the organizers provided one neural and one nonneural baseline system. the neural system, neural, is a character-level transformer @xcite . it is identical to the system chr-trm which was used in the 2021 task. the non-neural system, nonneur, is identical to the non-neural baseline made available in 2020 and 2021. 8 6 submitted systems cluzh @xcite : the cluzh team adapted their earlier model, character-level neural transducer, to work on large datasets @xcite . the model has previously shown superior performance, especially in low-resource scenarios. this year, the team optimized the training procedure using minibatches. they only relied on the teacher-forcing approach, i.e., using gold labels rather than what was predicted during the training phase. morphosyntactic features were treated individually, and their embeddings were summed. the team explored performance of the model across various task settings and demonstrated its ability to capture feature behaviour better than other team's models, especially in the small training condition. the system is identical to the one submitted to this year's acquisitioninspired subtask @xcite . osu @xcite ): osu's system is identical to the one submitted to this year's acquisition-inspired subtask. this inflection system is a transformer whose input is augmented with an analogical exemplar model showing how to inflect a different word into the target cell. in addition, alignment-based heuristic features indicate how well the exemplar is likely to match the output. the system works only when examples of the target cell are present in the training set and can serve as exemplars; otherwise, it outputs the 8 available here: https://github.com/sigmorphon/ 2022inflectionst/tree/main/baselines/nonneural lemma as a placeholder. thus, the system's scores are expected to be higher for the feature overlap and both overlap evaluation categories and very low when the target cell is unknown. tümorph-main @xcite : tü-morph's neural system is a modification of the character-level adaptation of transformer to morphology from @xcite . in particular, the team trained the transformer to predict a distribution over states of fst (whose states are characters) rather than character sequences themselves. the model is scored third on both the small and large training settings. tümorph-fst @xcite : as their second submission, the team manually developed fsts using grammars and corresponding unmorph repositories. since that requires more human labour and linguistic competence, the team focused only on three languages: chukchi, kholosi, and upper sorbian. the resulting fst models outperformed all other submitted systems on two of three languages. the authors confirm earlier observations from beemer et al. (2020) that such systems are able to reach superior results compared to neural ones, especially in low-resource scenarios and high morphological complexity, but require substantially more human working hours. ubc @xcite : the ubc team proposed enriching the character-level transformer of @xcite with reverse positional embeddings to better account for suffixing, one of the most common word formation processes. in addition, the team explored a synthetic data augmentation technique proposed by @xcite and student-forcing (nicolai and silfverberg, 2020), a training strategy where the model outputs are replaced with gold labels for some percentage of samples to alleviate exposure bias. data augmentation leads to significant improvements, especially in the small training condition, confirming its utility. the student forcing training also provides a certain accuracy gain but presents mixed results when used together with data hallucination. flexica @xcite is a modified version of the non-neural system submitted to the sigmorphon 2020 shared task on morphological reinflection @xcite . the system is based on refined alignment patterns between lemmas and inflected forms. in this year's submission, grammatical tag interchangeability learning was added to address smaller fea- ## results and evaluation performance was evaluated by exact match accuracy. macro-averages across languages on the entire test set and partitioned over the four overlap types are provided in a few points stand out immediately. first, overall performance is much lower this year compared to last year's similar task. during the 2021 iteration, all systems achieved over 90% accuracy on most of languages, while this year, no system achieves over 72% average in either training condition. this task was designed to be particularly challenging because the test set required systems to make predictions with only partial information. the results bore out this expectation. flexica, the only general non-neural submitted system, surpasses the non-neural baseline, but does not surpass 40% overall accuracy in either training condition. being a hand-built system, tümorph-fst outperformed all other systems on two of three languages that it was developed for. as expected, all systems that submitted full or nearly full predictions for both the small and large training conditions performed substantially better with more training data. cluzh, tümorph-main, ubc, and the neural baseline each improved by over ten points, while flexica and the non-neural baseline showed smaller gains of around four points. ubc achieved the highest performance of any system in either training condition. to understand why this is, it is necessary to look at a breakdown of performance by overlap type. the system is more resilient to novel feature sets than any other except for the hand-built fsts. ## error analysis by language this section contains qualitative error analysis for six languages from five different top-level families. ## discussion this year's shared task investigated two dimensions of generalization in morphological inflection: generalization over lemmas and generalization over inflectional categories. test items with lemmas or feature sets that were attested in training were evaluated separately from those with novel lemmas or feature sets to gain a better understanding of generalization. this proved to be a challenging version of the task, as performance is substantially lower across systems compared to previous years. we carried on the tradition of including a range of typologically diverse languages in the task. from the perspective of the two dimensions of generalization, different morphological paradigms could prove more or less challenging. in particular, it is more reasonable to expect a system to generalize to an unseen feature set if the form of the corresponding inflectional category is in some way derived from forms associated with each of the member features. similarly, a language with relatively invariant stem forms and little unpredictable stem-conditioned realization of inflectional categories should be conducive to generalization across lemmas, while a language with more stem changes or lexically arbitrary inflectional classes should prove more challenging. two major patterns emerged which held across systems. first, overall averages were lower than previous years in which overlaps between lemmas and features in training and test were left uncontrolled. the task was challenging. second, performance test items with novel feature sets was almost uniformly weaker than performance on test items with novel lemmas. this was true for all systems and still held true for agglutinative languages which stood the best chance of generalization across feature sets.