SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages (2021)

Abstract: This year’s iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features. In terms of the task, we enrich UniMorph with new data for 32 languages from 13 language families, with most of them being under-resourced: Kunwinjku, Classical Syriac, Arabic (Modern Standard, Egyptian, Gulf), Hebrew, Amharic, Aymara, Magahi, Braj, Kurdish (Central, Northern, Southern), Polish, Karelian, Livvi, Ludic, Veps, Võro, Evenki, Xibe, Tuvan, Sakha, Turkish, Indonesian, Kodi, Seneca, Asháninka, Yanesha, Chukchi, Itelmen, Eibela. We evaluate six systems on the new data and conduct an extensive error analysis of the systems’ predictions. Transformer-based models generally demonstrate superior performance on the majority of languages, achieving >90% accuracy on 65% of them. The languages on which systems yielded low accuracy are mainly under-resourced, with a limited amount of data. Most errors made by the systems are due to allomorphy, honorificity, and form variation. In addition, we observe that systems especially struggle to inflect multiword lemmas. The systems also produce misspelled forms or end up in repetitive loops (e.g., RNN-based models). Finally, we report a large drop in systems’ performance on previously unseen lemmas.

## task description in this shared task, the participants were told to design a model that learns to generate morphological inflections from both a lemma and a set of morphosyntactic features of the target form. specifically, each language in the task had its own training, development, and test splits. the training and development splits contained triples, with a lemma, a set of morphological features, and the target inflected form, while test splits only provided lemmas and morphological tags: the participants' models needed to predict the missing target form-making this a standard supervised learning task. the target of the task, however, was to analyse how well the current state-of-the-art reinflection models could generalise across a typologically diverse set of languages. these models should, in theory, be general enough to work for natural languages of any typological patterning. 2 as such, we designed the task in three phases: a development phase, a generalization phase, and an evaluation phase. as the phases advanced, more data and more languages were released. in the development phase, we provided training and development splits that should be used by participants to develop their systems. model development, evaluation, and hyper-parameter tuning were, thus, mainly performed on these sets of languages. we will refer to these as the development languages. in the generalization phase, we provided training and development splits for new languages where approximately half were genetically related (belonged to the same family) and half were genetically unrelated (either isolates or belonging to different families) to the development languages. these languages (and their families) were kept as a surprise throughout the first (development) phase and were only announced later on. as the participants were only given a few days with access to these languages before the submission deadline, we expected that the systems couldn't be radically improved to work on them-as such, these languages allowed us to evaluate the generalization capacity of the re-inflection models, and how well they performed on new typologically unrelated languages. finally, in the evaluation phase, the participants' models were evaluated on held-out test forms from all of the languages of the previous phases. the languages from the development phase and the generalization phase were evaluated simultaneously. the only difference between the development and generalization languages was that participants had more time to construct their models for the languages released in the development phase. it follows that a model could easily favor or overfit to the phenomena that are more frequent in the languages presented in the development phase, especially if the parameters were shared across languages. for instance, a model based on the morphological patterning of indo-european languages may end up with a bias towards suffixing and would struggle to learn prefixing or circumfixation, and the degree of the bias only becomes apparent during experimentation on other languages whose inflectional morphology patterns differ. further, the model architecture itself could also explicitly or implicitly favor certain word formation types @xcite . ## data preparation as in the previous editions, each instance in the provided training and development is in a form of a triple (lemma, tag, inflected form). the test set, on the other hand, was released with only lemmas and tags (i.e. without the target inflections). producing these data sets required a few extra steps, which we discuss in this section. conversion into the unimorph schema. after the data collection was finalised for the above languages, we converted them to the unimorph schema-canonicalising them in the process. 14 this process consisted mainly of typo corrections (e.g. removing an incorrectly placed space in a tag, "priv " → "priv"), removing redundant tags (e.g. duplicated verb annotation, "v;v.ptcp" → "v.ptcp"), and fixing tags to conform to the uni-morph schema (e.g. "2;incl" → "2+incl"). these changes were implemented via languagespecific bash scripts. given this freshly converted data, we canonicalised its tag annotations, making use of https://github.com/unimorph/ um-canonicalize . this process sorts the inflection tags into their canonical order and verifies that all the used tags are present in the ground truth unimorph schema, flagging potential data issues in the process. ## evaluation following the evaluation procedure established in the previous shared task iterations, we compare all systems in terms of their test set accuracy. in addition, we perform an extensive error analysis for most languages. ## results as this is likely due to the data being derived from morphological transducers where certain parts of verbal paradigms were excluded (see section 3.8.1). on the other hand, the accuracy on classical syriac, chukchi, itelmen, kunwinjku, braj, ludic, eibela, evenki, xibe is low overall. most of them are under-resourced and have very limited amounts of data-indeed, the spearman correlation between the transformer model's performance and a language's training set size is of roughly 77%. ## error analysis as @xcite note, accuracy-level evaluation might be sufficient to compare model variants but does not provide much insight into the understanding of morphological systems and their learnability. therefore, we now turn to a more detailed analysis of mispredictions made by the systems. for the purpose of this study, we will rely on the error type taxonomy proposed by @xcite and @xcite . ## conclusion in the development of this shared task we added new data for 32 languages (13 language families) to unimorph-most of which are under-resourced. further, we evaluated the performance of morphological reinflection systems on a typologically diverse set of languages and performed fine-grained analysis of their error patterns in most of these languages. the main challenge for the morphological reinflection systems is still (as expected) handling low-resource scenarios (where there is little training data). we further identified a large gap in these systems' performance between the test lemmas present in the training set and the previously unseen lemmas-the latter are naturally hard test cases, but the work on reinflection models could focus on improving these results going forward, following, for instance, the work of @xcite . further, allomorphy, honorificity and multiword lemmas also pose challenges for the current models. we hope that the analysis presented here, together with the new expansion of the unimorph resources, will help drive further improvements in morphological reinflection. following @xcite , we would like to emphasize that linguistic analyses using unimorph should be performed with some degree of caution, since for many languages it might not provide an exhaustive list of paradigms and variants.