SIGMORPHON – U ni M orph 2023 Shared Task 0: Typologically Diverse Morphological Inflection (2023)

Abstract: The 2023 SIGMORPHON–UniMorph shared task on typologically diverse morphological inflection included a wide range of languages: 26 languages from 9 primary language families. The data this year was all lemma-split, to allow testing models’ generalization ability, and structured along the new hierarchical schema presented in (Batsuren et al., 2022). The systems submitted this year, 9 in number, showed ingenuity and innovativeness, including hard attention for explainability and bidirectional decoding. Special treatment was also given by many participants to the newly-introduced data in Japanese, due to the high abundance of unseen Kanji characters in its test set.

## introduction as a long-running shared task, the sigmorphon-unimorph task on morphological inflection is a major engine behind the surging interest in computational morphology, as it facilitated both the building of unimorph as a large multilingual morphological dataset, and the development and testing of morphological inflection models. in its first few installments @xcite @xcite the focus of the task was first and foremost on generalization across languages, with the their number raising steadily from 10 languages in the task of 2016 to 90 languages in 2020. later studies, both in the 2021 shared task @xcite and otherwise @xcite , discovered that the impressive results achieved by systems submitted to these tasks were in large part due the presence of test lemmas in the train set. as a result, the 2022 shared task @xcite focused on generalization to both unseen lemmas and unseen feature combinations. in this task we continue to test systems on the challenging lemma-split setting while circling back to the inclusivity objective that guided the task from its inception. to this end, we employ the hierarchical annotation schema of unimorph 4.0 @xcite that allows more natural annotation of languages with complex morphological structures such as case stacking and polypersonal agreement. this year we include 26 languages from 9 primary language families: albanian, amharic, ancient greek, arabic (egyptian and gulf), armenian, belarusian, danish, english, finnish, french, georgian, german, hebrew, hungarian, italian, japanese, khaling, macedonian, navajo, russian, sámi, sanskrit, spanish, swahili and turkish. the inclusion of japanese, written in kanji characters that are rarely shared across lemmas, compelled all systems this year to find ways to deal with unseen characters in the test set. in total, 9 systems were submitted by 3 teams, both neural and non-neural models, and they were compared against 2 baselines, neural and nonneural as well. the submitted systems experimented with innovative ideas for morphological inflection as well as for sequence-to-sequence modeling in general. @xcite introduced an elaborate attention mechanism between static representations for explainability, and canby and hockenmaier (2023) experimented with a new type of decoder for transformer models that is able to decode from both left to right and vise versa simultaneously. lastly, @xcite improved the non-neural affixing system used as a baseline. the results show that although on average systems achieve impressive results in inflecting unseen lemmas, some languages still present a substantial challenge, mostly extinct languages like ancient greek and sanskrit or low resourced languages like navajo and sámi. in addition, the results point to a dependency on the writing system that could be further explored in future shared tasks. 2 task description this year's task was organized in a very similar fashion to previous iterations. participants were asked to design supervised learning systems which could predict an inflected form given a lemma and a morphological feature set corresponding to an inflectional category, or a cell in a morphological paradigm. they were provided with a training set of several thousands of examples, as well as a development set and test set for each language. the training data consisted of (lemma, feature set, inflected form) triples, while the inflected forms were held out from the test set. the development set was provided in both train-and test-like formats. data was made available to participants in two phases. in the first phase, the training and development sets were provided for most languages. in the second phase, training and development sets were released for some extra ("surprise") languages and the test sets were provided for all languages. 2 schema differences the data this year followed the hierarchical annotation schema that was suggested by @xcite and adopted in uni-morph 4.0 @xcite . the difference that was most pronounced in the data was the replacement of opaque tags that grouped several features such as ac3sm(a 3rd person singular masculine accusative argument) with the hierarchically combined features acc(3,sg,masc), i.e. without introducing a new tag for each feature combination in the cases of polypersonal agreement. ## the languages the selection of languages used in this year's task is varied at almost any dimension. in terms of language genealogy we have representatives of 9 language families, some are widely used, like english and spanish, and others are endangered or extinct, like khaling and sanskrit. the languages employ a wide variety of orthographic systems with varying degrees of transparency @xcite : alphabets (e.g., german), abugidas (e.g., sanskrit), abjads (e.g., hebrew), and even one logographs using language (japanese). in light of the new annotation schema, many languages in this year's selection employ forms that refer to multiple arguments. possessors are marked on nouns in 6 of the languages: hebrew, hungarian, amharic, turkish, armenian and finnish. in addition, polypersonal agreement appears in verbs of 5 of the languages: georgian, spanish, hungarian, khaling and swahili. 3 other notable morphological characteristics include, among others, the ablaut-extensive semitic languages and prefixinclined navajo. all in all, languages new to unimorph a couple of languages, namely swahili and sanskrit, have seen their respective unimorph data increased substantially in size for this task. the swahili data, that previously had partial inflection tables, was expanded using the clause morphology data of @xcite , so a swahili verbal inflection table includes more that 14,000 forms rather than mere 180. the sanskrit data was massively expanded, mostly in terms of the number of lemmas, by incorporating data from gérard huet's sanskrit inflector. 4 in addition, one previously unrepresented language was introduced to unimorph -japanese. the data was crawled from wiktionary and canonicalized to match the unimorph 4.0 format. the usage of kanji characters, logograms of chinese origin that are completely unrepresentative of the pronunciation and almost uniquely used per lemma, can pose an interesting challenge to inflection systems that will have to deal with many unseen characters. ## data preparation all data for this task is provided in standard unimorph format, with training items consisting of (lemma, morphosyntactic features, inflected form) triples. since the goal of the task is to predict inflected forms, the test set was presented as (lemma, features) pairs. the data for all languages was lemma-split @xcite . for each language, a number of inflection tables (i.e., lemmas) were sampled from the entire uni-morph dataset. 80% of the tables were used for the train set, and the rest were split between the validation and the test sets, then 10,000 forms were sampled from the inflection tables of the train set, and 1,000 forms were sampled for the validation and test sets from the respective tables. the number of inflection tables used was capped at 500, in cases where the tables were too small to generate enough data more tables were added until it was sufficient. ## results and analysis ## conclusions this year's shared task further promoted the goals of the recurring unimorph inflection task: we tested innovative inflection systems on a challenging lemma-split data, and did so in an inclusive fashion both in terms of typological diversity of the languages included and the annotation schema that allows treatment of more complex morphological phenomena. we received 9 submitted systems and tested them on 16 typologically diverse languages. the most interesting pattern arising from our results is the greatly varied performance between languages, with the best performing system ranging from 55.6 to 99.4 accuracy percentage. we thus conclude that further research is needed to close this gap. moreover, this year's task gave a prominent role to the orthographic systems of the languages selected, both by including for the first time a logographically written language and by analysing the role of abjad-vocalization in semitic languages. we believe that this direction is a promising lead for promoting the understanding of the factors influencing the performance of inflection models.