SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection (2020)

Abstract: A broad goal in natural language processing (NLP) is to develop a system that has the capacity to process any natural language. Most systems, however, are developed using data from just one language such as English. The SIGMORPHON 2020 shared task on morphological reinflection aims to investigate systems’ ability to generalize across typologically distinct languages, many of which are low resource. Systems were developed using data from 45 languages and just 5 language families, fine-tuned with data from an additional 45 languages and 10 language families (13 in total), and evaluated on all 90 languages. A total of 22 systems (19 neural) from 10 teams were submitted to the task. All four winning systems were neural (two monolingual transformers and two massively multilingual RNN-based models with gated attention). Most teams demonstrate utility of data hallucination and augmentation, ensembles, and multilingual training for low-resource languages. Non-neural learners and manually designed grammars showed competitive and even superior performance on some languages (such as Ingrian, Tajik, Tagalog, Zarma, Lingala), especially with very limited data. Some language families (Afro-Asiatic, Niger-Congo, Turkic) were relatively easy for most systems and achieved over 90% mean accuracy while others were more challenging.

## introduction human language is marked by considerable diversity around the world. though the world's languages share many basic attributes (e.g., @xcite , grammatical features, and even abstract implications (proposed in @xcite , each language nevertheless has a unique evolutionary trajectory that is affected by geographic, social, cultural, and other factors. as a result, the surface form of languages varies substantially. the morphology of languages can differ in many ways: some exhibit rich grammatical case systems (e.g., 12 in erzya and 24 in veps) and mark possessiveness, others might have complex verbal morphology (e.g., oto-manguean languages; palancar and léonard, 2016) or even "decline" nouns for tense (e.g., tupi-guarani languages). linguistic typology is the discipline that studies these variations by means of a systematic comparison of languages @xcite . typologists have defined several dimensions of morphological variation to classify and quantify the degree of crosslinguistic variation. this comparison can be challenging as the categories are based on studies of known languages and are progressively refined with documentation of new languages @xcite . nevertheless, to understand the potential range of morphological variation, we take a closer look at three dimensions here: fusion, inflectional synthesis, and position of case affixes @xcite . fusion, our first dimension of variation, refers to the degree to which morphemes bind to one another in a phonological word @xcite . languages range from strictly isolating (i.e., each morpheme is its own phonological word) to concatenative (i.e., morphemes bind together within a phonological word); nonlinearities such as ablaut or tonal morphology can also be present. from a geographic perspective, isolating languages are found in the sahel belt in west africa, southeast asia and the pacific. ablaut-concatenative morphology and tonal morphology can be found in african languages. tonal-concatenative morphology can be found in mesoamerican languages (e.g., oto-manguean). concatenative morphology is the most common system and can be found around the world. inflectional synthesis, the second dimension considered, refers to whether grammatical categories like tense, voice or agreement are expressed as affixes (synthetic) or individual words (analytic) @xcite . analytic expressions are common in eurasia (except the pacific rim, and the himalaya and caucasus mountain ranges), whereas synthetic expressions are used to a high degree in the americas. finally, affixes can variably surface as prefixes, suffixes, infixes, or circumfixes @xcite . most eurasian and australian languages strongly favor suffixation, and the same holds true, but to a lesser extent, for south american and new guinean languages @xcite . in mesoamerican languages and african languages spoken below the sahara, prefixation is dominant instead. these are just three dimensions of variation in morphology, and the cross-linguistic variation is already considerable. such cross-lingual variation makes the development of natural language processing (nlp) applications challenging. as @xcite notes, many current architectures and training and tuning algorithms still present language-specific biases. the most commonly used language for developing nlp applications is english. along the above dimensions, english is productively concatenative, a mixture of analytic and synthetic, and largely suffixing in its inflectional morphology. with respect to languages that exhibit inflectional morphology, english is relatively impoverished. 1 importantly, english is just one morphological system among many. a larger goal of natural language processing is that the system work for any presented language. if an nlp system is trained on just one language, it could be missing important flexibility in its ability to account for cross-linguistic morphological variation. in this year's iteration of the sigmorphon shared task on morphological reinflection, we specifically focus on typological diversity and aim to investigate systems' ability to generalize across typologically distinct languages many of which are low-resource. for example, if a neural network architecture works well for a sample of indo-european languages, should the same architecture also work well for tupi-guarani languages (where nouns are "declined" for tense) or austronesian languages (where verbal morphology is frequently prefixing)? ## task description the 2020 iteration of our task is similar to conll-sigmorphon 2017 @xcite and 2018 @xcite in that participants are required to design a model that learns to generate inflected forms from a lemma and a set of morphosyntactic features that derive the desired target form. for each language we provide a separate training, development, and test set. more historically, all of these tasks resemble the classic "wug"-test that @xcite developed to test child and human knowledge of english nominal morphology. unlike the task from earlier years, this year's task proceeds in three phases: a development phase, a generalization phase, and an evaluation phase, in which each phase introduces previously unseen data. the task starts with the development phase, which was an elongated period of time (about two months), during which participants develop a model of morphological inflection. in this phase, we provide training and development splits for 45 languages representing the austronesian, niger-congo, oto-manguean, uralic and indo-european language families. finally, test splits for all 90 languages are released in the evaluation phase. during this phase, the models are evaluated on held-out forms. importantly, the languages from both previous phases are evaluated simultaneously. this way, we evaluate the extent to which models (especially those with shared parameters) overfit to the development data: a model based on the morphological patterning of the indo-european languages may end up with a bias towards suffixing and will struggle to learn prefixing or infixation. ## meet our languages in the 2020 shared task we cover 15 language families: afro-asiatic, algic, austronesian, dravidian, indo-european, niger-congo, oto-manguean, sino-tibetan, siouan, songhay, southern daly, tungusic, turkic, uralic, and uto-aztecan. 2 five language families were used for the development phase while ten were held out for the generalization phase. tab. 1 and tab. 2 provide information on the languages, their families, and sources of data. in the following section, we provide an overview of each language family's morphological system. ## baseline systems the organizers provided two types of pre-trained baselines. their use was optional. ## competing systems as tab. 3 shows, 10 teams submitted 22 systems in total, out of which 19 were neural. some teams such as eth zurich and uiuc built their models on top of the proposed baselines. in particular, eth zurich enriched each of the (multilingual) neural baseline models with exact decoding strategy that uses dijkstra's search algorithm. uiuc enriched the transformer model with synchronous bidirectional decoding technique @xcite in order to condition the prediction of an affix character on its environment from both sides. (the authors demonstrate positive effects in oto-manguean, turkic, and some austronesian languages.) a few teams further improved models that were among top performers in previous shared tasks. ims and flexica re-used the hard monotonic attention model from @xcite . ims developed an ensemble of two models (with left-to-right and right-to-left generation or-der) with a genetic algorithm for ensemble search @xcite and iteratively provided hallucinated data. flexica submitted two neural systems. the first model (flexica-02-1) was multilingual (family-wise) hard monotonic attention model with improved alignment strategy. this model is further improved (flexica-03-1) by introducing a data hallucination technique which is based on phonotactic modelling of extremely low-resource languages @xcite . lti focused on their earlier model (anastasopoulos and neubig, 2019), a neural multi-source encoder-decoder with two-step attention architecture, training it with hallucinated data, cross-lingual transfer, and romanization of scripts to improve performance on low-resource languages. deepspin reimplemented gated sparse two-headed attention model from @xcite and trained it on all languages at once (massively multilingual). the team experimented with two modifications of the softmax function: sparsemax (martins and astudillo, 2016, deepspin-02-1) and 1.5-entmax @xcite , deepspin-01-1). many teams based their models on the transformer architecture. nyu-cuboulder experimented with a vanilla transformer model (nyu-cuboulder-04-0), a pointer-generator transformer that allows for a copy mechanism (nyu-cuboulder-02-0), and ensembles of three (nyu-cuboulder-01-0) and five (nyu-cuboulder-03-0) pointer-generator transformers. for languages with less than 1,000 training samples, they also generate hallucinated data. culing developed an ensemble of three (monolingual) transformers with identical architecture but different input data format. the first model was trained on the initial data format (lemma, target tags, target form). for the other two models the team used the idea of lexeme's principal parts @xcite and augmented the initial input (that only used the lemma as a source form) with entries corresponding to other (non-lemma) slots available for the lexeme. the cmu tartan team compared performance of models with transformer-based and lstm-based encoders and decoders. the team also compared monolingual to multilingual training in which they used several (related and unrelated) high-resource languages for low-resource language training. team description system model features neural ensemble multilingual hallucination baseline wu and cotterell (2019) mono-single mono-aug-single mono-shared mono-aug-shared wu et al. (2020) trm-single trm-aug-single trm-shared trm-aug-shared cmu tartan jayarao et al. (2020) cmu_tartan_00-0 cmu_tartan_00-1 cmu_tartan_01-0 cmu_tartan_01-1 cmu_tartan_02-1 cu7565 beemer et al. (2020) cu7565-01-0 cu7565-02-0 culing liu and hulden (2020) culing-01-0 deepspin peters and martins (2020) deepspin-01-1 deepspin-02-1 eth zurich forster and meister (2020) ethz00-1 ethz02-1 flexica scherbakov (2020) flexica-01-0 flexica-02-1 flexica-03-1 ims yu et al. (2020) ims-00-0 lti murikinati and anastasopoulos (2020) lti-00-1 nyu-cuboulder singer and kann (2020) nyu-cuboulder-01-0 nyu-cuboulder-02-0 nyu-cuboulder-03-0 nyu-cuboulder-04-0 uiuc canby et al. (2020) uiuc-01-0 ## evaluation this year, we instituted a slightly different evaluation regimen than in previous years, which takes into account the statistical significance of differences between systems and allows for an informed comparison across languages and families better than a simple macro-average. the process works as follows: ## results this year we had four winning systems (i.e., ones that outperform the best baseline): culing-01-0, deepspin-02-1, uiuc-01-0, and deepspin-01-1, all neural. as tab. 5 shows, they achieve over 90% accuracy. although culing-01-0 and uiuc-01-0 are both monolingual transformers that do not use any hallucinated data, they follow different strategies to improve performance. the strategy proposed by culing-01-0 of enriching the input data with extra entries that included non-lemma forms and their tags as a source form, enabled their system to be among top performers on all language families; uiuc-01-0, on the other hand, did not modify the data but rather changed the decoder to be bidirectional and made family-wise fine-tuning of each (monolingual) model. the system is also among the top performers on all language families except iranian. the third team, deepspin, trained and fine-tuned their models on all language data. both models are ranked high (although the sparsemax model, deepspin-02-1, performs better overall) on most language groups with exception of algic. sparsemax was also found useful by cmu-tartan. the neural ensemble model with data augmentation from ims team shows superior performance on languages with smaller data sizes (under 10,000 samples). lti and flexica teams also observed positive effects of multilingual training and data hallucination on low-resource languages. the latter was also found useful in the ablation study made by nyu-cuboulder team. several teams aimed to address particular research questions; we will further summarize their results. is developing morphological grammars manually worthwhile? this was the main question asked by cu7565 who manually designed finitestate grammars for 25 languages. paradigms of some languages were relatively easy to describe but neural networks also performed quite well on them even with a limited amount of data. for lowresource languages such as ingrian and tagalog the grammars demonstrate superior performance but this comes at the expense of a significant amount of person-hours. what is the best training strategy for lowresource languages? teams that generated hallucinated data highlighted its utility for lowresource languages. augmenting the data with tuples where lemmas are replaced with nonlemma forms and their tags is another technique that was found useful. in addition, multilingual training and ensembles yield extra gain in terms of accuracy. are the systems complementary? to address this question, we evaluate oracle scores for baseline systems, submitted systems, and all of them together. typically, as tables 8-21 in the appendix demonstrate, the baselines and the submissions are complementary -adding them together increases the oracle score. furthermore, while the full systems tend to dominate the partial systems (that were designed for a subset of languages, such as cu7565-01-0), there are a number of cases where the partial systems find the solution when the full systems don't -and these languages often then get even bigger gains when combined with the baselines. this even happens when the accuracy of the baseline is very high -finnish has baseline oracle of 99.89; full systems oracle of 99.91; submission oracle of 99.94 and complete oracle of 99.96, so an ensemble might be able to improve on the results. the largest gaps in oracle systems are observed in algic, oto-manguean, sino-tibetan, southern daly, tungusic, and uto-aztecan families. foot_6 ## error analysis in our error analysis we follow the error type taxonomy proposed in @xcite . first, we evaluate systematic errors due to inconsistencies in the data, followed by an analysis of whether having seen the language or its family improved accuracy. we then proceed with an overview of accuracy for each of the language families. for a select number of families, we provide a more detailed analysis of the error patterns. tab. 6 and tab. 7 provide the number of samples in the training, development, and test sets, percentage of inconsistent entries (the same lemma-tag pair has multiple infected forms) in them, percentage of contradicting entries (same lemma-tag pair occurring in train and development or test sets but assigned to different inflected forms), and percentage of entries in the development or test sets containing a lemma observed in the training set. the train, development and test sets contain 2%, 0.3%, and 0.6% inconsistent entries, respectively. azerbaijani (aze), old english (ang), cree (cre), danish (dan), middle low german (gml), kannada (kan), norwegian bokmål (nob), chichimec (pei), and veps (vep) had the highest rates of inconsistency. these languages also exhibit the highest percentage of contradicting entries. the inconsistencies in some finno-ugric languages (such as veps and ludic) are due to dialectal variations. the overall accuracy of system and language pairings appeared to improve with an increase in the size of the dataset (fig. 6 ; see also fig. 7 for accuracy trends by language family and fig. 8 for accuracy trends by system). overall, the variance was considerable regardless of whether the language family or even the language itself had been observed during the development phase. a linear mixed-effects regression was used to assess variation in accuracy using fixed effects of language category, the size of the training dataset (log count), and their interactions, as well as random intercepts for system and language family accuracy. 10 language category was sum-coded with three levels: development language-development family, surprise language-development family, or surprise language-surprise family. a significant effect of dataset size was observed, such that a one unit increase in log count corresponded to a 2% increase in accuracy (β = 0.019, @xmath0.001). language category type also significantly influenced accuracy: both development languages and surprise languages from development families were less accurate on average (β dev-@xmath1.145, β sur-@xmath2.167, each @xmath3.001). these main effects were, however, significantly modulated by interactions with dataset size: on top of the main effect of dataset size, accuracy for development languages increased an additional ≈ 1.7% (β dev-dev×@xmath4.017, @xmath5.001) and accuracy for surprise languages from development families increased an additional ≈ 2.9% (β sur-dev×@xmath6.029, @xmath7.001). ## conclusion this years's shared task on morphological reinflection focused on building models that could generalize across an extremely typologically diverse set of languages, many from understudied language families and with limited available text resources. as in previous years, neural models performed well, even in relatively low-resource cases. submissions were able to make productive use of multilingual training to take advantage of commonalities across languages in the dataset. data augmentation techniques such as hallucination helped fill in the gaps and allowed networks to generalize to unseen inputs. these techniques, combined with architecture tweaks like sparsemax, resulted in excellent overall performance on many languages (over 90% accuracy on average). however, the task's focus on typological diversity revealed that some morphology types and language families (tungusic, oto-manguean, south-ern daly) remain a challenge for even the best systems. these families are extremely low-resource, represented in this dataset by few or a single language. this makes cross-linguistic transfer of similarities by multilanguage training less viable. they may also have morphological properties and rules (e.g., evenki is agglutinating with many possible forms for each lemma) that are particularly difficult for machine learners to induce automatically from sparse data. for some languages (ingrian, tajik, tagalog, zarma, and lingala), optimal performance was only achieved in this shared task by hand-encoding linguist knowledge in finite state grammars. it is up to future research to imbue models with the right kinds of linguistic inductive biases to overcome these challenges.