U niversity of I llinois Submission to the SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection (2020)

Abstract: The objective of this shared task is to produce an inflected form of a word, given its lemma and a set of tags describing the attributes of the desired form. In this paper, we describe a transformer-based model that uses a bidirectional decoder to perform this task, and evaluate its performance on the 90 languages and 18 language families used in this task.

## introduction the world's languages vary greatly in the richness and complexity of their morphological inflection systems. indo-european languages such as latin or german tend to inflect words by adding suffixes to a meaning-bearing root, while austronesian languages like malay or tagalog use circumfixes to change the forms of nouns and verbs. it is important that natural language processing (nlp) systems be able to generate inflected forms for a variety of languages, which can be used in downstream tasks such as language modeling or machine translation. task 0 of the sigmorphon 2020 shared task @xcite encourages the development of morphological transduction models for a variety of the world's language families. since the task features such a diverse set of languages, it is important to create a generalized model that is not overly biased toward certain language typologies. in this paper, we present the university of illinois submission to the task. we have modified the baseline transformer model @xcite to use bidirectional decoding, following the work in @xcite . we believe the additional attention provided by the right-to-left decoding direction improves performance on many of the languages in the dataset. our model outperforms the baseline transformer model on average rank and is among the best performing submissions for this year's task. ## task the objective of task 0 of the sigmorphon 2020 shared task @xcite is to build a system that learns to generate morphological inflections. the model takes a lemma and a group of morphosyntactic tags as input and outputs the word inflected in the desired form. the following example comes from the german dataset: here, we want to inflect predigen in the form specified by the tags v;imp;sg;2, a 2nd person singular imperative verb. the desired output is predig. ## experimental results acc. avg. edit dist. ≥ > ≤ < development 27 18 30 14 surprise 29 13 33 15 it is clear that by either metric, our model equals or outperforms the baseline on more than half of the languages, demonstrating that our model generally does not perform worse than the baseline. tables 5 and 6 in appendix a.2 present full results on every language in the dataset. it is interesting to consider the l2r column, which indicates the percentage of test examples on which an l2r hypothesis was selected over an r2l hypothesis. there is considerable spread in the values of this column; this demonstrates that some languages strongly prefer one direction over the other, while others do not favor one direction in particular. it is important to remember that even though the inference algorithm returns only the best l2r or r2l hypothesis, the chosen direction is conditioned on the opposite direction; therefore, a language that appears to strongly prefer one direction may still gain important insight from the opposite direction. ## conclusion & future work the promising results of our experiments demonstrate that some languages may be amenable to bidirectional decoding; however, more investigation is required to fully understand the merits of such an approach. for example, our results show that some languages strongly favor l2r or r2l hypotheses while others are less preferential. we would like to determine if there are particular linguistic features that make one direction more valuable than the otherfor example, do inflected forms with suffixes prefer l2r decoding while in-flected forms with prefixes favor r2l decoding? we propose performing this analysis by exploring correlations with linguistic features in the wals database @xcite . we would also like to investigate how often each direction produces the correct form, as well as the percentage of examples on which the two directions agree with each other. a high disagreement could indicate a higher value in one direction with respect to the other for a particular language. it would also be informative to compare the bidirectional decoding approach with a purely r2l transformer baseline, in addition to the l2r baseline provided by the organizers. we also suspect that the bidirectional beam search algorithm can be improved if the hypotheses in one direction are paired with each of the hypotheses in the opposite direction when fed to the decoder at each time step. furthermore, once the halfway-point of the target form is passed in the decoding, we should expect lots of overlap between the l2r and r2l forms. we would like to see if this information can be used to join the l2r and r2l predictions to produce a better inflected form. in initial experiments we noticed that on some languages the bidirectional decoding model converges in considerably fewer epochs than the baseline transformer model, despite the same number of parameters. we want to fully investigate this phenomenon because, if it holds for many languages, it means that the model can gain insight more quickly with both directions than with just one. finally, in this work our models were trained from scratch on each individual language. we would like to investigate multilingual approaches by training separate models on individual language families or a single model for every involved language. in these ways, we hope to demonstrate the merits of bidirectional decoding and its implications for a morphological transduction task.