Training Strategies for Neural Multilingual Morphological Inflection (2021)

Abstract: This paper presents the submission of team GUCLASP to SIGMORPHON 2021 Shared Task on Generalization in Morphological Inflection Generation. We develop a multilingual model for Morphological Inflection and primarily focus on improving the model by using various training strategies to improve accuracy and generalization across languages.

## introduction morphological inflection is the task of transforming a lemma to its inflected form given a set of grammatical features (such as tense or person). different languages have different strategies, or morphological processes such as affixation, circumfixation, or reduplication among others @xcite . these are all ways to make a lemma express some grammatical features. one way to characterize how languages express grammatical features is a spectrum of morphological typology ranging from agglutinative to isolating. in agglutinative languages, grammatical features are encoded as bound morphemes attached to the lemma, while in isolating languages each grammatical feature is represented as a lemma. thus, languages in different parts of this spectrum will have different strategies for expressing information given by grammatical features. in recent years, statistical and neural models have been performing well on the task of morphological inflection @xcite @xcite . we follow this tradition and implement a neural multilingual model for morphological inflection. in a multilingual system, a single model is developed to handle input from several different languages: we can give the model either a word in evenk or russian and it perform inflection. this is a challenging problem for several reasons. for many languages resources are scarce, and a multilingual system must balance the training signals from both high-resource and low-resource languages such that the model learns something about both. additionally, different languages employ different morphological processes to inflect words. in addition to languages employing a variety of different morphological processes, different languages use different scripts (for example arabic, latin, or cyrillic), which can make it hard to transfer knowledge about one language to another. to account for these factors a model must learn to recognize the different morphological processes, the associated grammatical features, the script used, and map them to languages. in this paper, we investigate how far these issues can be tackled using different training strategies, as opposed to focusing on model design. of course, in the end, an optimal system will be a combination of a good model design and good training strategies. we employ an lstm encoder-decoder architecture with attention, based on the architecture of @xcite , as our base model and consider the following training strategies: the data released cover 38 languages of varying typology, genealogy, grammatical features, scripts, and morphological processes. the data for the different languages vary greatly in size, from 138 examples (ludic) to 100310 (turkish). for the low-resourced languages foot_0 we extend the original dataset with hallucinated data @xcite to train on. with respect to the work of anastasopoulos and neubig (2019), we make the following changes. we identify all subsequences of length 3 or more that overlap in the lemma and inflection. we then randomly sample one of them, denoted r, as the sequence to be replaced. for each language, we compile a set c l containing all (1,2,3,4)-grams in the language. we construct a string g to replace r with by uniformly sampling n-grams from c l and concatenating them @xmath0, ..., g m ) until we have a sequence whose length satisfy: |r| -2 ≤ |g| ≤ |r| + 2. additionally, we do not consider subsequences which include a phonological symbol. 2 a schematic of the hallucination process is shown in figure 1 . ## method in this section, the multilingual model and training strategies used are presented. 3 we employ a single model with shared parameters across all languages. ## results the results from our system using the four straining strategies presented earlier are presented in two of the worst languages in our model is classical syriac (syc) and xibe (sjo). an issue with classical syriac is that the language uses a unique script, the syriac abjad, which makes it difficult for the model to transfer information about operations and common character combinations/transformations into classical syriac from related languages such as modern standard arabic (spoken in the region). for xibe there is a similar story: it uses the sibe alphabet which is a variant of manchu script, which does not occur elsewhere in our dataset. ## language similarities our model process many languages simultaneously, thus it would be encouraging if the model also was able to find similarities between languages. to explore this we investigate whether the language embeddings learned by the model produce clusters of language families. a t-sne plot of the language embeddings is shown in figure 4 . the plot shows that the model can find some family resemblances between languages. for example, we have a uralic cluster consisting of the languages veps (vep), olonets (olo), and karelian (krl) which are all spoken in a region around russia and finland. however, ludic (lud) and võro (vro) are not captured in this cluster, yet they are spoken in the same region. we can see that the model seem to separate language families somewhat depending on the script used. the afro-asiatic languages are split into two smaller clusters, one cluster containing the languages that use standard arabic (ara, afv and arz) script and one cluster that use amharic and hebrew (amh, heb) script. as mentioned earlier classical syriac uses its another script and seems to consequently appear in another part of the map. in general, our model's language embeddings appear to learn some relationships between languages, but certainly not all of them. however, that we find some patterns in encouraging for future work. ## scheduled sampling we note that during the development all of our training strategies showed a stronger performance for the task, except one: scheduled sampling. we hypothesize this is because the low-resource languages benefit from using the gold as input when predicting the next character, while high-resource languages do not need this as much. the model has seen more examples from high-resource languages and thus can model them better, which makes using the previous hidden state more reliable as input when predicting the next token. indeed, the scheduled sampling degrade the overall performance by 3.04 percentage points, increasing our total average accuracy to 83.3 percentage points, primarily affecting low-resource languages. ## conclusions and future work we have presented a single multilingual model for morphological inflection in 38 languages enhanced with different training strategies: curriculum learning, multi-task learning, scheduled sampling and language-wise label smoothing. the results indicate that our model to some extent capture similarities between the input languages, however, languages that use different scripts appears problematic. a solution to this would be to employ transliteration @xcite . in future work, we plan on exploring curriculum learning in more detail and move away from estimating the competence of our model linearly, and instead, estimate the competence using the accuracy on the batches. another interesting line of work here is instead of scoring the examples by model loss alone, but combine it with insights from language acquisition and teaching, such as sorting lemmas based on their frequency in a corpus @xcite . we also plan to investigate language-wise label smoothing more closely, specifically how the value of α should be fine-tuned with respect to the number of characters and languages.