BME Submission for SIGMORPHON 2021 Shared Task 0. A Three Step Training Approach with Data Augmentation for Morphological Inflection (2021)

Abstract: We present the BME submission for the SIGMORPHON 2021 Task 0 Part 1, Generalization Across Typologically Diverse Languages shared task. We use an LSTM encoder-decoder model with three step training that is first trained on all languages, then fine-tuned on each language family and finally fine-tuned on individual languages. We use a different type of data augmentation technique in the first two steps. Our system outperformed the only other submission. Although it remains worse than the Transformer baseline released by the organizers, our model is simpler and our data augmentation techniques are easily applicable to new languages. We perform ablation studies and show that the augmentation techniques and the three training steps often help but sometimes have a negative effect. Our code is publicly available.

## introduction morphological inflection is the task of generating inflected word forms given a lemma and a set of morphosyntactic tags. inflection plays a key role in natural language generation, particularly in languages with rich morphology. the sigmor-phon shared tasks are yearly competitions for inflection tasks @xcite @xcite @xcite . this paper describes the bme team's submission for part 1 of the 2021 sigmorphon-unimorph shared task on generalization in morphological inflection generation. there were only two submissions to this subtask and our team outperformed the other submission by a large margin. the task was about type-level morphological inflection in 38 typologically diverse languages from 12 language families. our model builds on the work of @xcite . we use a sequence-to-sequence (seq2seq) model with a bidirectional lstm @xcite encoder and a unidirectional lstm decoder with attention. we perform a small hyperparameter search and we find that the most important parameters are the choice of the embedding size and the hidden size. we use two data augmentation techniques; a simple copy mechanism, a stem modification method. we add these methods at the first two steps of our three-step training regime. we describe a third data augmentation technique, a template induction method, that did not improve the overall results in the end so we did not use it in our submission. we first train a single model on all languages and then fine-tune the model on each language family and then on each language. our main contributions are: ## related work seq2seq neural networks were first popularized in machine translation @xcite and since the addition of the attention mechanism @xcite and the transformer @xcite , it has been the dominant approach in the field. the first major success of seq2seq models in morphological inflection was the submission by @xcite to the 2016 edition of the sigmorphon shared task. this was followed by an extensive study by @xcite on lstm-based encoder-decoder models for morphological inflection. we used the augmentation techniques introduced by @xcite . inspired by @xcite we attempted to extract different morphological properties of the languages and used them to generate data. @xcite used a two step training method that first trains on the language family and then on the individual languages. we use a similar procedure but we augment the data with a different technique in each step. ## data the shared task covered 38 languages from 12 language families. 35 of these languages were available from the beginning while 3 surprise languages, turkish, vibe and võro, were released one week before the submission deadline. each language had a train and a development split of varying size. each sample consists of a lemma, an inflected form and a list of morphosyntactic tags in the following format: the amount of data varies widely among the languages: while the language veps has more than 100000 examples, ludic, the most underresourced language, has only 128 train samples. ## augmentation techniques in this section we describe the data augmentation techniques we used. we applied the same steps for each language with varying effect. we performed ablation studies (section 7.1) on some languages to investigate the individual effect of these techniques. ## experiments we first experimented with a single large model for all languages. this model worked reasonably well for large languages and some of the medium ones as well, however, its performance was quite low on low resourced languages. to offset this, we introduce a three-step training method. step 1: general training. in the first phase, we train a model on all languages with copy augmentation. this step creates a general language model. using copy alone considerably improves the results on most languages. we further explore the effect of copy in section 7.1. we trained a small and a large model as described in section 4. step 2: language family training. we finetuned both model for each language family but this time we also added 10000 example to each language with using the stem modification augmentation. this resulted in 24 models, one small and one large for each of the 12 language families. step 3: language specific training. we then further fine-tune the family models on each language of the particular family. we use template based augmentation in this step to further increase the number of training samples. ## results our three step model achieves 81.51% accuracy on average with 0.627 average levenshtein distance between the output and the ground truth. this results is lower than the baseline (84.57% accuracy and 0.448 edit distance) but it is important to note that our lstm seq2seq model is simpler than the transformer baseline, while it remains competitive. we chose our model based on the accuracy of the development dataset. this way we submitted the results of 7 smaller and 31 larger models. ## conclusion we presented the bme team's submission for the sigmorphon 2021 task 0 part 1 shared task. we used a standard lstm encoder-decoder model with attention. we augmented the data with three different techniques and then used a three-step training method that first trains on all languages, then fine-tunes on the language families and finally fine-tunes on the individual languages. our submission outperformed the other team by a large margin. although our results did not outperform the baseline, we presented a simpler model with multiple data augmentation techniques that do not require any external resource or linguistic expertise.