Leveraging Principal Parts for Morphological Inflection (2020)

Abstract: This paper presents the submission by the CU Ling team from the University of Colorado to SIGMORPHON 2020 shared task 0 on morphological inflection. The task is to generate the target inflected word form given a lemma form and a target morphosyntactic description. Our system uses the Transformer architecture. Our overall approach is to treat the morphological inflection task as a paradigm cell filling problem and to design the system to leverage principal parts information for better morphological inflection when the training data is limited. We train one model for each language separately without external data. The overall average performance of our submission ranks the first in both average accuracy and Levenshtein distance from the gold inflection among all submissions including those using external resources.

## introduction the task of morphological inflection is to generate a target inflected word form (henceforth tgtform) given a lemma form (henceforth lemma) and a target morphosyntactic description (henceforth tgtmsd). in the sigmorphon 2020 shared task 0 on morphological inflection @xcite and previous years' sigmorphon shared tasks on morphological inflection @xcite @xcite , the training data is provided in the format of tabseparated lemma-tgtmsd-tgtform triples, and participating systems are expected to predict the missing target forms in the test data released shortly before prediction submission. the sequence-to-sequence (henceforth seq2seq) architecture has been very successful in dealing with morphological inflection, especially when there are abundant labeled data for training. the accuracies and levenshtein distances on the development data inflected by 9 baseline models are provided for the 45 typologically and genealogically diversified development languages: a non-neural model based on lemma-tgtform alignment and transformation, a per-language transformer model, a per-language-family transformer model, a perlanguage transformer model with data augmentation, a per-language-family transformer model with data augmentation, lstm seq2seq models with exact hard monotonic attention @xcite trained per language, per language family, per language with data augmentation, and per language family with data augmentation respectively. the data augmentation method used by the baseline models is from @xcite . the baseline numbers indicate that the transformer model for character-level transduction @xcite ) is very competitive, achieving the highest average accuracy and lowest average edit distance and best performance on most lan-guages (33 out of 45) when the model is trained per language. therefore, we adopt the transformer architecture @xcite for all three of our models (see figure 1 ) which are different from each other by the input and output to the transformer model, as will be presented in section 3. though not explicitly organized as a paradigm cell filling problem (pcfp) @xcite task, the shared task is closely related to and can largely be seen as a computational instance of it @xcite @xcite @xcite , where some slots are given in the paradigms as training data and others are to be inflected as development data or test data. 1 the data format of the shared task privileges the lemma as the source form (henceforth srcform) which all tgtforms are inflected from. however, the lemma form may not be the only and the most informative srcform to inflect all other slots from in the same paradigm. morphologists refer to a lexeme's principal parts @xcite as the minimum subset of paradigm slots which, if known, provide all the information needed to generate the other slots in its paradigm. the principal parts which best predict an inflected form in a lexeme's paradigm do not necessarily include the lemma, and more than one of the principal parts may be needed to generate an inflected form reliably (see examples in to generate inflected forms for the test data for submission, our system uses the same inputoutput format as the baselines for high-resource languages, and includes two slightly different approaches of leveraging principal parts information for low-resource languages. the evaluation results indicate that the transformer model augmented with principal parts information can handle morphological inflection very well for typologically and genealogically diverse languages, whether it has been tuned on the language or not, even when the training data is limited. the sigmorphon 2020 shared task 0 (vylomova et al., 2020) is a typical morphological inflection task. compared to previous years' sigmor-phon shared tasks on morphological inflection, this year's task highlights the distinction between development languages and surprise languages and the inflection model's ability to generalize to new languages that may be genetically related or unrelated to the languages according to which it is developed. in the development phase, 45 languages from 5 language families were provided, and these languages are development languages. in the generalization phase, 45 surprise languages from 16 language families were released. in the evaluation phase, test data include both development languages and surprise languages. deviating from previous years' tasks, this year's task did not feature different (low/medium/high) data settings for the languages @xcite or manipulate the data size of genetically related language pairs @xcite . instead, each language comes with different amount of training, development and test data, corresponding to the reality of data availability for the language. of the total 90 languages from 18 language families, 44 have 5,000 or more lemmatgtmsd-tgtform training triples and 46 have fewer than 5,000. of the 45 development languages, 24 have fewer than 5,000 training examples. in this paper, we refer to languages with 5,000 or more training triples as high-resource and those with fewer than 5,000 training triples as low-resource. ## system description all our models use the self-attention transformer architecture @xcite as implemented in the fairseq @xcite tool, a pytorchbased sequence modeling toolkit. both the encoder and decoder have 4 layers with 4 attention heads, an embedding size of 256 and hidden layer size of 1024. models are trained with the adam algorithm (kingma and ba, 2014) for optimization with an initial learning rate of 0.001, a batch size of 400, 0.1 label smoothing, the gradient clip threshold as 1.0, and 4,000 warmup updates. the models are trained for a maximum of 20,000 or 30,000 optimizer updates depending on the amount of inputoutput tuples for training, with checkpoints saved every 10 epochs. the checkpoint with the smallest loss and the last checkpoint are also saved. the model with the best parameters was selected from all the saved checkpoints based on the accuracy on the development data. beam search is used at decoding time with a beam width of 5. our submission is an ensemble of predictions from three types of models: baseline (fairseq), 1src, and 2-src. these three types of models have identical model architecture for inflection and are different from each other in the input and output. as the varied baseline results trained per language family provided by the organizers did not show consistent improvements compared to training languages separately, we train all the models per language without using external resources. we made our code publicly available. foot_1 ## experiments considering the time constraints and the already strong performance of the baseline modelsespecially when training data is abundant-we focused our experiments on the 24 low-resource development languages in the development phase, for which we attempted to augment the transformer model for inflection by reorganizing the data into paradigms and making use of the principal parts morphology idea in different ways. in addition to the 1-src and 2-src models described in section 3.2, other approaches we experimented with included 2-random-src, 3-random-src and 4-random-src models where we randomly pick two, three or four given slots as input which will be translated to the tgtform corresponding to the tgtmsd, as well as all-src-tgtform and all-src-allform models, where the concatenation of all given slots followed by the tgtmsd are input to the inflection model which predicts the corresponding tgtform or all srcforms and the tgtform. though these models produced better performance for one or two languages that we experimented with initially, we did not see consistent performance improvement proportional to the increasing model complexity over the 1-src and 2-src models. we also experimented with warming up the 1-src model with an additional copying task following the practice suggested by @xcite , but did not see improvements. therefore, we focused exclusively on the 1-src and 2-src models after initial experiments. further experiments with the 1-src model were conducted on the 24 development languages with fewer than 5,000 training triples, and further experiments with the 2-src model were conducted on the 17 development languages, each of which has fewer than 2,000 training triples. the performance of the two selected models will be presented and discussed in the next section. ## results and discussion the average inflection accuracy of development data for the 24 languages by the 1-src model is 91.72%, which is 1.3% higher than the unaugmented per-language transformer baseline and 0.55% higher than the best performance of all baseline models. the 1-src model achieved higher or equal accuracy on 18 languages compared to the unaugmented per-language transformer baseline and 17 languages compared to the best performance of all baselines. the 2-src models for the 17 languages we experimented with achieve an average accuracy of 91.63% and their performance on 7 languages (czn, frr, gsw, izh, mlt, mwf, zpv) is better than the 1-src model. figure 4 plots the difference in the accuracy on the development set for each language by the 1-src for 2-src model from that by the unaugmented perlanguage transformer baseline. figures 4(a ) and 4(c) depict the relationship between this difference and the number of training triples. figures 4(b ) and 4(d) show the relationship between this difference and the completeness of the paradigms seen in training. the filled percentage of each paradigm is calculated by dividing the number of given slots by the number of all slots in the paradigm, and the 0 500 1000 1500 2000 2500 3000 3500 number of training examples 6 4 2 0 2 4 6 8 10 acc difference from best baseline for each language (%) aka ceb cly ctp czn frr gmh hil izh liv lug mao tgl vot xty zpv zul ceb czn frr gmh hil izh mao tgl vot zpv zul (a) 1-src 2-src 1-src 2-src 50 55 60 65 70 75 80 average filled percentage of paradigms (%) 6 4 2 0 2 4 6 8 10 acc difference from best baseline for each language (%) aka ceb cly ctp czn frr gmh hil izh liv lug mao tgl vot xty zpv zul ceb czn frr gmh hil izh mao tgl vot zpv zul (b) 1-src 2-src 1-src 2-src 0 500 1000 1500 2000 2500 3000 3500 number of training examples 4 2 0 2 4 6 8 10 acc difference from best baseline for each language (%) aka ceb cly ctp czn frr gmh hil izh liv lug mao tgl vot xty zpv zul (c) 1-src austronesian germanic niger-congo oto-manguean uralic 50 55 60 65 70 75 80 average filled percentage of paradigms (%) 4 2 0 2 4 6 8 10 acc difference from best baseline for each language (%) aka ceb cly ctp czn frr gmh hil izh liv lug mao tgl vot xty zpv zul (d) 1-src austronesian germanic niger-congo oto-manguean uralic figure 5: scatter plots with trend lines for the difference in accuracy between the 1-src or 2-src model and the best performance of all baselines on low-resource dev languages: (a) 1-src vs 2-src: size vs ∆acc 3 or ∆acc 4 , (b) 1-src vs 2-src: percentage vs ∆acc 3 or ∆acc 4 , (c) 1-src and genealogy: size vs ∆acc 3 , (d) 1-src and genealogy: percentage vs ∆acc 3 . (size: training data size, percentage: average percentage of slots per paradigm in training data, ∆acc i 3 = acc i 1-src -acc i best-baseline , ∆acc j 4 = acc j 2-src -acc j best-baseline ) ## related work morphological inflection is one of the natural language processing tasks which achieve great improvement by applying neural network models, especially sequence to sequence models, which initially outperformed other approaches by a large margin on high-resource languages @xcite @xcite and have been improved and augmented later to achieve state-of-the-art performance on low-resource languages as well @xcite @xcite @xcite @xcite . subtask 2 of the conll-sigmorphon 2017 shared task @xcite was about paradigm cell filling, and received submissions of neural network systems @xcite . there is also other work which targets the paradigm cell filling problem @xcite @xcite . @xcite models the principal parts idea with graphical models to generate all the missing slots in paradigms. our 1-src model has an input-output format similar to @xcite . our work is also closely related to @xcite on multisource inflection which is also motivated by a principal parts analysis. @xcite use an explicit neural model that organizes paradigm slots in their most predictable order to investigate measures of morphological complexity, an instantiation of the principal parts idea in another context. ## conclusion we have presented the system for our submission to the sigmorphon 2020 shared task 0 on mor-phological inflection. it achieved the highest average accuracy and smallest average levenshtein distance across all the 90 languages from 18 language families. the standard deviation of our submission is the lowest for accuracy and the second lowest (0.004 higher than the lowest) for edit distance. our work indicates that the self-attention transformer architecture can perform well for the morphological inflection task for a genealogically and typologically diverse group of languages. the architecture has a strong generalization ability and can inflect new languages as effectively as the languages it is tuned on. we augment the transformer model by converting the morphological inflection task to the paradigm cell filling problem and leveraging the principal parts of paradigms in indirect ways, which turns out to be helpful, especially when the training data is limited and the reconstructed paradigms have a high completion rate. our primary strategy to incorporate principal parts information in this work is to use each given slot in the reconstructed paradigm to predict the target form and select the final prediction from predictions generated by different slots by highest average score or majority vote. another strategy is to use all possible two-slot combinations to predict the target form. according to principal parts morphology, the number of principal parts may vary between paradigms and languages, and different slots may require different numbers of principal parts to inflect correctly, indicating that uniformly using every slot individually or every two-slot combination may not always be the best choice. future work is needed to explore how to use principal parts information more effectively, perhaps tuning the number and choice of forms on a per-language basis or developing strategies to explicitly determine principal parts for the paradigms.