Backtranslation in Neural Morphological Inflection (2021)

Abstract: Backtranslation is a common technique for leveraging unlabeled data in low-resource scenarios in machine translation. The method is directly applicable to morphological inflection generation if unlabeled word forms are available. This paper evaluates the potential of backtranslation for morphological inflection using data from six languages with labeled data drawn from the SIGMORPHON shared task resource and unlabeled data from different sources. Our core finding is that backtranslation can offer modest improvements in low-resource scenarios, but only if the unlabeled data is very clean and has been filtered by the same annotation standards as the labeled data.

## introduction both machine translation (mt) and morphological inflection generation are string transduction tasks: mt is typically treated as word-level (or subwordlevel) string transduction while morphological inflection generation can be treated as character-level string transduction. mt models and techniques can usually be naturally applied to morphological inflection, as is shown in recent work on morphological inflection @xcite @xcite @xcite @xcite @xcite @xcite . backtranslation @xcite has become a common practice in machine translation in low-resource scenarios @xcite @xcite @xcite @xcite . there has been work on data augmentation for morphological generation in low-resource scenarios @xcite @xcite , but no previous work has applied the backtranslation technique. in this paper, we propose to apply backtranslation as a inpu ( input inflected enchained bristle enforcing meagred remarried … (b) morphological inflection lemma enchain bristle enforce meagre remarry … inflected form enchained bristle enforcing meagred remarried … msd tag v;v.ptcp;pst v;nfin v;v.ptcp;prs v;v.ptcp;pst v;pst … (a) labeled data lemma enchain bristle enforce meagre remarry … msd tag v;v.ptcp;pst v;nfin v;v.ptcp;prs v;v.ptcp;pst v;pst … input output inflected form enchained bristle enforcing meagred remarried … (c) morphological analysis lemma enchain bristle enforce meagre remarry … msd tag v;v.ptcp;pst v;nfin v;v.ptcp;prs v;v.ptcp;pst v;pst … input output inflected form enchained bristle enforcing meagred remarried … (b) morphological inflection lemma enchain bristle enforce meagre remarry … inflected form enchained bristle enforcing meagred remarried … msd tag v;v.ptcp;pst v;nfin v;v.ptcp;prs v;v.ptcp;pst v;pst … (a) labeled data lemma enchain bristle enforce meagre remarry … msd tag v;v.ptcp;pst v;nfin v;v.ptcp;prs v;v.ptcp;pst v;pst … input output inflected form enchained bristle enforcing meagred remarried … (c) morphological analysis lemma enchain bristle enforce meagre remarry … msd tag v;v.ptcp;pst v;nfin v;v.ptcp;prs v;v.ptcp;pst v;pst … input output inflected form enchained bristle enforcing meagred remarried … figure 1: data example for morphological inflection and morphological analysis. ## method the backtranslation method comes from machine translation. suppose we need to translate from language a to language b, and we have a parallel text corpus. suppose further that we have additional monolingual data for b. the idea in backtransla- train a morphological analyzer label more words add the newly labeled data to the original data train a morphological generator tion is to train an mt model for b-to-a translation using parallel text, use this mt model to translate our monolingual b data into a, and then add that translated data to the a-to-b parallel text corpus to (re)train an a-to-b translation model. for morphological inflection, the labeled data is usually of the type shown in figure 1 (a), where we are provided with triplets consisting of lemma, inflected form, and a morphosyntactic description (msd) tag corresponding to the inflected form. in a morphological inflection task, the input is the lemma and the msd, while the expected output is the inflected form, as shown in figure 1(b ). to apply the backtranslation technique to the morphological inflection task, we can follow a pipeline like the one illustrated in figure 2 : leverage the labeled data to train a morphological analyzer instead of a generator, apply the morphological analyzer to tag more unlabeled words with msds, and then add the newly labeled data to the original data to train models for morphological inflection. when training the morphological analyzer, the input is the inflected form and the output is the lemma and the msd, as is illustrated in figure 1(c ). ## experiments and results we conduct several experiments to evaluate the performance of morphological inflection with the backtranslation data augmentation technique. the deep learning architecture we use is the transformer model @xcite as implemented in fairseq @xcite . for our experiments, we use the same hyperparameter settings as the best-performing system @xcite in the sigmorphon 2020 shared task on inflection @xcite . all models have been trained with a single nvidia tesla p100 gpu. data our experiments cover six languages: czech, finnish, german, russian, spanish and turkish. these languages are selected to include variety in morphological inflection complexity and difficulty. finnish and turkish are agglutinative languages, both of which have vowel harmony and extensive agglutination. spanish has a rich inflec-tional system, but is quite regular. czech is a slavic language that uses a latin writing system, and is a fusional language with rich morphology. russian is also a slavic language with a rich fusional morphological inflection system, and is written in cyrillic script. german has a relatively limited inflectional system, but is challenging due to a high rate of syncretism. as it has been noted that the quality of the backtranslation model (in our case, the morphological analyzer) is positively correlated to the ability of backtranslation data augmentation to yield improvements @xcite , we present the morphological analyzer accuracy in figure 4 . the development and test data for the morphological analyzer is created by simply reversing the input and output of the development and test set data for morphological inflection. the reported accuracy for each morphological inflection model and each morphological analysis model are the average of five runs with different random initializations to ensure a good representation of the model performance. ## discussion and conclusion though backtranslation has become a common technique in machine translation for data augmentation, our experiments indicate that it is not significantly helpful-at least not by itself-for morphological inflection generation, a character-level string transduction task closely related to mt. we find small improvements when the backtranslated data is drawn from exactly the same source as the evaluation data, i.e. the sigmorphon shared task data. when other sources are used, such as ud or wikipedia text, backtranslation degrades performance across all data sizes. though we have controlled the part-of-speech of ud words to match the original training data distribution, adding backtranslated ud words is still unhelpful. considering that ud data set is labeled with different annotation standards and may also contain some noise, this indicates that unlabeled words used for backtranslation need to be noise-free and have been filtered with the same annotation standards as the labeled data in order to be helpful. such a strict requirement of data correctness probably renders it unpractical to apply backtranslation to morphological inflection generation in most scenarios. further, we did not find any significant difference between the techniques of standard backtranslation and tagged backtranslation in our experiments for morphological inflection.