Adobe AMPS ’s Submission for Very Low Resource Supervised Translation Task at WMT 20 (2020)

Abstract: In this paper, we describe our systems submitted to the very low resource supervised translation task at WMT20. We participate in both translation directions for Upper Sorbian-German language pair. Our primary submission is a subword-level Transformer-based neural machine translation model trained on original training bitext. We also conduct several experiments with backtranslation using limited monolingual data in our post-submission work and include our results for the same. In one such experiment, we observe jumps of up to 2.6 BLEU points over the primary system by pretraining on a synthetic, backtranslated corpus followed by fine-tuning on the original parallel training data.

## introduction this paper describes our submissions to the shared task on very low resource supervised machine translation at wmt 2020. the task involved a single language pair: upper sorbian-german. we submit supervised neural machine translation (nmt) systems for both translation directions, upper @xmath0. nmt models @xcite @xcite have achieved stateof-the-art performance on benchmark datasets for multiple language pairs. a big advantage of such systems over phrase-based statistical machine translation (pbsmt) @xcite models is that they can be trained end-to-end. the bulk of the development, however, has been limited to a handful of high-resource language pairs. the primary reason is that training a well-performing nmt system requires a large amount of parallel training data, which means a lot of equivalent investment in terms of resources. @xcite show that when compared to pbsmt approaches, nmt models need more training data to achieve the same level of performance. 1 one of the most popular ways to increase the amount of parallel training data for supervised training is backtranslation @xcite . we utilize this approach to improve upon the performance of our baseline models. all of our systems follow the transformer architecture @xcite . our primary system is a supervised nmt model trained on the original training bitext. we also report our results on experiments with backtranslation, which were completed post the shared task and hence not a part of our primary submissions. we use the backtranslated data in two distinct ways -as a standalone parallel corpus, and to create a combined parallel corpus by mixing in a 1:1 ratio with the provided training data. we also report the performance of fine-tuned models originally trained only on the backtranslated data. in the following sections, we begin by briefly describing the transformer architecture and backtranslation. we then discuss our experimental setup as well as our experiments with backtranslation. we conclude with a discussion of our results and possible future work. ## related work the transformer model is the dominant architecture within current nmt models due to its superior performance on several language pairs. while still a sequence-to-sequence @xcite model composed of an encoder and a decoder, transformer models are highly parallelizable thanks to being composed purely of feedforward and self-attention layers rather than recurrent layers @xcite . the reader is encouraged to read the original paper @xcite to gain a deeper understanding of the model. we adopt the transformer base architecture available under the fairseq foot_2 @xcite library for all our models. however, nmt models are known to be datahungry @xcite ; their performance improves sharply with the availability of more parallel training data. except for a few language pairs (e.g. english-german), most have little to no such data available. on the other hand, a far greater number of languages have a decent amount of monolingual data available online (e.g. wikipedia). to address this issue of lack of parallel data, sennrich et al. (2016a) introduced the concept of backtranslation. it involves creating a synthetic parallel corpus by translating sentences from the target-side monolingual data to the source language and making corresponding pairs. a baseline @xmath1, trained with limited data, is generally used for this purpose. it enables the use of large corpora of monolingual data for several languages, the size of which is typically orders of magnitude larger than any corresponding bitext available. what is notable is that only the sourceside data is synthetic in such a scenario and the target-side still corresponds to original monolingual data. some studies @xcite ) have investigated the effects of varying the amount of backtranslated data as a proportion of the total training corpus, including training only on the synthetic dataset as a standalone corpus. we follow some of the related experiments conducted by @xcite on gujarati-english (another low-resource pair) with a few exceptions. besides, we also report performance when pretraining solely on the synthetic corpus following by finetuning on either original or mixed data. while not quite the same, one could think of this approach as having some similarities with transfer learning @xcite as well as domain adaptation @xcite ) for machine translation. there has also been work on using sampling @xcite for generating backtranslations, but we stick to using beam search in this work. ## additional backtranslation experiments in this section, we report our post-submission work on using monolingual data for backtranslation. we took the raw monolingual data that we describe in section 3.1 and backtranslated with our primary submission models for the respective translation directions, i.e., @xmath2. we used fairseq-generate function with a beam size of 5 for this purpose. once again, we limited the number of subwords in each sentence to 250. finally, we took all sentence pairs for backtranslated upper sorbian corpus and the first two million sentence pairs for the german corpus. we define the following systems for making use of the backtranslated data. note that the first system only differs from the primary system in the number of training epochs completed. fine-tuning was carried out by loading pretrained checkpoints and adding extra training flags in reset-optimizer and reset-lr-scheduler. ## results the systems were evaluated on the blind test set (newstest2020) using automated metrics; no human evaluation was done. in addition to evaluating on blind test sets, we also report bleu scores on the development test set in the same table. two outcomes are worth highlighting: ## conclusion in this paper, we described our transformer model for supervised machine translation for upper sorbian-german language pair. we take note of relatively high bleu scores achieved by our primary systems (and those of other participants) on this low-resource language pair, which could relate to the high quality of the training corpus. we also report results and takeaways from several experiments with backtranslated data completed post the shared task. a key result is matching the performance of a system trained on the original bitext with one trained on a limited amount of synthetic, backtranslated data. domain mismatch and a difference in the quality of monolingual corpus might have prevented the system from achieving a similar result in the other direction. we notice big improvements in performance over the primary systems by following a "pretraining then fine-tuning" regime. an interesting future work would be to measure the applicability of this approach to other lowresource language pairs. additional systems could be added as well. for instance, models trained on mixed data and fine-tuned on auth data might provide a meaningful comparison. prior work @xcite has shown that the number of bpe merge operations has a significant effect on the performance of nmt systems. this work was pointed out during the review process and should be an avenue for further improvement of the model performance.