The IICT -Yverdon System for the WMT 2021 Unsupervised MT and Very Low Resource Supervised MT Task (2021)

Abstract: In this paper, we present the systems submitted by our team from the Institute of ICT (HEIG-VD / HES-SO) to the Unsupervised MT and Very Low Resource Supervised MT task. We first study the improvements brought to a baseline system by techniques such as back-translation and initialization from a parent model. We find that both techniques are beneficial and suffice to reach performance that compares with more sophisticated systems from the 2020 task. We then present the application of this system to the 2021 task for low-resource supervised Upper Sorbian (HSB) to German translation, in both directions. Finally, we present a contrastive system for HSB-DE in both directions, and for unsupervised German to Lower Sorbian (DSB) translation, which uses multi-task training with various training schedules to improve over the baseline.

## introduction in this paper, we present the systems submitted to the wmt 2021 task on unsupervised mt and very low resource supervised mt. we first build a series of baseline systems, driven mostly by considerations of simplicity, trained on data from the 2020 edition of the task, for translation between upper sorbian (hsb) and german (de). these systems, described in section 3, enable us to quantify the merits of using additional back-translated data @xcite and of initializing the system for a low-resource pair with parameters learned on a high-resource pair (same target language and related source language). the systems described above serve as the basis for our 2021 baseline submitted to the shared task, for @xmath0, presented in section 4, which improves upon our 2020 baseline with the addition of more parallel data, and achieves competitive performance with the use of back-translation and parent-initialization only. however, this approach does not lead to an effective baseline for unsupervised german to lower sorbian (dsb) translation (section 5). in section 6, we present experiments with a contrastive system that implements multi-task learning, with several schedules, in which denoising tasks together with translation are presented to the systems in increasing order of complexity, leading to more robust hsb↔de systems, together with a strategy of diverse ensembling. we also use our @xmath1, although in this case the performance is not competitive. ## datasets we use various upper sorbian datasets from the 2020 edition of the task, and additional wmt data, as presented in ## baseline hsb↔de low resource systems for 2021 given the results of the previous section, we choose the transformer-big for our 2021 baseline. we change the dropout level from 0.3 to 0.1 since our experiments revealed an increase in performance with the latter value. furthermore, we add the 87,502 sentences of additional parallel hsb-de training data provided in 2021 to the datasets used in our 2020 baseline. we use the same sentence-piece model with de, hsb, and cs data that we used for our 2020 baseline system, with approximately 700k lines for each language. at translation time, after observing a number of out-of-vocabulary tokens, we replace the unknown tokens with the source token that has the highest attention weight. we do not make any further changes regarding our 2020 transformer-big model. the scores of our baseline systems on 2020 and 2021 data are synthesized in after the submission to the 2021 shared task, we continued training the above @xmath4,000 steps-ensembling the last 4 saved checkpoints, bleu scores were close to the ones shown in the last line of ## baseline for unsupervised @xmath5 moreover, we studied the same techniques for translating lower sorbian (dsb), for which no parallel resources are provided. we translated the monolingual dsb data provided by the organizers with our @xmath6, hypothesizing that the differences between dsb and hsb are small enough to obtain an acceptable dsb-de pseudo-parallel corpus, with high-quality text on the dsb side, following insights from our experience with swiss-german dialects @xcite . we use the parameters from our best @xmath7. when ensembling the best 4 checkpoints, we reach bleu scores of 8.25 / 8.22 without observing any significant increase of the scores during training. in fact, the initial score, which is the performance of a @xmath8, is even slightly higher. an even lower bleu score was reached when using our @xmath9, thus confirming the finding that this approach does not lead to pseudo-parallel corpora of sufficient quality. therefore, we did not submit these translations to the 2021 shared task. ## contrastive hsb↔de and @xmath10 in contrast to the baseline systems presented above, we study an innovative approach, in which we train multitask systems with denoising auxiliary tasks that are presented in order of increasing complexity. this insight is drawn from curriculum learning @xcite . we thus test whether increasing the complexity of the tasks makes it easier for an nmt model to learn the simple tasks first, and the harder ones later in training. as @xcite showed, source-tosource pre-training and multitasking improves translation, but not enough to compete with stateof-the-art setups. therefore, instead, we perform target-to-target and source+target-to-target denoising. considering their findings, we decide not to introduce special tokens into our vocabulary, such as mask tokens (instead just deleting the tokens with wish to mask), or sentence and language separators. finally, due to computational constraints, we use the transformer-base as our architecture. ## conclusion in this work, we showed that non-iterative backtranslation and parent-model transfer learning provide improvements for translation in a low-resource setting. furthermore, multi-task scheduled training with monolingual or cross-lingual tasks also resulted in better models. in particular, child models starting with translation language modeling tasks and masking tasks improved over the baseline in all translation directions. finally, our strategy of ensembling diverse models also produced higher scores than a mere checkpoint ensemble strategy.