A Simplified Training Pipeline for Low-Resource and Unsupervised Machine Translation (2023)

Abstract: Training neural MT systems for low-resource language pairs or in unsupervised settings (i.e. with no parallel data) often involves a large number of auxiliary systems. These may include parent systems trained on higher-resource pairs and used for initializing the parameters of child systems, multilingual systems for neighboring languages, and several stages of systems trained on pseudo-parallel data obtained through back-translation. We propose here a simplified pipeline, which we compare to the best submissions to the WMT 2021 Shared Task on Unsupervised MT and Very Low Resource Supervised MT. Our pipeline only needs two parents, two children, one round of back-translation for low-resource directions and two for unsupervised ones and obtains better or similar scores when compared to more complex alternatives.

## introduction several known techniques enable the design of neural mt systems with little or no parallel data for the source and target languages. among them are the initialization with a parent model trained on parallel data from related languages @xcite and repeated cycles of back-translation of monolingual data that create pseudo-parallel corpora used for training @xcite . when designing a very low-resource or unsupervised system, many practitioners rightfully consider as a guideline the best-performing systems found in several shared tasks, such as wmt shared task on unsupervised mt and very low resource supervised mt @xcite @xcite , where teams compete in order to obtain the highest scores among them. while these systems typically do obtain very high scores, in this paper we show that the pipelines of the highest-scoring systems in this task may be unnecessarily complex, and they can be substantially simplified while still achieving comparable results. to solve this shared task, high-resource parent models have been leveraged to initialize child models for low-resource languages, which in turn have been used to warm-start the training for unsupervised directions. however, the submissions to the above-mentioned shared task typically developed several dozen models, with numerous parent/child models in both directions as well as increasingly better models trained on several rounds of backtranslated data. these models were finally ensembled for best results. for the 2021 edition of the task, the unsupervised language pair was lower sorbian / german (dsb/de), with parallel data only available for testing, while the low-resource pair was upper sorbian / german (hsb/de). a large amount of german / czech (de/cs) parallel or monolingual data is available to train parent models, due to the similarity of sorbian dialects to czech. moreover, given the similarity of the two sorbian dialects, child low-resource models can become parents of "grandchild" systems for the unsupervised task. as a result, these systems are quite complex, which raises the question: up to which point can these architectures be simplified with virtually no loss of performance? our study answers this question by presenting a simpler pipeline than the ones submitted to the shared task, which reaches superior or comparable scores to the ones from the highest-scoring teams. in our pipeline, we apply the same selection and filtering of data as the best-performing team for comparability. we train high-resource parent models on authentic parallel data in two directions (cs↔de), and then use them to initialize child low-resource models (hsb↔de). we improve these systems with one round of backtranslated monolingual data, and finally use them to initialize systems and to produce back-translated data for the unsupervised pair (dsb↔de). more specifically, our simplifications are the following: we make public the configuration files that create these systems in the opennmt-py framework. 1 2 related work ## proposed pipeline we propose a simplified training pipeline represented in figure 1 , which reaches comparable or better results than the above systems. the pipeline is minimal, in the sense that only eight systems are trained for hsb↔de and dsb↔de translation, including parent systems for initialization. we show that one round of back-translation for lowresource directions and two for unsupervised ones are sufficient. in comparison with the numerous rounds and checkpoints of the nrc-cnrc and lmu systems, our pipeline is an order of magnitude smaller. we start by training from scratch parent models @xmath0. from their best-performing checkpoint, we respectively initialize @xmath1, which we train only on authentic parallel data. we then use their bestperforming checkpoints to generate synthetic parallel data (back-translations) by translating monolingual target data (resulting in synthetic datasets hsb bt -de mono and de bt -hsb mono ). we initialize from the best-performing checkpoints of the previous systems new models @xmath2 de-hsb authentic de hsb src tgt hsb bt de mono de-hsb authentic+bt src tgt hsb de de bt de-dsb unsupervised de bt1 dsb mono src tgt de dsb bt1 de mono dsb bt2 (2x) de-cs parent de cs src tgt cs-de parent cs de src tgt hsb-de authentic hsb de src tgt de bt hsb mono hsb-de authentic+bt src tgt de bt1 de de mono dsb mono hsb hsb bt dsb-de unsupervised src tgt de de mono dsb bt1 dsb bt2 (2x) de bt2 dsb mono hsb mono figure 1: pipeline of implemented systems. solid arrows represent the parent systems used, and dashed arrows represent creation of synthetic data through back-translation. the datasets in color are those presented in to evaluate our systems, we use the 'newstest2019-csde' as a test set for our cs↔de models. for our hsb↔de and dsb↔de models we use the 'devel' set from the wmt20 task during development, and 'devel_test' for final evaluations. since the official scores of the task are calculated on an undisclosed subset of the blind test set, we cannot compare our results with the final official ones. we will thus compare them with the scores on 'devel_test' reported by each team in their articles. our two evaluation metrics are the same as in the shared task. we use the sacrebleu library @xcite to compute bleu @xcite . 3 we also use bertscore foot_2 @xcite , with the xlm-roberta-large model @xcite for translations into german, as provided with the bertscore toolkit. we test the statistical significance of differences in scores at the 95% confidence level using paired bootstrap resampling from sacrebleu. ## discussion and conclusion we show in compared to the nrc-cnrc submission, our pipeline uses the same data selection and filtering, a single vocabulary for the tokenizer, trains from a single random initialization for each of the translation direction, does not train multitask or multilingual models, uses a much simpler filtering for back-translated sentence pairs, and sets a single set of values for hyper-parameters such as learning rate and label smoothing. compared to lmu, our pipeline uses a smaller amount of authentic parallel data for the parent cs↔de models, does not use monolingual data back-translated for these parent models, and uses an architecture with fewer parameters (transformer-base instead of big). moreover, we use only one round of back-translation instead of four for the child hsb↔de systems and two instead of eight for the grandchild dsb↔de systems submitted by lmu. noahnmt also produced high scores on the supervised tasks, although with the use of a pretrained bert model @xcite , vast amounts of monolingual data (100m lines), and dual parent transfer. cl_rug scored well in the unsupervised tasks, but made use of sequence masking, denoising auto-encoding, cross-lingual back-translation, and vocabulary alignment between hsb and dsb with vecmap @xcite . iict-yverdon applied a scheduled multitask training to both the supervised and unsupervised directions, which appeared to be particularly ineffective for the unsupervised task. we now provide some hypotheses on why our simplified pipeline produces scores that are comparable with those from more complex ones. firstly, a much better trained parent model does not necessarily result in noticeable better child models. whatever the cause of the improvement of the parent models (additional parent training data, parent back-translation, or additional parent pairs), when several stages in the training pipeline can be found afterwards (such as training on authentic data, then children back-translation, then grandchildren backtranslation, etc.), the initial benefit may be lost later in the pipeline. this is particularly exacerbated when child systems are later trained with data of dubious quality, such as back-translations. @xcite , for instance, showed that when performing iterative back-translation, the quality of the initial system has minimal effect on the final performance, as systems tend to converge to scores dictated by the monolingual data. this first hypothesis feeds into a second hypothesis: large amounts of parent parallel or monolingual data make it reasonable for practitioners to choose larger architectures, which must then be carried over to the lower-resource children, since pruning rarely happens mid-pipeline. although there is evidence that fitting large models to very small amounts of data is not necessarily detrimental @xcite and can even be beneficial @xcite , it is unclear if this still holds with a more complex training pipeline. in any case, a smaller architecture in a low-resource setting, while still over-parameterized, can perform as well as a larger one. as a third hypothesis, and on a more practical note, since it is necessary to carry out the full pipeline to obtain the final results, some practitioners may choose to introduce elements into the pipeline without empirically measuring the extent to which they improve the scores, since that sometimes may require re-training the entire pipeline. finally, modern transformer-based systems are robust, and there seems to be a large area of "acceptable results" which is relatively easy to access, as we have empirically shown with our comparison to five different submissions to the wmt shared task. however, our pipeline is only trained on a group of similar languages (czech, upper sorbian, and lower sorbian) to and from german, which may not generalize in the same manner to other languages or domains. to sum up, although the competition to achieve first place in shared tasks such as the one discussed here leads participants towards increasingly com-plex pipelines, we have shown that competitive or even better results can be achieved with a much simpler training pipeline. we hope this will encourage practitioners to further participate in shared tasks such as these, while minimizing entry constraints regarding time, training strategy, or computing resources.