Leveraging Synthetic Targets for Machine Translation (2023)

Abstract: In this work, we provide a recipe for training machine translation models in a limited resource setting by leveraging synthetic target data generated using a large pre-trained model. We show that consistently across different benchmarks in bilingual, multilingual, and speech translation setups, training models on synthetic targets outperforms training on the actual ground-truth data. This performance gap grows bigger with increasing limits on the amount of available resources in the form of the size of the dataset and the number of parameters in the model. We also provide preliminary analysis into whether this boost in performance is linked to ease of optimization or more deterministic nature of the predictions, and whether this paradigm leads to better out-of-distribution performance across different testing domains.

## introduction neural machine translation (nmt) @xcite @xcite relies on deep learning models to train end-to-end translation systems. with the advent of deep recurrent models like lstms @xcite @xcite and their attention-augmented improvements @xcite , these models outperformed traditional statistical @xcite and rule-based @xcite approaches. recently, with the introduction of fully attentionbased networks networks @xcite @xcite @xcite @xcite and increase in compute and data, large-scale transformer networks have dominated the field of natural language process- † work done during an internship at nvidia. @xcite we used the nemo codebase @xcite for all our experiments. while it has been established that scaling up data and compute boosts the performance of the largescale nmt systems @xcite @xcite , there is still a need to focus on budget models that can run on mobile and edge computing devices. in other tasks, like end-to-end speech translation, training data is scarce and expensive. inspired from these needs, we provide a recipe for training textto-text and text-to-speech translation systems in a limited resource setting at the modest overhead of running inference of pre-trained models on the source sentences. though in theory, increasing the amount of data provides a relatively simple methodology for bolstering the performance of current ai systems, it is difficult to do so when obtaining new data is costly, especially because of the labeling process in supervised learning. on the other hand, there have been a variety of approaches leveraging synthetic data to either improve the robustness of the systems or to boost their performance. this can be achieved by introducing adversarial examples in the training set @xcite , considering knowledge distillation when provided access to large models but not their pre-training data @xcite @xcite @xcite @xcite , or using forward and back translation techniques @xcite @xcite @xcite when additional monolingual data, which is easily available, is leveraged to generate synthetic targets to augment the amount of data. in this work, we use large, often pre-trained, nmt systems to provide synthetic targets which can be leveraged to train high performing lowcompute and low-data models. our findings show that using these synthetic translations to train different nmt systems leads to considerable improvements, even if we remove all ground-truth feedback from the training set. we also test models trained with synthetic targets on out-of-distribution settings through translations on different domains in a zero-shot manner as well as finetuning them on a different domain using additional synthetic data from an existing finetuned model, and further highlight the improvements obtained in both. we also find additional evidence to support that the ease of optimization resulting from training on synthetic targets does not completely explain its superior performance @xcite . instead, we showcase that models trained with synthetic targets are more deterministic than their counterparts which are trained on real targets. ## related work synthetic targets have been consistently used in the past to either augment the amount of data available or to boost performance through knowledge transfer between models. in the domain of machine translation, a popular way of augmenting data is by considering monolingual data that is available in abundance and obtaining its paired counterpart using a trained model. back translation. back translation @xcite @xcite relies on translating unseen sentences from the target side back to the source side using an existing trained model to provide new paired samples which can then be leveraged to train a model in the forward direction.. for example, if the task is to translate sentences from language s to t (@xmath0, one can obtain a corpus of monolingual data from t and translate it backwards to s using an existing trained @xmath1. this would then provide additional paired data that can be combined with the existing ground-truth data to train a @xmath2. ## method our aim is to perform machine translation from a source language s to a target language t given some training data d @xmath3, t i )} n @xmath4. further, we assume that we have access to a teacher network f @xmath5, using which we obtain synthetic targets f @xmath6 for our experiments, we consider different dataset sizes for the student models by subsampling from d @xmath7. all the models considered in this work rely on the encoder-decoder transformer architecture @xcite with the teacher network generally having 24 encoder and 6 decoder layers (24×6). we consider different model sizes for the student, ranging from small models to matching the teacher's size. typically, knowledge distillation considers the same input data for training both the teacher and the student. instead, we perform analysis where the student has access to different amounts of data. also, unlike knowledge distillation where knowledge is transferred from a bigger teacher to a smaller student network, we additionally consider setups where the student network is as big as the teacher, and showcase the benefits in this regime. ## experiments for all our experiments, we only considered transformer models for both teachers and students. unless specified, we used the pre-layer normalization variant where layernorm is applied before the respective attention and residual computations @xcite . in our experiments, we consider two text-to-text machine translation setups: bilingual and multilingual, and one speechto-text setup. for the text-to-text machine translation experiments, we considered byte-pair encoding @xcite for bilingual experiments and sentence-piece encoding @xcite for multilingual experiments. ## ablations our key ablations into understanding the phenomena presented involve analysing whether the results can be solely explained from the lens of optimization, or are there other reasons at play (eg. stochasticity of the predictive model). ## conclusion inspired by the recent advances in knowledge distillation and the need for better performing low- figure 3 : degradation of machine translation performance with increasing k in top-k sampling for the decoding procedure for both english to german and english to russian translations using student 24 × 6 models trained with real and synthetic targets respectively. this improvement in performance of small or low-resource models comes at the additional inference costs tied with the large teacher model. however, this is a one-time data curating cost based on which training of multiple different smaller models can be accomplished which will enjoy not only faster inference (owing to their smaller size) but also better performance than if they were trained on the original data, sometimes even better than larger models trained on real data. we believe this is an exciting avenue of research as low-compute high-performance models are important when deployed in constrained settings like edge computing and mobile devices.