Exploiting large pre-trained models for low-resource neural machine translation (2023)

Abstract: Pre-trained models have drastically changed the field of natural language processing by providing a way to leverage large-scale language representations to various tasks. Some pre-trained models offer general-purpose representations, while others are specialized in particular tasks, like neural machine translation (NMT). Multilingual NMT-targeted systems are often fine-tuned for specific language pairs, but there is a lack of evidence-based best-practice recommendations to guide this process. Moreover, the trend towards even larger pre-trained models has made it challenging to deploy them in the computationally restrictive environments typically found in developing regions where low-resource languages are usually spoken. We propose a pipeline to tune the mBART50 pre-trained model to 8 diverse low-resource language pairs, and then distil the resulting system to obtain lightweight and more sustainable models. Our pipeline conveniently exploits back-translation, synthetic corpus filtering, and knowledge distillation to deliver efficient, yet powerful bilingual translation models 13 times smaller than the original pre-trained ones, but with close performance in terms of BLEU.

## introduction in the field of natural language processing (nlp), most of the so called pre-trained or foundation models @xcite fall into one of three categories, based on whether the underlying architecture corresponds to the encoder of the transformer @xcite , the decoder or both. encoder-like models consist of a number of bidirectional self-attention layers that learn deep general-purpose representations with self-supervised denoising learning objectivessuch as predicting the original token for masked or perturbed tokens in the input-and can then be adapted to a wide range of downstream tasks. monolingual models such as bert @xcite and cross-lingual variations like mbert or xlm-r @xcite have been obtained this way. decoder-like pre-trained models -such as gpt-3 @xcite or llama @xcite -are trained to auto-regressively predict the next token in the sequence by using causal self-attention layers. pre-trained models involving the whole encoder-decoder transformer architecture -e.g. deltalm @xcite , bart @xcite and its crosslingual variation mbart @xcite -are also pre-trained to denoise perturbations in the input, and then fine-tuned for particular text-to-text downstream tasks such as neural machine translation (nmt). in addition to models pre-trained to obtain general-purpose neutral representations, there exist a number of multilingual encoder-decoder models specifically pre-trained to translate between many different language pairs. well-known systems in this group include mbart50 @xcite , or @xcite . all these pre-trained models attain high translation quality @xcite because they leverage information from multiple language pairs, thus becoming an interesting realization of the possibilities of transfer learning. in this paper, we focus on mbart50 and leave the exploration of other pretrained models to future work. mbart50 @xcite was obtained by additionally training mbart in a supervised manner to translate between english and 49 languages, and vice versa. 1 as a consequence of the relatively recent release of pre-trained models specifically aimed at nmt, there are just a few studies (see sect. 5) on how to adapt them to a certain language pair. in this paper we focus on low-resource languages in lowresource settings, since low-resource languages are usually spoken in impoverished or conflicting areas with limited computational resources. we propose a pipeline to tune the english-tomany mbart50 model for the translation between english and a specific low-resource language (or vice versa with the many-to-english pretrained model) and, afterwards, distill the knowledge in the fine-tuned mbart50 teacher model to build a lightweight student model that has a much smaller number of parameters. in this regard, our pipeline considers mbart50 as an initial resource-hungry model which is conveniently exploited to generate synthetic parallel sentences that are conveniently filtered before training a smaller student nmt system that can then be run on low-end devices. we prove that filtering is beneficial in most cases, without being detrimental in any of them. we chose mbart50 for our experiments based on its performance in the literature @xcite @xcite , as it has been shown to provide comparable or better bleu scores than alternatives like m2m-100, mt5, criss, and sixt, at least for language pairs including english. our pipeline is evaluated on eight translation tasks involving four low-resource languages and english. in order to evaluate the transferability of the pre-trained model to unseen languages, two of our languages were not considered during mbart50's pre-training. languages were chosen so that each one belongs to a different language family. the results show that when english is the source language, our student models outperform the teacher models or perform comparably. however, when english is the target language, the teachers perform better that the students. in either case, the student models are 92% faster than the teacher models when they are executed on a cpu. the rest of the paper is organized as follows. next section describes our pipeline for fine-tuning and knowledge distillation of pre-trained nmt models. sect. 3 then presents the experimental setown, as it results from adapting a pre-trained model to a particular task, or as a pre-trained model used as the seed to obtain specific bilingual machine translation (mt) models as we do in this paper. parallel corpus [en-xx] fine tune mbart50 @xmath0, whereas sect. 4 reports the main results and discusses the most relevant observed patterns. the paper ends with a review of related work, followed by some concluding remarks and future work plans. foot_1 ## approach our pipeline consists of two different stages: a first stage aimed at improving the pre-trained models by combining iterative back-translation, parallel corpus filtering and fine-tuning; and a second stage aimed at distilling the knowledge from the fine-tuned models to train a student model with far fewer parameters but comparable performance. fine-tuning of pre-trained models. this process, depicted in figure 1 , combines finetuning of the pre-trained models with backtranslation @xcite and synthetic parallel corpus filtering via a fine-tuned xlm-r model @xcite . for our englishcentric scenario and a particular low-resource language, this consists of the following steps: to filter the synthetic corpora generated in each iteration, a threshold in the interval [0,1] is used to discretize the output of bicleaner-ai. this threshold is set in the first iteration of the backtranslation process -step 3(b)-by exploring all thresholds in [0.0, 0.9] at steps of 0.1. the threshold for the remaining iterations is the one that produces the synthetic corpus that leads to the best mbart50 models on the development set. we start the iterative back-translation with 1 million monolingual sentences in each language (or the whole corpus if the amount is smaller) and we add 1 million sentences in each language (if available) after step 3(d). training of student models. knowledge distillation is usually implemented in nlp at token level, but in tasks like nmt performing it at sequence level @xcite is usually equivalent and easier to implement: the student is trained on a synthetic corpus obtained by translating with the teacher the source segments of the original training parallel corpus, if available. however, in the case of third-party-developed pretrained models, this corpus may not be available. we hypothesize that, in its absence, as well as for languages never seen by pre-trained models, we can generate synthetic training samples by translating monolingual data with the teacher model and then filtering the synthetic data generated to discard low-quality or noisy sentence pairs. once the pre-trained models have been properly fine-tuned, we train a student model by performing standard sentence-level knowledge distillation @xcite . to this end, monolingual english data is automatically translated into the low-resource language with the best fine-tuned english-to-many mbart50 system and the resulting synthetic bilingual corpus (opportunely cleaned with the same bicleaner-ai model) together with the true bilingual corpus are used to train the student model translating the low-resource language into english. conversely, monolingual data available for the low-resource language is automatically translated into english with the best fine-tuned many-to-english mbart50 model and the resulting cleaned corpus together with the bilingual corpus are used to train the system translating from english into the lowresource language. in addition to this approach based on back-translation, we will also explore two other approaches to student training: using forward-translated texts @xcite and using both, forward-and back-translated ones. ## experimental settings selection of low-resource languages. we conducted experiments for the translation from four low-resource languages into english, and vice versa. these low-resource languages are swahili (sw), kyrgyz (ky), burmese (my) and macedonian (mk). 3 they belong to different language families and use different alphabets. swahili belongs to the niger-congo language family and is written in the latin script. kyrgyz is a turkic language written in a cyrillic alphabet in kyrgyzstan, and in a perso-arabic alphabet in xinjiang. burmese is a sino-tibetan language that has its own writing system. the presence of blank spaces between words is optional in burmese, but they are commonly used in a non-standard manner to ease legibility. finally, macedonian is a slavic language using the cyrillic alphabet, but differs in some characters from other languages with the same script. model architecture. the pre-trained model exploited in this paper is mbart50 @xcite , a multilingual sequence-to-sequence encoder-decoder pre-trained on large-scale monolingual corpora using the bart denoising objective @xcite and then fine-tuned for multilingual mt. mbart50 was trained on a set of 50 languages, including english, burmese and macedonian, but neither swahili nor kyrgyz. mbart50 uses a standard transformer architecture @xcite with 12 layers for both the encoder and the decoder, embedding dimension of 1024, feed-forward inner-layer dimension of 4096, and 16 attention heads. this adds up to approximately 680m parameters. our bilingual baselines and student models consist of a transformer architecture with 6 layers for both the encoder and the decoder, embedding dimension of 512, feed-forward inner-layer dimension of 2048, and 8 attention heads. these models have near 50m parameters, approximately 13 times fewer parameters than the mbart50 models. all our models were trained or fine-tuned using the fairseq toolkit. 4 data. most of the training corpora used for each language pair comes from opus. 5 in addition, parallel corpora from gourmet foot_5 and jw300 were also used. the alt corpora foot_6 was additionally used for burmese and sawa @xcite for swahili. we used monolingual texts from newscrawl, except for burmese, for which we used oscar @xcite . we added the monolingual corpora available in gourmet to kyrgyz and macedonian. for macedonian, an in-house corpus was used, representing 48% of the macedonian monolingual sentences shown in for development and testing, we used the flores-101 @xcite contains the same set of sentences translated by professional translators across 101 languages. we use the 927 sentences in the dev directory for development and the 1,012 sentences in the devtest directory for testing. foot_8 sub-word splitting. when using mbart50, sentences in all languages are tokenized with the sentencepiece model @xcite provided with mbart50 (same model for all languages). to be consistent with mbart, whose parameters are used to initialize mbart50 before pre-training, mbart50 uses mbart's sentencepiece model, which in turn was obtained using monolingual data for the 101 languages in the xlm-r pre-trained model @xcite . consequently, this senten-cepiece model (with a vocabulary of 250k tokens) already supports languages beyond the 50 languages in mbart50 pre-training, including swahili and kyrgyz. sub-word tokens for these languages are thus present in the embedding table of mbart50, but their parameters were not updated during mbart50's pre-training 10 except for those tokens shared with some of the 50 languages. moreover, as the sentencepiece model is jointly computed for 101 languages, it may split words in swahili or kyrgyz in sub-optimal ways. to avoid these issues, we obtained two new joint sentence-piece models of 10,000 tokens each for english-swahili and english-kyrgyz. we then filtered the embedding table of mbart50 out by removing those tokens that were not included in the new sen-tencepiece vocabulary. finally, we extended the embedding table to include every new token in the sentencepiece vocabulary. 11 the already learned embeddings are thus kept for those tokens already included in the original token set. this procedure may also be applied to new languages not in the original mbart50's sentencepiece model, even if they have a new alphabet. as regards the students and the baseline bilingual models, we computed a different joint bilingual sentencepiece model for each language pair using the bilingual training corpora and a vocabulary of 10,000 tokens. training. when training and fine-tuning, we used a learning rate of 0.0007 with the adam (kingma and ba, 2015) optimizer (β 1 =0.9, β 2 =0.98), 8,000 warm-up updates and 4,000 max tokens. we trained with a dropout of 0.1 and updated the model every 5,000 steps. validation-based early stopping on the flores-101 development set was carried out as a form of regularization to prevent over-fitting. the crossentropy loss with label smoothing was computed on the development set after every epoch and the best checkpoint was selected after 6 validation steps with no improvement. ## results and discussion as can be seen, when english is the target language, the student models lag further behind the teacher models as compared to when english is the source language: the difference with the best student models ("student all" in all cases) is around 3 bleu points, being the minimum difference of 1.82 bleu points (ky-en) and the maximum difference of 3.80 bleu points (my-en). this is clearly motivated by the fact that the english-tomany mbart50 translates from one language to 50 languages, whereas the many-to-english model only generates english. the latter is therefore specialized in generating english texts. as the student models have been trained on much less english corpora than mbart50, they are not able to match the performance of mbart50 when translating into english. alternative evaluation metrics, such as chrf @xcite or spbleu (see below), show the same trend; consequently, only bleu scores are reported in the best student models consistently improve the results of the bilingual baselines by a wide margin, thus confirming the appropriateness of considering large pre-trained models as the seed for nmt models and the effectiveness of our pipeline. as regards the low bleu scores attained by the bilingual baseline models involving kyrgyz, our results match the pattern described by @xcite , who observed that 8 out of 9 low-resource nmt systems for african languages trained on jw300 generalized very poorly in human evaluations when shifting to domains such as ted talks or covid-19 surveys; they concluded that the validation score on the jw300 test set was misleading as it overestimated the model quality. ## related work multilingual nmt models. a large amount of pre-trained multilingual nmt models 15 have been 14 as good tokenizers are not always available for lowresource languages, spbleu @xcite has been proposed as an evaluation metric. spbleu applies senten-cepiece @xcite to both the output and the reference translation before computing bleu. as all our languages are part of flores-101, the pre-computed sen-tencepiece model of 256k tokens provided by its developers at https://github.com/facebookresearch/ flores\#spm-bleu has been used. 15 we omit discussion of general multilingual text-to-text models such as deltalm @xcite , mt5 @xcite or mt6 @xcite that were not specifically de-developed in the last years: nllb-200 @xcite , criss @xcite , deltalm @xcite , m2m-100 @xcite , m2m-124 16 @xcite , mbart50 @xcite , sixt @xcite , and sixt+ @xcite , to name but a few. in most cases, their encoders and decoders are initialized from cross-lingual encoderlike pre-trained models, mainly xlm-r @xcite , or full cross-lingual models such as mbart @xcite . the number of supported languages varies, ranging from a few to around 100, mainly those in the opus-100 17 or flores-101 @xcite corpora. recently, larger models supporting up to 200 @xcite or even around 1000 @xcite languages have appeared. mbart50 can be seen as a medium-size english-centric model supporting 50 languages. a number of common training techniques such as iterative back-translation are exploited by most models. additionally, every model incorporates distinctive elements: language-specific layers @xcite ; removing of residual connections in the encoder to minorate language-specific representations by reducing the influence of positional information @xcite ; adding a mixture of experts sublayer to significantly improve the representability of low-resource languages while maintaining the same inference and training efficiency @xcite ; modification of the decoder to have interleaved layers with self-attention and cross-attention so that the former are randomly initialized but the latter can be paired with the corresponding layers in an encoder-like pre-trained model @xcite ; or rescaling the gradients so that performance for low-resource languages improves @xcite . pre-training is based on monolingual masking/corruption and, optionally, translation pair masking/corruption, but for some models, such as deltalm+zcode @xcite , this kind of denoising tasks are learned at the same time they are fine-tuned for mt. deltalm+zcode @xcite is based on deltalm @xcite and can be considered as one of the best current signed for mt, although they could be fine-tuned to do so. multilingual nmt systems, 18 translating all directions across the 101 languages in the flores-101 dataset. its training process exploits multiple factors such as an incremental architecture, generation of pseudo-parallel synthetic data, curriculum learning to progressively reduce the influence of the denoising tasks, and iterative back-translation. fine-tuning of multilingual models. @xcite fine-tuned mbart50 via curriculum learning and back-translation to obtain competitive english-pashto nmt systems. @xcite evaluated mbart50 on 10 languages, all disjoint with ours. @xcite improved mbart's performance on nmt with new languages by pretraining with a denoising task on mixed-language sentences containing masked tokens, removed tokens, or words replaced by their english counterparts obtained from unsupervised bilingual dictionaries @xcite . similar mixedlanguage sentences that allow the system to align representations between english and the new languages were also used in the mrasp2 @xcite fine-tuned m2m-100 for african languages by mapping the codes of languages not included in the pre-training to the codes of already included languages. a parallel line of research @xcite adds language-specific information for unseen languages in the form of adapters which are pre-trained with monolingual data and then fine-tuned with bilingual data. the nmt-adapt method @xcite initializes the transformer with mbart and then jointly optimizes a combination of tasks including high-resource translation, low-resource back-translation, monolingual denoising of all languages, and adversarial training 18 deltalm+zcode won the task on large-scale multilingual machine translation of wmt 2021 @xcite . to obtain universal representations. finally, @xcite perform monolingual fine-tuning of pre-trained multilingual models on unseen representative african languages. ## concluding remarks in this paper, we have presented a pipeline to tune large nmt pre-trained models, and distill the knowledge in the fine-tuned teachers to build student models using far fewer parameters. in order to fine-tune the teacher model we apply an iterative back-translation procedure that integrates a bicleaner-ai classifier based on xlm-r to discard poor quality translations. we have demonstrated that filtering yields benefits in the majority of cases, without causing harm in any instance. our approach has been tested on the englishcentric mbart50 pre-trained model and on four different low-resource languages, translating to and from english. the languages belong to different language families and two of them were not part of the pre-training stage of mbart50. the results show two clear trends, depending on whether english is the source or the target language. when translating from english, our student models outperform the teacher models or perform comparably. when translating into english, the teacher models clearly outperform the student models. in any case, the student models have 13 times fewer parameters and are 92% faster when translating on a regular cpu, which makes them suitable for affordable computational devices. we leave the in-depth exploration of alternative models such as sixt+, nllb-200 or deltalm as future work. we also plan to extend our pipeline with monolingual and bilingual denoising tasks, especially for unseen languages, as well as to explore a larger number of language combinations.