Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation (2021)

Abstract: There has been recent success in pre-training on monolingual data and fine-tuning on Machine Translation (MT), but it remains unclear how to best leverage a pre-trained model for a given MT task. This paper investigates the benefits and drawbacks of freezing parameters, and adding new ones, when fine-tuning a pre-trained model on MT. We focus on 1) Fine-tuning a model trained only on English monolingual data, BART. 2) Fine-tuning a model trained on monolingual data from 25 languages, mBART. For BART we get the best performance by freezing most of the model parameters, and adding extra positional embeddings. For mBART we match or outperform the performance of naive fine-tuning for most language pairs with the encoder, and most of the decoder, frozen. The encoder-decoder attention parameters are most important to fine-tune. When constraining ourselves to an out-of-domain training set for Vietnamese to English we see the largest improvements over the fine-tuning baseline.

## introduction machine translation (mt) has recently seen significant advances, with improvements in modeling, especially since the advent of neural models @xcite , and the availability of large parallel corpora for training such systems @xcite @xcite . however, often standard neural systems do not perform well on low-resource language pairs @xcite , especially when the language pairs are only distantly related. since these languages are spoken by a large fraction of the world's population, reducing the gap in performance between high and low-resource mt could have a large impact. an explosion of interest in large-scale pretraining in natural language processing has led to increased performance on smaller datasets, by simple fine-tuning of large pre-trained models on downstream tasks. the typical approach is to train a large model on text from the web (for example english wikipedia), with a common objective predicting masked out tokens using the unmasked context. for natural language generation (for example summarization of text), performance can be improved by pre-training a sequence-to-sequence model @xcite . however previous work has shown that on nlp tasks such as natural language inference, the relative performance of fine-tuning vs. keeping the pretrained model frozen depends on the similarity of the pre-training and downstream tasks @xcite . we observe empirically that simple finetuning of a monolingual model for mt can result in worse performance than training from scratch (e.g. multilingual pre-training offers a solution, by modifying the pre-training objective to include many languages. using a multilingual pre-trained model for mt gives good performance, especially on lower-resource language directions @xcite . however it is challenging to balance the training data so that higher-resource languages do not overwhelm lower-resource ones @xcite . for a particular language it may be hard to source monolingual data, or it may be simply not included in training. we also consider multilingual mt (training on many language pairs and sharing all or most model parameters) as a downstream task. sharing 'knowledge' across language directions can improve performance on low-resource language pairs by transfer from other pairs included in training. previous work observed problems of performance degradation, often on high-resource languages, due to interference and constrained capacity @xcite . and when initialising from a pre-trained model, we want to avoid 'catastrophic forgetting', where by fine-tuning on a particular language pair we lose the knowledge about another language pair that is stored in the model weights. previous work has explored how to improve on simple fine-tuning, by freezing pre-trained model parameters @xcite and using lightweight 'adapter modules' @xcite which are inserted between the layers of the pretrained network. we aim to explore and improve on these approaches for both bilingual and multilingual mt (in contrast to previous work largely focusing on text classification). we explore freezing different subsections of the pre-trained model.we expect freezing to be particularly useful when the parallel data is of low quality, in which case naive fine-tuning may, for example, over-specify the pretrained model to a particular domain. our main contributions are: ## methods because bart has been trained on only english input, we need to use different techniques when fine-tuning bart and mbart for mt, with a schematic overview shown in figure 1 and figure 2. bart and mbart are standard sequenceto-sequence models, where an encoder consumes a sequence of source-side tokens, and a decoder acts as a conditional language model, generating target tokens given a source sequence. intuitively, we want the encoder and decoder to be performing roughly the same tasks during fine-tuning as they were during pre-training. for bart this means the input to the encoder should be similar to (embedding vectors of) noisy english text. therefore when training on say, vietnamese to english, we first transform the vietnamese source sentence into a representation useful for bart. we introduce new parameters (the 'input module') that consume the source sentence and produce hidden vectors we can feed into the bart encoder. we describe the input module architecture in section 3.1. ## experimental settings we use the fairseq @xcite library for all experiments. the final models are selected based on validation likelihood, except for multilingual fine-tuning where we evaluate the models after 10000 training steps. we use beam-search with beam size 5 for decoding, and evaluate all bleu scores using sacrebleu @xcite 1 . we use iso 693-2 language codes in this work for convenience, and use the same parallel data as @xcite , both listed in listed in we fine-tune frozen bart and an input module on bilingual parallel text, feeding the source language into the input module. for mbart we feed the source language into the encoder, and use the same hyper-parameters as @xcite . when using adapters we use 0.1 dropout in the adapter bottleneck layer (z in section 3.3), and a hidden dimension of either 128, or 2/3 • 128 when using a gated linear unit adapter. we use the adam @xcite optimizer. hyperparameters are listed in appendix b, and we use the same hyper-parameter search space for frozen and non-frozen models. ## conclusion we recommend: for a language with high quality parallel data but without a pre-trained model trained on monolingual data from that language, using a frozen (english-only) bart model with additional parameters at the source side (the 'input module') improves performance over a randomly initialised baseline. for this approach it is important to freeze the pre-trained model. we also give the model both learned positional embeddings at the embedding layer, and fixed sinusoidal positional embeddings at each layer of the input module. for a multilingual pre-trained model, we found performance improvements on some (mostly distantly related) languages for multilingual manyto-one fine-tuning. for bilingual @xmath0, although the performance drops are small, and by freezing parameters we need less memory at training time compared to fine-tuning. for @xmath1, and keep the rest of the decoder frozen. this can improve on simple fine-tuning, especially for distantly-related language pairs or those with out-of-domain training data. we recommend fine-tuning layer-norm parameters as a parameter-efficient complement to adapter layers. for our mbart experiments we found it was necessary to fine-tune the token embeddings, which correspond to a large number of parameters, and future work could remove this cost by working out a subset of the vocabulary to fine-tune, or another method.