When does Parameter-Efficient Transfer Learning Work for Machine Translation? (2022)

Abstract: Parameter-efficient fine-tuning methods (PEFTs) offer the promise of adapting large pre-trained models while only tuning a small number of parameters. They have been shown to be competitive with full model fine-tuning for many downstream tasks. However, prior work indicates that PEFTs may not work as well for machine translation (MT), and there is no comprehensive study showing when PEFTs work for MT. We conduct a comprehensive empirical study of PEFTs for MT, considering (1) various parameter budgets, (2) a diverse set of language-pairs, and (3) different pre-trained models. We find that ‘adapters’, in which small feed-forward networks are added after every layer, are indeed on par with full model fine-tuning when the parameter budget corresponds to 10% of total model parameters. Nevertheless, as the number of tuned parameters decreases, the performance of PEFTs decreases. The magnitude of this decrease depends on the language pair, with PEFTs particularly struggling for distantly related language-pairs. We find that using PEFTs with a larger pre-trained model outperforms full fine-tuning with a smaller model, and for smaller training data sizes, PEFTs outperform full fine-tuning for the same pre-trained model.

## introduction there has been enormous progress on scaling up neural machine translation (nmt) in the recent years, resulting in 'massively multilingual' models that are capable of translating across many languages @xcite . most successful applications rely on sequence-to-sequence pretraining that (1) leverages web-scale monolingual * both authors contributed the paper equally and the order is determined by coin flip. 1 our code and scripts for reproducing the experiments are available at https://github.com/ahmetustun/ fairseq data with a masking objective to build a multilingual backbone (parent) model @xcite , or (2) directly targets a manyto-many nmt system by mining parallel corpora @xcite . standard practice is to fine-tune every parameter of a particular a pre-trained model to specialize it to a language pair (or domain) of interest @xcite . however, if we require specialization to many language pairs or domains, the storage and time costs of full fine-tuning may become prohibitive. moreover, as models grow ever larger, more efficient methods become attractive. as an alternative to full model fine-tuning, several parameter-efficient fine-tuning methods (pefts) have been proposed. such methods only fine-tune a small number of parameters, reducing storage cost, and avoid calculating the gradients for every model parameter, reducing training time and memory cost. examples include adapters @xcite and prefixtuning @xcite , which introduce a few extra parameters to fine-tune, keeping the pretrained model fixed. others like bitfit @xcite tune only the bias vectors of the backbone model and similarly @xcite update only cross-attention layers. pefts can produce results that are competitive with full fine-tuning. for instance, adapters can match full fine-tuning performance on the glue benchmark using only 2-4% additional parameters @xcite . however their potential for mt has not been fully explored. prior studies indicate that pefts designed for classification tasks can fail for mt @xcite , and it is not known how source and target language characteristics affect pefts' performance. in this work, we provide a comprehensive analysis of pefts for mt. for our analysis, we consider: (1) different pre-trained models which vary in size from 484 million to 1.2 billion total parameters, (2) several pefts, and (3) typographically and geographically diverse languages. moreover, we vary the number of tuned parameters, resulting in different parameter 'budgets', ranging from 0.03% to 10% of total model parameters. our main research questions are: rq1: for a given parameter budget, which peft works best? rq2: how does language similarity affect the performance of pefts for different parameter 'budgets'? rq3: how does (i) the pre-training objective, and (ii) the size of the parent model affect the performance of pefts? rq4: do pefts work better than fine-tuning for small dataset sizes? key findings 1) we found methods which introduce new parameters to a pre-trained model, namely adapters and prefix tuning, give us the best performance ( § 5.1). as we increase the number of new parameters, adapters retain good performance, while prefix-tuning falls behind. 2) we found a large variation in pefts' performance across language pairs. specifically, the distance between the source and target languages is negatively correlated with performance, especially for methods tuning the smallest number of parameters and methods tuning a subset of existing parameters (like bias terms or cross attention) ( § 5.2). 3) we observe that increasing model size, but keeping the same number of fine-tuned parameters, substantially increases mt performance ( § 5.3). finally, 4) we observe that adapters perform better than full fine-tuning for small datasets, with the advantage for adapters increasing as dataset size gets smaller ( § 5.4). ## background this section briefly describes the two multilingual pre-trained models that we focus on in this work, namely mbart and m2m-100. multilingual denoising pre-training multilingual bart, mbart @xcite , is a sequence-to-sequence transformer model @xcite that consists of an encoder and an autoregressive decoder. it is pre-trained with a denoising objective, reconstructing a document from a noisy version. mbart uses span masking and sentence permutation to noise the original document. it consists of 12 encoder and 12 decoder layers, with hidden dimension of 1024 and 16 attention heads. mbart is trained entirely on monolingual data that includes multiple languages and it has a large multilingual vocabulary of 250k tokens. in our experiments, we use mbart-50 @xcite which was pre-trained on 50 languages. ## parameter efficient fine-tuning methods all of our experiments fall under the umbrella of specialising a pre-trained sequence-to-sequence transformer model for mt of a particular language pair, with source language x and target language y. if the pre-training task was mt, and x and y were included, then a lower bound will be simply applying the pre-trained model without any changes. conversely an upper bound is fine-tuning 100% of the pre-trained model parameters ('full fine-tuning'). in between full fine-tuning and directly using the pre-trained model, we consider the following parameter-efficient fine-tuning methods (pefts) in this work: adapter-tuning @xcite 'adapter layers' are lightweight, learnable units inserted between transformer layers. they typically take the form of a feedforward network inserted as the final operation in a transformer layer. formally, we follow the architecture introduced by @xcite for mt: prefix-tuning (li and liang, 2021) prepends a sequence of continuous task-specific vectors ('prefixes') to the model input, in analogy to natural language prompts (e.g. 'translate this sentence:') which the transformer can attend to, but the prefix consists entirely of free parameters. for each transformer layer, the prefix is replaced with a new set of vectors, increasing expressiveness. concretely, we replace token embeddings by with @xmath0, v 0 ∈ r p×d the prefix vectors, and l the original sequence length, p the prefix length and d model dimension. before transformer layer ℓ we additionally set the first p hidden states to a new prefix vector, i.e. h ℓ [:p, :] = v ℓ with @xmath1. bitfit @xcite bias term fine-tuning was introduced in the context of fine-tuning bert for classification tasks, and consists of training only the bias terms and the task-specific classification layer. for mt we additionally fine-tune all decoder bias terms, and do not need the classification head. we introduce a simple improvement to bitfit, based on replacing redundant parameters with ones that increase expressiveness. note that bitfit finetunes bias parameters in layer-norm (ln) modules @xcite , since the layer-norm contains the following affine transformation: notice the same space of functions is available by only updating the b ℓ m term in w ℓ m • β + b ℓ m . we simply switch to updating γ instead of β, i.e. unfreezing the ln weight and freezing the bias, in order to increase expressiveness (confirmed empirically in § 5.1). we use this version of bitfit throughout this work unless stated otherwise. x-attention tuning @xcite refers to fine-tuning only cross-attention (x-attention) and corresponding layer-norm parameters located in each decoder layer of a transformer model. this method is based on the importance of crossattention for mt. ## experiments datasets we conduct experiments with a selection of 12 typologically and geographically diverse languages, paired with english. in our experiments, we fine-tune the pre-trained model on only one language pair and translation direction at a time (e.g. @xmath2. the parallel data for all languages is from ted talks in order to factor out the impact of the domain differences (except finnish and estonian which we only use for a separate control experiment). to pick these languages, we consider variation in language families and scripts. more details of the datasets are given in appendix a. experimental settings we used mbart-50 @xcite and m2m-100 @xcite as our multilingual pre-trained models, and all the languages we experiment with are included in their pre-training data. mbart needs to learn machine translation with parallel data, but m2m-100 can also be used without finetuning, since it is initially pre-trained for mt (see § 2). we conduct experiments with both the base and the medium size m2m-100, to measure the impact of parent model size. for all fine-tuning methods, we fine-tuned models with a maximum learning rate of 1e-4 with 2500 warm-up steps for 100k training updates. we picked the best model based on dev set perplexity. we used a maximum batch size of 1024 tokens for mbart and 600 tokens for m2m-100, with a gradient accumulation step (update-frequency) of 2 for both models. all experiments are performed with the fairseq @xcite library. additional details including dataset splits are in appendix a. we use bleu scores to estimate mt quality, calculated from sacrebleu 2 @xcite . to compare fine-tuning methods across different languages, we often report relative performance with respect to full fine-tuning (ft) for each language by calculating the ratio of each method's bleu score w.r.t. the full ft bleu score. 3 on the recommendation of @xcite we report chrf @xcite in appendix c for each fine-tuning method. parameter budget selection in order to fairly compare different methods, we selected a series of parameter 'budgets', and adjusted the settings of each method such that they update the same number of parameters. to determine the parameter budgets, we used the number of trainable parameters for the cross-attention update and bitfit since these numbers are constant (unlike adapters and prefix-tuning, where we have an adjustable bottleneck dimension). additionally, when comparing adapters and prefix-tuning, we start from the parameter size of the smallest adapter where the bottleneck dimension is 1. foot_0 ## results and discussion in this section, we first compare the performance of various pefts on two language directions for different parameter budgets § 5.1. we then select a subset of these methods to test on ten language directions, in order to evaluate the effect of language similarity on the performance of pefts § 5.2. we use these results to explore the effect of parent model pre-training § 5.3 and parent model size § 5.3. we noticed that on the language directions with the smallest dataset size, adapter methods outperformed full fine-tuning, and therefore conducted control experiments showing that as dataset size decreases, adapters outperform full fine-tuning by a larger margin. ## related work pefts have been widely used for fine-tuning transformer models to new tasks, domains or languages. adapters @xcite have been used in multi-task learning @xcite @xcite , cross-lingual transfer @xcite and multilingual nmt @xcite @xcite . prefix-tuning (li and liang, 2021) and prompt-tuning @xcite (i.e. only using soft prompt tokens without prefix vectors in each layer), have a natural interpretation in terms of virtual tokens. they can be used as task embeddings for inter-task transferability @xcite . lora @xcite injects trainable low-rank matrices into query and value projection matrices of each transformer layer. @xcite present a unified framework that integrates the above methods. some of these methods have been compared in a controlled setting for english classification tasks @xcite or only a single language pair (english and romanian) for mt @xcite . @xcite test pefts for various english classification tasks and observe that on the tasks with the smallest dataset sizes, pefts outperform fine-tuning, but they do not conduct a control experiment varying dataset size and parent model for a single task as we do. aspects of efficiency and scale in mt in terms of inference cost @xcite , vocabulary size @xcite , model size @xcite and number of languages @xcite have been explored. other work aims to improve full ft for domain adaptation by mixing in different data @xcite , regularisation @xcite or many other methods @xcite . however, none of these works study pefts for mt, and we aim to fill this gap. ## conclusion do pefts work for mt? we found that the answer depends on multiple factors: the particular method, the backbone model, the number of tuned parameters and the fine-tuning language pair. adapters usually have the highest performance out of all pefts ( § 5.1), although for the smallest parameter budgets we consider, prefix tuning outperforms adapters for mbart. for large parameter budgets (≈50m parameters) adapters almost recover full fine-tuning performance, and even for lower budgets, if the pre-training task was mt, i.e. m2m-100, adapters can @xmath3. however pefts only outperform full ft for smaller dataset sizes ( § 5.4), less than around ≈2k examples for mbart and ≈128k for m2m. future work could explore in detail how the difference between pre-training objective and fine-tuning task affects this phenomenon. using peft with a larger model (m2m-100 medium size) can outperform full ft of a smaller model (m2m-100 base size). however when translating in the @xmath4.g. korean, full ft is superior ( § 5.3). more generally, distantly related language pairs require more parameters to be tuned to get close to full ft, for all methods ( § 5.2).