T OWARDS A U NIFIED V IEW OF P ARAMETER -E FFICIENT T RANSFER L EARNING
A BSTRACT
Fine-tuning large pretrained language models on downstream tasks has become the de-facto learning paradigm in NLP.
However, conventional approaches ﬁnetune all the parameters of the pretrained model, which becomes prohibitive as the model size and the number of tasks grow.
Recent work has proposed a variety of parameter-efﬁcient transfer learning methods that only ﬁne-tune a small number of (extra) parameters to attain strong performance.
While effective, the critical ingredients for success and the connections among the various methods are poorly understood.
In this paper we break down the design of state-of-the-art parameter-efﬁcient transfer learning methods and present a uniﬁed framework that establishes connections between them.
Speciﬁcally, we re-frame them as modiﬁcations to speciﬁc hidden states in pretrained models and deﬁne a set of design dimensions along which different methods vary, such as the function to compute the modiﬁcation and the position to apply the modiﬁcation.
Through comprehensive empirical studies across machine translation, text summarization, language understanding and text classiﬁcation benchmarks, we utilize the uniﬁed view to identify important design choices in previous methods.
Furthermore, our uniﬁed framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efﬁcient ﬁne-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to ﬁne-tuning all parameters on all four tasks.
I NTRODUCTION
Transfer learning from pre-trained language models (PLMs) is now the prevalent paradigm in natural language processing, yielding strong performance on many tasks (Peters et al., 2018;
Devlin et al., 2019; Qiu et al., 2020).
The most common way to adapt general-purpose PLMs to downstream tasks is to ﬁne-tune all the model parameters ( full ﬁne-tuning ).
However, this results in a separate copy of ﬁne-tuned model parameters for each task, which is prohibitively expensive when serving models that perform a large number of tasks.
This issue is particularly salient with the ever-increasing size of PLMs, which now range from hundreds of millions (Radford et al., 2019; Lewis et al., 2020) to hundreds of billions (Brown et al., 2020) or even trillions of parameters (Fedus et al., 2021).
To mitigate this issue, a few lightweight alternatives have been proposed to update only a small number of extra parameters while keeping most pretrained parameters frozen.
For example, adapter tuning (Houlsby et al., 2019) inserts small neural modules called adapters to each layer of the pretrained network and only the adapters are trained at ﬁne-tuning time.
Inspired by the success of prompting methods that control PLMs through textual prompts (Brown et al., 2020; Liu et al., 2021a), preﬁx tuning (Li & Liang, 2021) and prompt tuning (Lester et al., 2021) prepend an additional l tunable [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Figure]
The number of ﬁne-tuned parameters is relative to the tuned parameters in full ﬁne-tuning
[Figure]
preﬁx tokens to the input or hidden layers and only train these soft prompts when ﬁne-tuning on downstream tasks.
More recently, Hu et al.
(2021) learn low-rank matrices to approximate parameter updates.
We illustrate these methods in Figure 1.
These approaches have all been reported to demonstrate comparable performance to full ﬁne-tuning on different sets of tasks, often through updating less than 1% of the original model parameters.
Besides parameter savings, parameter-efﬁcient tuning makes it possible to quickly adapt to new tasks without catastrophic forgetting (Pfeiffer et al., 2021) and often exhibits superior robustness in out-of-distribution evaluation (Li & Liang, 2021).
However, we contend that the important ingredients that contribute to the success of these parameterefﬁcient tuning methods are poorly understood, and the connections between them are still unclear.
In this paper, we aim to answer three questions: (1) How are these methods connected?
(2) Do these methods share design elements that are essential for their effectiveness, and what are they?
(3) Can the effective ingredients of each method be transferred to others to yield more effective variants?
In order to answer these questions, we ﬁrst derive an alternative form of preﬁx tuning that reveals preﬁx tuning’s close connections with adapters (§3.1).
Based on this we then devise a uniﬁed framework that frames the aforementioned methods as different ways to modify the hidden representations of frozen PLMs (§3.2).
Our uniﬁed framework decomposes previous methods along a shared set of design dimensions, such as the function used to perform the modiﬁcation, the position in which to impose this modiﬁcation, and how to integrate the modiﬁcation.
This framework allows us to transfer design choices across approaches to propose new variants such as adapters with multiple heads (§3.3).
In experiments, we ﬁrst show that existing parameter-efﬁcient tuning methods still lag behind full ﬁne-tuning on higher-resource and challenging tasks (§4.2), as exempliﬁed in Figure 2.
Then we utilize the uniﬁed framework to identify critical design choices and validate the proposed variants empirically (§4.3-4.6).
Our experiments on four NLP benchmarks covering text summarization, machine translation (MT), text classiﬁcation, and general language understanding, demonstrate that the proposed variant uses less parameters than existing methods while being more effective, matching full ﬁne-tuning results on all four tasks.
P RELIMINARIES 2. R ECAP OF THE TRANSFORMER A RCHITECTURE
The transformer model (Vaswani et al., 2017) is now the workhorse architecture behind most stateof-the-art PLMs.
In this section we recap the equations of this model for completeness.
Transformer models are composed of L stacked blocks, where each block (Figure 1) contains two types of sub- layers: multi-head self-attention and a fully connected feed-forward network (FFN).
The conventional attention function maps queries Q ∈ R n × d k and key-value pairs K
[Equation]
where n and m are the number of queries and key-value pairs respectively.
Multi-head attention performs the attention function in parallel over N h heads, where each head is separately parameterized by W ( i ) q , W ( i ) k , W ( i ) v ∈ R d × d h to project inputs to queries, keys, and values.
Given a sequence of m vectors C ∈ R m × d over which we would like to perform attention and a query vector x ∈ R d , multi-head attention (MHA) computes the output on each head and concatenates them:
[Equation]
where W o ∈ R d × d .
d is the model dimension, and in MHA d h is typically set to d/N h to save parameters, which indicates that each attention head is operating on a lower-dimensional space.
The other important sublayer is the fully connected feed-forward network (FFN) which consists of two linear transformations with a ReLU activation function in between:
[Equation]
where W 1 ∈ R d × d m , W 2 ∈ R d m × d .
Transformers typically use a large d m , e.g. d m = 4 d .
Finally, a residual connection is used followed by layer normalization (Ba et al., 2016).
2. O VERVIEW OF P REVIOUS P ARAMETER - EFFICIENT T UNING M ETHODS
Below and in Figure 1, we introduce several state-of-the-art parameter-efﬁcient tuning methods.
Unless otherwise speciﬁed, they only tune the added parameters while the PLM’s are frozen.
Adapters (Houlsby et al., 2019):
The adapter approach inserts small modules (adapters) between transformer layers.
The adapter layer generally uses a down-projection with W down ∈ R d × r to project the input h to a lower-dimensional space speciﬁed by bottleneck dimension r , followed by a nonlinear activation function f ( · ) , and a up-projection with W up ∈ R r × d .
These adapters are surrounded by a residual connection, leading to a ﬁnal form:
[Equation]
Houlsby et al.
(2019) places two adapters sequentially within one layer of the transformer, one after the multi-head attention and one after the FFN sub-layer.
Pfeiffer et al.
(2021) have proposed a more efﬁcient adapter variant that is inserted only after the FFN “add & layer norm” sub-layer.
Preﬁx Tuning (Li & Liang, 2021):
Inspired by the success of textual prompting methods (Liu et al., 2021a), preﬁx tuning prepends l tunable preﬁx vectors to the keys and values of the multihead attention at every layer.
Speciﬁcally, two sets of preﬁx vectors P k , P v ∈ R l × d are concatenated with the original key K and value V .
Then multi-head attention is performed on the new preﬁxed keys and values.
The computation of head i in Eq.
2 becomes:
[Equation]
P k and P v are split into N h head vectors respectively and P ( i ) k , P ( i ) v ∈ R l × d/N h denote the i -th head vector.
Prompt-tuning (Lester et al., 2021) simpliﬁes preﬁx-tuning by only prepending to the input word embeddings in the ﬁrst layer; similar work also includes P-tuning (Liu et al., 2021b).
LoRA (Hu et al., 2021): LoRA injects trainable low-rank matrices into transformer layers to approximate the weight updates.
For a pre-trained weight matrix W ∈ R d × k , LoRA represents its update with a low-rank decomposition W +∆ W = W + W down W up , where W down ∈ R d × r , W up ∈
R r × k are tunable parameters.
LoRA applies this update to the query and value projection matrices ( W q , W v ) in the multi-head attention sub-layer, as shown in Figure 1.
For a speciﬁc input x to the linear projection in multi-head attention, LoRA modiﬁes the projection output h as:
[Equation]
Below, we sometimes ignore the head index i to simplify notation when there is no confusion.
[Table]
/Xvuftf/9+OHj/Uf/Y2BMf/2rqs/frzk/H+P/B9Wg0mE=</latexit>
[Figure]
where s ≥ 1 is a tunable scalar hyperparameter.
Others: Other parameter-efﬁcient tuning methods include BitFit (Ben Zaken et al., 2021), which only ﬁne-tunes bias vectors in the pre-trained model, and diff-pruning (Guo et al., 2021), which learns a sparse parameter update vector.
B RIDGING THE G AP – A U NIFIED V IEW
We ﬁrst derive an equivalent form of preﬁx tuning to establish its connection with adapters.
We then propose a uniﬁed framework for parameter-efﬁcient tuning that includes several state-of-the-art methods as instantiations.
3. A C LOSER L OOK AT P REFIX T UNING
Eq.
5 describes the mechanism of preﬁx tuning which changes the attention module through prepending l learnable vectors to the original attention keys and values.
Here, we derive an equivalent form of Eq.
and provide an alternative view of preﬁx tuning:
[Equation]
where λ ( x ) is a scalar that represents the sum of normalized attention weights on the preﬁxes:
[Equation]
Note that the ﬁrst term in Eq. 7, Attn ( xW q , CW k , CW v ) , is the original attention without preﬁxes, whereas the second term is a position-wise modiﬁcation independent of C .
Eq.
7 gives an alternative view of preﬁx tuning that essentially applies a position-wise modiﬁcation to the original head attention output h through linear interpolation:
[Equation]
The Connection with Adapters: We deﬁne W [Equation] W q [Equation]
k [Equation] W 2 = P v , f =softmax, and rewrite Eq. 9:
[Equation]
(10) which reaches a very similar form to the adapter function in Eq. 4, except that preﬁx tuning is performing weighted addition while the adapter one is unweighted.
6 Figure 3b demonstrates the [Table] computation graph of preﬁx tuning from this view, which allows for abstraction of preﬁx tuning as a plug-in module like adapters.
Further, we note that
W 1 ∈ R d h × l and W 2 ∈ R l × d h are lowrank matrices when l is small [Table] and thus they function similarly to the W down and W up matrices in adapters.
This view also suggests that the number of preﬁx vectors, l , plays a similar role to the bottleneck dimension r in adapters: they both represent the rank limitation of computing the modiﬁcation vector ∆ h .
Thus we also refer l as the bottleneck dimension.
Intuitively, the rank limitation implies that ∆ h is a linear combination of the same l (or ≤ l ) basis vectors for any x .
The Difference from Adapters:
In addition to the gating variable λ , we emphasize three differences between preﬁx tuning and adapters.
(1) As demonstrated in Figure 3, preﬁx tuning uses x , the input of the PLM layer, to compute ∆ h , while adapters use h , the output of the PLM layer.
Thus, preﬁx tuning can be thought of as a “parallel” computation to the PLM layer, whereas the typical adapter is “sequential” computation.
(2) Adapters are more ﬂexible with respect to where they are inserted than preﬁx tuning: adapters typically modify attention or FFN outputs, while preﬁx tuning only modiﬁes the attention output of each head.
Empirically, this makes a large difference as we will show in §4.4.
(3) Eq.
10 applies to each attention head, while adapters are always single-headed, which makes preﬁx tuning more expressive: head attention is of dimension d/N h – basically we have full rank updates to each attention head if l ≥ d/N h , but we only get full-rank updates to the whole attention output with adapters if r ≥ d .
Notably, preﬁx tuning is not adding more parameters than adapters when l = r .
7 We empirically validate such multi-head inﬂuence in §4.4.
3. T HE U NIFIED F RAMEWORK
Inspired by the connections between preﬁx tuning and adapters, we propose a general framework that aims to unify several state-of-the-art parameter-efﬁcient tuning methods.
Speciﬁcally, we cast them as learning a modiﬁcation vector ∆ h , which is applied to various hidden representations.
Formally, we denote the hidden representation to be directly modiﬁed as h , and the direct input to the PLM sub-module that computes h as x (e.g. h and x can be the attention output and input respectively).
To characterize this modiﬁcation process, we deﬁne a set of design dimensions, and different methods can be instantiated by varying values along these dimensions.
We detail the design dimensions below, and illustrate how adapters, preﬁx tuning, and LoRA fall along them in Table 1:
Functional Form is the speciﬁc function that computes ∆ h .
We have detailed the functional form for adapters, preﬁx tuning, and LoRA in Eq. 4, 6, and 10 respectively.
The functional forms of all these methods are similar with a proj down → nonlinear → proj up architecture, while “nonlinear” degenerates to the identity function in LoRA.
Modiﬁed Representation indicates which hidden representation is directly modiﬁed.
Insertion Form is how the added module is inserted into the network.
As mentioned in the previous section and shown in Figure 3, traditionally adapters are inserted at a position in a sequential manner, where both the input and output are h .
Preﬁx tuning and LoRA – although not originally described in this way – turn out to be equivalent to a parallel insertion where x is the input.
Composition Function is how the modiﬁed vector ∆ h is composed with the original hidden representation h to form the new hidden representation.
For example, adapters perform simple additive composition, preﬁx tuning uses a gated additive composition as shown in Eq.
10, and LoRA scales ∆ h by a constant factor and adds it to the original hidden representation as in Eq.
We note that many other methods not present in Table 1 ﬁt into this framework as well.
For example, prompt tuning modiﬁes the head attention in the ﬁrst layer in a way similar to preﬁx tuning, and various adapter variants (Pfeiffer et al., 2021; Mahabadi et al., 2021) can be represented in a similar way as adapters.
Critically, the uniﬁed framework allows us to study parameter-efﬁcient tuning methods along these design dimensions, identify the critical design choices, and potentially transfer design elements across approaches, as in the following section.
3. T RANSFERRING D ESIGN E LEMENTS
Here, and in Figure 3, we describe just a few novel methods that can be derived through our uniﬁed view above by transferring design elements across methods: (1) Parallel Adapter is the variant by transferring the parallel insertion of preﬁx tuning into adapters.
Interestingly, while we motivate the parallel adapter due to its similarity to preﬁx tuning, concurrent work (Zhu et al., 2021) independently proposed this variant and studied it empirically; (2) Multi-head Parallel Adapter is a further step to make adapters more similar to preﬁx tuning: we apply parallel adapters to modify head attention outputs as preﬁx tuning.
This way the variant improves the capacity for free by utilizing the multi-head projections as we discuss in §3.1.
(3) Scaled Parallel Adapter is the variant by transferring the composition and insertion form of LoRA into adapters, as shown in Figure 3e.
Our discussion and formulation so far raise a few questions: Do methods varying the design elements above exhibit distinct properties?
Which design dimensions are particularly important?
Do the novel methods described above yield better performance?
We answer these questions next.
E XPERIMENTS 4. G ENERAL S ETUP
Datasets: We study four downstream tasks:
(1) XSum (Narayan et al., 2018) is an English summarization dataset where models predict a summary given a news article; (2) English to Romanian translation using the WMT 2016 en-ro dataset (Bojar et al., 2016); (3) MNLI (Williams et al., 2018) is an English natural language inference dataset where models predict whether one sentence entails, contradicts, or is neutral to another.
(4) SST2 (Socher et al., 2013) is an English sentiment classiﬁcation benchmark where models predict whether a sentence’s sentiment is positive or negative.
Setup: We use BART LARGE (Lewis et al., 2020) and a multilingual version of it, mBART LARGE (Liu et al., 2020a), as the underlying pretrained models for XSum and en-ro translation respectively, and we use RoBERTa BASE (Liu et al., 2019) for MNLI and SST2.
We vary the bottleneck dimension within { 1 , 30 , 200 , 512 , 1024 } if needed.
9 We mainly study adapters, preﬁx tuning (preﬁx), and
LoRA which greatly outperform bitﬁt and prompt tuning in our experiments.
In the analysis sections (§4.3-4.5) we insert adapters either at the attention or FFN layers for easier analysis, but include the results of inserting at both places in the ﬁnal comparison (§4.6).
We re-implement these methods based on their respective public code.
10 We use the huggingface transformers library (Wolf et al., 2020) for our implementation.
Complete setup details can be found in Appendix A.
Evaluation: We report ROUGE 1/2/L scores (R-1/2/L, Lin (2004)) on the XSum test set, BLEU scores (Papineni et al., 2002) on the en-ro test set, and accuracy on the MNLI and SST2 dev set.
For MNLI and SST2, we take the median of ﬁve random runs.
We also report the number of tuned parameters relative to that in full ﬁne-tuning (#params).
Number of Tunable Parameters: BART and mBART have an encoder-decoder structure that has three types of attention: encoder self-attention, decoder self-attention, and decoder cross-attention.
RoBERTa only has encoder self-attention.
For each attention sub-layer, the number of parameters
[Figure]
[Table]
[Table]
[Table]
used of each method is: (1) preﬁx tuning prepends l vectors to the keys and values and uses 2 × l × d parameters; (2) adapter has W down and W up thus uses 2 × r × d parameters; (3) LoRA employs a pair of W down and W up for query and value projections, hence uses 4 × r × d parameters. For the adapter modiﬁcation at ffn, it uses 2 × r × d parameters which is the same as adapter at attention. Therefore, for a speciﬁc value of r or l , preﬁx tuning uses the same number of parameters as adapters, while
LoRA uses more parameters.
More details can be found in Appendix B.
4. T HE R ESULTS OF E XISTING M ETHODS
We ﬁrst overview the results of existing methods on the four tasks.
As shown in Figure 4 and Table 2, while existing methods can achieve competitive performance on MNLI and SST2 by tuning fewer than 1% parameters, a large gap is still present if we add 5% parameters in XSum and en-ro.
The gap remains signiﬁcant even though we increase the relative parameter size to > 10%.
Even larger gaps have been observed in Raffel et al.
(2020) on high-resource MT tasks.
This shows that many methods that claimed comparable results to full ﬁne-tuning on the GLUE benchmark with an encoder-only model (Guo et al., 2021; Ben Zaken et al., 2021; Mahabadi et al., 2021), or on relatively simple generation benchmarks such as E2E (Novikova et al., 2017) with an encoder-decoder model (Li & Liang, 2021), may not generalize well to other standard benchmarks.
The inﬂuencing factors could be complicated including the number of training samples, task complexity, or model architecture.
We thus advocate for future research on this line to report results on more diverse benchmarks to exhibit a more complete picture of their performance proﬁle.
Below, our analysis will mainly focus on the XSum and en-ro datasets to better distinguish different design choices.
We note that these two benchmarks are relatively high-resource performed with an encoder-decoder model (BART), while we will discuss the results on MNLI and SST2 with an encoder-only model (RoBERTa) in §4.6.
4. W HICH I NSERTION F ORM – S EQUENTIAL OR P ARALLEL ?
We ﬁrst study the insertion form design dimension, comparing the proposed parallel adapter (PA) variant to the conventional sequential adapter (SA) over both the attention (att) and FFN modiﬁcation.
We also include preﬁx tuning as a reference point.
As shown in Table 3, preﬁx tuning, which uses parallel insertion, outperforms attention sequential adapters.
Further, the parallel adapter is able to beat sequential adapters in all cases, 11 with PA (ffn) outperforming SA (ffn) by 1.7 R-2 points on
[Figure]
XSum and 0.8 BLEU points on en-ro respectively.
Given the superior results of parallel adapters over sequential adapters, we focus on parallel adapter results in following sections.
4. W HICH M ODIFIED R EPRESENTATION – A TTENTION OR FFN?
Setup: We now study the effect of modifying different representations.
We mainly compare attention and FFN modiﬁcation.
For easier analysis we categorize methods that modiﬁes any hidden representations in the attention sub-layer (e.g. the head output, query, etc) as modifying the attention module.
We compare parallel adapters at attention and FFN and preﬁx tuning.
We also transfer the FFN modiﬁcation to LoRA to have a LoRA (ffn) variant for a complete comparison.
Speciﬁcally, we use LoRA to approximate the parameter updates for the FFN weights W 1 ∈ R d × d m and
W 2 ∈ R
d m × d .
In this case W up in LoRA for W 1 (similar for W down of W 2 ) would have dimensions of r × d m , where d m = 4 d as described in §2.1.
Thus we typically use smaller r for LoRA (ffn) than other methods to match their overall parameter size in later experiments.
Results: As shown in Figure 5, any method with FFN modiﬁcation outperforms all the methods with attention modiﬁcation in all cases (the red markers are generally above all the blue ones, the only exception is ffn-PA with 2.4% params), often with fewer parameters.
Second, the same method applied at FFN always improves over its attention counterpart.
For example, LoRA (ffn) improves
LoRA (attn) by 1 R-2 points on XSum.
We also highlight that preﬁx tuning does not keep improving when we further increase the capacity, which is also observed in Li & Liang (2021).
These results suggest that FFN modiﬁcation can utilize the added parameters more effectively than attention, no matter what the functional form or composition function is .
We hypothesize that this is because the FFN learns task-speciﬁc textual patterns (Geva et al., 2021), while attention learns pairwise positional interactions which do not require large capacity for adapting to new tasks.
Is the story different when we use 0 .
1% parameters?
In §3.1 we reason that preﬁx tuning is more expressive than adapters (attn), which, however, is not reﬂected in Figure 5.
We conjecture that this is because multi-head attention is only superior when the parameter budget is small.
To validate this hypothesis, we compare preﬁx tuning to parallel adapters when they add 0 .
1% of the pretrained parameters.
To ablate the impact of the composition function, we also report the results of removing the gating in preﬁx tuning as h + ∆ h .
We include the results of the multi-head parallel adapter variant (MH PA) described in §3.3.
As shown in Table 4, the multi-head methods – preﬁx tuning and MH PA (attn) – outperform all others by at least 1.6 BLEU points when using 0.1% of the parameters.
Surprisingly, reducing l from 200 to 30 only causes 0.4 BLEU loss for preﬁx tuning while PA (attn) loses 1.9 points.
The gating composition function in preﬁx tuning slightly helps the results by 0.3 points.
We highlight that the MH parallel adapter improves the single-headed version by 1.6 points, which again veriﬁes the effectiveness of the multi-head formulation.
Combining the results in Figure 5 and Table 4, we conclude that modifying head attention shows the best results when the parameter budget is very small, while the FFN can better utilize modiﬁcations at larger capacities.
This suggests that it may be effective to allocate a larger parameter budget to
FFN modiﬁcation instead of treating attention and FFN equally as in Houlsby et al.
4. W HICH C OMPOSITION F UNCTION ?
We have presented three composition functions in §3.2: simple addition (adapter), gated addition (preﬁx tuning) and scaled addition (LoRA).
As it is unnatural to incorporate the exact gated addition into methods whose functional form does not use softmax, we examine the other two by [Table] ablating on LoRA and comparing with the proposed scaled parallel adapter (Scaled PA)
[Table]
we constrain modiﬁed representation to be FFN since it is generally more effective as shown in §4.4.
[Table]
Table 5 reports the results on XSum.
We set r asfor adapters and 102 for LoRA so that their tuned parameter sizes are the same.
We select s based on the R-2 score on the dev set.
We observe that
LoRA ( s = 4 ) performs better than parallel adapter.
However, the advantage disappears if we remove the scaling by setting s = 1 .
Through plugging the composition function of LoRA into parallel adapter, the resulted Scaled PA improves the vanilla parallel adapter by 0.56 ROUGE-2 points.
We also experiment with a learned scalar which does not give better results.
Therefore, we conclude that the scaling composition function is better than the vanilla additive one while being easily applicable.
4. A N E FFECTIVE I NTEGRATION BY T RANSFERRING F AVORABLE D ESIGN E LEMENTS
We ﬁrst highlight three ﬁndings in previous sections: (1) Scaled parallel adapter is the best variant to modify FFN; (2) FFN can better utilize modiﬁcation at larger capacities; and (3) modifying head attentions like preﬁx tuning can achieve strong performance with only 0.1% parameters.
Inspired by them, we mix and match the favorable designs behind these ﬁndings: speciﬁcally, we use preﬁx tuning with a small bottleneck dimension ( l = 30 ) at the attention sub-layers and allocate more parameter budgets to modify FFN representation using the scaled parallel adapter ( r = 512 ).
Since preﬁx tuning can be viewed as a form of adapter in our uniﬁed framework, we name this variant as Mix-And-Match adapter (MAM Adapter) .
In Table 6, we compare MAM adapter with various parameter-efﬁcient tuning methods.
For completeness, we also present results of other combination versions in Table 6: using parallel adapters at both attention and FFN layers and combining preﬁx tuning (attn) with LoRA (ffn) – both of these combined versions can improve over their respective prototypes.
However, MAM Adapter achieves the best performance on both tasks and is able to match the results of our full ﬁne-tuning by only updating 6 .
7% of the pre-trained parameters.
In
Table 2, we present the results of MAM Adapter on MNLI and SST2 as well, where MAM Adapter achieves comparable results to full ﬁne-tuning by adding only 0.5% of pretrained parameters.
D ISCUSSION
We provide a uniﬁed framework for several performant parameter-tuning methods, which enables us to instantiate a more effective model that matches the performance of full ﬁne-tuning method through transferring techniques across approaches.
We hope our work can provide insights and guidance for future research on parameter-efﬁcient tuning.
E THICS S TATEMENT
Our work proposes a method for efﬁcient ﬁne-tuning of pre-trained models, in particular language models.
Pre-trained language models have a wide variety of positive applications, such as the applications to summarization, translation, or language understanding described in our paper.
At the same time, there are a number of ethical concerns with language models in general, including concerns regarding the generation of biased or discriminative text (Bordia & Bowman, 2019), the leakage of private information from training data (Carlini et al., 2020), and environmental impact of training or tuning them (Strubell et al., 2019).
Our method attempts to train language models making minimal changes to their pre-existing parameters.
While it is an interesting research question whether parameter-efﬁcient ﬁne-tuning methods exacerbate, mitigate, or make little change to issues such as bias or information leakage, to our knowledge no previous work has examined this topic.
It is an interesting avenue for future work.
With respect to environmental impact, the methods proposed in this paper add a small number of extra parameters and components to existing models, and thus they have a nominal negative impact on training and inference time – for example, the ﬁnal MAM Adapter needs 100% - 150% training time of full ﬁne-tuning in our four benchmarks since parameter-efﬁcient tuning typically needs more epochs to converge; the inference time is roughly the same as the model obtained by full ﬁne-tuning.
On the other hand, as the methods proposed in this paper may obviate the need for full ﬁne-tuning, this may also signiﬁcantly reduce the cost (in terms of memory/deployed servers) of serving models.
Notably, the great majority of the experimentation done for this paper was performed on a data center powered entirely by renewable energy.
R EPRODUCIBILITY S TATEMENT
In addition to the setup description in §4.1, we have detailed the complete experiments setup such as batch size, optimizer, learning rates in Appendix A. Besides, we have publicized our source code.
These resources should be sufﬁcient to reproduce results of the paper.
A CKNOWLEDGEMENT
We thank the anonymous reviewers for their comments.
This work was supported in part by the
CMU-Portugal MAIA Project, a Baidu PhD Fellowship for Junxian He, and a CMU Presidential
Fellowship for Chunting Zhou.
A E XPERIMENTS
A.
S ETUPS
[Table]
We implement all the parameter-efﬁcient tuning methods using the huggingface transformers library (Wolf et al., 2020).
We use BART LARGE (Lewis et al., 2020) and mBART LARGE (Liu et al., 2020b) (mBART-cc25) for the summarization and machine translation tasks respectively, and we use RoBERTa BASE (Liu et al., 2019) for MNLI and SST2.
BART LARGE and mBART LARGE have the same encoder-decoder architectures.
mBART LARGE is pre-trained on 25 languages.
We use their public checkpoints from the transformers library in experiments.
For MT and classiﬁcations tasks, the max token lengths of training data are set to be 150 and 512 respectively.
For XSum, we set the max length of source articles to be 512 and the max length of the target summary to be 128.
The detailed dataset statistics is present in Table 7.
In our summarization experiments, we only useexamples for validation to save time.
While we vary the bottleneck dimension within { 1 , 30 , 512 , 1024 } as mentioned in §4.1, we test bottleneck dimension 1024 only when the modiﬁed representation is FFN, because the training of preﬁx tuning does not ﬁt into 48GB GPU memory when l = 1024 .
While other methods do not have memory issues, we keep the bottleneck dimension of attention modiﬁcation at most 512 to have a relatively fair comparison with preﬁx tuning.
For LoRA we always tune its scaling hyperparameters s on the dev set.
A. T RAINING AND E VALUATION
We present some training hyperparameters of parameter-efﬁcient tuning methods in Table 8.
For all the tasks, we train with the Adam optimizer (Kingma & Ba, 2015), and use a polynomial learning rate scheduler that linearly decays the learning rate throughout training.
We set the warm up steps of learning rate to be 0 for both MT and summarization tasks, and for the classiﬁcation tasks, learning rate is linearly warmed up from 0 for the ﬁrst 6% of the total training steps before decay.
For full ﬁne-tuning we set these training hyperparameters following Lewis et al.
(2020) (XSum), Liu et al.
(2020b) (en-ro), and (Liu et al., 2019) (MNLI and SST2).
We also did hyperparameter search in the full ﬁne-tuning case to try to reproduce their results.
We set dropout rate to be 0.1 for all the tasks.
We use ROUGE-2 and perplexity as the validation metrics for summarization and MT respectively.
For MT and text summarization, we use beam search for decoding and set the number of beams to be 6 and 5 following previous work (Li & Liang, 2021; Liu et al., 2020b).
The min and max generation lengths for summarization and MT are set to be (10, 60) and (1, 200) respectively.
A. O THER E XPERIMENTAL D ETAILS
Preﬁx Tuning:
Following Li & Liang (2021), we reparameterize the preﬁx vectors by a MLP network which is composed of a small embedding matrix and a large feedforward neural network.
This is conducive for learning due to the shared parameters across all layers.
LoRA: LoRA and adapter employ different parameter initialization methods: LoRA uses a random Kaiming uniform (He et al., 2015) initialization for W down and zero for W up (LoRA init), while adapters use the same initialization as BERT (Devlin et al., 2019).
We found it beneﬁcial to use the same initialization method as LoRA in scaled PA.
[Table]
B C OMPUTATION OF T UNABLE P ARAMETERS
[Table]
[Table]
We compute the number of tunable parameters based on where the tunable module is inserted into and how it is parameterized.
The pretrained-models for summarization or MT have an encoderdecoder structure and each has L layers, whereas RoBERTa BASE for classiﬁcation tasks only has
L encoder layers.
To simplify the computation of tunable parameters, we compute the sum of parameter used in one encoder layer and one decoder layer as the parameter overhead of one single layer of the pre-trained encoder-decoder model.
Each layer has N attn sub-layers and N ﬀn sublayers.
For the encoder-decoder models, N attn = 3 : the encoder self-attention, the decoder selfattention and the decoder cross-attention.
For the classiﬁcation tasks, RoBERTa BASE only has the encoder self-attention, thus N attn = 1 .
We present the number of attention and ffn sub-layers for different pre-trained models in Table 10.
For modiﬁcations applied at the attention sub-layers, the number of tunable parameters is computed by | Θ | attn =
N attnW × N attn × L , where N attnW denotes the number of parameters ( W down or W up ) used for one attention sub-layer.
Similarly, the number of tunable parameters for the FFN sub-layers is computed by | Θ | ﬀn = N ﬀnW × N ﬀn ×
L .
In Table 10, we show the number of parameters for one sub-layer.
As we have explained in §4.4, LoRA approximates the update of each weight matrix with a pair of W down and W up , thus
LoRA typically uses more parameters with the same r as other methods.
Finally, the total number of tunable parameters for preﬁx tuning, adapter variants and LoRA is | Θ | = | Θ | attn + | Θ | ﬀn as applicable.
Prompt tuning prepends l tunable vectors at the input layer and uses l × d number of parameters.
Using MBART/BART as an example, we present the number of parameters used by several representative methods throughout our paper in Table 11, where adapter variants include sequential adapter, parallel adapter, scaled adapter and multi-head adapter.
[Table]
C F ULL R ESULTS ON D IFFERENT B OTTLENECK D IMENSIONS
[Table]