Original: L O RA : L OW -R ANK A DAPTATION OF L ARGE L AN - GUAGE M ODELS
Revised: L O RA: L OW -R ANK A DAPTATION OF L ARGE L ANGUAGE M ODELS

Original: Anonymous authors
Revised: Edward Hu Yelong Shen Phillip Wallis Allen-Zhu

Original: Paper under double-blind review
Revised: Yuanzhi Li Shean Wang Wang Chen

Revised: Microsoft

Original: A BSTRACT
Revised: A BSTRACT

Original: An important paradigm of natural language processing consists of large-scale pretraining on general domain data and adaptation to particular tasks or domains.
Revised: An important paradigm of natural language processing consists of large-scale pretraining on general domain data and adaptation to particular tasks or domains.

Original: As we pre-train larger models, full ﬁne-tuning, which retrains all model parameters, becomes less feasible.
Revised: As we pre-train larger models, full ﬁne-tuning, which retrains all model parameters, becomes less feasible.

Original: Using GPT-3 175B as an example – deploying independent instances of ﬁne-tuned models, each with 175B parameters, is prohibitively expensive.
Revised: Using GPT-3 175B as an example – deploying independent instances of ﬁne-tuned models, each with 175B parameters, is prohibitively expensive.

Original: We propose Lo
Revised: We propose Lo

Original: w- R ank
Revised: w- R ank

Original: A daptation, or LoRA, which freezes the pretrained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.
Revised: A daptation or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.

Original: Compared to GPT-3 175B ﬁne-tuned with Adam, LoRA can reduce the number of trainable parameters by  10,000times and the GPU memory requirement by 3 times.
Revised: Compared to GPT-3 175B ﬁne-tuned with Adam, LoRA can reduce the number of trainable parameters by a factor of 10,000 and the GPU memory requirement by a factor of 3.

Original: LoRA performs on-par or better than ﬁnetuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and no additional inference latency.
Revised: LoRA performs on-par or better than ﬁnetuning in model quality on RoBERTa, DeBERTa, GPT- and GPT-3, despite having fewer trainable parameters, a higher training throughput and, unlike adapters, no additional inference latency .

Original: We also providein the appendix an empirical investigation into rank-deﬁciency in language model adaptation, which sheds light on the efﬁcacy of LoRA.
Revised: We also provide an empirical investigation into rank-deﬁciency in language model adaptation, which sheds light on the efﬁcacy of LoRA.

Revised: We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa,

Revised: DeBERTa and GPT-2 at https github.com microsoft/LoRA

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Pretrained
Revised: Pretrained

Original: Weights 𝑊 ∈ ℝ 𝑑×𝑑 x
Revised: Weights 𝑊 ∈ ℝ 𝑑×𝑑 x

Original: h 𝐵 =
Revised: h 𝐵 =

Original: 𝐴 = 𝒩(0,𝜎 2 ) 𝑑 𝑟
Revised: 𝐴 = 𝒩(0,𝜎 2 ) 𝑑 𝑟

Original: Pretrained
Revised: Pretrained

Original: Weights 𝑊 ∈ ℝ 𝑑×𝑑 x f(x) 𝑑
Revised: Weights 𝑊 ∈ ℝ 𝑑×𝑑 x f(x) 𝑑

Original: Many applications in natural language processing rely on adapting one large-scale, pre-trained language model to multiple downstream applications.
Revised: Many applications in natural language processing rely on adapting one large-scale, pre-trained language model to multiple downstream applications.

Original: Such adaptation is usually done via ﬁne-tuning , which updates all the parameters of the pre-trained model.
Revised: Such adaptation is usually done via ﬁne-tuning , which updates all the parameters of the pre-trained model.

Original: The major downside of ﬁne-tuning is that the new model contains as many parameters as in the original model.
Revised: The major downside of ﬁne-tuning is that the new model contains as many parameters as in the original model.

Original: As larger models are trained every few months, this changes from a mere “inconvenience” for
Revised: As larger models are trained every few months, this changes from a mere “inconvenience” for

Original: GPT-2 (Radford et al., b) or RoBERTa large (Liu et al., 2019) to a critical deployment challenge for GPT-3 (Brown et al., 2020) with 175 billion trainable parameters.
Revised: GPT-2 (Radford et al., b) or RoBERTa large (Liu et al., 2019) to a critical deployment challenge for GPT-3 (Brown et al., 2020) with 175 billion trainable parameters.

Original: Many sought to mitigate this by adapting only some parameters or learning external modules for new tasks.
Revised: Many sought to mitigate this by adapting only some parameters or learning external modules for new tasks.

Original: This way, we only need to store and load a small number of task-speciﬁc parameters in addition to the pre-trained model for each task, greatly boosting the operational efﬁciency when deployed.
Revised: This way, we only need to store and load a small number of task-speciﬁc parameters in addition to the pre-trained model for each task, greatly boosting the operational efﬁciency when deployed.

Original: However, existing techniques often introduce inference latency (Houlsby et al., 2019;
Revised: However, existing techniques often introduce inference latency (Houlsby et al., 2019;

Original: Rebufﬁ et al., 2017) by extending model depth or reduce the model’s usable sequence length (Li & Liang, 2021; Lester et al., 2021; Hambardzumyan et al., 2020;
Revised: Rebufﬁ et al., 2017) by extending model depth or reduce the model’s usable sequence length (Li & Liang, 2021; Lester et al., 2021; Ham- bardzumyan et al., 2020;

Original: Liu et al., 2021)
Revised: Liu et al., 2021)

Original: (Section 3).
Revised: (Section 3).

Original: More importantly, these method often fail to match the ﬁne-tuning baselines, posing a trade-off between efﬁciency and model quality.
Revised: More importantly, these method often fail to match the ﬁne-tuning baselines, posing a trade-off between efﬁciency and model quality.

Original: We take inspiration from Li et al.
Revised: We take inspiration from Li et al.

Original: (2018a);
Revised: (2018a);

Original: Aghajanyan et al.
Revised: Aghajanyan et al.

Original: (2020) which show that the learned over-parametrized models in fact reside on a low intrinsic dimension.
Revised: (2020) which show that the learned over-parametrized models in fact reside on a low intrinsic dimension.

Original: We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed
Revised: We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed

Original: Lo w-
Revised: Lo w-

Original: R ank
Revised: R ank

Original: A daptation (LoRA) approach.
Revised: A daptation (LoRA) approach.

Original: LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen, as shown in Figure 1.
Revised: LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen, as shown in Figure 1.

Original: Using GPT175B as an example, we show that a very low rank (i.e., r in Figure 1 can be one or two) sufﬁces even when the full rank (i.e., d ) is as high as 12,288, making LoRA both storage- and compute-efﬁcient.
Revised: Using GPT175B as an example, we show that a very low rank (i.e., r in Figure 1 can be one or two) sufﬁces even when the full rank (i.e., d ) is as high as 12,288, making LoRA both storage- and compute-efﬁcient.

Original: LoRA possesses several key advantages.
Revised: LoRA possesses several key advantages.

Original: • A pre-trained model can be shared and used to build many small LoRA modules for different tasks.
Revised: • A pre-trained model can be shared and used to build many small LoRA modules for different tasks.

Original: We can freeze the shared model and efﬁciently switch tasks by replacing the matrices A and B in Figure 1, reducing the storage requirement and task-switching overhead signiﬁcantly.
Revised: We can freeze the shared model and efﬁciently switch tasks by replacing the matrices A and B in Figure 1, reducing the storage requirement and task-switching overhead signiﬁcantly.

Original: • LoRA makes training more efﬁcient and lowers the hardware barrier to entry by up totimes when using adaptive optimizers since we do not need to calculate the gradients or maintain the optimizer states for most parameters.
Revised: • LoRA makes training more efﬁcient and lowers the hardware barrier to entry by up totimes when using adaptive optimizers since we do not need to calculate the gradients or maintain the optimizer states for most parameters.

Original: Instead, we only optimize the injected, much smaller low-rank matrices.
Revised: Instead, we only optimize the injected, much smaller low-rank matrices.

Original: • Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing no inference latency compared to ﬁne-tuning, by construction.
Revised: • Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing no inference latency compared to a fully ﬁne-tuned model, by construction.

Original: • LoRA is orthogonal to many prior methods and can be combined with many of them, such as preﬁx-tuning.
Revised: • LoRA is orthogonal to many prior methods and can be combined with many of them, such as preﬁx-tuning.

Original: We provide an example in Appendix F.
Revised: We provide an example in Appendix H.

Original: Terminologies and Conventions We make frequent references to the Transformer architecture and use the conventional terminologies for its dimensions.
Revised: Terminologies and Conventions We make frequent references to the Transformer architecture and use the conventional terminologies for its dimensions.

Original: We call the input and output dimension size of a Transformer layer d model .
Revised: We call the input and output dimension size of a Transformer layer d model .

Original: We use W q , W k , W v , and W o to refer to the query/key/value/output projection matrices in the self-attention module.
Revised: We use W q , W k , W v , and W o to refer to the query/key/value/output projection matrices in the self-attention module.

Original: W or W 0 refers to a pretrained weight matrix and ∆ W its accumulated gradient update during adaptation.
Revised: W or W 0 refers to a pretrained weight matrix and ∆ W its accumulated gradient update during adaptation.

Original: We use r to denote the rank of a LoRA module.
Revised: We use r to denote the rank of a LoRA module.

Original: We follow the conventions set out by (Vaswani et al., 2017;
Revised: We follow the conventions set out by (Vaswani et al., 2017;

Original: Brown et al., 2020) and use Adam (Loshchilov & Hutter, 2019; Kingma & Ba, 2017) for model optimization and use a Transformer MLP feedforward dimension d ffn = 4 × d model .
Revised: Brown et al., 2020) and use Adam (Loshchilov & Hutter, 2019; Kingma & Ba, 2017) for model optimization and use a Transformer MLP feedforward dimension d ffn = 4 × d model .

Original:  P ROBLEM S TATEMENT
Revised:  P ROBLEM S TATEMENT

Original: While our proposal is agnostic to training objective, we focus on  language modeling as our motivating use case.
Revised: While our proposal is agnostic to training objective, we focus on conditional language modeling as our motivating use case.

Original: Below is a brief description of the language modeling problem and, in particular, the maximization of conditional probabilities given a task-speciﬁc prompt.
Revised: Below is a brief description of the language modeling problem and, in particular, the maximization of conditional probabilities given a task-speciﬁc prompt.

Original: Suppose we are given a pre-trained autoregressive language model P Φ ( y | x ) parametrized by Φ .
Revised: Suppose we are given a pre-trained autoregressive language model P Φ ( y | x ) parametrized by Φ .

Original: For instance, P Φ ( y | x ) can be a generic multi-task learner such as GPT (Radford et al.
Revised: For instance, P Φ ( y | x ) can be a generic multi-task learner such as GPT (Radford et al.

Original: , b; Brown et al., 2020) based on the Transformer architecture (Vaswani et al., 2017).
Revised: , b; Brown et al., 2020) based on the Transformer architecture (Vaswani et al., 2017).

Original: Consider adapting this pre-trained model to downstream conditional text generation tasks, such as summarization, machine reading comprehension (MRC), and natural language to SQL (NL2SQL).
Revised: Consider adapting this pre-trained model to downstream conditional text generation tasks, such as summarization, machine reading comprehension (MRC), and natural language to SQL (NL2SQL).

Original: Each downstream task is represented by a training dataset of context-target pairs: Z = { ( x i , y i ) } i =1 ,..,N , where both x i and y i are sequences of tokens.
Revised: Each downstream task is represented by a training dataset of context-target pairs: Z = { ( x i , y i ) } i =1 ,..,N , where both x i and y i are sequences of tokens.

Original: For example, in NL2SQL, x i is a natural language query and y i its corresponding SQL command; for summarization, x i is the content of an article and y i its summary.
Revised: For example, in NL2SQL, x i is a natural language query and y i its corresponding SQL command; for summarization, x i is the content of an article and y i its summary.

Original: During full ﬁne-tuning, the model is initialized to pre-trained weights Φ 0 and updated to Φ 0 + ∆Φ by repeatedly following the gradient to maximize the conditional language modeling objective:
Revised: During full ﬁne-tuning, the model is initialized to pre-trained weights Φ 0 and updated to Φ 0 + ∆Φ by repeatedly following the gradient to maximize the conditional language modeling objective:

Original: [Equation]
Revised: [Equation]

Original: One of the main drawbacks for full ﬁne-tuning is that for each downstream task, we learn a different set of parameters ∆Φ whose dimension | ∆Φ | equals | Φ 0 | .
Revised: One of the main drawbacks for full ﬁne-tuning is that for each downstream task, we learn a different set of parameters ∆Φ whose dimension | ∆Φ | equals | Φ 0 | .

Original: Thus, if the pre-trained model is large (such as GPT-3 with | Φ 0 | ≈ 175 Billion), storing and deploying many independent instances of ﬁne-tuned models can be challenging, if at all feasible.
Revised: Thus, if the pre-trained model is large (such as GPT-3 with | Φ 0 | ≈ 175 Billion), storing and deploying many independent instances of ﬁne-tuned models can be challenging, if at all feasible.

Original: In this paper, we adopt a more parameter-efﬁcient approach, where the task-speciﬁc parameter increment ∆Φ = ∆Φ(Θ) is further encoded by a much smaller-sized set of parameters Θ with | Θ | (cid:28) | Φ 0 | .
Revised: In this paper, we adopt a more parameter-efﬁcient approach, where the task-speciﬁc parameter increment ∆Φ = ∆Φ(Θ) is further encoded by a much smaller-sized set of parameters Θ with | Θ | (cid:28) | Φ 0 | .

Original: The task of ﬁnding ∆Φ thus becomes optimizing over Θ :
Revised: The task of ﬁnding ∆Φ thus becomes optimizing over Θ :

Original: [Equation]
Revised: [Equation]

Original: In the subsequent sections, we propose to use a low-rank representation to encode ∆Φ that is both compute- and memory-efﬁcient.
Revised: In the subsequent sections, we propose to use a low-rank representation to encode ∆Φ that is both compute- and memory-efﬁcient.

Original: When the pre-trained model is GPT-3 175B, the number of trainable parameters | Θ | can be as small as 0 .
Revised: When the pre-trained model is GPT-3 175B, the number of trainable parameters | Θ | can be as small as 0 .

Original: 01% of | Φ 0 | .
Revised: 01% of | Φ 0 | .

Original:  A REN ’ T E XISTING S OLUTIONS G OOD E NOUGH ?
Revised:  A REN ’ T E XISTING S OLUTIONS G OOD E NOUGH ?

Original: The problem we set out to tackle is by no means new.
Revised: The problem we set out to tackle is by no means new.

Original: Since the inception of transfer learning, dozens of works have sought to make model adaptation more parameter- and compute-efﬁcient.
Revised: Since the inception of transfer learning, dozens of works have sought to make model adaptation more parameter- and compute-efﬁcient.

Original: See Section 6 for a survey of some of the well-known works.
Revised: See Section 6 for a survey of some of the well-known works.

Original: Using language modeling as an example, there are two prominent strategies when it comes to efﬁcient adaptations: adding adapter layers (Houlsby et al., 2019; Rebufﬁ et al., 2017; Pfeiffer et al., 2021; R¨uckl´e et al., 2020) or optimizing some forms of the input layer activations (Li & Liang, 2021; Lester et al., 2021; Hambardzumyan et al., 2020;
Revised: Using language modeling as an example, there are two prominent strategies when it comes to efﬁcient adaptations: adding adapter layers (Houlsby et al., 2019; Rebufﬁ et al., 2017; Pfeiffer et al., 2021; R ¨ uckl ´ e et al., 2020) or optimizing some forms of the input layer activations (Li & Liang, 2021; Lester et al., 2021; Hambardzumyan et al., 2020;

Original: Liu et al., 2021).
Revised: Liu et al., 2021).

Original: However, both strategies have their limitations, especially in a large-scale and latency-sensitive production scenario.
Revised: However, both strategies have their limitations, especially in a large-scale and latency-sensitive production scenario.

Original: Adapter Layers Introduce Inference Latency There are many variants of adapters.
Revised: Adapter Layers Introduce Inference Latency There are many variants of adapters.

Original: We focus on the original design by Houlsby et al.
Revised: We focus on the original design by Houlsby et al.

Original: (2019) which has two adapter layers per Transformer block and a more recent one by Lin et al.
Revised: (2019) which has two adapter layers per Transformer block and a more recent one by Lin et al.

Original: (2020) which has only one per block but with an additional
Revised: (2020) which has only one per block but with an additional

Original: LayerNorm (Ba et al., 2016).
Revised: LayerNorm (Ba et al., 2016).

Original: While one can reduce the overall latency by pruning layers or exploiting multi-task settings (R ¨ uckl ´ e et al., 2020; Pfeiffer et al., 2021), there is no direct ways to bypass the extra compute in adapter layers.
Revised: While one can reduce the overall latency by pruning layers or exploiting multi-task settings (R¨uckl´e et al., 2020; Pfeiffer et al., 2021), there is no direct ways to bypass the extra compute in adapter layers.

Original: This seems like a non-issue since adapter layers are designed to have few parameters (sometimes < 1% of the original model) by having a small bottleneck dimension, which limits the FLOPs they can add.
Revised: This seems like a non-issue since adapter layers are designed to have few parameters (sometimes < 1% of the original model) by having a small bottleneck dimension, which limits the FLOPs they can add.

Original: However, large neural networks rely on hardware parallelism to keep the latency low, and adapter layers have to be processed sequentially.
Revised: However, large neural networks rely on hardware parallelism to keep the latency low, and adapter layers have to be processed sequentially.

Original: This makes a difference in the online inference setting where the batch size is typically as small as one.
Revised: This makes a difference in the online inference setting where the batch size is typically as small as one.

Original: In a generic scenario without model parallelism, such as running inference on GPT-2 (Radford et al., b) medium on a single GPU, we see a noticeable increase in latency when using adapters, even with a very small bottleneck dimension (Table 1).
Revised: In a generic scenario without model parallelism, such as running inference on GPT-2 (Radford et al., b) medium on a single GPU, we see a noticeable increase in latency when using adapters, even with a very small bottleneck dimension (Table 1).

Original: [Table]
Revised: [Table]

Original: This problem gets worse when we need to shard the model as done in Shoeybi et al.
Revised: This problem gets worse when we need to shard the model as done in Shoeybi et al.

Original: Lepikhin et al.
Revised: Lepikhin et al.

Original: (2020), because the additional depth requires more synchronous GPU operations such as
Revised: (2020), because the additional depth requires more synchronous GPU operations such as

Original: AllReduce and Broadcast , unless we store the adapter parameters redundantly many times.
Revised: AllReduce and Broadcast , unless we store the adapter parameters redundantly times

Original: Directly Optimizing the Prompt is Hard The other direction, as exempliﬁed by preﬁx tuning (Li & Liang, 2021), faces a different challenge.
Revised: Directly Optimizing the Prompt is Hard The other direction, as exempliﬁed by preﬁx tuning (Li & Liang, 2021), faces a different challenge.

Original: We observe that preﬁx tuning is difﬁcult to optimize and that its performance changes non-monotonically in trainable parameters, conﬁrming similar observations in the original paper.
Revised: We observe that preﬁx tuning is difﬁcult to optimize and that its performance changes non-monotonically in trainable parameters, conﬁrming similar observations in the original paper.

Original: More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods.
Revised: More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt underperform other methods.

Original: We defer the study on task performance to Section 5.
Revised: We defer the study on task performance to Section 5.

Original:  O UR M ETHOD
Revised:  O UR M ETHOD

Original: We describe the simple design of LoRA and its practical beneﬁts.
Revised: We describe the simple design of LoRA and its practical beneﬁts.

Original: The principles outlined here apply to any dense layers in deep learning models, though we only focus on certain weights in Transformer language models in our experiments as the motivating use case.
Revised: The principles outlined here apply to any dense layers in deep learning models, though we only focus on certain weights in Transformer language models in our experiments as the motivating use case.

Original: 4. L OW -R ANK -P ARAMETRIZED U PDATE M ATRICES
Revised: 4. L OW -R ANK -P ARAMETRIZED U PDATE M ATRICES

Original: A neural network contains many dense layers which perform matrix multiplication.
Revised: A neural network contains many dense layers which perform matrix multiplication.

Original: The weight matrices in these layers typically have full-rank.
Revised: The weight matrices in these layers typically have full ranks.

Original: When adapting to a speciﬁc task, Aghajanyan et al.
Revised: When adapting to a speciﬁc task, Aghajanyan et al.

Original: (2020) shows that the pre-trained language models have a low “instrisic dimension” and can still learn efﬁciently despite a random projection to a smaller subspace.
Revised: (2020) shows that the pre-trained language models have a low “instrisic dimension” and can still learn efﬁciently despite a random projection to a smaller subspace.

Original: Inspired by this, we hypothesize the updates to the weights also have a low “intrinsic rank” during adaptation.
Revised: Inspired by this, we hypothesize the updates to the weights also have a low “intrinsic rank” during adaptation.

Original: For a pre-trained weight matrix W 0 ∈ R d × k , we constrain its update by representing the latter with a low-rank decomposition W 0 + ∆ W = W 0 + BA , where B ∈ R d × r , A ∈ R r × k , and the rank r (cid:28) min( d, k ) .
Revised: For a pre-trained weight matrix W 0 ∈ R d × k , we constrain its update by representing the latter with a low-rank decomposition W 0 + ∆ W = W 0 + BA , where B ∈ R d × r , A ∈ R r × k , and the rank r (cid:28) min( d, k ) .

Original: During training, W 0 is frozen and does not receive gradient updates, while A and B contain trainable parameters.
Revised: During training, W 0 is frozen and does not receive gradient updates, while A and B contain trainable parameters.

Original: Note both W 0 and ∆ W = BA are multiplied with the same input, and their respective output vectors are summed coordinate-wise.
Revised: Note both W 0 and ∆ W = BA are multiplied with the same input, and their respective output vectors are summed coordinate-wise.

Original: For h = W 0 x , our modiﬁed forward pass yields:
Revised: For h = W 0 x , our modiﬁed forward pass yields:

Original: [Equation]
Revised: [Equation]

Original: We illustrate our reparametrization in Figure 1.
Revised: We illustrate our reparametrization in Figure 1.

Original: We use a random Gaussian initialization for A and zero for B , so ∆ W = BA is zero at the beginning of training.
Revised: We use a random Gaussian initialization for A and zero for B , so ∆ W = BA is zero at the beginning of training.

Revised: We then scale ∆ Wx by αr , where α is a constant in r .

Revised: When optimizing with Adam, tuning α is roughly the same as tuning the learning rate if we scale the initialization appropriately.

Revised: As a result, we simply set α to the ﬁrst r we try and do not tune it.

Revised: This scaling helps to reduce the need to retune hyperparameters when we vary r (Yang & Hu, 2021).

Original: A Generalization of Full Fine-tuning.
Revised: A Generalization of Full Fine-tuning.

Original: A more general form of ﬁne-tuning allows the training of a subset of the pre-trained parameters.
Revised: A more general form of ﬁne-tuning allows the training of a subset of the pre-trained parameters.

Original: LoRA takes a step further and does not require the accumulated gradient update to weight matrices to have full-rank during adaptation.
Revised: LoRA takes a step further and does not require the accumulated gradient update to weight matrices to have full ranks during adaptation.

Original: This means that when applying LoRA to all weight matrices and training all biases 2 , we roughly recover the expressiveness of full ﬁne-tuning by setting the LoRA rank r to the rank of the pre-trained weight matrices.
Revised: This means that when applying LoRA to all weight matrices and training all biases 2 , we roughly recover the expressiveness of full ﬁne-tuning by setting the LoRA rank r to the rank of the pre-trained weight matrices.

Original: In other words, as we increase the number of trainable parameters 3 , training LoRA roughly converges to training the original model, while adapter-based methods converges to an MLP and preﬁx-based methods to a model that cannot take long input sequences.
Revised: In other words, as we increase the number of trainable parameters 3 , training LoRA roughly converges to training the original model, while adapter-based methods converges to an MLP and preﬁx-based methods to a model that cannot take long input sequences.

Original: No Additional Inference Latency.
Revised: No Additional Inference Latency.

Original: When deployed in production, we can explicitly compute and store W = W 0 + BA and perform inference as usual.
Revised: When deployed in production, we can explicitly compute and store W = W 0 + BA and perform inference as usual.

Original: Note that both W 0 and BA are in R d × k .
Revised: Note that both W 0 and BA are in R d × k .

Original: When we need to switch to another downstream task, we can recover W 0 by subtracting BA and then adding a different B (cid:48)
Revised: When we need to switch to another downstream task, we can recover W 0 by subtracting BA and then adding a different B (cid:48)

Original: A (cid:48) , a quick operation with very little memory overhead.
Revised: A (cid:48) , a quick operation with very little memory overhead.

Original: Critically, this guarantees that we do not introduce any additional latency during inference compared to a ﬁne-tuned model by construction.
Revised: Critically, this guarantees that we do not introduce any additional latency during inference compared to a ﬁne-tuned model by construction.

Original: 4. A PPLYING L O RA TO T RANSFORMER
Revised: 4. A PPLYING L O RA TO T RANSFORMER

Original: In principle, we can apply LoRA to any subset of weight matrices in a neural network to reduce the number of trainable parameters.
Revised: In principle, we can apply LoRA to any subset of weight matrices in a neural network to reduce the number of trainable parameters.

Revised: In the Transformer architecture, there are four weight matrices in 2 They represent a negligible number of parameters compared to weights.

Original: In the Transformer architecture, there are four weight matrices in the self-attention module ( W q , W k , W v , W o ) and two in the MLP module.
Revised: the self-attention module ( W q , W k , W v , W o ) and two in the MLP module.

Original: We treat W q (or W k , W v ) as a single matrix of dimension d model × d model , even though the output dimension is usually sliced into attention heads.
Revised: We treat W q (or W k , W v ) as a single matrix of dimension d model × d model , even though the output dimension is usually sliced into attention heads.

Original: We limit our study to only adapting the attention weights for downstream tasks and freeze the MLP modules (so they are not trained in downstream tasks) both for simplicity and parameter-efﬁciency.
Revised: We limit our study to only adapting the attention weights for downstream tasks and freeze the MLP modules (so they are not trained in downstream tasks) both for simplicity and parameter-efﬁciency.

Original: We further study the effect on adapting different types of attention weight matrices in a Transformer in Section A.1.
Revised: We further study the effect on adapting different types of attention weight matrices in a Transformer in Section B.1.

Original: We leave the empirical investigation of adapting the MLP layers, LayerNorm layers, and biases to a future work.
Revised: We leave the empirical investigation of adapting the MLP layers, LayerNorm layers, and biases to a future work.

Original: Practical Beneﬁts and Limitations.
Revised: Practical Beneﬁts and Limitations.

Original: The most signiﬁcant beneﬁt comes from the reduction in memory and storage usage.
Revised: The most signiﬁcant beneﬁt comes from the reduction in memory and storage usage.

Original: For a large Transformer trained with Adam, we reduce that VRAM usage by up to 2 / 3 if r (cid:28) d model as we do not need to store the optimizer states for the frozen parameters.
Revised: For a large Transformer trained with Adam, we reduce that VRAM usage by up to 2 / 3 if r (cid:28) d model as we do not need to store the optimizer states for the frozen parameters.

Original: On GPT-3 175B, we reduce the VRAM consumption during training from 1.2TB to 350GB.
Revised: On GPT-3 175B, we reduce the VRAM consumption during training from 1.2TB to 350GB.

Original: With r = 4 and only the query and value projection matrices being adapted, the checkpoint size is reduced by roughly 10,000 × (from 350GB to 35MB) 4 .
Revised: With r = 4 and only the query and value projection matrices being adapted, the checkpoint size is reduced by roughly 10,000 × (from 350GB to 35MB) 4 .

Original: This allows us to train with signiﬁcantly fewer GPUs and avoid I/O bottlenecks.
Revised: This allows us to train with signiﬁcantly fewer GPUs and avoid I/O bottlenecks.

Original: Another beneﬁt is that we can switch between tasks while deployed at a much lower cost by only swapping the LoRA weights as opposed to all the parameters.
Revised: Another beneﬁt is that we can switch between tasks while deployed at a much lower cost by only swapping the LoRA weights as opposed to all the parameters.

Original: This allows for the creation of many customized models that can be swapped in and out on the ﬂy on machines that store the pre-trained weights in VRAM.
Revised: This allows for the creation of many customized models that can be swapped in and out on the ﬂy on machines that store the pre-trained weights in VRAM.

Original: We also observe a 25% speedup during training on GPT-3 175B compared to full ﬁne-tuning 5 as we do not need to calculate the gradient for the vast majority of the parameters.
Revised: We also observe a 25% speedup during training on GPT-3 175B compared to full ﬁne-tuning 5 as we do not need to calculate the gradient for the vast majority of the parameters.

Revised: See Appendix L for how to calculate gradient when using LoRA.

Original: LoRA also has its limitations.
Revised: LoRA also has its limitations.

Original: For example, it is not straightforward to batch inputs to different tasks with different A and B in a single forward pass, if one chooses to absorb A and B into W to eliminate additional inference latency.
Revised: For example, it is not straightforward to batch inputs to different tasks with different A and B in a single forward pass, if one chooses to absorb A and B into W to eliminate additional inference latency.

Original: Though it is possible to not merge the weights and dynamically choose the LoRA modules to use for samples in a batch for scenarios where latency is not critical.
Revised: Though it is possible to not merge the weights and dynamically choose the LoRA modules to use for samples in a batch for scenarios where latency is not critical.

Original:  E MPIRICAL E XPERIMENTS
Revised:  E MPIRICAL E XPERIMENTS

Original: We evaluate the downstream task performance of LoRA on RoBERTa (Liu et al., 2019), DeBERTa
Revised: We evaluate the downstream task performance of LoRA on RoBERTa (Liu et al., 2019), DeBERTa

Original: (He et al., 2021), and GPT-2 (Radford et al., b), before scaling up to GPT-3 175B (Brown et al., 2020).
Revised: (He et al., 2021), and GPT-2 (Radford et al., b), before scaling up to GPT-3 175B (Brown et al., 2020).

Original: Our experiments cover a wide range of tasks, from natural language understanding (NLU) to generation (NLG).
Revised: Our experiments cover a wide range of tasks, from natural language understanding (NLU) to generation (NLG).

Original: Speciﬁcally, we evaluate on the GLUE (Wang et al., 2019) benchmark for RoBERTa and DeBERTa.
Revised: Speciﬁcally, we evaluate on the GLUE (Wang et al., 2019) benchmark for RoBERTa and DeBERTa.

Original: We follow the setup of Li & Liang (2021) on GPT-2 for a direct comparison and add WikiSQL (Zhong et al., 2017)
Revised: We follow the setup of Li & Liang (2021) on GPT-2 for a direct comparison and add WikiSQL (Zhong et al., 2017)

Original: (NL to SQL queries) and SAMSum (Gliwa et al., 2019)
Revised: (NL to SQL queries) and SAMSum (Gliwa et al., 2019)

Original: (conversation summarization) for large-scale experiments on GPT-3.
Revised: (conversation summarization) for large-scale experiments on GPT-3.

Original: See Appendix D for more details on the datasets we use.
Revised: See Appendix D for more details on the datasets we use.

Original: We use NVIDIA Tesla V100 for all experiments.
Revised: We use NVIDIA Tesla V100 for all experiments.

Original: 5. B ASELINES
Revised: 5. B ASELINES

Original: To compare with other baselines broadly, we replicate the setups used by prior work and reuse their reported numbers whenever possible.
Revised: To compare with other baselines broadly, we replicate the setups used by prior work and reuse their reported numbers whenever possible.

Original: This, however, means that some baselines might only appear in certain experiments.
Revised: This, however, means that some baselines might only appear in certain experiments.

Revised: We point out that for most tasks, our goal is not to compete with

Revised: SOTA because 1) for tasks such as WikiSQL, SOTA uses specialized SQL-aware decoders, and; 2)

Revised: SOTA makes frequent use of techniques such as ensembling and adversarial training, which might confound our study on adaptation.

Revised: Thus, we focus on a scenario where we are given a powerful pre-trained autoregressive language model and use as few task-speciﬁc modiﬁcations as possible.

Revised: See Appendix E for how we count the number of trainable parameters for different baselines.

Original: Fine-Tuning (FT) is a common approach for adaptation.
Revised: Fine-Tuning (FT) is a common approach for adaptation.

Original: During ﬁne-tuning, the model is initialized to the pre-trained weights and biases, and all model parameters undergo gradient updates.
Revised: During ﬁne-tuning, the model is initialized to the pre-trained weights and biases, and all model parameters undergo gradient updates.

Original: A simple variant is to update only some layers while freezing others.
Revised: A simple variant is to update only some layers while freezing others.

Original: We include one such baseline reported in prior work (Li & Liang, 2021) on GPT-2, which adapts just the last two layers ( FT Top2 ).
Revised: We include one such baseline reported in prior work (Li & Liang, 2021) on GPT-2, which adapts just the last two layers ( FT Top2 ) [Table]

Original: Bias-only or BitFit is a baseline where we only train the bias vectors while freezing everything else.
Revised: Bias-only or BitFit is a baseline where we only train the bias vectors while freezing everything else.

Original: Contemporarily, this baseline has also been studied by BitFit (Zaken et al., 2021).
Revised: Contemporarily, this baseline has also been studied by BitFit (Zaken et al., 2021).

Original: Preﬁx-embedding tuning (PreEmbed) inserts special tokens among the input tokens.
Revised: Preﬁx-embedding tuning (PreEmbed) inserts special tokens among the input tokens.

Original: These special tokens have trainable word embeddings and are generally not in the model’s vocabulary.
Revised: These special tokens have trainable word embeddings and are generally not in the model’s vocabulary.

Original: Where to place such tokens can have an impact on performance.
Revised: Where to place such tokens can have an impact on performance.

Original: We focus on “preﬁxing”, which prepends such tokens to the prompt, and “inﬁxing”, which appends to the prompt; both are discussed in Li &
Revised: We focus on “preﬁxing”, which prepends such tokens to the prompt, and “inﬁxing”, which appends to the prompt; both are discussed in Li &

Original: Liang (2021).We use l p (resp.
Revised: Liang (2021).

Original: l i ) denote the number of preﬁx (resp.

Original: inﬁx) tokens. The number of trainable parameters is | Θ | = d model × ( l p + l i ) . Preﬁx-layer tuning (PreLayer) is an extension to preﬁx-embedding tuning.
Revised: Preﬁx-layer tuning (PreLayer) is an extension to preﬁx-embedding tuning.

Original: Instead of just learning the word embeddings (or equivalently, the activations after the embedding layer) for some special tokens, we learn the activations after every Transformer layer.
Revised: Instead of just learning the word embeddings (or equivalently, the activations after the embedding layer) for some special tokens, we learn the activations after every Transformer layer.

Original: The activations computed from previous layers are simply replaced by trainable ones.The resulting number of trainable parameters is | Θ | =
Revised: The activations computed from previous layers are simply replaced by trainable ones.

Original: L × d model × ( l p + l i ) , where L is the number of Transformer layers.

Original: [Table] Adapter tuning as proposed in Houlsby et al.
Revised: Adapter tuning as proposed in Houlsby et al.

Original: (2019) inserts adapter layers between the selfattention module (and the MLP module) and the subsequent residual connection.
Revised: (2019) inserts adapter layers between the selfattention module (and the MLP module) and the subsequent residual connection.

Original: There are two fully connected layers with biases in an adapter layer with a nonlinearity in between.
Revised: There are two fully connected layers with biases in an adapter layer with a nonlinearity in between.

Original: We call this original design Adapter H .
Revised: We call this original design Adapter H .

Original: Recently, Lin et al.
Revised: Recently, Lin et al.

Original: (2020) proposed a more efﬁcient design with the adapter layer applied only after the MLP module and after a LayerNorm.
Revised: (2020) proposed a more efﬁcient design with the adapter layer applied only after the MLP module and after a LayerNorm.

Original: We call it Adapter L .
Revised: We call it Adapter L .

Revised: This is very similar to another deign proposed in Pfeiffer et al.

Original:  We also include another baseline call AdapterDrop (R ¨ uckl ´ e et al., 2020) which drops some adapter layers for greater efﬁciency ( Adapter D ).
Revised: (2021), which we call Adapter P . We also include another baseline call AdapterDrop (R¨uckl´e et al., 2020) which drops some adapter layers for greater efﬁciency ( Adapter D ).

Original: We cite numbers from prior works whenever possible to maximize the number of baselines we compare with; they are in rows with an asterisk (*) in the ﬁrst column.In all cases, we have | Θ | = ˆ L Adpt × (2 × d model × r + r + d model ) + 2 × ˆ L LN ×
Revised: We cite numbers from prior works whenever possible to maximize the number of baselines we compare with; they are in rows with an asterisk (*) in the ﬁrst column.

Original: d model where ˆ L Adpt is the number of adapter layers and ˆ L LN the number of trainable LayerNorms (e.g., in

Original: Adapter L ). LoRA adds trainable pairs of rank decomposition matrices in parallel to existing weight matrices.
Revised: LoRA adds trainable pairs of rank decomposition matrices in parallel to existing weight matrices.

Original: As mentioned in Section 4.2, we only apply LoRA to W q and W v in most experiments for simplicity.The number of trainable parameters is determined by the rank r and the shape of the original weights: | Θ | = 2 × ˆ L LoRA ×
Revised: As mentioned in Section 4.2, we only apply LoRA to W q and W v in most experiments for simplicity.

Original: d model × r , where ˆ L LoRA is the number of weight matrices we apply LoRA to.

Original: 5. R O BERT A BASE / LARGE
Revised: 5. R O BERT A BASE / LARGE

Original: RoBERTa (Liu et al., 2019) optimized the pre-training recipe originally proposed in BERT (Devlin et al., 2019a) and boosted the latter’s task performance without introducing many more trainable parameters.
Revised: RoBERTa (Liu et al., 2019) optimized the pre-training recipe originally proposed in BERT (Devlin et al., 2019a) and boosted the latter’s task performance without introducing many more trainable parameters.

Original: While RoBERTa has been overtaken by much larger models on NLP leaderboards such as the GLUE benchmark (Wang et al., 2019) in recent years, it remains a competitive and popular pre-trained model for its size among practitioners.
Revised: While RoBERTa has been overtaken by much larger models on NLP leaderboards such as the GLUE benchmark (Wang et al., 2019) in recent years, it remains a competitive and popular pre-trained model for its size among practitioners.

Original: We take the pre-trained RoBERTa base (125M) and RoBERTa large (355M) from the HuggingFace
Revised: We take the pre-trained RoBERTa base (125M) and RoBERTa large (355M) from the HuggingFace

Original: Transformers library (Wolf et al., 2020)  and evaluate the performance of different efﬁcient adaptation approaches on tasks from the GLUE benchmark. 
Revised: Transformers library (Wolf et al., 2020) [Table] and evaluate the performance of different efﬁcient adaptation approaches on tasks from the GLUE benchmark. We also replicate Houlsby et al.

Revised: and Pfeiffer et al.

Revised: according to their setup.

Revised: To ensure a fair comparison, we make two crucial changes to how we evaluate LoRA when comparing with adapters.

Original: The result is presented in Table 2 (Top Two Sections).
Revised: First, we use the same batch size for all tasks and use a sequence length of 128 to match the adapter baselines. Second, we initialize the model to the pre-trained model for MRPC, RTE, and STS-B, not a model already adapted to MNLI like the ﬁne-tuning baseline. Runs following this more restricted setup from Houlsby et al.

Revised: (2019) are labeled with † .

Revised: The result is presented in Table 2 (Top Three Sections).

Original: See Section E.1 for details on the hyperparameters used.
Revised: See Section F.1 for details on the hyperparameters used.

Original: 5. D E BERT A XXL
Revised: 5. D E BERT A XXL

Original: DeBERTa (He et al., 2021) is a more recent variant of BERT that is trained on a much larger scale and performs very competitively on benchmarks such as GLUE (Wang et al., 2019) and SuperGLUE (Wang et al., 2020).
Revised: DeBERTa (He et al., 2021) is a more recent variant of BERT that is trained on a much larger scale and performs very competitively on benchmarks such as GLUE (Wang et al., 2019) and SuperGLUE (Wang et al., 2020).

Original: We evaluate if LoRA can still match the performance of a fully ﬁne-tuned DeBERTa XXL (1.5B) on GLUE.
Revised: We evaluate if LoRA can still match the performance of a fully ﬁne-tuned DeBERTa XXL (1.5B) on GLUE.

Original: The result is presented in Table 2 (Bottom Section).
Revised: The result is presented in Table 2 (Bottom Section).

Original: See Section E.2 for details on the hyperparameters used.
Revised: See Section F.2 for details on the hyperparameters used.

Original: [Table] 5. GPT-2 MEDIUM / LARGE
Revised: 5. GPT-2 MEDIUM / LARGE

Original: Having shown that LoRA can be a competitive alternative to full ﬁne-tuning on NLU [Table] we hope to answer if LoRA still prevails on NLG models, such as GPT-2 medium and large (Radford et al.
Revised: Having shown that LoRA can be a competitive alternative to full ﬁne-tuning on NLU, we hope to answer if LoRA still prevails on NLG models, such as GPT-2 medium and large (Radford et al.

Original: , b).
Revised: , b).

Original: We keep our setup as close as possible to Li & Liang (2021) for a direct comparison.
Revised: We keep our setup as close as possible to Li & Liang (2021) for a direct comparison.

Original: Due to space constraint, we only present our result on E2E NLG Challenge (Table 3) in this section.
Revised: Due to space constraint, we only present our result on E2E NLG Challenge (Table 3) in this section.

Original: See Section G.1 for results on WebNLG (Gardent et al., 2017) and DART (Nan et al., 2020).
Revised: See Section I.1 for results on WebNLG (Gardent et al., 2017) and DART (Nan et al., 2020).

Original: We include a list of the hyperparameters used in Section E.3. [Table]
Revised: We include a list of the hyperparameters used in Section F.3.

Original: 5. S CALING UP TO GPT-3 175B
Revised: 5. S CALING UP TO GPT-3 175B

Original: As a ﬁnal stress test for LoRA [Table] we scale up to GPT-3 with 175 billion parameters.
Revised: As a ﬁnal stress test for LoRA, we scale up to GPT-3 with 175 billion parameters.

Original: Due to the high training cost, we only report the typical standard deviation for a given task over random seeds, as opposed to providing one for every entry.
Revised: Due to the high training cost, we only report the typical standard deviation for a given task over random seeds, as opposed to providing one for every entry.

Original: See Section E.4 for details on the hyperparameters used.
Revised: See Section F.4 for details on the hyperparameters used.

Original: As shown in Table [Table] LoRA matches or exceeds the ﬁne-tuning baseline on all three datasets.
Revised: As shown in Table 4, LoRA matches or exceeds the ﬁne-tuning baseline on all three datasets.

Original: Note that not all methods beneﬁt monotonically from having more trainable parameters, as shown in Fig- ure 2.
Revised: Note that not all methods beneﬁt monotonically from having more trainable parameters, as shown in Figure 2.

Original: We observe a signiﬁcant performance drop when we use more than 256 special tokens for preﬁx-embedding tuning or more than 32 special tokens for preﬁx-layer tuning. 
Revised: We observe a signiﬁcant performance drop when we use more than 256 special tokens for preﬁx-embedding tuning or more than 32 special tokens for preﬁx-layer tuning. This corroborates [Table]

Original: This corroborates similar observations in Li & Liang (2021).
Revised: similar observations in Li & Liang (2021).

Original: While a thorough investigation into this phenomenon is out-of-scope for this work, we suspect that having more special tokens causes the input distribution to shift further away from the pre-training data distribution.
Revised: While a thorough investigation into this phenomenon is out-of-scope for this work, we suspect that having more special tokens causes the input distribution to shift further away from the pre-training data distribution.

Original: Separately, we investigate the performance of different adaptation approaches in the low-data regime in Section G.3.
Revised: Separately, we investigate the performance of different adaptation approaches in the low-data regime in Section I.3.

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Transformer Language Models.
Revised: Transformer Language Models.

Original: Transformer (Vaswani et al., 2017) is a sequence-to-sequence architecture that makes heavy use of self-attention.
Revised: Transformer (Vaswani et al., 2017) is a sequence-to-sequence architecture that makes heavy use of self-attention.

Original: Radford et al.
Revised: Radford et al.

Original: (a) applied it to autoregressive language modeling by using a stack of Transformer decoders.
Revised: (a) applied it to autoregressive language modeling by using a stack of Transformer decoders.

Original: Since then, Transformer-based language models have dominated NLP, achieving the state-of-the-art in many tasks.
Revised: Since then, Transformer-based language models have dominated NLP, achieving the state-of-the-art in many tasks.

Original: A new paradigm emerged with BERT (Devlin et al., 2019b) and GPT-2 (Radford et al., b) – both are large Transformer language models trained on a large amount of text – where ﬁne-tuning on task-speciﬁc data after pretraining on general domain data provides a signiﬁcant performance gain compared to training on task-speciﬁc data directly.
Revised: A new paradigm emerged with BERT (Devlin et al., 2019b) and GPT-2 (Radford et al., b) – both are large Transformer language models trained on a large amount of text – where ﬁne-tuning on task-speciﬁc data after pretraining on general domain data provides a signiﬁcant performance gain compared to training on task-speciﬁc data directly.

Original: Training larger
Revised: Training larger

Original: Transformers generally results in better performance and remains an active research direction.
Revised: Transformers generally results in better performance and remains an active research direction.

Original: GPT-3 (Brown et al., 2020) is the largest single Transformer language model trained to-date with 175B parameters.
Revised: GPT-3 (Brown et al., 2020) is the largest single Transformer language model trained to-date with 175B parameters.

Original: Prompt Engineering and Fine-Tuning.
Revised: Prompt Engineering and Fine-Tuning.

Original: While GPT-3 175B can adapt its behavior with just a few additional training examples, the result depends heavily on the input prompt (Brown et al., 2020).
Revised: While GPT-3 175B can adapt its behavior with just a few additional training examples, the result depends heavily on the input prompt (Brown et al., 2020).

Original: This necessitates an empirical art of composing and formatting the prompt to maximize a model’s performance on a desired task, which is known as prompt engineering or prompt hacking.
Revised: This necessitates an empirical art of composing and formatting the prompt to maximize a model’s performance on a desired task, which is known as prompt engineering or prompt hacking.

Original: Fine-tuning retrains a model pre-trained on general domains to a speciﬁc task Devlin et al.
Revised: Fine-tuning retrains a model pre-trained on general domains to a speciﬁc task Devlin et al.

Original: (2019b);
Revised: (2019b);

Original: Radford et al.
Revised: Radford et al.

Original: (a).
Revised: (a).

Original: Variants of it include learning just a subset of the parameters Devlin et al.
Revised: Variants of it include learning just a subset of the parameters Devlin et al.

Original: (2019b);
Revised: (2019b);

Original: Collobert & Weston (2008), yet practitioners often retrain all of them to maximize the downstream performance.
Revised: Collobert & Weston (2008), yet practitioners often retrain all of them to maximize the downstream performance.

Original: However, the enormity of GPT-3 175B makes it challenging to perform ﬁne-tuning in the usual way due to the large checkpoint it produces and the high hardware barrier to entry since it has the same memory footprint as pre-training.
Revised: However, the enormity of GPT-3 175B makes it challenging to perform ﬁne-tuning in the usual way due to the large checkpoint it produces and the high hardware barrier to entry since it has the same memory footprint as pre-training.

Original: Parameter-Efﬁcient Adaptation.
Revised: Parameter-Efﬁcient Adaptation.

Original: Many have proposed inserting adapter layers between existing layers in a neural network (Houlsby et al., 2019; Rebufﬁ et al., 2017; Lin et al., 2020).
Revised: Many have proposed inserting adapter layers between existing layers in a neural network (Houlsby et al., 2019; Rebufﬁ et al., 2017; Lin et al., 2020).

Original: Our method uses a similar bottleneck structure to impose a low-rank constraint on the weight updates.
Revised: Our method uses a similar bottleneck structure to impose a low-rank constraint on the weight updates.

Original: The key functional difference is that our learned weights can be merged with the main weights during inference, thus not introducing any latency, which is not the case for the adapter layers (Section 3).
Revised: The key functional difference is that our learned weights can be merged with the main weights during inference, thus not introducing any latency, which is not the case for the adapter layers (Section 3).

Original: A comtenporary extension of adapter is COMPACTER (Mahabadi et al., 2021), which essentially parametrizes the adapter layers using Kronecker products with some predetermined weight sharing scheme.
Revised: A comtenporary extension of adapter is COMPACTER (Mahabadi et al., 2021), which essentially parametrizes the adapter layers using Kronecker products with some predetermined weight sharing scheme.

Original: Similarly, combining LoRA with other tensor product-based methods could potentially improve its parameter efﬁciency, which we leave to future work.
Revised: Similarly, combining LoRA with other tensor product-based methods could potentially improve its parameter efﬁciency, which we leave to future work.

Original: More recently, many proposed optimizing the input word embeddings in lieu of ﬁne-tuning, akin to a continuous and differentiable generalization of prompt engineering (Li & Liang, 2021;
Revised: More recently, many proposed optimizing the input word embeddings in lieu of ﬁne-tuning, akin to a continuous and differentiable generalization of prompt engineering (Li & Liang, 2021;

Original: Lester et al., 2021;
Revised: Lester et al., 2021;

Original: Hambardzumyan et al., 2020; Liu et al., 2021).
Revised: Hambardzumyan et al., 2020; Liu et al., 2021).

Original: We include comparisons with Li & Liang (2021) in our experiment section.
Revised: We include comparisons with Li & Liang (2021) in our experiment section.

Original: However, this line of works can only scale up by using more special tokens in the prompt, which take up available sequence length for task tokens when positional embeddings are learned.
Revised: However, this line of works can only scale up by using more special tokens in the prompt, which take up available sequence length for task tokens when positional embeddings are learned.

Original: Low-Rank Structures in Deep Learning.
Revised: Low-Rank Structures in Deep Learning.

Original: Low-rank structure is very common in machine learning.
Revised: Low-rank structure is very common in machine learning.

Original: A lot of machine learning problems have certain intrinsic low-rank structure (Li et al., 2016;
Revised: A lot of machine learning problems have certain intrinsic low-rank structure (Li et al., 2016;

Original: Cai et al., 2010; Li et al., 2018b; Grasedyck et al., 2013).
Revised: Cai et al., 2010; Li et al., 2018b; Grasedyck et al., 2013).

Original: Moreover, it is known that for many deep learning tasks,especially those with a heavily over-parametrized neuralnetwork, the learned neural network will enjoy low-rank properties after training (Oymak et al., 2019).
Revised: Moreover, it is known that for many deep learning tasks, a heavily over-parametrized neural network will exhibit low-rank properties after training (Oymak et al., 2019).

Original: Some prior works even explicitly impose the low-rank constraint when training the original neural network (Sainath et al., 2013; Povey et al., 2018; Zhang et al., 2014; Jaderberg et al., 2014; Zhao et al., 2016; Khodak et al., 2021; Denil et al., 2014); however, to the best of our knowledge, none of these works considers low-rank update to a frozen model for adaptation to downstream tasks .
Revised: Some prior works even explicitly impose the low-rank constraint when training the original neural network (Sainath et al., 2013; Povey et al., 2018; Zhang et al., 2014; Jaderberg et al., 2014; Zhao et al., 2016; Khodak et al., 2021; Denil et al., 2014); however, to the best of our knowledge, none of these works considers low-rank update to a frozen model for adaptation to downstream tasks .

Original: In theory literature, it is known that neural networks outperform other classical learning methods, including the corresponding (ﬁnite-width) neural tangent kernels (Allen-Zhu et al., 2019; Li & Liang, 2018) when the underlying concept class has certain low-rank structure (Ghorbani et al., 2020; Allen-Zhu & Li, 2019; Allen-Zhu & Li, 2020a).
Revised: In theory literature, it is known that neural networks outperform other classical learning methods, including the corresponding (ﬁnite-width) neural tangent kernels (Allen-Zhu et al., 2019; Li & Liang, 2018) when the underlying concept class has certain low-rank structure (Ghorbani et al., 2020; Allen-Zhu & Li, 2019; Allen-Zhu & Li, 2020a).

Original: Another theoretical result in Allen-Zhu & Li (2020b) suggests that low-rank adaptations can be useful for adversarial training.
Revised: Another theory result in Allen-Zhu & Li (2020b) suggests that low-rank adaptations can be useful for adversarial training.

Original: In sum, we believe that our proposed low-rank adaptation update is well-motivated by the literature.
Revised: In sum, we believe that our proposed low-rank adaptation update is well-motivated by the literature.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: Fine-tuning enormous language models is prohibitively expensive in terms of the hardware required and the storage/switching cost for hosting independent instances for different tasks.
Revised: Fine-tuning enormous language models is prohibitively expensive in terms of the hardware required and the storage/switching cost for hosting independent instances for different tasks.

Original: We propose
Revised: We propose

Original: LoRA, an efﬁcient adaptation strategy that neither introduces inference latency nor reduces input sequence length while retaining high model quality.
Revised: LoRA, an efﬁcient adaptation strategy that neither introduces inference latency nor reduces input sequence length while retaining high model quality.

Original: Importantly, it allows for quick task-switching when deployed as a service by sharing the vast majority of the model parameters.
Revised: Importantly, it allows for quick task-switching when deployed as a service by sharing the vast majority of the model parameters.

Original: While we focused on Transformer language models, the proposed principles are generally applicable to any neural networks with dense layers.
Revised: While we focused on Transformer language models, the proposed principles are generally applicable to any neural networks with dense layers.

Original: There are many directions for future works.
Revised: There are many directions for future works.

Original: 1) LoRA can be combined with other efﬁcient adaptation methods, potentially providing orthogonal improvement.
Revised: 1) LoRA can be combined with other efﬁcient adaptation methods, potentially providing orthogonal improvement.

Original: The mechanism behind ﬁne-tuning or LoRA is far from clear – how are features learned during pre-training transformed to do well on downstream tasks?
Revised: The mechanism behind ﬁne-tuning or LoRA is far from clear – how are features learned during pre-training transformed to do well on downstream tasks?

Original: We believe that LoRA makes it more tractable to answer this than full ﬁnetuning.
Revised: We believe that LoRA makes it more tractable to answer this than full ﬁnetuning.

Original: We mostly depend on heuristics to select the weight matrices to apply LoRA to.
Revised: We mostly depend on heuristics to select the weight matrices to apply LoRA to.

Original: Are there more principled ways to do it?
Revised: Are there more principled ways to do it?

Original: Finally, the rank-deﬁciency of ∆ W suggests that W could be rank-deﬁcient as well, which can also be a source of inspiration for future works.A U NDERSTANDING THE L OW -R ANK U PDATES
Revised: Finally, the rank-deﬁciency of ∆ W suggests that W could be rank-deﬁcient as well, which can inspire future works.

Original: Given the empirical advantage of LoRA we hope to further explain the properties of the low-rank adaptation learned from downstream tasks.

Original: Note that the low-rank structure not only lowers the hardware barrier to entry which allows us to run multiple experiments in parallel, but also gives better interpretability of how the update weights are correlated with the pre-trained weights.

Original: We focus our study on GPT-3 175B, where we achieved the largest reduction of trainable parameters (up to 10,000 × ) without adversely affecting task performances.

Original: We perform a sequence of empirical studies to answer the following questions: 1) Given a parameter budget constraint, which subset of weight matrices in a pre-trained Transformer should we adapt to maximize downstream performance?

Original: Is the “optimal” adaptation matrix ∆ W really rankdeﬁcient ?

Original: If so, what is a good rank to use in practice?

Original: What is the connection between ∆ W and

Original: W ?

Original: Does ∆ W highly correlate with W ?

Original: How large is ∆ W comparing to W ?

Original: We believe that our answers to question (2) and (3) shed light on the fundamental principles of using pre-trained language models for downstream tasks, which is a critical topic in NLP.

Original: A. W HICH W EIGHT M ATRICES IN T RANSFORMER S HOULD W E A PPLY L O RA TO ?

Original: Given a limited parameter budget, which types of weights should we adapt with LoRA to obtain the best performance on downstream tasks?

Original: As mentioned in Section 4.2, we only consider weight matrices in the self-attention module.

Original: We set a parameter budget of 18M (roughly 35MB if stored in FP16) on GPT-3 175B, which corresponds to r = 8 if we adapt one type of attention weights or r = 4 if we adapt two types, for all 96 layers.

Original: The result is presented in Table 5.

Original: Note that putting all the parameters in ∆ W q or ∆ W k results in signiﬁcantly lower performance, while adapting both W q and W v yields the best result.

Original: This suggests that even a rank of four captures enough information in ∆ W such that it is preferable to adapt more weight matrices than adapting a single type of weights with a larger rank.

Original: A. W HAT IS THE O PTIMAL R ANK r FOR L O RA?

Original: We turn our attention to the effect of rank r on model performance.

Original: We adapt { W q , W v } , { W q , W k , W v , W c } , and just W q for a comparison.

Original: [Table]

Original: [Table]

Original: on GPT-2 in Section I.2.

Original: Table 6 shows that, surprisingly, LoRA already performs competitively with a very small r (more so for { W q , W v } than just W q ).

Original: This suggests the update matrix ∆ W could have a very small “intrinsic rank”.

Original: 6 To further support this ﬁnding, we check the overlap of the subspaces learned by different choices of r and by different random seeds.

Original: We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufﬁcient.

Original: Subspace similarity between different r .

Original: Given A r =8 and A r =64 which are the learned adaptation matrices with rank r = 8 and 64 using the same pre-trained model , we perform singular value decomposition and obtain the right-singular unitary matrices

Original: [-U A r =
-]
Revised: [-U A r =
-]

Original: and U A r =64 .

Original: 7 We hope to answer: how much of the subspace spanned by the top i singular vectors in U A r =8 (for 1 ≤ i ≤ 8 ) is contained in the subspace spanned by top j singular vectors of U A r =64 (for 1 ≤ j ≤ 64 )?

Original: We measure this quantity with a normalized subspace similarity based on the Grassmann distance (See Appendix H for a more formal discussion)

Original: [Equation]

Original: where U iA r =8 represents the columns of U A r =8 corresponding to the top- i singular vectors.

Original: φ ( · ) has a range of [0 , 1] , where 1 represents a complete overlap of subspaces and 0 a complete separation.

Original: See Figure 3 for how φ changes as we vary i and j .

Original: We only look at the 48th layer (out of 96) due to space constraint, but the conclusion holds for other layers as well, as shown in Section I.1.

Original: We make an important observation from Figure 3.

Original: 16121823293540465258 j 1357i

Original: W q 16121823293540465258 j

Original: W v 1 2 3 4 5 6 7 8 j

Original: W q 1 2 3 4 5 6 7 8 j

Original: W v ( A r =64 , A r =8 , i , j )

Original: Directions corresponding to the top singular vector overlap signiﬁcantly between

Original: A r =8 and A r =64 , while others do not.

Original: Speciﬁcally, ∆ W v (resp.

Original: ∆ W q ) of A r = and ∆ W v (resp.

Original: ∆ W q ) of A r =64 share a subspace of dimension 1 with normalized similarity > 0 .

Original: 5 , providing an explanation of why r = 1 performs quite well in our downstream tasks for GPT-3.

Original: Since both A r =8 and A r =64 are learned using the same pre-trained model, Figure 3 indicates that the top singular-vector directions of A r =8 and A r =64 are the most useful, while other directions potentially contain mostly random noises accumulated during training.

Original: Hence, the adaptation matrix can indeed have a very low rank.

Original: j        i

Original: W q

Original: [Figure]

Original: [Figure]

Original: Subspace similarity between different random seeds.

Original: We further conﬁrm this by plotting the normalized subspace similarity between two randomly seeded runs with r = 64 , shown in Figure 4.

Original: ∆ W q appears to have a higher “intrinsic rank” than ∆ W v , since more common singular value directions are learned by both runs for ∆ W q , which is in line with our empirical observation in Table 6.

Original: As a comparison, we also plot two random Gaussian matrices, which do not share any common singular value directions with each other.

Original: A. H OW D OES THE A DAPTATION M ATRIX ∆ W C OMPARE TO W ?

Original: We further investigate the relationship between ∆ W and W .

Original: In particular, does ∆ W highly correlate with W ?

Original: (Or mathematically, is ∆ W mostly contained in the top singular directions of W ?)

Original: Also, how “large” is ∆ W comparing to its corresponding directions in W ?

Original: This can shed light on the underlying mechanism for adapting pre-trained language models.

Original: To answer these questions, we project W onto the r -dimensional subspace of ∆ W by computing U (cid:62) WV (cid:62) , with U / V being the left/right singular-vector matrix of ∆ W .

Original: Then, we compare the Frobenius norm between (cid:107) U (cid:62) WV (cid:62) (cid:107) F and (cid:107) W (cid:107) F .

Original: As a comparison, we also compute (cid:107) U (cid:62) WV (cid:62) (cid:107) F by replacing U, V with the top r singular vectors of W or a random matrix.

Original: [Equation]

Original: [Equation]

Original: We draw several conclusions from Table 7.

Original: First, ∆ W has a stronger correlation with W compared to a random matrix, indicating that ∆ W ampliﬁes some features that are already in W .

Original: Second, instead of repeating the top singular directions of W , ∆ W only ampliﬁes directions that are not emphasized in W .

Original: Third, the ampliﬁcation factor is rather huge: 21 .

Original: 5 ≈ 6 . 91 / 0 .

Original: 32 for r = 4 .

Original: See Section I.4 for why r = 64 has a smaller ampliﬁcation factor.

Original: We also provide a visualization in Section I.3 for how the correlation changes as we include more top singular directions from W q .

Original: This suggests that the low-rank adaptation matrix potentially ampliﬁes the important features for speciﬁc downstream tasks that were learned but not emphasized in the general pre-training model .

Original: B L ARGE L ANGUAGE M ODELS S TILL N EED P ARAMETER U PDATES

Original: Few-shot learning, or prompt engineering, is very advantageous when we only have a handful of training samples.

Original: However, in practice, we can often afford to curate a few thousand or more training examples for performance-sensitive applications.

Original: As shown in Table 8, ﬁne-tuning improves the model performance drastically compared to few-shot learning on datasets large and small.

Original: We take the GPT-3 few-shot result on RTE from the GPT-3 paper (Brown et al., 2020).

Original: For MNLI-matched, we use two demonstrations per class and six in-context examples in total.

Original: [Table]

Original: C I NFERENCE L ATENCY I NTRODUCED BY A DAPTER L AYERS

Original: Adapter layers are external modules added to a pre-trained model in a sequential manner, whereas our proposal, LoRA, can be seen as external modules added in a parallel manner.

Original: Consequently, adapter layers must be computed in addition to the base model, inevitably introducing additional latency.

Original: While as pointed out in R¨uckl´e et al.

Original: (2020), the latency introduced by adapter layers can be mitigated when the model batch size and/or sequence length is large enough to full utilize the hardware parallelism.

Original: We conﬁrm their observation with a similar latency study on GPT-2 medium and point out that there are scenarios, notably online inference where the batch size is small, where the added latency can be signiﬁcant.

Original: We measure the latency of a single forward pass on an NVIDIA Quadro RTX8000 by averaging over 100 trials.

Original: We vary the input batch size, sequence length, and the adapter bottleneck dimension r .

Original: We test two adapter designs: the original one by Houlsby et al.

Original: (2019), which we call Adapter H , and a recent, more efﬁcient variant by Lin et al.

Original: (2020), which we call Adapter L .

Original: See Section 5.

Original: for more details on the designs.

Original: We plot the slow-down in percentage compared to the no-adapter baseline in Figure 5.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: r

Original: Seq Len =

Original: Seq Len =

Original: Seq Len =

Original: [Figure]

Original: [Figure]

Original: a p t e r L r

Original: [Figure]

Original: D D ATASET D ETAILS

Original: GLUE Benchmark is a wide-ranging collection of natural language understanding tasks.

Original: It includes

Original: MNLI (inference, Williams et al.

Original: (2018)), SST-2 (sentiment analysis, Socher et al.

Original: , MRPC (paraphrase detection, Dolan & Brockett (2005)), CoLA (linguistic acceptability, Warstadt et al.

Original: (2018)), QNLI (inference, Rajpurkar et al.

Original: , QQP 8 (question-answering), RTE (inference), and STS-B (textual similarity, Cer et al.

Original: The broad coverage makes GLUE benchmark a standard metric to evaluate NLU models such as RoBERTa and DeBERTa.

Original: The individual datasets are released under different permissive licenses.

Original: WikiSQL is introduced in Zhong et al.

Original: and contains 56 , 355 / 8 , 421 training/validation examples.

Original: The task is to generate SQL queries from natural language questions and table schemata.

Original: We encode context as x = { table schema , query } and target as y = { SQL } .

Original: The dataset is release under the BSD 3-Clause License.

Original: SAMSum is introduced in Gliwa et al.

Original: and contains 14 , 732 / 819 training/test examples.

Original: It consists of staged chat conversations between two people and corresponding abstractive summaries written by linguists.

Original: We encode context as ” \ n” concatenated utterances followed by a ” \ n \ n”, and target as y = { summary } .

Original: The dataset is released under the non-commercial licence:

Original: Creative

Original: Commons BY-NC-ND 4.0.

Original: E2E NLG Challenge was ﬁrst introduced in Novikova et al.

Original: (2017) as a dataset for training end-toend, data-driven natural language generation systems and is commonly used for data-to-text evaluation.

Original: The E2E dataset consists of roughly 42 , 000 training, 4 , 600 validation, and 4 , 600 test examples from the restaurant domain.

Original: Each source table used as input can have multiple references.

Original: Each sample input ( x, y ) consists of a sequence of slot-value pairs, along with a corresponding natural language reference text.

Original: The dataset is released under Creative Commons BY-NC-SA 4.0.

Original: DART is an open-domain data-to-text dataset described in Nan et al.

Original: DART inputs are structured as sequences of ENTITY — RELATION — ENTITY triples.

Original: With 82 K examples in total, DART is a signiﬁcantly larger and more complex data-to-text task compared to E2E. The dataset is released under the MIT license.

Original: WebNLG is another commonly used dataset for data-to-text evaluation (Gardent et al., 2017).

Original: With 22 K examples in total WebNLG comprises 14 distinct categories, nine of which are seen during training.

Original: Since ﬁve of the 14 total categories are not seen during training, but are represented in the test set, evaluation is typically broken out by “seen” categories (S), “unseen” categories (U) and “all” (A).

Original: Each input example is represented by a sequence of SUBJECT — PROPERTY —

Original: OBJECT triples.

Original: The dataset is released under Creative Commons BY-NC-SA 4.0.

Original: E H YPERPARAMETERS U SED IN E XPERIMENTS E. R O BERT A

Original: We train using AdamW with a linear learning rate decay schedule.

Original: We sweep learning rate, number of training epochs

Original: [Table]

Original: and batch size for LoRA.

Original: Following Liu et al.

Original: (2019), we initialize the LoRA modules to our best MNLI checkpoint when adapting to MRPC, RTE, and STS-B, instead of the usual initialization; the pre-trained model stays frozen for all tasks.

Original: We report the median overrandom seeds; the result for each run is taken from the best epoch.

Original: See the hyperparameters used in our runs in Table

Original: [Table]

Original: E. D E BERT A

Original: We again train using AdamW with a linear learning rate decay schedule.

Original: Following He et al.

Original: , we tune learning rate, dropout probability, warm-up steps, and batch size.

Original: We use the same model sequence length used by (He et al., 2021) to keep our comparison fair.

Original: Following He et al.

Original: (2021), we initialize the LoRA modules to our best MNLI checkpoint when adapting to MRPC, RTE, and

Original: STS-B, instead of the usual initialization; the pre-trained model stays frozen for all tasks.

Original: We report the median over 5 random seeds; the result for each run is taken from the best epoch.

Original: See the hyperparameters used in our runs in Table 10.

Original: E. GPT-

Original: We train all of our GPT-2 models using AdamW (Loshchilov & Hutter, 2017) with a linear learning rate schedule for 5 epochs.

Original: We use the batch size, learning rate, and beam search beam size described in Li & Liang (2021).

Original: Accordingly, we also tune the above hyperparameters for LoRA.

Original: We report the mean over 3 random seeds; the result for each run is taken from the best epoch.

Original: The hyperparameters used for LoRA in GPT-2 are listed in Table 11.

Original: For those used for other baselines, see Li & Liang (2021).

Original: [Table]

Original: [Table]

Original: For all GPT-3 experiments

Original: [Table]

Original: we train using AdamW (Loshchilov & Hutter, 2017) for 2 epochs with a batch size of 128 samples and a weight decay factor of 0.1.

Original: We use a sequence length of 384 for

Original: WikiSQL (Zhong et al., 2017), 768 for MNLI (Williams et al., 2018), and 2048 for SAMSum (Gliwa et al., 2019).

Original: We tune learning rate for all method-dataset combinations.

Original: See Section E.4 for more details on the hyperparameters used.

Original: For preﬁx-embedding tuning, we ﬁnd the optimal l p and l i to be 256 and 8, respectively, totalling 3 .

Original: 2 M trainable parameters.

Original: We use l p = 8 and l i = 8 for preﬁx-layer tuning with 20 .

Original: 2 M trainable parameters to obtain the overall best performance.

Original: We present two parameter budgets for LoRA: 4.7M ( r q = r v = 1 or r v = 2 ) and 37.7M ( r q = r v =or r q = r k = r v = r o = 2 ).

Original: We report the best validation performance from each run.

Original: The training hyperparameters used in our GPT-3 experiments are listed in Table 12.

Original: F C OMBINING L O RA WITH P REFIX T UNING

Original: LoRA can be naturally combined with existing preﬁx-based approaches.

Original: In this section, we evaluate two combinations of LoRA and variants of preﬁx-tuning on WikiSQL and MNLI.

Original: [Table]

Original: LoRA+PreﬁxEmbed (LoRA+PE) combines LoRA with preﬁx-embedding tuning, where we insert l p + l i special tokens whose embeddings are treated as trainable parameters.

Original: For more on preﬁxembedding tuning, see Section 5.1.

Original: LoRA+PreﬁxLayer (LoRA+PL) combines LoRA with preﬁx-layer tuning.

Original: We also insert l p + l i special tokens; however, instead of letting the hidden representations of these tokens evolve naturally, we replace them after every Transformer block with an input agnostic vector.

Original: Thus, both the embeddings and subsequent Transformer block activations are treated as trainable parameters.

Original: For more on preﬁx-layer tuning, see Section 5.1.

Original: In Table 15, we show the evaluation results of LoRA+PE and LoRA+PL on WikiSQL and MultiNLI.

Original: First of all, LoRA+PE signiﬁcantly outperforms both LoRA and preﬁx-embedding tuning on

Original: WikiSQL, which indicates that LoRA is somewhat orthogonal to preﬁx-embedding tuning.

Original: On

Original: MultiNLI, the combination of LoRA+PE doesn’t perform better than LoRA, possibly because LoRA on its own already achieves performance comparable to the human baseline.

Original: Secondly, we notice that LoRA+PL performs slightly worse than LoRA even with more trainable parameters.

Original: We attribute this to the fact that preﬁx-layer tuning is very sensitive to the choice of learning rate and thus makes the optimization of LoRA weights more difﬁcult in LoRA+PL.

Original: G A DDITIONAL E MPIRICAL E XPERIMENTS G. A DDITIONAL E XPERIMENTS ON GPT-

Original: We also repeat our experiment on DART (Nan et al., 2020) and WebNLG (Gardent et al., 2017) following the setup of Li & Liang (2021).

Original: The result is shown in Table 13.

Original: Similar to our result on E2E NLG Challenge, reported in Section 5, LoRA performs better than or at least on-par with preﬁx-based approaches given the same number of trainable parameters.

Original: G. A DDITIONAL E XPERIMENTS ON GPT-

Original: We present additional runs on GPT-3 with different adaptation methods in Table 15.

Original: The focus is on identifying the trade-off between performance and the number of trainable parameters.

Original: G. L OW -D ATA R EGIME

Original: To evaluate the performance of different adaptation approaches in the low-data regime.

Original: we randomly sample 100, 1k and 10k training examples from the full training set of MNLI to form the low-data

Original: MNLI- n tasks.

Original: In Table 16, we show the performance of different adaptation approaches on MNLIn .

Original: To our surprise, PreﬁxEmbed and PreﬁxLayer performs very poorly on MNLI-100 dataset, with

Original: PreﬁxEmbed performing only slightly better than random chance (37.6% vs. 33.3%).

Original: PreﬁxLayer performs better than PreﬁxEmbed but is still signiﬁcantly worse than Fine-Tune or LoRA on MNLI100.

Original: The gap between preﬁx-based approaches and LoRA/Fine-tuning becomes smaller as we increase the number of training examples, which might suggest that preﬁx-based approaches are not suitable for low-data tasks in GPT-3.

Original: LoRA achieves better performance than ﬁne-tuning on both

Original: MNLI-100 and MNLI-Full, and comparable results on MNLI-1k and MNLI-10K considering the ( ± 0 . 3 ) variance due to random seeds.

Original: [Table]

Original: [Table]

Original: The training hyperparameters of different adaptation approaches on MNLI-n are reported in Table 17.

Original: We use a smaller learning rate for PreﬁxLayer on the MNLI-100 set, as the training loss does not decrease with a larger learning rate.

Original: H M EASURING S IMILARITY B ETWEEN S UBSPACES

Original: In this paper we use the measure φ ( A, B, i, j )

Original: =

Original: ψ ( U iA , U jB )

Original: = (cid:107)

Original: U

Original: i (cid:62) A U B (cid:107) 2 F min { i,j } to measure the subspace similarity between two column orthonormal matrices U iA ∈ R

Original: d

Original: × i and U jB ∈ R

Original: d × j , obtained by taking columns of the left singular matrices of A and B .

Original: We point out that this similarity is simply a reverse of the standard Projection Metric that measures distance between subspaces Ham & Lee (2008).

Original: [Table]

Original: 175B

Original: [Table]

Original: [Table]

Original: M

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: M

Original: [Table]

Original: [Table]

Original: [Table]

Original: M

Original: [Table]

Original: [Table]

Original: [Table]

Original: M 63.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: M

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: M

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: M

Original: [Table]

Original: [Table]

Original: Adapter

Original: [Table]

Original: [Table]

Original: M

Original: [Table]

Original: [Table]

Original: [Table]

Original: 2 M 73.

Original: [Table]

Original: [Table]

Original: M 73.

Original: r =77.9 M 73.

Original: r =304.4 M 72.

Original: LoRA r v =4.7 M 73.

Original: r q = r v =4.7 M 73.

Original: r q = r v =9.4 M 73.

Original: r q = r k = r v = r o =9.4 M 74.

Original: r q = r v =18.8 M 73.

Original: r q = r k = r v = r o =18.8 M 73.

Original: r q = r v =37.7 M 73.

Original: r q = r k = r v = r o =37.7 M 74.

Original: r q = r v =301.9 M 73.

Original: r q = r k = r v = r o =603.8 M 73.

Original: LoRA+PE r q = r v = 8 , l p = 8 , l i =37.8 M 75.

Original: r q = r v = 32 , l p = 8 , l i =151.1 M 75.

Original: r q = r v = 64 , l p = 8 , l i =302.1 M 76.

Original: LoRA+PL r q = r v = 8 , l p = 8 , l i =52.8 M 72.

Original: [Table]

Original: performs exhibits favorable sample-efﬁciency compared to other methods, including ﬁne-tuning.

Original: To be concrete, let the singular values of U i (cid:62) A U j B to be σ 1 , σ 2 , · · · , σ p where p =

Original: min { i, j } .

Original: We know that the Projection Metric Ham & Lee (2008) is deﬁned as:

Original: [Equation]

Original: [Table]

Original: where our similarity is deﬁned as:

Original: [Equation]

Original: This similarity satisﬁes that if U iA and U jB share the same column span, then φ ( A, B, i, j )

Original: = 1 .

Original: If they are completely orthogonal, then φ ( A, B, i, j ) = 0 .

Original: Otherwise, φ ( A, B, i, j ) ∈ (0 , 1) .

Original: I A DDITIONAL E XPERIMENTS ON L OW -R ANK M ATRICES

Original: We present additional results from our investigation into the low-rank update matrices.

Original: I.

Original: C ORRELATION BETWEEN L O RA M ODULES

Original: See Figure 6 and Figure 7 for how the results presented in Figure 3 and Figure 4 generalize to other layers.

Original: I.

Original: E FFECT OF

Original: r ON GPT-

Original: We repeat our experiment on the effect of r (Section A.2) in GPT-2.

Original: Using the E2E NLG Challenge dataset as an example, we report the validation loss and test metrics achieved by different choices of r after training for 26,000 steps.

Original: We present our result in Table 18.

Original: The optimal rank for GPTMedium is between 4 and 16 depending on the metric used, which is similar to that for GPT-3 175B.

Original: Note that the relationship between model size and the optimal rank for adaptation is still an open question.

Original: I. C ORRELATION BETWEEN W AND ∆ W

Original: See Figure 8 for the normalized subspace similarity between W and ∆ W with varying r .

Original: Note again that ∆ W does not contain the top singular directions of W , since the similarity between the top 4 directions in ∆ W and the top-10% of those in W barely exceeds 0.2.

Original: This gives evidence that ∆ W contains those “task-speciﬁc” directions that are otherwise not emphasized in W .

Original: An interesting next question to answer, is how “strong” do we need to amplify those task-speciﬁc directions, in order for the model adaptation to work well?

Original: L a y e r 1 i

Original: W q

Original: W v

Original: W q

Original: [-W v 
-]
Revised: [-W v 
-]

Original: [-L a y e r 32 i 
-]
Revised: [-L a y e r 32 i 
-]

Original: [-L a y e r 64 i 16121823293540465258 j 
-]
Revised: [-L a y e r 64 i 16121823293540465258 j 
-]

Original: [-L a y e r 
-]
Revised: [-L a y e r 
-]

Original: i

Original: [Figure]

Original: ( A r =8 , A r =64 , i , j )

Original: I. A MPLIFICATION F ACTOR

Original: One can naturally consider a feature ampliﬁcation factor as the ratio (cid:107) ∆ W (cid:107) F (cid:107) U (cid:62) WV (cid:62) (cid:107) F , where U and V are the left- and right-singular matrices of the SVD decomposition of ∆ W .

Original: (Recall UU (cid:62) WV (cid:62) V gives the “projection” of W onto the subspace spanned by ∆ W .)

Original: Intuitively, when ∆ W mostly contains task-speciﬁc directions, this quantity measures how much of them are ampliﬁed by ∆ W .

Original: As shown in Section A.3, for r = 4 , this ampliﬁcation factor is as large as 20.

Original: In other words, there are (generally speaking) four feature directions in each layer (out of the entire feature space from the pre-trained model W ), that need to be ampliﬁed by a very large factor 20, in order to achieve our reported accuracy for the downstream speciﬁc task.

Original: And, one should expect a very different set of feature directions to be ampliﬁed for each different downstream task.

Original: One may notice, however, for r = 64 , this ampliﬁcation factor is only around 2, meaning that most directions learned in ∆ W with r = 64 are not being ampliﬁed by much.

Original: This should not be surprising, and in fact gives evidence (once again) that the intrinsic rank needed to represent the “task-speciﬁc directions” (thus for model adaptation) is low.

Original: In contrast, those directions in the rank-4 version of ∆ W (corresponding to r = 4 ) are ampliﬁed by a much larger factor 20.

Original: 17131925313743495561 L a y e r 1 i

Original: W q

Original: W v

Original: L a y e r

Original: W q

Original: W v 161116212631364146515661 j

Original: [-L a y e r 64 i 
-]
Revised: [-L a y e r 64 i 
-]

Original: j 161116212631364146515661 j

Original: L a y e r 161116212631364146515661 j ( A r =64 , A 0 r =64 , i , j )

Original: Rank r val loss

Original: BLEU

Original: NIST

Original: METEOR

Original: ROUGE L

Original: CIDEr 1.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]