LoRA: Low-Rank Adaptation of Large Language Models  Compared to V1, this draft includes better baselines, experiments on GLUE, and more on adapter latency. Abstract  An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA. Introduction  Many applications in natural language processing rely on adapting one large-scale, pre-trained language model to multiple downstream applications. Such adaptation is usually done via fine-tuning, which updates all the parameters of the pre-trained model. The major downside of fine-tuning is that the new model contains as many parameters as in the original model. As larger models are trained every few months, this changes from a mere "inconvenience" for GPT-2 or RoBERTa large to a critical deployment challenge for GPT-3 with 175 billion trainable parameters. While GPT-3 175B achieves non-trivial performance with few-shot learning, fine-tuning boosts its performance significantly. Many sought to mitigate this by adapting only some parameters or learning external modules for new tasks. This way, we only need to store and load a small number of task-specific parameters in addition to the pre-trained model for each task, greatly boosting the operational efficiency when deployed. However, existing techniques often introduce inference latency by extending model depth or reduce the model's usable sequence length. More importantly, these method often fail to match the fine-tuning baselines, posing a trade-off between efficiency and model quality. We take inspiration from Li et al. and Aghajanyan et al. which show that the learned over-parametrized models in fact reside on a low intrinsic dimension. We hypothesize that the change in weights during model adaptation also has a low "intrinsic rank", leading to our proposed Low-Rank Adaptation (LoRA) approach. LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers' change during adaptation instead, while keeping the pre-trained weights frozen. Using GPT-3 175B as an example, we show that a very low rank (i.e., r can be one or two) suffices even when the full rank (i.e., d) is as high as 12,288, making LoRA both storage- and compute-efficient. LoRA possesses several key advantages. A pre-trained model can be shared and used to build many small LoRA modules for different tasks. We can freeze the shared model and efficiently switch tasks by replacing the matrices A and B, reducing the storage requirement and task-switching overhead significantly. LoRA makes training more efficient and lowers the hardware barrier to entry by up to 3 times when using adaptive optimizers since we do not need to calculate the gradients or maintain the optimizer states for most parameters. Instead, we only optimize the injected, much smaller low-rank matrices. Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing no inference latency compared to a fully fine-tuned model, by construction. LoRA is orthogonal to many prior methods and can be combined with many of them, such as prefix-tuning. Terminologies and Conventions. We make frequent references to the Transformer architecture and use the conventional terminologies for its dimensions. We call the input and output dimension size of a Transformer layer d model. We use Wq, Wk, Wv, and Wo to refer to the query/key/value/output projection matrices in the self-attention module. W or W0 refers to a pre-trained weight matrix and delta W its accumulated gradient update during adaptation. We use r to denote the rank of a LoRA module. We follow the conventions set out by Vaswani et al. and Brown et al. and use Adam for model optimization and use a Transformer MLP feedforward dimension d ffn equals 4 times d model. Problem Statement  While our proposal is agnostic to training objective, we focus on language modeling as our motivating use case. Below is a brief description of the language modeling problem and, in particular, the maximization of conditional probabilities given a task-specific prompt. Suppose we are given a pre-trained autoregressive language model P Phi of y given x parametrized by Phi. For instance, P Phi of y given x can be a generic multi-task learner such as GPT based on the Transformer architecture. Consider adapting this pre-trained model to downstream conditional text generation tasks, such as summarization, machine reading comprehension (MRC), and natural language to SQL (NL2SQL). Each downstream task is represented by a training dataset of context-target pairs: Z equals the set of (xi, yi) from i equals 1 to N, where both xi and yi are sequences of tokens. For example, in NL2SQL, xi is a natural language query and yi its corresponding SQL command; for summarization, xi is the content of an article and yi its summary. During full fine-tuning, the model is initialized to pre-trained weights Phi 0 and updated to Phi 0 plus delta Phi by repeatedly following the gradient to maximize the conditional language modeling objective: the sum over all context-target pairs in Z of the sum from t equals 1 to the length of y of the log of the probability P Phi of yt given x and y less than t. One of the main drawbacks for full fine-tuning is that for each downstream task, we learn a different set of parameters delta Phi whose dimension, the dimension of delta Phi, equals the dimension of Phi 0. Thus, if the pre-trained model is large (such as GPT-3 with the dimension of Phi 0 approximately 175 Billion), storing and deploying many independent instances of fine-tuned models can be challenging, if at all feasible. In this paper, we adopt a more parameter-efficient approach, where the task-specific parameter increment delta Phi equals delta Phi of Theta is further encoded by a much smaller-sized set of parameters Theta with the dimension of Theta much less than the dimension of Phi 0. The task of finding delta Phi thus becomes optimizing over Theta: max over Theta of the sum over all context-target pairs in Z of the sum from t equals 1 to the length of y of the log of the probability p with subscript Phi 0 plus delta Phi of Theta of yt given x and y less than t. In the subsequent sections, we propose to use a low-rank representation to encode delta Phi that is both compute- and memory-efficient. When the pre-trained model is GPT-3 175B, the number of trainable parameters, the dimension of Theta, can be as small as 0.01 percent of the dimension of Phi 0. Aren't Existing Solutions Good Enough? The problem we set out to tackle is by no means new. Since the inception of transfer learning, dozens of works have sought to make model adaptation more parameter- and compute-efficient. Using language modeling as an example, there are two prominent strategies when it comes to efficient adaptations: adding adapter layers or optimizing some forms of the input layer activations. However, both strategies have their limitations, especially in a large-scale and latency-sensitive production scenario. Adapter Layers Introduce Inference Latency. There are many variants of adapters. We focus on the original design by Houlsby et al. which has two adapter layers per Transformer block and a more recent one by Lin et al. which has only one per block but with an additional LayerNorm. While one can reduce the overall latency by pruning layers or exploiting multi-task settings, there is no direct ways to bypass the extra compute in adapter layers. This seems like a non-issue since adapter layers are designed to have few parameters (sometimes less than 1% of the original model) by having a small bottleneck dimension, which limits the FLOPs they can add. However, large neural networks rely on hardware parallelism to keep the latency low, and adapter layers have to be processed sequentially. This makes a difference in the online inference setting where the batch size is typically as small as one. In a generic scenario without model parallelism, such as running inference on GPT-2 medium on a single GPU, we see a noticeable increase in latency when using adapters, even with a very small bottleneck dimension. This problem gets worse when we need to shard the model as done in Shoeybi et al. and Lepikhin et al., because the additional depth requires more synchronous GPU operations such as AllReduce and Broadcast, unless we store the adapter parameters redundantly many times. Directly Optimizing the Prompt is Hard. The other direction, as exemplified by prefix tuning, faces a different challenge. We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters, confirming similar observations in the original paper. More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods. Our Method  We describe the simple design of LoRA and its practical benefits. The principles outlined here apply to any dense layers in deep learning models, though we only focus on certain weights in Transformer language models in our experiments as the motivating use case. Low-Rank-Parametrized Update Matrices  A neural network contains many dense layers which perform matrix multiplication. The weight matrices in these layers typically have full-rank. When adapting to a specific task, Aghajanyan et al. shows that the pre-trained language models have a low "instrisic dimension" and can still learn efficiently despite a random projection to a smaller subspace. Inspired by this, we hypothesize the updates to the weights also have a low "intrinsic rank" during adaptation.