Since both A with subscript r equals 8 and A with subscript r equals 64 are learned using the same pre-trained model, this indicates that the top singular-vector directions of A with subscript r equals 8 and A with subscript r equals 64 are the most useful, while other directions potentially contain mostly random noises accumulated during training. Hence, the adaptation matrix can indeed have a very low rank. Subspace similarity between different random seeds. We further confirm this by plotting the normalized subspace similarity between two randomly seeded runs with r=64. delta Wq appears to have a higher "intrinsic rank" than delta Wv, since more common singular value directions are learned by both runs for delta Wq, which is in line with our empirical observation. As a comparison, we also plot two random Gaussian matrices, which do not share any common singular value directions with each other. How Does the Adaptation Matrix delta W Compare to W? We further investigate the relationship between delta W and W. In particular, does delta W highly correlate with W? (Or mathematically, is delta W mostly contained in the top singular directions of W?) Also, how "large" is delta W comparing to its corresponding directions in W? This can shed light on the underlying mechanism for adapting pre-trained language models. To answer these questions, we project W onto the r-dimensional subspace of delta W by computing U transpose W V transpose, with U and V being the left and right singular-vector matrices of delta W. Then, we compare the Frobenius norm between the Frobenius norm of U transpose W V transpose and the Frobenius norm of W. As a comparison, we also compute the Frobenius norm of U transpose W V transpose by replacing U,V with the top r singular vectors of W or a random matrix. We draw several conclusions. First, delta W has a stronger correlation with W compared to a random matrix, indicating that delta W amplifies some features that are already in W. Second, instead of repeating the top singular directions of W, delta W only amplifies directions that are not emphasized in W. Third, the amplification factor is rather huge: 21.5, which is approximately 6.91 divided by 0.32, for r equals 4. This suggests that the low-rank adaptation matrix potentially amplifies the important features for specific downstream tasks that were learned but not emphasized in the general pre-training model. Conclusion and Future Work  Fine-tuning enormous language models is prohibitively expensive in terms of the hardware required and the storage/switching cost for hosting independent instances for different tasks. We propose LoRA, an efficient adaptation strategy that neither introduces inference latency nor reduces input sequence length while retaining high model quality. Importantly, it allows for quick task-switching when deployed as a service by sharing the vast majority of the model parameters. While we focused on Transformer language models, the proposed principles are generally applicable to any neural networks with dense layers. There are many directions for future works. 1) LoRA can be combined with other efficient adaptation methods, potentially providing orthogonal improvement. 2) The mechanism behind fine-tuning or LoRA is far from clear -- how are features learned during pre-training transformed to do well on downstream tasks? We believe that LoRA makes it more tractable to answer this than full fine-tuning. 3) We mostly depend on heuristics to select the weight matrices to apply LoRA to. Are there more principled ways to do it? 4) Finally, the rank-deficiency of delta W suggests that W could be rank-deficient as well, which can also be a source of inspiration for future works. Appendix  Large Language Models Still Need Parameter Updates  Few-shot learning, or prompt engineering, is very advantageous when we only have a handful of training samples. However, in practice, we can often afford to curate a few thousand or more training examples for performance-sensitive applications. Fine-tuning improves the model performance drastically compared to few-shot learning on datasets large and small. We take the GPT-3 few-shot result on RTE from the GPT-3 paper. For MNLI-matched, we use two demonstrations per class and six in-context examples in total. Inference Latency Introduced by Adapter Layers  Adapter layers are external modules added to a pre-trained model in a sequential manner, whereas our proposal, LoRA, can be seen as external modules added in a parallel manner. Consequently, adapter layers must be computed in addition to the base model, inevitably introducing additional latency. While as pointed out, the latency introduced by adapter layers can be mitigated when the model batch size and/or sequence length is large enough to full utilize the hardware parallelism. We confirm their observation with a similar latency study on GPT-2 medium and point out that there are scenarios, notably online inference where the batch size is small, where the added latency can be significant. We measure the latency of a single forward pass on an NVIDIA Quadro RTX8000 by averaging over 100 trials. We vary the input batch size, sequence length, and the adapter bottleneck dimension r. We test two adapter designs: the original one by Houlsby et al., which we call Adapter H, and a recent, more efficient variant by Lin et al., which we call Adapter L. We plot the slow-down in percentage compared to the no-adapter baseline. Dataset Details  GLUE Benchmark is a wide-ranging collection of natural language understanding tasks. It includes MNLI (inference), SST-2 (sentiment analysis), MRPC (paraphrase detection), CoLA (linguistic acceptability), QNLI (inference), QQP (question-answering), RTE (inference), and STS-B (textual similarity). The broad coverage makes GLUE benchmark a standard metric to evaluate NLU models such as RoBERTa and DeBERTa. The individual datasets are released under different permissive licenses. WikiSQL is introduced and contains 56,355 training and 8,421 validation examples. The task is to generate SQL queries from natural language questions and table schemata. We encode context as the set of table schema and query and target as the SQL. The dataset is release under the BSD 3-Clause License. SAMSum is introduced and contains 14,732 training and 819 test examples. It consists of staged chat conversations between two people and corresponding abstractive summaries written by linguists. We encode context as newline concatenated utterances followed by a double newline, and target as the summary. The dataset is released under the non-commercial licence: Creative Commons BY-NC-ND 4.0. E2E NLG Challenge was first introduced as a dataset for training end-to-end, data-driven natural language generation systems and is commonly used for data-to-text evaluation. The E2E dataset consists of roughly 42,000 training, 4,600 validation, and 4,600 test examples from the restaurant domain. Each source table used as input can have multiple references. Each sample input (x, y) consists of a sequence of slot-value pairs, along with a corresponding natural language reference text. The dataset is released under Creative Commons BY-NC-SA 4.0. DART is an open-domain data-to-text dataset. DART inputs are structured as sequences of ENTITY, RELATION, ENTITY triples. With around 82K examples in total, DART is a significantly larger and more complex data-to-text task compared to E2E. The dataset is released under the MIT license. WebNLG is another commonly used dataset for data-to-text evaluation. With around 22K examples in total WebNLG comprises 14 distinct categories, nine of which are seen during training. Since five of the 14 total categories are not seen during training, but are represented in the test set, evaluation is typically broken out by "seen" categories (S), "unseen" categories (U) and "all" (A). Each input example is represented by a sequence of SUBJECT, PROPERTY, OBJECT triples. The dataset is released under Creative Commons BY-NC-SA 4.0. Hyperparameters Used in Experiments  RoBERTa  We train using AdamW with a linear learning rate decay schedule. We sweep learning rate, number of training epochs, and batch size for LoRA. Following Liu et al., we initialize the LoRA modules to our best MNLI checkpoint when adapting to MRPC, RTE, and STS-B, instead of the usual initialization; the pre-trained model stays frozen for all tasks. We report the median over 5 random seeds; the result for each run is taken from the best epoch. For a fair comparison with the setup in Houlsby et al. and Pfeiffer et al., we restrict the model sequence length to 128 and used a fixed batch size for all tasks. Importantly, we start with the pre-trained RoBERTa large model when adapting to MRPC, RTE, and STS-B, instead of a model already adapted to MNLI. The runs with this restricted setup are marked with a dagger. DeBERTa  We again train using AdamW with a linear learning rate decay schedule. Following He et al., we tune learning rate, dropout probability, warm-up steps, and batch size. We use the same model sequence length used by He et al. to keep our comparison fair. Following He et al., we initialize the LoRA modules to our best MNLI checkpoint when adapting to MRPC, RTE, and STS-B, instead of the usual initialization; the pre-trained model stays frozen for all tasks. We report the median over 5 random seeds; the result for each run is taken from the best epoch. GPT-2  We train all of our GPT-2 models using AdamW with a linear learning rate schedule for 5 epochs. We use the batch size, learning rate, and beam search beam size described in Li and Liang. Accordingly, we also tune the above hyperparameters for LoRA. We report the mean over 3 random seeds; the result for each run is taken from the best epoch. GPT-3  For all GPT-3 experiments, we train using AdamW for 2 epochs with a batch size of 128 samples and a weight decay factor of 0.1. We use a sequence length of 384 for WikiSQL, 768 for MNLI, and 2048 for SAMSum. We tune learning rate for all method-dataset combinations. For prefix-embedding tuning, we find the optimal lp and li to be 256 and 8, respectively, totalling 3.2M trainable parameters. We use lp=8 and li=8 for prefix-layer tuning with 20.2M trainable parameters to obtain the overall best performance. We present two parameter budgets for LoRA: 4.7M and 37.7M. We report the best validation performance from each run. Combining LoRA with Prefix Tuning  LoRA can be naturally combined with existing prefix-based approaches. In this section, we evaluate two combinations of LoRA and variants of prefix-tuning on WikiSQL and MNLI. LoRA+PrefixEmbed (LoRA+PE) combines LoRA with prefix-embedding tuning, where we insert lp plus li special tokens whose embeddings are treated as trainable parameters. LoRA+PrefixLayer (LoRA+PL) combines LoRA with prefix-layer tuning. We also insert lp plus li special tokens; however, instead of letting the hidden representations of these tokens evolve naturally, we replace them after every Transformer block with an input agnostic vector. Thus, both the embeddings and subsequent Transformer block activations are treated as trainable parameters. We show the evaluation results of LoRA+PE and LoRA+PL on WikiSQL and MultiNLI. First of all, LoRA+PE significantly outperforms both LoRA and prefix-embedding tuning on WikiSQL, which indicates that LoRA is somewhat orthogonal to prefix-embedding tuning. On MultiNLI, the combination of LoRA+PE doesn't perform better than LoRA, possibly because LoRA on its own already achieves performance comparable to the human baseline. Secondly, we notice that LoRA+PL performs slightly worse than LoRA even with more trainable parameters. We attribute this to the fact that prefix-layer tuning is very sensitive to the choice of learning rate and thus makes the optimization of LoRA weights more difficult in LoRA+PL. Additional Empirical Experiments  Additional Experiments on GPT-2  We also repeat our experiment on DART and WebNLG following the setup of Li and Liang. The result is shown.