LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models  Abstract We present LongLoRA, an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs), with limited computation cost. Typically, training LLMs with long context sizes is computationally expensive, requiring extensive training hours and GPU resources. For example, training on the context length of 8192 needs 16 times the computational costs in self-attention layers as that of 2048. In this paper, we speed up the context extension of LLMs in two aspects. On the one hand, although dense global attention is needed during inference, fine-tuning the model can be effectively and efficiently done by sparse local attention. The proposed shifted sparse attention (S2-Attn) effectively enables context extension, leading to non-trivial computation saving with similar performance to fine-tuning with vanilla attention. Particularly, it can be implemented with only two lines of code in training, while being optional in inference. On the other hand, we revisit the parameter-efficient fine-tuning regime for context expansion. Notably, we find that LoRA for context extension works well under the premise of trainable embedding and normalization. LongLoRA combines this improved LoRA with S2-Attn. LongLoRA demonstrates strong empirical results on various tasks on Llama2 models from 7B/13B to 70B. LongLoRA extends Llama2 7B from 4k context to 100k, or Llama2 70B to 32k on a single 8 by A100 machine. LongLoRA extends models' context while retaining their original architectures, and is compatible with most existing techniques, like Flash-Attention2. In addition, we further conduct supervised fine-tuning with LongLoRA and our long instruction-following LongAlpaca dataset. All our code, models, dataset, and demo are available at github.com/dvlab-research/LongLoRA. Section 1: Introduction Large language models (LLMs) are typically trained with a pre-defined context size, such as 2048 tokens for LLaMA and 4096 tokens for Llama2. However, the pre-defined size limits LLMs in many applications, like summarizing long documents or answering long questions. To resolve this limitation, some recent works train or fine-tune LLMs to longer context. However, training an LLM from scratch with long sequences poses computational challenges, and fine-tuning an existing pre-trained LLM is also considerably expensive. For instance, Position Interpolation spent 32 A100 GPUs to extend LLaMA models from 2k to 8k context, and 128 A100 GPUs for longer context fine-tuning. FOT used 32 TPUs for standard transformer training and 128 TPUs for LongLLaMA. These computation resources are typically unaffordable for common researchers, which naturally leads us to question: can we extend the context window of LLMs efficiently? One straightforward approach is to fine-tune a pre-trained LLM via low-rank adaptation (LoRA). LoRA modifies the linear projection layers in self-attention blocks by utilizing low-rank matrices, which are generally efficient and reduce the number of trainable parameters. However, our empirical findings indicate that training long context models in this manner is neither sufficiently effective nor efficient. In terms of effectiveness, plain low-rank adaptation results in a high perplexity in long context extension. Increasing the rank to a higher value, for example, rank = 256, does not alleviate this issue. In terms of efficiency, regardless of whether LoRA is employed or not, computational cost increases dramatically as the context size expands, primarily due to the standard self-attention mechanism. As shown in Figure 1, even with LoRA, the training hours for the standard Llama2 model increase substantially when the context window expands. In this work, we introduce LongLoRA, an efficient fine-tuning approach that extends the context windows of pre-trained LLMs, for example, Llama2. LoRA uses low-rank weight updates to approximate full fine-tuning. Similarly, we find that short attention is also able to approximate long context during training. We present shifted sparse attention (S2-Attn) as an efficient substitute for standard self-attention. As shown in Figure 2, we split context length into several groups and conduct attention in each group individually. In half attention heads, we shift the tokens by half group size, which ensures the information flow between neighboring groups. For example, we use S2-Attn with group size 2048 to approximate the total 8192 context length training. This shares a high-level spirit with Swin Transformer. Models fine-tuned via S2-Attn retain the original attention architecture during inference. This facilitates most existing optimization and infrastructure. Techniques for common LLMs can also be applied to ours. For example, Flash-Attention2 is compatible with our method in both training and inference time. The reason behind this is that short attention resembles the attention scheme in the pre-training stage of LLMs. Other efficient attentions, for example, dilated or sparse attention, have a large gap to the standard style and do not work well like ours. We empirically show that learnable embedding and normalization layers are the key to unlocking long context LoRA fine-tuning. Embedding and normalization layers take up a small proportion of parameters in the entire LLM. For example, embedding has less than 2 percent parameters, and normalization has less than or equal to 0.004 percent parameters in Llama2 7B. This ratio decreases for even larger LLMs. In experiments, we show that LongLoRA is effective and efficient. We present experimental results of extending the context window for Llama2 7B, 13B, and 70B. Following the experimental settings of Position Interpolation, we fine-tune models with proper position embeddings. The trained models achieve comparable performance to the full-attention and fully fine-tuned results, while the computational cost is much less as shown in Figure 1. LongLoRA can fine-tune Llama2 7B up to 100k context, or a 70B model up to 32k, on a single 8 by A100 machine. In addition, we present a solution for supervised fine-tuning (SFT) with our self-collected long instruction-following dataset, LongAlpaca. Our LongLoRA models are further fine-tuned with long questions and the corresponding answers. We design various types of questions for technical papers, science fiction, and other books. SFT is important for improving the chat ability of LLMs. We introduce our SFT settings in Section B.5 in the appendix. Section 2: Related Work  Long-context Transformers. A large body of research has been developed to increase the context length of transformers.