Our model achieves reasonable passkey retrieval accuracy until 33k or 34k. Without further fine-tuning, We modify the max position embeddings to 48k in the position interpolation, which is the Ours 7B (extended PI). We show that this model can handle longer documents by simply extending the position interpolation. As the dashed orange line in Figure 4, the model, fine-tuned on 32k context length, presents moderate retrieval ability (60 percent to 90 percent accuracy) in the range of 33k to 45k. Even with the position interpolation extended, Llama2 7B suffers from a sharp accuracy degradation (dashed blue line) after the 4k context length. Subsection 4.3: Ablation Study In this section, we introduce ablation studies on the number of fine-tuning steps and attention patterns. Other experimental results including ablations on group sizes, attention variants, and efficiency analysis are Section B in the appendix. Ablation on Fine-tuning Steps. We report the relationship between perplexity and fine-tuning steps for a Llama2 7B model extending to the 8192 context length on the PG19 validation set. We see that without fine-tuning, at step 0, the model has a limited long context capability, for example, 15.82 perplexity. We show that the perplexity drops quickly. Full fine-tuning converges faster than low-rank training. They come closer after 200 steps, without a large gap at the end. Attention Patterns. We show the effects of different attention patterns during fine-tuning. We fine-tune a Llama2 7B model to 32768 context length on Redpajama datasets and evaluate the perplexity on PG19 validation set. We first examine the manner of swapping among various settings. For the shift operation we used in LongLoRA, there are three choices: disabling it, shifting between sequential layers, and shifting among attention heads. We show that shifting between layers is acceptable but not the best. In addition, setting all attention heads as pattern 1 or pattern 2 does not work. In addition, we empirically find that shifting left or right has little difference in performance. We then test other types of efficient attention designs, including dilated attention, block sparse attention, and stride sparse attention. For dilated attention, we vary the dilate rate from 1 to 2 evenly among attention heads. For block sparse attention, we use n=4 block-wise masking matrices in attention heads and move the block left to make it causal. Stride sparse attention contains both local and stride patterns. These settings share similar computational costs. We visualize these patterns in Figure 7 in the appendix. These attention patterns are invented in training-from-scratch transformers. This experiment is to examine their capability of fine-tuning on pre-trained LLMs, toward long context adaptation. Dilated attention performs well in full fine-tuning but is not well with low-rank adaptation. Fine-tuning with stride sparse attention is harmful. They have a large gap to full attention, which is applied in the pre-training stage. Section 5: Conclusion In this work, we propose LongLoRA that can efficiently extend the context length of LLMs to be significantly larger. LongLoRA has less GPU memory cost and training time than standard full fine-tuning, with minimal accuracy compromise. At the architecture level, we propose S2-Attn to approximate the standard self-attention pattern during training. S2-Attn is easy to implement, requiring only two lines of code. Moreover, models trained via S2-Attn retain the original standard attention architecture during inference, making most pre-existing infrastructure and optimization reusable. At the training level, we bridge the gap between LoRA and full fine-tuning with trainable normalization and embedding. Our method can extend Llama2 7B to 100k context length and 70B model to 32k context length, on a single 8 by A100 machine. We also present a long instruction-following dataset, LongAlpaca and conducted supervised fine-tuning with LongLoRA.