We believe that LongLoRA is a general method that could be compatible with more types of LLMs and position encodings. We plan to investigate these in future work. Appendix Section A: Settings Subsection A.1: Environments All our experiments are conducted on an 8 by A100 machine. We train all models using PyTorch with the DeepSpeed and Flash-Attention2. By default, we use DeepSpeed in stage 2 and use stage 3 for the maximum context length experiments. Gradient checkpoint is used by default, which is a common technique in the Peft codebase. Note that sometimes, 8 by A100 GPUs might not be necessary and 3090 Ti GPUs are acceptable, like fine-tuning 7B models to 8192 context size. Subsection A.2: Format of Passkey Retrieval We follow existing literature for the document format of passkey retrieval. The document has the following format: It begins with the instruction, "There is an important info hidden inside a lot of irrelevant text. Find it and memorize them. I will quiz you about the important information there." This is followed by a block of filler text like "The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again." repeated M times. Then, the key information is presented, for example, "The pass key is 12362. Remember it. 12362 is the pass key." This is followed by another block of filler text repeated N times. Finally, the prompt asks, "What is the pass key? The pass key is". The document length varies with the value of M and N. The passkey number to retrieve is randomly sampled and varies at each testing time. Section B: Experiments Subsection B.1: Evaluation perplexity on PG19 test split. We present the evaluation results on the PG19 test split. We use the same settings as the models on proof-pile evaluation in the paper. Similarly, for a model trained on a certain context length, as the evaluation context length increases, our models achieve better perplexity. Note that the perplexity is higher than that in the proof-pile dataset, as PG19 has very different writing styles. Subsection B.2: Ablation on Group Sizes. We provide an ablation study on the group size of the S2-Attn. We experimented with fine-tuning Llama2 7B to 8192 and 16384 context lengths via LongLoRA. The group size varies from 1/2, 1/4, 1/6, to 1/8 of the target context length. For example, the group size is 1024 for 1/8 of the context length 8192. We find that the 1/2 and 1/4 settings have minor gaps to full attention fine-tuning. Group sizes less than 1/4 would be not good enough. We set the group size as 1/4 of the context length in experiments by default. Subsection B.3: Ablation on the variants of S2-Attn. We ablate some variants of S2-Attn. Variant 1 is to change the shifting direction from down to up. It shows that the shifting direction has no effect on the perplexity. One concern about S2-Attn is that it moves the last tokens to the front into one group, which might be inconsistent with causal masks. Variant 2 uses individual groups for the shifted tokens, which ablates this concern. Variant 3 swaps the shifted and the original front tokens, which can also ablate the concern. We show that these variants present similar perplexity to ours. We suppose that although there are communications among the front and last tokens, they are originally far away from others while it is limited in the local group. Moreover, S2-Attn is only used for fine-tuning, while we use standard causal masks and full attention during inference. Variant 2 and 3 also work well but involve additional steps to ours. Subsection B.4: Evaluation on Long-context Benchmarks. We evaluate our method on long-context benchmarks, LongBench and LEval. We fine-tune Llama2 7B to 16384 context length, with the supervised fine-tuning method and data introduced in Section B.5. We compare our model with GPT-3.5-Turbo and other Llama2-based long-context models, like Vicuna and LongChat models. It shows that our 7B model presents comparable or even better performance than these Llama2-based long-context models, while ours only takes about 4 hours, about 0.3 billion tokens, on a single 8 by A100 machine. Subsection B.5: Efficiency Analysis. We break down the FLOPs of Llama2 7B into various types of layers, including FFN - feed-forward layers, Proj - projection for queries, values, keys, and attention outputs, Attn - self-attention computation, Others - other layers like embedding, normalization, LLM head. For full attention, the proportion of Attn sharply increases as the context length increases. For example, Attn has 24.5 percent of the total FLOPs at the 8192 context length while it increases to 72.2 percent at the 65536 context length.