Effective Long-Context Scaling of Foundation Models (2024)

Abstract: We present an effective recipe to train strong long-context LLMs that are capable of utilizing massive context windows of up to 32,000 tokens. Our models are built through continual pretraining from Llama 2 checkpoints with longer text sequences and on a dataset where long texts are upsampled. We perform extensive evaluation using language modeling, synthetic context probing tasks, and a wide range of downstream benchmarks. Across all evaluations, our models achieve consistent improvements on most regular-context tasks and significant improvements on long-context tasks over Llama 2. Moreover, with a cost-effective instruction tuning procedure that is free of expensive annotation, the presented models can already surpass \texttt{gpt-3.5-turbo-16k} ‘s overall performance on long-context benchmarks. Alongside these results, we provide an in-depth analysis on each individual component of our method. We delve into Llama’s position encodings and discuss its key limitation in modeling long data. We examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths – ablation results suggest that having abundant long texts in the pretrain dataset is \textit{not} the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.

## introduction large language models (llms), trained with an unprecedented magnitude of data and compute, hold the promise of fundamentally improving the way we interact with the digital world. as llms get rapidly deployed and continue to evolve through scaling, we envision these models to serve more intricate and complex use cases, such as analyzing figure 1 : we show that our model's validation loss can be fit as a function of the context length: l(c) = ( α c ) β + γ with a different set of α, β, γ for each model size. this power-law relationship suggests that context length is another important axis of scaling llms and our model can continually improve its performance as we increase the context length up to 32,768 tokens. llms with robust long-context capabilities are primarily provided through proprietary llm apis @xcite and there is no open recipe for building long-context model that can demonstrate on-par downstream performance as these proprietary models. moreover, existing open-sourced long-context models @xcite @xcite often fall short on evaluations and primarily measure long-context capabilities with the language modeling loss and synthetic tasks, which do not comprehensively demonstrate their effectiveness in diverse, real-world scenarios. additionally, these models often overlook the necessity of maintaining strong performance on standard short-context tasks, either bypassing the evaluations or reporting degenerated perfor-mance @xcite . in this work, we present a method to build longcontext llms with superior performance over existing open-sourced models. we build our models by continually pretraining from llama 2 checkpoints with additional 400 billion tokens formed as long training sequences. among the model series, the smaller 7b/13b variants are trained with 32,768-token sequences while the 34b/70b variants with 16,384-token sequences. in contrast to the limited evaluation performed by existing studies, we extensively evaluate our models across various types of benchmarks. on language modeling, our model demonstrates a clear power-law scaling behavior with respect to context lengths. this scaling behavior, as shown in figure 1 , not only shows our models' ability to consistently benefit from more contexts but also suggest that context length is another importance axis of scaling llms. when comparing our models to llama 2 on standard benchmarks, we not only observe significant improvements on long-context tasks but also modest improvements on standard short-context tasks, especially on coding, math, and knowledge benchmarks. we explored using a simple and costeffective procedure to instruction finetune our continually pretrained long models without any humanannotated data. the end result is a chat model that can achieve stronger overall performance than gpt-3.5-turbo-16k on a series of long-context benchmarks covering question answering, summarization, and multi-document aggregation tasks. in the remaining part of this paper, we begin by presenting the continual pretraining approach and a lightweight instruction tuning procedure, followed by detailed results on a range of short and long context tasks. to facilitate future research, we complement our results with a thorough analysis discussing how the design of positional encodings, the length distribution of the dataset, and the training curriculum contribute to the final performance. ## conclusion we present a series of long-context llms that leverage a simple yet necessary position encoding refinement and continual pretraining to achieve strong long-context performance. our long context scaling is performed by continually pretraining from llama 2 with additional 400b tokens and outperform llama 2 on both short and longcontext tasks. our models also demonstrate superior performance compared to existing open-source long-context models and compare favorably against gpt-3.5-turbo-16k on a suite of long-context tasks after a simple instruction finetuning procedure without human supervision. we complement our results with a comprehensive analysis, providing insights on the influences of various factors including the nuances of position encodings, the data mix, and the pretraining curriculum on the final performance. we hope our study could make long-context llms more accessible and facilitate further advancements in this field.