Papers
arxiv:2309.16039

Effective Long-Context Scaling of Foundation Models

Published on Sep 27, 2023
· Featured in Daily Papers on Sep 29, 2023
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,

Abstract

We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. We perform extensive evaluation on language modeling, synthetic context probing tasks, and a wide range of research benchmarks. On research benchmarks, our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2. Notably, with a cost-effective instruction tuning procedure that does not require human-annotated long instruction data, the 70B variant can already surpass gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks. Alongside these results, we provide an in-depth analysis on the individual components of our method. We delve into Llama's position encodings and discuss its limitation in modeling long dependencies. We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.

Community

The paper mentions the decaying effect of RoPE for distant tokens.

Is there any evidence that may be the reason behind the lack of attention to the system prompt after a few turns, which leads to the application of Gatt, in the original llama 2 paper?

It was really interesting to see the challenges they face when scaling in redteaming, hallucinations, tokenization, RLHF, etc

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2309.16039 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2309.16039 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2309.16039 in a Space README.md to link it from this page.

Collections including this paper 14