E2- LLM : Efficient and Extreme Length Extension of Large Language Models (2024)

Abstract: Training Large Language Models (LLMs) to process extensive context lengths incurs prohibitive computational costs. Prevailing techniques for extending context capabilities in LLMs typically require not only additional training procedures but also access to datasets with long context (e.g., sequences of 32K tokens), presupposing substantial GPU expenditures. To address the aforementioned issues, we introduce a novel solution named Efficient and Extreme length extension for Large Language Models (E2-LLM). E2-LLM entails a singular training process over considerably short sequences (e.g., 4K tokens), which greatly mitigates the cost of continual-pretraining or fine-tuning. Within the training phase, we incorporate a dual augmentation strategy with Rotary Position Embeddings (RoPE) that adjusts the scale and position indices across distinct training samples. E 2 -LLM is meticulously designed to enhance the model’s robustness to diverse relative positions. The experimental results on multiple benchmark datasets demonstrate the superior performance of E 2 -LLM on demanding tasks of processing long contexts.

## introduction large language models are inherently constrained by a predetermined context window length. for instance, in the case of the llama architecture @xcite , input sequences are restricted to a maximum of either 2,048 or 4,096 tokens. this preset context window limit could be readily surpassed in applications such as multi-turn dialogues, summarizations of long documents, or intricate reasoning chains @xcite . thus, llms capable of accommodating extensive context windows are thereby highly demanding. nevertheless, the process of training an llm with long context windows from scratch * first two authors contributed equally. ## 50k 100k 150k 200k figure 5 : generalization abilities on the unseen scales. ## conclusion in this study, we propose e 2 -llm to extend the context windows with a singular training phase and minimal computational overhead. e 2 -llm harnesses rope-based position embeddings to implement a novel dual augmentation strategy that manipulates the interpolation scale and positional offset in training examples of short lengths (e.g., 4k/8k). comprehensive experimental results demonstrate the superior performance of e 2 -llm on long-context processing tasks.