Extending Context Window of Large Language Models via Semantic Compression (2024)

Abstract: Transformer based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses due to the quadratic complexity. These constraints restrict their applicability in long text scenarios. In this paper, we propose a novel semantic compression method that enables generalization to texts that are 6-8 times longer without incurring significant computational costs or requiring fine-tuning. Our proposed framework draws inspiration from source coding in information theory and employs a pre-trained model to reduce the semantic redundancy of long inputs before passing them to the LLMs for downstream tasks. Experimental results demonstrate that our method effectively extends the context window of LLMs across a range of tasks including question answering, summarization, few-shot learning, and information retrieval. Furthermore, the proposed semantic compression method exhibits consistent fluency in text generation while reducing the associated computational overhead.

## introduction the recent successful release of large language models (llms) such as chatgpt @xcite and llama @xcite have sparked significant research efforts from both industry and academia. these llms have demonstrated the ability to engage in fluent and coherent conversations with human users, and have shown exceptional performance across various tasks, including document summarization, question answering, dialogue bots, and code generation. one critical issue faced by state-of-the-art (sota) llms is the restriction on the length of text that can be inputted into the model at once. when the input context exceeds the limit of the context * correspondence to: niuxueyan3@huawei.com. the limitation on the context window primarily stems from the quadratic computation of the selfattention mechanism in the transformer. handling lengthy texts significantly increases the computational costs in terms of both memory and time. typically, models are trained on shorter contexts, so the maximum sequence length (i.e., the context window) is fixed. if the models are compelled to generate contexts that exceed the context window, they tend to compromise the quality of the output due to the lack of position encoding information that is not learned from the training. furthermore, generating long sequences imposes substantial memory requirements on the computational device. this accumulation of memory requirements and the lack of effective position encoding can result in length generalization failure @xcite , where the models struggle to generate meaningful and coherent text beyond a certain context window size. some techniques have been developed to address the aforementioned challenges. one approach is to devise architectures with nearly linear complexity, which enables efficient scaling to handle very long sequences. however, training a large model from scratch incurs substantial costs. another strategy involves employing interpolation and fine-tuning techniques to adapt the position encoding to unseen sequence lengths. while this method has the potential to improve the overall performance of llms on long sequences, it still demands significant time and gpu resources for fine-tuning and inference. therefore, methods that do not necessitate altering the parameters of the pre-trained models are more efficient and resource-friendly. while most previous algorithms involve modifying the pre-trained models, we instead exploit the statistical properties of the natural language input. one empirical phenomenon, known as zipf's law @xcite , observes that a small set of the most frequent word tokens in a large corpus of natural language account for almost all occurrences. this pattern arises from the tendency of language users to minimize effort in their daily conversations. consequently, by utilizing an expanded vocabulary, sentences can be significantly shortened while preserving the semantic meaning. moreover, it is common for language users to include redundant words during communication @xcite . these language habits are prevalent among users, and we propose to include a semantic compression module to mitigate the associated redundancies. our proposed semantic compression method, reminiscent of lossy source coding in information theory, extends the context window by equivalently shortening the long text input while preserving the semantic meaning. this procedure is conducted before inputting the tokens into the pre-trained llms. as illustrated in fig. 1 , the input undergoes compression before being transmitted to the llm for potential downstream tasks. the semantic compression method can be customized and optimized for downstream tasks, taking into consideration practical constraints such as time and memory resources. the implementation of the semantic compression module is straightforward and can be easily incorporated into other interpolation-based context window extension methods and black box apis. our method demonstrates enhanced performance compared to sota interpolation-based methods on a range of tasks, including single-document question answering, multi-document question answering, summarization, few-shot learning, and information retrieval, using real-world datasets while incurring no extra parameter updates or memory consumption. empirically, the proposed method is computationally efficient and achieves 6-8 times context window extension. ## methodology we now introduce our semantic compression method for extending the context window. the core idea is to compress the input into shorter texts without losing the key information or important details. this enables us to effectively include more content within the fixed input length constrained by the base llm. fig. 2 provides an overview of our method, which leverages pre-trained summarization models commonly used in natural language processing (nlp). like llms, existing summarization methods also have limitations regarding the length of the input. here, we propose a divide-and-conquer approach that takes into account the topic structure of the text. by identifying distinct topics in the lengthy text and dividing the text into blocks that exhibit a certain level of mutual independence, the content within each block can be compressed efficiently owing to their statistical correlation. the blocks can then be processed in parallel using pretrained models, and the results are combined to create a condensed textual input that can be processed by the base llm. this approach provides a computationally efficient way of removing redundancies in long texts which effectively extend the context window of the input. ## experiments we demonstrate that the proposed method of semantic compression can effectively extend the context window by up to 7-8 times without modifying the parameters of the pre-trained models. furthermore, the semantic compression module can be seamlessly integrated into existing methods, allowing for further extension of the context window. this versatility enables our approach to be adapted and combined with other techniques such as yarn, enhancing the overall performance and flexibility. to evaluate the performance of our method, we conduct experiments on several language tasks that require an understanding of long contexts. these tasks include passkey retrieval, single-document question answering, multi-document question answering, summarization, and few-shot learning. in each task, the model is provided with a sequence of context c (typically lengthy texts) and a sequence of text q (e.g., a prompt), and it is expected to generate the output answer a. then we conduct the ablation study of the summary model to explore how much parameter work. we conducted an ablation study on the semantic compression module to investigate the parameter size at which the module becomes effective. additionally, we also investigate the perplexity of generated text by our method in appendix e. ## results in this section, we report the results of the above experiments along with a comprehensive analysis. passkey retrieval we present the results of the passkey retrieval task in fig. 4 . we also showcase the compression results of our method for this task in fig. 3 . when employing llama2 for passkey retrieval, we observe a rapid drop in accuracy to zero once the input length surpasses the window size of 4096. however, by utilizing our method, the retrieval accuracy of the llama2 model remains above 90% even for inputs with lengths general nlp tasks we present our results on various general nlp tasks in ablation study the summary model in the semantic compression module plays a crucial role. the larger the parameter size of the model, the better its ability to generalize, and the same principle applies to the task of summarization as well. in this section, we investigate the effect of parameter size on the performance of the semantic compression module. we adopt three models with parameter sizes of 406m for bart-largecnn, 320m for distilbart-cnn-12-6, and 60m for text_summarization, which we refer to as "large", "medium", and "small", respectively. by only changing the summary model, we evaluate their performance on the english tasks in long-bench dataset. the results in ## conclusion in this work, we propose a novel approach to addressing the limitation of context window length in large language models using semantic compression. by leveraging the statistical properties of natural language and exploiting redundancy in communication, we are able to significantly shorten texts while preserving their semantic meaning. this allows for a 6-8 time extension of the context window without the need for modifying the parameters of the pre-trained model or incurring additional computational costs. the experiments demonstrate that our proposed semantic compression method outperforms current sota approaches on the multitask long context understanding benchmark, long-bench. furthermore, the implementation of our semantic compression module is straightforward and can be easily integrated into other interpolationbased methods and black box apis. this provides flexibility and adaptability to different downstream tasks, considering practical constraints such as time and memory resources. we believe our work can lead to a simpler context window extension method to be used in practice, thereby reducing the cost of large language models.