SCA : Selective Compression Attention for Efficiently Extending the Context Window of Large Language Models (2024)

Abstract: Large language models (LLMs) have achieved impressive performance across various domains, but the limited context window and the expensive computational cost of processing long texts restrict their more comprehensive application. In this paper, we propose Selective Compression Attention (SCA), a general and effective method to expand the context window and reduce memory footprint by compressing the KV cache of LLMs. Specifically, through preliminary experiments, we found that the KV cache contains many similar vectors, resulting in information redundancy, which can be compressed by retaining representative vectors and discarding others. Therefore, SCA continuously selects the most distinctive vectors to keep through a greedy algorithm, reducing information loss during compression. Extensive experiments on various tasks verify the effectiveness of our method. Compared with existing methods, SCA can significantly reduce the impact on model performance under the same compression ratio. Furthermore, the context window of LLMs can be efficiently expanded using SCA without any training, which can even achieve better performance than specially fine-tuned long context models.

## introduction transformer-based @xcite large language models (llms) have excellent capabilities, which have extensively promoted the development of various natural language processing applications @xcite @xcite and provided a possibility for artificial general intelligence. however, due to their huge size, their deployment is very expensive. in particular, the quadratic cost of attention layers and the growing kv cache make the overhead of llms unacceptable when processing long texts, which limits the application and development of llms in long context scenarios. significant efforts have been made to improve the efficiency and extend the context window for llms. for example, some methods @xcite @xcite use a manually set sparse attention mode to limit the maximum size of the attention calculation window. however, they will lose valuable information, causing the performance to decrease significantly. there are some other methods @xcite ) that only use the retrieved most relevant chunks to calculate attention but still need to keep the complete kv cache. another works improve efficiency by changing the model structure @xcite . however, such methods require retraining or fine-tuning the model, making their application costly. therefore, this paper aims to propose a method that can overcome the shortcomings of previous approaches. specifically, 1. it can effectively compress the kv cache and expand the context window of llms. we first conducted a preliminary experiment to explore the feasibility. fortunately, we found that the kv cache has many redundant vectors that could be deleted. specifically, the cosine similarity between many vectors in the kv cache is extremely high. the similar vectors provide similar information when calculating attention. therefore, we can compress the kv cache and retain its original information by reserving representative vectors and removing similar redundant vectors. based on the preliminary experimental results, we propose the selective compression attention (sca) method, which can effectively compress the kv cache, improve the efficiency of llms, and extend their context window. specifically, our method uses a greedy algorithm to select the least redundant vector based on the current retained result to reserve, ensuring that more different information can be kept at each step during compression. when the kv cache length reaches a given maximum threshold, it can be compressed using the sca approach to provide free space, allowing llms to receive more context. moreover, unlike the recently proposed autocompressors @xcite , our method does not require fine-tuning and can be easily applied to any llms. to verify the effectiveness of our proposed method, we conduct extensive experiments on different llms and datasets. on the one-shot and zero-shot short text tasks, the performance after using sca to compress the kv cache is almost the same as the original full attention, verifying that sca can retain most of the original information during compression. for the long context tasks, our method can effectively extend the llms' original context window and ensure the fluency and accuracy of the generated results. especially, sca can still achieve 100% accuracy on the passkey retrieval task after extending the context window size of llama2-13b-chat @xcite to 12k. furthermore, using sca to extend the context length of vicuna1.5-7b @xcite to 16k can even perform better than the fine-tuned vicuna1.5-7b-16k on real long context tasks. in summary, our main contributions are the following: (1) we analyze and verify the feasibility of compressing the kv cache. by exploring the similarities between vectors, the preliminary ex-periment demonstrates that the kv cache contains much redundant information. (2) we propose an efficient and plug-and-play approach, which can compress the kv cache and keep most of the original information by retaining the representative vectors. (3) we conduct extensive experiments to show the powerful potential of our method, which can effectively extend the context window and reduce the memory footprint for different llms. ## related work extensive research has been done on efficient inference and context window extension of llms. an intuitive idea is manually setting sparse attention to limit computational complexity @xcite @xcite . for example, streamingllm @xcite only retains the most recent tokens and several initial tokens for stable attention computation. streamingllm can perform language modeling of millions of tokens. however, it loses much original information and cannot truly enhance llms' ability to remember and use long contexts. recently, @xcite proposed h 2 o, a heuristic kv cache eviction policy. h 2 o compresses the kv cache by evicting tokens with the smallest accumulated attention score. however, the score calculated only based on the current kv cache is one-sided, which may cause it to discard tokens needed in the future. unlike the previous methods, our approach selects the most representative vectors based on the vector distribution of the kv cache so that more different information can be retained, significantly reducing the information loss during the compression process. the second type of method retrieves the most relevant chunk in the kv cache for the attention calculation @xcite @xcite . although these methods can reduce the overhead of attention calculation, they still need to store the complete kv cache. therefore, they can not solve the problem of the kv cache increasing linearly as the context length increases. when the context is very long, they need to offload the kv cache to the cpu, increasing communication overhead between the gpu and the cpu. in contrast, sca can ensure the kv cache size does not exceed a given threshold, significantly reducing the memory footprint of llms when processing long contexts. another type of work changes the model structure to make it more efficient @xcite @xcite . for example, transformer-xl (dai et al., 2019) uses a segmentlevel recurrence mechanism to expand its receptive field and capture longer dependencies while fixing the attention window size. reformer @xcite proposes a new attention module that uses locality sensitive hashing attention to reduce the computational cost from quadratic to superlinear complexity. however, such methods require retraining, making their deployment on llms costly. in contrast, our approach is plug-and-play and can be easily adapted to any llms. ## preliminary experiment in this section, we carefully explore the characteristics of the kv cache in llms. specifically, we conducted experiments to answer two questions: (1) is there information redundancy in the kv cache? ## method this section details the proposed approach. first, we present the problem definition in 4.1, then introduce the design ideas of our method in 4.2, and give the implementation details in 4.3. ## experiments in this section, we conduct extensive experiments to verify the effectiveness of our proposed method. specifically, we first verify whether using sca to compress the kv cache affects the performance of the llms in section 5.1. then, we verify the context window extension capability of our method based on a variety of tasks, including language modeling tasks (seciton 5.2), passkey retrieval tasks (section 5.3), and real long context tasks in the l-eval benchmark (section 5.4). finally, we conduct fine-grained ablation experiments in section 5.5 to further analyze our approach. we use a single nvidia rtx a6000 48gb gpu for experiments. during inference, we use the greedy search for llms to generate results. we mainly compare several advanced baselines, including: considering the token's integrity, sca is performed in units of tokens during compression. specifically, we concatenate the vectors of all attention heads in the kv cache to construct the token vector for sca. moreover, based on the experimental results in section 5.5, we only use the sca algorithm for the last layer and let all layers share the selection results to further improve efficiency. ## conclusion in this paper, we first explore the characteristics of the kv cache and verify the feasibility of compressing it by retaining representative vectors and discarding others. based on these experimental results, we propose a general and plug-and-play method called sca, which adopts a greedy algorithm to minimize the information loss during the compression process. extensive experiments on various tasks demonstrate the effectiveness of our approach, which can compress the kv cache with little impact on the model performance. furthermore, sca can easily and efficiently expand the context window of llms, and its performance is even better than the fine-tuned long context llms.