In-Context Former: Lightning-fast Compressing Context for Large Language Model (2024)

Abstract: With the rising popularity of Transformer-based large language models (LLMs), reducing their high inference costs has become a significant research focus. One effective approach to mitigate these costs is compressing the long input contexts. Existing methods typically leverage the self-attention mechanism of the large model itself for context compression. While these methods have achieved notable results, the compression process still entails quadratic complexity. To mitigate this limitation, we propose the In-Context Former (IC-Former). This method does not rely on the target large model but instead utilizes cross-attention mechanisms to extract and condense information from the contextual embeddings. The computational overhead of our method grows linearly with the compression range. Experimental results indicate that our method requires only 1/32 of the floating-point operations of the baseline during compression and improves processing speed by 68 to 112 times while achieving 90% of the baseline performance on evaluation metrics. Additionally, IC-Former demonstrates strong regularity in its interactions with the context, enhancing its interpretability. Overall, IC-Former significantly reduces compression costs, making real-time compression scenarios feasible.

## introduction in recent years, transformer-based @xcite language models especially large language models (llms) have made significant strides in the field of natural language processing, demonstrating exceptional performance across a wide range of tasks. however, the self-attention mechanism in llms leads to high inference costs. previous work @xcite @xcite @xcite @xcite et @xcite has explored various approaches to reduce computational complexity by improving the self-attention mechanism of language models. although these strategies mitigate the overhead of long context processing, they inevitably introduce modifications to the original structure of llms, potentially impacting the capabilities of the original model @xcite . to better avoid modifications to the llm structure, a more intuitive approach is to introduce a preliminary context compression process. these methods are based on a core assumption: most natural language texts contain redundant information, which makes context compression feasible. in early exploration, @xcite have attempted to compress the instructions into short soft prompts. this method offers a novel perspective but still has limitations in long context compression. later works @xcite aim to further extend compression abilities for document-level long contexts, and achieved considerable results. as illustrated in figure 1 , these methods design compression models to condense lengthy contexts into short, context-rich soft prompts, which then serve as substitutes for the original context when input into the llm. however, these methods still suffer the issue of expensive time costs during the compression process. this limitation restricts their application in realtime compression scenarios, such as compressing retrieved @xcite or real-time internet documents @xcite immediately. by reviewing previous works on compressors, we find that existing methods typically utilize the llm as the encoder. while these methods fully utilize the powerful semantic understanding capabilities of llm, they also suffer from rapidly increasing quadratic time complexity as the context lengthens. so is there a way to significantly reduce the theoretical complexity of compressors, with an acceptable decrease in performance? driven by this motivation, we design an efficient context compression model, the in-context former (ic-former), which aims at optimizing resource consumption during the compression of long context in existing models. this model is based on two assumptions regarding semantic content compression: (1) word embeddings already contain sufficient semantic information @xcite , suggesting that interactions between embeddings may not be necessary prior to the extraction process. (2) learnable tokens within an elaborate structure can effectively aggregate information to a certain extent @xcite . based on these assumptions, we try to discard the costly self-attention interaction of text content in previous models. instead, we leverage the efficiency of the cross-attention mechanism for information extraction. this innovative strategy ensures that the computational overhead of compression grows linearly with the context length within the compression range, significantly enhancing compression efficiency compared to the previous methods. specifically, our ic-former consists of a few cross-attention blocks and some learnable digest tokens. through this structure, the ic-former leverages the digest tokens to extract information from lengthy contextual content and refine it into compact digest vectors. subsequently, these digest vectors directly replace the original, verbose context and serve as input to llms while ensuring that the generated texts are faithful to the original context. in the training phase, to effectively compress context, we follow the previous training paradigm @xcite , employing a strategy that combines pre-training and fine-tuning to optimize the ic-former. during the pre-training phase, the ic-former engages in a context reconstruction task. it generates digest vectors from which an llm can reconstruct the original context. in the fine-tuning phase, we train the ic-former on instruction data to ensure the generated digest vectors correctly respond to various context-related prompts. additionally, through theoretical calculations, we demonstrate that at a compression ratio of 4x, our ic-former achieves only 1/32 of the floatingpoint operations required by the baseline. experimental results further show that our method achieves a compression speed that is 68 to 112 times faster than the baseline while maintaining over 90% of the baseline performance on evaluation metrics. this indicates a higher costeffectiveness. overall, our contributions can be summarized in the following three points: they align the model predictions that are based on the original prompt and those conditioned on the soft prompt by optimizing kl divergence (hershey and olsen, 2007). as a result, wingate et al. (2022) discover that the trained causal cross attention digest embeddings context embeddings digest vectors feed forward network 𝒅 𝟐 𝒅 𝟏 …… 𝒅 𝒌 x n 𝒆(𝑤 ) …… 𝒆(𝑤 ) 𝒆(𝑤 ) k v q 𝒆 (𝑑 ) …… 𝒆 (𝑑 ) 𝒆 (𝑑 ) :concatenate 𝑑 𝑑 𝑑 𝑤 𝑤 𝑤 𝑤 𝑤 𝑑 𝑑 𝑑 … … … attention mask :masked query key :unmasked 𝒆(𝑝 ) 𝒆(𝑝 ) …… 𝒆(𝑝 ) 𝒅 𝟐 𝒅 𝟏 …… 𝒅 𝒌 large language model in-context former 𝒆 (𝑑 ) …… 𝒆 (𝑑 ) 𝒆 (𝑑 ) 𝒚 output context embeddings digest embeddings prompt embeddings 𝒆(𝑤 ) …… 𝒆(𝑤 ) 𝒆(𝑤 ) soft prompt retain high-level semantic information and can be utilized to control generation. however, this approach suffers high computational costs as it requires retraining a new soft prompt for each new context. in contrast, our method can predict the soft prompt corresponding to the input context. context distillation another related work @xcite focuses on distilling the contextual information such as instruction into a student model without prompting. @xcite propose gist to compress prompts into gist tokens, which can be viewed as key-value attention prefixes. nonetheless, this approach did not address the long context issue as it is limited to compressing short prompts. in addition, this method requires updating the parameters of language model, which differs from our method. our method keeps the language model fixed and therefore preserves its capability. @xcite propose autocompressors to compress long text into summary vectors recursively. however, the compression procedure is sophisticated and llms are still required to be fine-tuned to generate summary vectors. icae @xcite is the most closely related study to our research. icae compresses context into short memory slots, with a small number of additional parameters by the lora @xcite approach with a fixed llm. however, both autocompressors and icae employ self-attention to integrate contextual information, resulting in a quadratic complexity with respect to the length of context. instead, our model does not incorporate contextual interactions and reduces both time and space complexities, striking a balance between efficiency and performance. ## conclusion in this paper, we propose the in-context former (ic-former), a novel context compression model, which can efficiently condense contextual information into digest vectors in a linear complexity by removing irrelevant interaction processing. moreover, our proposed ic-former utilizes the cross-attention mechanism to enhance the extraction ability of digest tokens. our experimental results demonstrate that ic-former significantly reduces time and space complexity while preserving contextual semantics, thereby supporting broader applications requiring extensive context.