L ong LLML ingua: Accelerating and Enhancing LLM s in Long Context Scenarios via Prompt Compression (2024)

Abstract: In long context scenarios, large language models (LLMs) face three main challenges: higher computational cost, performance reduction, and position bias. Research indicates that LLM performance hinges on the density and position of key information in the input prompt. Inspired by these findings, we propose LongLLMLingua for prompt compression towards improving LLMs’ perception of the key information to simultaneously address the three challenges. Our extensive evaluation across various long context scenarios demonstrates that LongLLMLingua not only enhances performance but also significantly reduces costs and latency. For instance, in the NaturalQuestions benchmark, LongLLMLingua boosts performance by up to 21.4% with around 4x fewer tokens in GPT-3.5-Turbo, leading to substantial cost savings. It achieves a 94.0% cost reduction in the LooGLE benchmark. Moreover, when compressing prompts of about 10k tokens at ratios of 2x-6x, LongLLMLingua can accelerate end-to-end latency by 1.4x-2.6x.

## introduction large language models (llms) have revolutionized user-oriented language technologies and are serving as crucial components in more and more applications. carefully designing prompts is necessary to achieve better performance in specific downstream tasks. the commonly used technologies such as in-context learning (icl) @xcite , retrieval augment generation (rag) @xcite , and multi-turn agent @xcite @xcite are driving prompts to be increasingly longer, even reaching thousands of tokens. scenarios such as multi-document question answering, code completion, and document summarization also necessitate the processing of long contexts. 1 our code is available at https://aka.ms/llmlingua . there are three main challenges when llms are used in long context scenarios: (1) higher computational costs, encompassing both financial and latency expenses. (2) longer prompts introduce irrelevant and redundant information, which can weaken llms' performance @xcite , as illustrated in figure 1a . (3) llms exhibit position bias @xcite , also known as the "lost in the middle" issue @xcite , suggesting that the placement of key information within the prompt significantly affects llms' performance. this is demonstrated by the purple curve in figure 1b . inspired by these observations, we propose longllmlingua to address the three challenges. specifically, we use llmlingua @xcite as the backbone for prompt compression to address the first challenge, i.e., reduce cost and latency. however, in the case of long contexts, the distribution of question-relevant key information in the prompt is generally dynamic and sparse. existing prompt compression methods like llmlingua @xcite and selective-context @xcite that often fail to consider question during compression, resulting in retention of excessive noise and decreased performance. longllm-lingua aims to improve llms' perception of key information pertinent to the question, thereby overcoming the noise and position bias issues in long contexts, shown in figure 1b . the underlying principle of longllmlingua is that small lm are inherently capable of capturing the distribution of key information relevant to a given question. our main contributions are five-fold: (1) we propose a question-aware coarse-to-fine compression method to improve the key information density in the prompt (sec. 4.1); (2) we introduce a document reordering strategy to minimize position bias in llms. (sec. 4.2); (3) we establish dynamic compression ratios for precise control between coarse and fine compression levels (sec. 4.3); (4) we propose a post-compression in this case, we keep k most relevant documents/paragraphs based on the ground-truth or longllmlingua r k . a larger k implies more noise introduced into the prompt. to improve the key information density in the prompt, we present question-aware coarse-to-fine compression. (b) llms' ability to capture the relevant information depends on their positions in the prompt. to reduce information loss in the middle, we introduce a document reordering mechanism. ## problem formulation following llmlingua @xcite , we use @xmath0, x doc 1 , • • • , x doc k , x que ) to represent a prompt, including the instruction x ins , k documents x doc i , and the question x que . however, this definition can be adjusted for specific scenarios. the objective of a prompt compression system can be formulated as: ## preliminary: llmlingua llmlingua @xcite utilizes a small language model m s to evaluate the perplexity of each prompt token, removing those with lower perplexities. this method is premised on the idea that tokens with lower perplexities have a negligible effect on the language model's overall entropy gain, implying their removal slightly impacts the llms' contextual understanding. this process is viewed as an application of "lm is compression" @xcite . llmlingua include three key components: budget controller, iterative token-level prompt compression, and distribution alignment, highlighted by italic text in figure 2 . the budget controller assigns varying compression ratios to different parts of the prompt (i.e., instruction, demonstrations, question), implementing coarse-level prompt compression. subsequent steps involve dividing intermediate results into segments and applying token-level compression iteratively, where each token's perplexity based on preceding compressed segments. to aware different target llms, llmlingua fine-tunes m s using data from the target llm. ## longllmlingua longllmlingua builds on llmlingua to better compress prompts in long context scenorias. it tackles three main issues in handling lengthy contexts, as introduced in sec. 1. this approach focuses on making llms more effective at recognizing key original prompt longllmlingua compressed prompt instruction: answer the question based on the given passages. only give me the answer and do not output any other words. the following are given passages. document 1: alberic iii of dammartin alberic iii of dammartin (aubry de dammartin) (c. 1138 -19 september 1200) was a french count and son of alberic ii, count of dammartin, and clé mence de bar, daughter of reginald i, count of bar… document 2: … document n: pope agapetus ii pope agapetus ii (died 8 november 955) was the bishop of rome and ruler of the papal states from 10 may 946 to his death. a nominee of the princeps of rome, alberic ii of spoleto, his pontificate occurred during … question: who gave the mother of alberic ii of spoleto the title "patricia" of rome? small model black-box llms ~13k tokens answer the question based on the given passages. …passage 4:was a roman noblewo who was the alleged mistress of pope sergius iii and was given the unprecedented titles senatrix ("senatoress") and patricia of rome by pope john x. , when ottosys, the of and, were to to discusss rome and other more important, were.us was to the dispute the the of re… who gave the mother of alberic ii of spoleto the title "patricia" of rome? ~2k tokens i budget controller question-aware coarse-grained compression w/ document reordering ii iterative token-level question-aware fine-grained compression w/ dynamic compression ratio 0 distribution alignment iii execution with compressed prompt subsequence recovery response iv figure 2: framework of longllmlingua. gray italic content: as in llmlingua. ## experiments here, we investigate: (1) how effective is longllmlingua? (2) how efficient is longllm-lingua? implementation details in this paper, we use gpt-3.5-turbo-0613 6 and longchat-13b-16k as the target llms, both accessible via openai 7 and huggingface 8 . to ensure stable and reproducible results, we employ greedy decoding and set the temperature to 0 in all experiments. for the small language models used for compression, we apply llama-2-7b-chat foot_6 , which has been aligned by supervised fine-tuning and rlhf. we implement our approach with pytorch 1.13.1 and hugging-face transformers. we set up hyperparameters following llmlingua except for the segment size used in iterative token-level compression set to 200 here. more details are provided in appendix b. ## conclusion we propose longllmlingua to address the three challenges, i.e., higher computational cost, performance reduction, and position bias for llms in long context scenarios. we develop longllmlingua from the perspective of efficient prompt compression, thus reducing computational cost. we further design four components, i.e., a questionaware coarse-to-fine compression method, a document reordering mechanism, dynamic compression ratios, and a subsequence recovery strategy to improve llms' perception of the key informa-tion, with which longllmlingua demonstrate superior performance. experiments on the multidocument qa, multi-hop qa, and long context benchmarks demonstrate that longllmlingua compressed prompt can derive higher performance than original prompts while both api costs for inference and the end-to-end system latency are largely reduced.