LLML ingua: Compressing Prompts for Accelerated Inference of Large Language Models (2023)

Abstract: Large language models (LLMs) have been applied in various applications due to their astonishing capabilities. With advancements in technologies such as chain-of-thought (CoT) prompting and in-context learning (ICL), the prompts fed to LLMs are becoming increasingly lengthy, even exceeding tens of thousands of tokens. To accelerate model inference and reduce cost, this paper presents LLMLingua, a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity under high compression ratios, a token-level iterative compression algorithm to better model the interdependence between compressed contents, and an instruction tuning based method for distribution alignment between language models. We conduct experiments and analysis over four datasets from different scenarios, i.e., GSM8K, BBH, ShareGPT, and Arxiv-March23; showing that the proposed approach yields state-of-the-art performance and allows for up to 20x compression with little performance loss.

## introduction the widespread adoption of chatgpt has transformed numerous scenarios by harnessing the powerful generalization and reasoning capabilities of large language models (llms). in practical applications, crafting suitable prompts is crucial and usually involves techniques such as chain-ofthought, in-context learning, and retrieving related documents or historical conversations @xcite . while these methods can elicit highly effective generations by activating llms' domain-specific knowledge, they often require longer prompts. therefore, striking a balance between the massive computational demands of llms and the need for longer prompts has become an urgent issue. some studies attempt to accelerate model inference by modifying the parameters of 1 our code is available at https://aka.ms/llmlingua . llms through quantization @xcite , compression @xcite , etc. however, these approaches may be not suitable when the llms can be accessed via apis only. approaches that attempt to reduce the length of original prompts while preserving essential information have emerged lately. these approaches are grounded in the concept that natural language is inherently redundant @xcite and thus can be compressed. @xcite also indicate that llms can effectively reconstruct source code from compressed text descriptions while maintaining a high level of functional accuracy. therefore, we follow this line of studies to compress a long prompt into a shorter one without any gradient flow through the llms to support applications based on a larger range of llms. in terms of information entropy, tokens with lower perplexity (ppl) contribute less to the overall entropy gains of the language model. in other words, removing tokens with lower perplexity has a relatively minor impact on the llm's comprehension of the context. motivated by this, @xcite propose selective-context, which first employs a small language model to compute the selfinformation of each lexical unit (such as sentences, phrases, or tokens) in original prompts, and then drops the less informative content for prompt compression. however, this method not only ignores the interdependence between the compressed contents but also neglects the correspondence between the llm being targeted and the small language model used for prompt compression. this paper proposes llmlingua, a coarseto-fine prompt compression method, to address the aforementioned issues. specifically, we first present a budget controller to dynamically allocate different compression ratios to various components in original prompts such as the instruction, demonstrations, and the question, and meanwhile, perform coarse-grained, demonstration-level compression to maintain semantic integrity under high compression ratios. we further introduce a tokenlevel iterative algorithm for fine-grained prompt compression. compared with selective context, it can better preserve the key information within the prompt by taking into account the conditional dependencies between tokens. additionally, we pose the challenge of distribution discrepancy between the target llm and the small language model used for prompt compression, and further propose an instruction tuning based method to align the distribution of both language models. we validate the effectiveness of our approach on four datasets from different domains, i.e., gsm8k and bbh for reasoning and icl, sharegpt for conversation, and arxiv-march23 for summarization. the results show that our method yields state-ofthe-art performance across the board. furthermore, we conduct extensive experiments and discussions to analyze why our approach attains superior performance. to our best knowledge, we are the first to evaluate reasoning and icl capabilities in the domain of efficient llms. ## problem formulation a prompt compression system is designed to generate a compressed prompt @xmath0, x dems , x que ), where in this section, we elaborate on the proposed coarseto-fine prompt compression approach, llmlingua. first, we introduce a budget controller to dynamically allocate different compression ratios to various components in prompts and meanwhile, perform coarse-grained, demonstration-level compression to maintain semantic integrity under high compression ratios. next, we describe the proposed iterative prompt algorithm designed to retain knowledge from the prompt while compressing. finally, we introduce alignment to address the distribution gap between the small model and black-box large models. figure 1 show the framework. ## conclusion we introduce a coarse-to-fine algorithm for prompt compression, named llmlingua, which is based on the small lm's ppl for black-box llms. our approach consists of three modules: budget controller, iterative token-level compression, and alignment. we validate the effectiveness of our approach on 4 datasets from different domains, i.e., gsm8k, bbh, sharegpt, and arxiv-march23, demonstrating that our method achieves state-ofthe-art performance across all datasets, with up to 20x compression with only a 1.5 point performance drop. moreover, we observe that llms can effectively restore compressed prompts, and prompt compression contributes to a reduction in generated text length. our approach holds substantial practical implications, as it not only reduces computational costs but also offers a potential solution for accommodating longer contexts in llms. the method of compressing prompts has the potential to enhance downstream task performance by compressing longer prompts and to improve the llms's inference efficiency by compressing the kv cache.