Style-Compress: An LLM -Based Prompt Compression Framework Considering Task-Specific Styles (2024)

Abstract: Prompt compression condenses contexts while maintaining their informativeness for different usage scenarios. It not only shortens the inference time and reduces computational costs during the usage of large language models, but also lowers expenses when using closed-source models. In a preliminary study, we discover that when instructing language models to compress prompts, different compression styles (e.g., extractive or abstractive) impact performance of compressed prompts on downstream tasks. Building on this insight, we propose Style-Compress, a lightweight framework that adapts a smaller language model to compress prompts for a larger model on a new task without additional training. Our approach iteratively generates and selects effective compressed prompts as task-specific demonstrations through style variation and in-context learning, enabling smaller models to act as efficient compressors with task-specific examples. Style-Compress outperforms two baseline compression models in four tasks: original prompt reconstruction, text summarization, multi-hop QA, and CoT reasoning. In addition, with only 10 samples and 100 queries for adaptation, prompts compressed by Style-Compress achieve performance on par with or better than original prompts at a compression ratio of 0.25 or 0.5.

## introduction the rapid advancement of large language models has revolutionized natural language processing, enhancing both text understanding and generation capabilities @xcite . despite these advancements, the increased size of these models has resulted in longer inference time, higher computational cost, and greater environmental impact. additionally, the token-based fees associated with closed-source models such as gpt-4 * these authors are co-corresponding authors. @xcite and claude (anthropic) further emphasize the need for more efficient solutions. these challenges highlight the necessity for research into efficient inference techniques, with prompt compression emerging as a promising approach to mitigate these issues by reducing the length of input prompts while still maintaining performance. previous research has explored compressing contexts into soft prompts @xcite , however this method is ineffective for black-box models. other studies have focused on task-agnostic prompt compression by identifying and pruning redundant tokens using perplexity as an important metric @xcite . this task-agnostic paradigm naturally encounters difficulties in adjusting to special characteristics of end-tasks. on the other hand, compression methods tailored for certain tasks @xcite require additional training and are challenging to transfer to other tasks. due to these challenges, we aim to propose a training-free prompt compression framework that can automatically adapt a generative prompt compressor to end tasks, with minimal data for adaption. our key intuition is that different tasks prefer compressed prompts with different styles, e.g., extractive or abstractive (as shown in figure 1 ), and the performance of prompt compression can be improved by iteratively learning to compress in the style beneficial to a specific task. based on it, we propose a novel, lightweight framework, style-compress, which adapts smaller language models and compresses prompts for larger models in different downstream tasks without additional training. our approach leverages the generative capabilities of smaller models and enhances their task-specific compression through iterative style variation and incontext learning. specifically, our method involves iteratively generating and selecting representative compressions as demonstrations, enabling smaller for example, the retrieval qa task prefers compressed prompts that are extractive and fluent, whereas cot reasoning favors compressed prompts that are abstractive and maintain a structured format. therefore, we aim to automatically tailor prompt compression to specific downstream tasks from the perspective of styles. our experimental results demonstrate that our method surpasses two baseline compression models across four tasks: original prompt reconstruction, text summarization, multi-hop qa, and cot reasoning. notably, prompts compressed by style-compress achieve performance that matches or exceeds that of original prompts at compression ratio of 0.25 or 0.5, with only 10 samples and 100 queries for adaptation. by improving prompt compression using smaller models, our research offers a valuable pathway towards more efficient and environmentally sustainable use of large language models. ## preliminary study: on the role of styles in prompt compression in this section, we conduct a pilot experiment to explore the following research question: if we instruct llms to compress prompts, would the style of compression affect the performance of compressed prompts on various downstream tasks? styles of compression. we consider five dimensions of styles 1 , and instruct an llm to do prompt compression with specific styles. we present the specific instructions to set these styles in appendix a.2: models and tasks. for starters, we use llama-2 7b @xcite as both the compression and evaluation model, and larger models will be used for evaluation in later. we concatenate the instruction and the original prompt, and feed it to the compression model. we also incorporate two baseline compression models, selective-context @xcite and llm-lingua @xcite , for comparison. we incorporate four tasks: reconstruction, summarization, multi-hop qa and reasoning. details of tasks, evaluation metrics and baseline models will be explained in section 4.1. results. the results of our preliminary study are shown in ## related work the goal of prompt compression is to make the input of language models shorter, therefore accelerate the inference time and reduce the computational cost. other researchers explore to empower efficient inference for some specific tasks: @xcite train extractive and abstractive summarization models to compress retrieved documents. tailered for the in-contect learning task, @xcite predict the number of examples needed for each sample to avoid redundancy. @xcite focus on the role of task definitions in prompts and propose to iteratively remove the unimportant contents in task definitions. ## conclusion this work aims to solve the challenge of enhancing prompt compression for diverse tasks. our preliminary study reveals that styles have an impact in prompt compression. in light of that, enabling the compression model to compress in styles beneficial for different tasks could be a solution for this challenge. we propose style-compress, a training-free framework that adapts a smaller lm to compress prompts for a larger lm across different tasks. we first generate and evaluate effective compressions with diverse styles, then use theses examples to guide the compression model. by learning taskrelevant styles, our model demonstrates improved performance across four tasks. with limited data for adaptation and queries for the larger lm, style-compress is able to compress at compression ratios like 0.25 and 0.5 with minimal or no performance drop on tasks including summarization, multi-hop qa and reasoning.