LLML ingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024)

Abstract: This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. Considering the redundancy in natural language, existing approaches compress prompts by removing tokens or lexical units according to their information entropy obtained from a causal language model such as LLaMa-7B. The challenge is that information entropy may be a suboptimal compression metric: (i) it only leverages unidirectional context and may fail to capture all essential information needed for prompt compression; (ii) it is not aligned with the prompt compression objective.To address these issues, we propose a data distillation procedure to derive knowledge from an LLM to compress prompts without losing crucial information, and meantime, introduce an extractive text compression dataset. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one, and use a Transformer encoder as the base architecture to capture all essential information for prompt compression from the full bidirectional context. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTa-large and mBERT.We evaluate our method on both in-domain and out-of-domain datasets, including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its small size, our model shows significant performance gains over strong baselines and demonstrates robust generalization ability across different LLMs. Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.6x-2.9x with compression ratios of 2x-5x.

## introduction recent years have witnessed the emergence of various prompting techniques for large language models (llms), such as chain-of-thought (cot) @xcite , in-context learning (icl) @xcite , and retrieval augmented generation (rag) @xcite . these techniques empower llms to handle complex and varied tasks through rich and informative prompts that may exceed tens of thousands of tokens. however, the benefits of such lengthy prompts come at a cost of increased computational and financial overhead, as well as the degraded information perception ability of llms. prompt compression is a straightforward solution to address these issues, which attempts to shorten the original prompts without losing essential information. several methods have been proposed to compress prompts in a task-aware manner @xcite @xcite . these techniques aim to generate compressed prompts tailored to the specific task or query, typically resulting in enhanced performance on downstream tasks, particularly in question answering. however, the dependency on task-specific features presents challenges in terms of efficiency and generalizability when deploying these methods. for example, in rag-style applications, it may become necessary to compress the same documents multiple times depending on the associated queries with task-aware prompt compression. more details are discussed in sec. 2. some works have explored task-agnostic prompt compression methods for better generalizability and efficiency @xcite . the underlying assumption is that natural language contains redundancy @xcite that may be useful for human understanding but might not be necessary for llms. therefore, they propose to compress prompts by removing tokens @xcite or lexical units @xcite according to their information entropy obtained from a causal small language model (slm), regardless of the downstream task or question information. however, these task-agnostic methods face two challenges: (i) information entropy is an empirical metric for prompt compression. relying on it for prompt trimming may be suboptimal, as it is not aligned with the prompt compression objective. (ii) causal lms only leverage unidirectional context, which may fail to capture all essential information needed for prompt compression within the context. the challenges lead to the following research questions: q1. how can we identify or build a suitable dataset to align the slm towards effective prompt compression? q2. how can we design a compression algorithm that effectively leverages the full bidirectional context for better performance? for q1, most text compression datasets are abstractive @xcite @xcite , meaning that they treat prompt compression as a generative task where the original prompts are rephrased into condensed ones. however, this autoregressive generation process is slow and it may produce hallucinated content @xcite . on the other hand, extractive compression datasets such as sent-comp @xcite and debatesum @xcite are usually created for the summarization task and often lack detailed information. in the case of prompt compression, this will hurt the performance of llm inference in downstream applications such as qa (see appendix g for some examples). therefore, it is necessary to construct an extractive text compression dataset that retains essential information. contributions. we present this paper to address the above challenges for task-agnostic prompt compression. we make the following contributions. ## related works depending on whether task information is used for compression, prompt compression methods can be categorized into task-aware and task-agnostic compression approaches. task-aware compression compresses the context based on the downstream task or the current query. for example, longllmlingua @xcite applies a question-aware coarse-to-fine compression approach to estimate the information entropy of the tokens and adapts the estimation according to the question. reinforcement learning (rl) based methods @xcite usually train a model for prompt compression with reward signals from downstream tasks. soft prompt tuning methods @xcite typically require fine-tuning for the specific task. @xcite trains a summarization model to compress the context depending on the question. task-aware compression approaches are usually tailored for specific tasks and compression ratios, which may limit their generalizability in real-world applications. task-agnostic methods compress the prompt without considering the specific task, making it more adaptable to a range of applications and blackbox llms. however, producing compressed text that can generalize well to different tasks is not trivial. typical methods involve using information entropy-based metrics to remove redundant information in the prompt @xcite . they employ a small language model to estimate token importance from the information metrics. despite being training-free, these methods may not effectively capture the token importance distribution optimized for specific llms and often entail high computation overhead. summarizationbased methods are also leveraged for task-agnostic compression @xcite . however, they often omit crucial details and do not generalize well. an alternative approach is to compress or trim the context hidden or kv caches @xcite @xcite @xcite . however, this is orthogonal to our work and cannot be easily applied to black-box llms. ## dataset construction in this section, we outline the process of dataset construction for prompt compression. we first introduce our data distillation procedure, which involves extracting knowledge from an llm (gpt-4 ) to compress texts without losing crucial information or introducing hallucinated content (sec. 3.1). leveraging the distilled knowledge from the llm, we explain our data annotation algorithm, which assigns labels to each word in the original text to indicate whether it should be preserved after compression (sec. 3.2). to ensure the dataset's quality, we propose two quality control metrics for filtering low-quality samples (sec. 3.3). ## compressor we formulate prompt compression as a binary token classification problem (i.e., preserve or discard) to guarantee the faithfulness of the compressed prompt to the original content, and meantime ensure the low latency of the compression model itself. for the token classification model, we employ a transformer encoder as the feature extractor to leverage information from the bidirectional contexts of each token. we train the classification model on the dataset constructed in sec. 3 from meetingbank @xcite . during inference, we determine whether to preserve or discard each token in the original prompt based on its probability calculated by our classification model. ## conclusion this paper targets task-agnostic prompt compression for better generalizability and efficiency. in this paper, we identify the challenges encountered in existing methods and address them accordingly. we conduct extensive experiments and analysis on five benchmarks across different tasks and domains. our model shows superiority over strong baselines in terms of performance and compression latency. we publicly release the dataset of text compression with no essential information loss in this paper.