From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compression (2024)

Abstract: Large language models (LLMs) have achieved significant performance gains using advanced prompting techniques over various tasks. However, the increasing length of prompts leads to high computational costs and often obscures crucial information. Prompt compression has been proposed to alleviate these issues, but it faces challenges in (i) capturing the global context and (ii) training the compressor effectively. To tackle these challenges, we introduce a novel prompt compression method, namely Reading To Compressing (R2C), utilizing the Fusion-in-Decoder (FiD) architecture to identify the important information in the prompt. Specifically, the cross-attention scores of the FiD are used to discern essential chunks and sentences from the prompt. R2C effectively captures the global context without compromising semantic consistency while detouring the necessity of pseudo-labels for training the compressor. Empirical results show that R2C retains key contexts, enhancing the LLM performance by 6% in out-of-domain evaluations while reducing the prompt length by 80%.

## introduction large language models (llms) have recently exhibited remarkable performance gains in various tasks owing to a wide variety of prompting, e.g., retrieval-augmented generation (rag) @xcite , chain-of-thought (cot) @xcite , and in-context learning (icl) @xcite . the use of rich prompts unlocks the abilities of llms, but prompts can be verbose to deliver sufficient information. the lengthy prompts not only increase computational costs but also make llms struggle to discern important information. although the input length limit has recently been extended to a million tokens @xcite , the quadratic increase in computation over the input length is still a substantial burden. however, existing prompt compression methods face two challenges in handling lengthy inputs. (i) how to extract essential information based on the global context? the key points in a single paragraph may differ significantly from the main topic of the lengthy document. however, existing works divide the prompt into multiple segments if exceeding the maximum input length of compressors and compress each segment independently @xcite . since the model iden-tifies crucial tokens only within the segment, it is limited in capturing essential information across the global context. (ii) how to train a compressor? since the ground truth of the compressed prompt is non-trivial to define, it is difficult to train the compressor. recent work detours the issue by utilizing state-of-theart llms, e.g., gpt-4, for generating the pseudocompressed prompts to train the compressor @xcite . however, as pointed out in @xcite ; @xcite , gpt-4 underperforms in prompt compression, suggesting that there is still room for improvement in training compressors. addressing these issues, we shed light on one prominent solution to handle multiple documents, i.e., fusion-in-decoder (fid) @xcite . fid is a question-answering model aggregating evidence from multiple documents to answer a question. it effectively captures the global context over multiple documents by leveraging crossattention to answer the question, as depicted in figure 1 . it is worth noting that fid effectively highlights salient parts based on the global context to generate the answer regardless of the length of the whole context. to this end, we propose a novel prompt compression method, reading to compressing (r2c), which fully utilizes the structure and training strategy of fid to align with prompt compression. first, the prompt compression is connected with the fid to capture the global context of lengthy input. specifically, lengthy prompts are divided into multiple chunks as input units of fid, and global semantics over chunks are aggregated in the decoder. r2c yields efficiency by utilizing the cross-attention scores computed in generating only the first token, avoiding auto-regressive generation. second, we utilize the question-answering task to train the compressor, a natural way to identify key information without relying on pseudo-labels. the cross-attention scores are trained to align with a question-answering process, providing the effective approximation of tokens that the target llm needs to focus on. we thoroughly conduct experiments on indomain, i.e., natural questions, and out-of-domain datasets, i.e., longbench, demonstrating the effectiveness and efficiency of r2c. notably, r2c yields up to 14.5 times faster compression than existing methods and reduces end-to-end latency by 26% with a 5x compression of the original prompt with minimal performance degradation. ## proposed method this section introduces r2c, a novel prompt compression method using the fusion-in-decoder (fid) architecture. in contrast to the existing compression methods, which consider the local context within each chunk, r2c effectively seizes the global context that lies across chunks in the lengthy prompt. the prompt p c consists of an instruction i, a context c, and a question q, i.e., p @xmath0; c; q), where the context is usually the longest component. r2c compresses the context c to ĉ and reduces the overall prompt length from |p c | to |p ĉ|. figure 2 depicts the overall framework of r2c. first, we divide context c into multiple chunks and feed them into fid to obtain the importance score for each token (section 3.1). next, token scores are aggregated in chunk-and sentence-level for multi-granularity compression (section 3.2). finally, we hierarchically compress the context in a coarse-to-fine manner, i.e., chunk-to-sentence order (section 3.3). note that the training process is omitted in this paper since r2c is not trained for prompt compression. instead, r2c utilizes the trained fid weights for the qa task. ## conclusion in this work, we explored the capabilities of multidocument readers for prompt compression and successfully bridged two different lines of research. we propose a novel prompt compression method, namely reading to compressing (r2c), leveraging the fusion-in-decoder. r2c effectively captures global context and identifies salient information across multiple segments. furthermore, by training the compressor using question-answering datasets, the most influential tokens are identified without using noisy pseudo-labels for compressed prompts. experimental results demonstrate that r2c outperforms existing prompt compression methods by preserving semantic integrity and even surpasses uncompressed prompts by removing ambiguity.