Learning to Compress Prompt in Natural Language Formats (2024)

Abstract: Large language models (LLMs) are great at processing multiple natural language processing tasks, but their abilities are constrained by inferior performance with long context, slow inference speed, and the high cost of computing the results. Deploying LLMs with precise and informative context helps users process large-scale datasets more effectively and cost-efficiently. Existing works rely on compressing long prompt contexts into soft prompts. However, soft prompt compression encounters limitations in transferability across different LLMs, especially API-based LLMs. To this end, this work aims to compress lengthy prompts in the form of natural language with LLM transferability. This poses two challenges: (i) Natural Language (NL) prompts are incompatible with back-propagation, and (ii) NL prompts lack flexibility in imposing length constraints. In this work, we propose a Natural Language Prompt Encapsulation (Nano-Capsulator) framework compressing original prompts into NL formatted Capsule Prompt while maintaining prompt utility and transferability. Specifically, to tackle the first challenge, the Nano-Capsulator is optimized by a reward function that interacts with the proposed semantics preserving loss. To address the second question, the Nano-Capsulator is optimized by a reward function featuring length constraints. Experimental results demonstrate that the Capsule Prompt can reduce 81.4% of the original length, decrease inference latency up to 4.5x, and save 80.1% of budget overheads while providing transferability across diverse LLMs and different datasets.

## introduction large language models (llms) have demonstrated substantial proficiency across a variety of natural language processing tasks. despite their significant potential and broad adoption, llms are fundamentally limited by the long context length input, which impairs their capability to understand lengthy documents and affects their efficiency during inference @xcite @xcite . as the demand for processing millions of tokens increases, it is progressively crucial to deploy llms that are adept at comprehending extended lengths while minimizing budgetary requirements. to help llms better process long context knowledge, recent advancements have focused on compressing long prompt contexts into concise soft prompts. this approach effectively transforms the original extensive prompt into a manageable series of short-length soft prompt tokens. generally, compression-oriented soft prompts are learned with the guarantee of semantics through selfinformation @xcite , instruction finetuning @xcite , and the performance alignment via knowledge distillation @xcite . however, with the rapid evolution and the growth of api-based accessibility of llms, soft prompts pose significant limitations in terms of transferability across different llms, implying that welltrained soft prompts can only be effectively adapted to the specific llms for which they were designed. this situation creates a critical need to achieve both transferability and utility effectively. a natural question is raised: can we compress lengthy prompts in a natural language format, yet still preserve utility and ensure transferability among various llms? compressing extended prompts into a shorter, natural language (nl) format continues to be a challenging and unresolved issue. as depicted in figure 1 , effective prompt compression entails preserving essential semantic information in a constrained length with successful performance preservation. however, unlike soft prompts, which can be directly optimized with a fixed length, compressing prompts into shorter nl prompts is challenging for several reasons: (i) nl prompts are incompatible with back-propagation, as the gradient cannot backward to a discrete raw text; (ii) nl prompts lack flexibility on imposing strict length constraints, where overly stringent limitations on generation length may lead to performance degradation. thus, it is nontrivial to compress lengthy prompts into shorter nl ones. to tackle the aforementioned problems, we propose a natural language prompt encapsulation (nano-capsulator) framework to effectively compress original prompts into a capsule prompt with the aid of a rewarding technique. our proposed nano-capsulator aims to encapsulate long prompts into shorter ones under specific generation length constraints, maintaining performance through an explicit semantic preservation objective with reward scores. specifically, we compress our prompt by employing a semantics-preserving summarization, and then monitor the optimization process using reward scores that reflect the remaining information relevant to the downstream task. notably, shorter capsule prompts, characterized by their concise nl formatting, preserve transferability and utility across diverse llms. capsule prompt enables two advantages: the preservation of prompt transferability and utility across different llms, and the reduction of inference time and budget overheads. additionally, nano-capsulator can be directly applied to unseen datasets without any further training, provided these new datasets encompass downstream tasks with similar domains. to assess the effectiveness of nano-capsulator, we conduct compression experiments with two different prompt types: few-shot demonstration chainof-thoughts (cot) and passage prompts of reading comprehension (i.e., content passages). capsule prompt exhibits strong transferability across different llms and similar but unseen downstream datasets. this enables effective adaptation without retraining the prompt compressor. our main contributions are concluded as follows: first, we introduce and formalize nano-capsulator framework, which can effectively generate highquality capsule prompt with prominent transferability across multiple llms and unseen datasets with similar downstream tasks. second, we effectively reduce the original prompts to 81.4% of their initial length and transform them into nl-formatted capsule prompt, which retains the prompt's transferability and utility across various llms. our compression mechanism significantly decreases up to 4.5× of the inference latency and saves 80.1% of the budget overheads for input sequences. third, experimental results demonstrate that capsule prompt can efficiently perform across diverse llms, which is applicable to both few-shot demonstration cot and input contextual prompts. ## long prompt encapsulation we systematically introduce the nano-capsulator framework in this section. figure 2 illustrates the overall pipeline of nano-capsulator. in particular, our pipeline initially compresses text into nlformatted capsule prompt and concurrently optimizes their utility using the proposed rewarding method. the design of the nl-formatted compression aims to maintain prompt utility and preserve transferability among different llms. ## experiments in this section, we conduct experiments to evaluate the performance of nano-capsulator, aiming to answer the following three research questions: ## conclusion our work introduces natural language prompt encapsulation (nano-capsulator), a framework for effectively compressing long prompts for lms while preserving essential information. nano-capsulator alleviates the context length limitations of llms, enhancing processing efficiency and cost-effectiveness. our results show that nano-capsulator reduces prompt lengths by 81.4%, decreases inference latency by up to 4.5×, and cuts budget overheads by 80.1% with almost identical accuracy and relevance. this demonstrates its significant potential for improving llm efficiency across various applications that utilize long input documents. future research will focus on refining nano-capsulator for broader domain applications and exploring its usage in data-intensive fields.