Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in a wide range of natural language processing tasks when leveraging in-context learning. To mitigate the additional computational and financial costs associated with in-context learning, several prompt compression methods have been proposed to compress the in-context learning prompts. Despite their success, these methods face challenges with transferability due to model-specific compression, or rely on external training data, such as GPT-4. In this paper, we investigate the ability of LLMs to develop a unified compression method that discretizes uninformative tokens, utilizing a self-supervised pre-training technique. By introducing a small number of parameters during the continual pre-training, the proposed Selection-p produces a probability for each input token, indicating whether to preserve or discard it. Experiments show Selection-p achieves state-of-the-art performance across numerous classification tasks, achieving compression rates of up to 10 times while experiencing only a marginal 0.8% decrease in performance. Moreover, it exhibits superior transferability to different models compared to prior work. Additionally, we further analyze how Selection-p helps maintain performance on in-context learning with long contexts.

## introduction in-context learning has shown remarkable success in various natural language processing tasks @xcite , such as classification task @xcite , and mathematical reasoning task @xcite , enabling large language models (llms) to tackle complex and diverse tasks using only few-shot samples. however, in-context learning also significantly extends the length of prompts, resulting in increased computational and financial costs. recently, a line of work has been focusing on prompt compression, which aims to compress the original prompts while minimizing information loss. they are categorized into discrete compression @xcite @xcite and continuous compression @xcite @xcite ; the former compresses the context into discrete tokens, while the latter compresses it into a short sequence of continuous vectors. observing redundant and repetitive content in a given input, discrete compression methods aim to eliminate less informative context without significantly compromising the model's performance. for example, llmlingua @xcite proposes to perform iterative token truncation based on the content perplexity, requiring multi-round decoding @xcite . llmlingua-2 @xcite , distilled from gpt-4 (openai, 2023), addresses the potential misalignment between entropy and the compression objective, as well as the distribution gap between the perplexity of the compression model and the target model. high costs are still involved in the training data construction. meanwhile, optimizing the distribution for specific llms (gpt-4) may, on the other hand, hinder the transferability of the compressed content to other llms. more details are discussed in section 4.4. continuous compression @xcite teaches pre-trained lms the ability to compress text into a short sequence of continuous vectors. autocompressors @xcite uses an unsupervised learning objective, which motivates the model to cache crucial information within the summary vectors. despite their success, these methods have poor generalization as they can only compress to the length specified during training. additionally, since the continuous vector cannot be transferred between models, a separate compressor must be trained for each model. this raises an interesting research question: "can llms learn to identify less informative tokens within a given context without external annotated signals?" to answer this question, we propose a pretraining strategy with a self-supervised signal, which enables the model to autonomously learn to predict the next token based on compressed context. with a small additional number of parameters, a forward pass on the proposed selection-p creates the probability vector p corresponding to each input token, indicating whether to preserve or discard the token. during inference, we can apply the detokenized compressed tokens to any downstream llms with only single-turn decoding and without reliance on any costly external resources as depicted in the main contributions of this work are fourfold: ## methodology under the intuition that redundant texts often exist and their removal does not hinder human understanding of the text, we assume that llms behave in a similar manner. to efficiently identify less informative tokens within a given context, we propose a simple pre-training objective encouraging the model to predict the same token both before and after discarding less informative tokens. areas in orange are learnable parameters. for the input context [x 1 , x 2 , . . . , x n-1 ], inference without parameters update is performed first to create the attention mask p. these subsequently form the model input for lora training and updating the parameters of the additional linear layer. ## conclusion we introduce a simple yet effective self-supervised approach in context compression and conduct evaluation across 10 classification tasks in both fewshot and long-context settings. our approach also demonstrated great transferability to both the open-