PISCO : Pretty Simple Compression for Retrieval-Augmented Generation (2025)

Abstract: Retrieval-Augmented Generation (RAG) pipelines enhance Large Language Models (LLMs) by retrieving relevant documents, but they face scalability issues due to high inference costs and limited context size. Document compression is a practical solution, but current soft compression methods often suffer from accuracy losses and require extensive pretraining. In this paper, we introduce PISCO, a novel method that achieves a 16x compression rate with minimal accuracy loss (0-3%) across diverse RAG-based question-answering (QA) tasks. Unlike existing approaches, PISCO requires no pretraining or annotated data, relying solely on sequence-level knowledge distillation from document-based questions. With the ability to fine-tune a 7-10B LLM in 24 hours on a single A100 GPU, PISCO offers a highly efficient and scalable solution. We present comprehensive experiments showing that PISCO outperforms existing compression models by 8% in accuracy.

## introduction retrieval-augmented generation (rag) @xcite @xcite pipelines have become a crucial component in addressing various natural language tasks. by incorporating documents retrieved from a selected collection, rag enhances large language models (llms) enabling them to provide more accurate, current, and domain-specific responses. the primary drawback is the increased inference cost, which scales quadratically with the number of tokens and, consequently, with the number of retrieved documents when using transformer-based architectures. in addition to inference costs, the limitations on llm context size restrict the number of documents-and thus the amount of information-that can be utilized. this constrains the potential scaling of inference time @xcite . 1 code and models will be released soon. compressing documents is a practical way to reduce the computational burden of processing large contexts. hard compression techniques focus on altering the surface structure of the documents, such as by pruning @xcite @xcite or summarization @xcite . these methods are easily interpretable and can typically be applied to any llm without requiring modifications. however, the compression rate is limited by the amount of information that can be effectively conveyed through text tokens, usually achieving a reduction of 2x-5x. soft compression techniques aim to condense documents into vector representations @xcite . they may also involve attention keyvalue pairs that the model attends to during generation, either through self-attention @xcite or dedicated cross-attention mechanisms @xcite . these methods trade off interpretability for efficiency, achieving higher compression rates while maintaining some performance levels. most existing soft compression approaches for rag follow a similar pipeline compression lora adapters ... ce loss decoding lora adapters query what is the colorless spirit produced in the winemaking regions of peru and chile? document collection teacher llm the colorless spirit produced in peru and chile is called pisco. distillation retrieval memory tokens documents embeddings ... compression ... ... rag prompt with docs and query student llm student llm a large quantity of a spirit known as pisco ... made by distilling fermented grape juice into.. ## related work ## experiments our experiments aim to measure the performance of pisco models §4.2, then to analyse the importance of training data, distillation and pretraining §4.3. furthermore, we evaluate pisco models generalization §4.4 to out-of-domain, multilingual data and large number of documents. finally, we investigate how information is stored within the document embeddings §4.5. ## conclusion we proposed pisco, the first compression method for rag which enables large compression rates with little to no accuracy loss. our analysis and ablations revealed the ineffective transfer between pretraining and the question answering task for compression models. we also showed the importance of training labels for compression: using an appropriate teacher llm for distillation is key. given the strong evidences of robustness and accuracy, pisco models may be used as drop-in replacement for existing rag systems currently relying on their uncompressed backbones. adopting pisco would reduce inference costs and latency with minimal performance impact.