Enhancing RAG Efficiency with Adaptive Context Compression (2025)

Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge but incurs significant inference costs due to lengthy retrieved contexts. While context compression mitigates this issue, existing methods apply fixed compression rates—over-compressing simple queries or under-compressing complex ones. We propose Adaptive Context Compression for RAG (ACC-RAG), a framework that dynamically adjusts compression rates based on input complexity, optimizing inference efficiency without loss of accuracy. ACC-RAG combines a hierarchical compressor (for multi-granular embeddings) with a context selector to retain minimal sufficient information, akin to human skimming. Evaluated on Wikipedia and five QA datasets, ACC-RAG outperforms fixed-rate methods and unlocks >4× faster inference versus standard RAG while maintaining or improving accuracy.

## introduction large language models (llms) are pre-trained on massive datasets, encoding vast knowledge within billions of parameters. while they excel at many tasks, their parametric knowledge often falls short for knowledge-intensive applications. retrieval-augmented generation (rag) addresses this limitation by extending the model's knowledge boundaries through external context retrieval @xcite . however, integrating lengthy retrieved contexts into prompts increases inference costs and may exceed llms' context window limits @xcite . context compression mitigates this issue by transforming long contexts into shorter input sequences. existing methods fall into two categories: (1) lexical-based compression, which reduces input length by preserving key tokens @xcite what album is sacrifice by elton john on? document: sacrifice (song) is a ballad performed by musician elton john. the song appears on the 1989 album "sleeping with the past" … output: sleeping with the past vanilla rag document: with context compression output: too low for zero document: output: sleeping with the past [x] [x] [x] [x] [x] [x] [x] [x] [x] document: output: sleeping with the past with adaptive context compression redundant document embeddings insufficient document embeddings perfect document embeddings too long document [x] [x] [x] [x] [x] [x] [x] [x] @xcite ; and (2) embedding-based compression, which encodes text into dense embeddings @xcite for inference. embedding-based methods have been proven more efficient and effective @xcite , typically employing a compressor trained in two phases: first through pretraining (e.g., via autoencoding or language modeling) to preserve contextual information @xcite , followed by fine-tuning (e.g., with instruction follow-up or self-distillation) to adapt to downstream tasks @xcite . existing embedding-based approaches fix the compression rate (i.e., token-to-embedding ratio), leading to trade-offs: high rates risk losing essential information, while low rates retain redundancies (figure 1 ). additionally, inconsistent evaluation benchmarks-varying training data scales and tasks across baselines-hinder fair comparisons. we propose an adaptive context compression framework for rag that dynamically adjusts the number of compressed embeddings during inference. the framework decouples offline compression (fixed-rate hierarchical embeddings) from on-line selection (dynamic embedding feeding, halted once sufficient context is accumulated), mimicking human selective reading. we establish a unified benchmark (wikipedia corpus + five qa datasets) to ensure fair evaluation. our method outperforms other compression techniques in effectiveness and efficiency, matches or exceeds standard rag accuracy on four datasets with 4× faster inference, and demonstrates significant potential for adaptive compression via compressor analysis. in summary, our contributions include: (1) a novel adaptive context compression framework for rag, improving inference efficiency while maintaining retrieval augmentation benefits. (2) training strategies for a hierarchical compressor and adaptive selector, enabling multi-granularity compressed embedding generation and automatic context selection. (3) a unified benchmark for context compression methods, resolving evaluation biases from inconsistent training data and task setups in prior works. (4) experimental results showing our method as the best under context compression, with comparable results to regular rag but with improved efficiency. (5) further analysis highlighting the potential of adaptive context compression. ## adaptive context compression for rag a rag framework typically consists of a retriever r and a decoder d. the retriever builds a search index i containing dense representations of all documents from a collection. for a given query q, r retrieve the top-k relevant documents @xmath0, d 2 , . . . , d k } by matching q's embedding against i via similarity search. the query q and documents d are concatenated and fed into the decoder d to generate a response r. this increases the input size to the decoder, reducing inference efficiency as |d| ≫ |q|. existing embedding-based compression methods employ a fixed-rate compressor to convert documents into embedding sequences, which are subsequently fed into the decoder for answer generation instead of the original documents. however, this fixed-ratio approach lacks adaptability to query complexity. the acc-rag framework … … … … compressor … decoder question … … … … selector … doc emb 1 who sings the song let me love you? answer justin bieber discard doc emb 2 "let me love you" is a song recorded by french dj and electronic music producer dj snake featuring vocals from canadian singer justin bieber. the song was released … document 1 document embeddings question embeddings last-layer hidden states concatenate trainable frozen insufficient sufficient retriever retriever figure 2: overview of adaptive context compression for rag pipeline. the workflow is as follow: context retrieval -→ hierarchical compression -→ adaptive context selection -→ response generation. ## conclusion in this paper, we propose adaptive context compression for rag, a framework that dynamically adjusts compressed embeddings required during inference. the framework effectively improves inference efficiency while maintaining the quality benefits of retrieval. experimental results under standardized benchmark show that our method achieves the best results under context compression and comparable or superior accuracy with over 4× inference efficiency compared to rag.