DAST : Context-Aware Compression in LLM s via Dynamic Allocation of Soft Tokens (2025)

Abstract: Large Language Models (LLMs) face computational inefficiencies and redundant processing when handling long context inputs, prompting a focus on compression techniques. While existing semantic vector-based compression methods achieve promising performance, these methods fail to account for the intrinsic information density variations between context chunks, instead allocating soft tokens uniformly across context chunks. This uniform distribution inevitably diminishes allocation to information-critical regions. To address this, we propose Dynamic Allocation of Soft Tokens (DAST), a simple yet effective method that leverages the LLM’s intrinsic understanding of contextual relevance to guide compression. DAST combines perplexity-based local information with attention-driven global information to dynamically allocate soft tokens to the informative-rich chunks, enabling effective, context-aware compression. Experimental results across multiple benchmarks demonstrate that DAST surpasses state-of-the-art methods.

## introduction large language models (llms) @xcite @xcite @xcite @xcite @xcite @xcite have demonstrated remarkable performance on long context tasks @xcite @xcite @xcite , excelling capturing complex dependencies and generating coherent responses over extended contexts @xcite @xcite . nevertheless, processing long contexts incurs high computational * indicates equal contribution. early approaches to context compression primarily relied on context pruning or summarization @xcite @xcite , which reduced input length through content removal or rephrasing. however, these methods often compromise semantic integrity through direct modification of the input sequence. recent semantic vector-based methods @xcite @xcite @xcite address this limitation by replacing the original context of length n with m compressed soft tokens (m n), preserving essential information in a more compact representation. although effective, these methods typically append soft tokens at the context terminus or distribute them uniformly, overlooking uneven information density across context chunks. this uniform distribution prevents optimal allocation of compression capacity to informationrich regions. notably, text-pruning-based approaches like longllmlingua @xcite attempt dynamic pruning using external models to estimate tokens importance. however, this external guidance fails to capture the llm's intrinsic understanding of information relevance, creating incompatibility with vector-based methods. ## experiments the experiment's detail are presented in appendix b. next, we want to answer two questions: ## conclusion in this paper, we propose dast, a simple yet effective method that dynamically allocates soft tokens by leveraging the llm's intrinsic perception of information density. by integrating perplexitybased local information and attention-driven global relevance, dast adaptively focuses compression capacity on high-information regions without re-