F lex Q uant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization (2025)

Abstract: The rapid advancement of large language models (LLMs) has exacerbated the memory bottleneck due to the widening gap between model parameter scaling and hardware capabilities. While post-training quantization techniques effectively reduce memory overhead, existing methods predominantly rely on static quantization strategies, which struggle to adapt to dynamic workloads. To address this, we propose FlexQuant, a dynamic precision-switching framework that optimizes the trade-off between inference speed and accuracy. Leveraging model perplexity entropy and Kullback-Leibler divergence, FlexQuant enables fine-grained, layer-wise mixed-precision quantization and dynamically adjusts bit-widths during each token generation. FlexQuant provides a comprehensive analysis of quantization strategies, introduces a precision requirement model for optimal switching, and implements efficient fine-grained precision management. Evaluations demonstrate that FlexQuant achieves a 1.3× end-to-end speedup across diverse language tasks with negligible accuracy loss introduced. This framework offers a flexible and adaptive solution for efficient LLM deployment.

## introduction over the past decade, artificial neural networks (anns), particularly large language models (llms) based on transformer architectures, have witnessed remarkable advancements, sparking profound transformations across various societal sectors. yet, these impressive strides in model capabilities have been achieved alongside an exponential growth in model parameters, which have been expanding at a rate of 240-fold every two years. this this work was done when jinhong xia was an intern at shanghai jiao tong university. junping zhao and li jiang are the corresponding authors. model quantization emerges as a promising technique to alleviate the memory demands, with a plethora of low bit-width quantization algorithms continuously being proposed @xcite . in general, model quantization methodologies fall into two primary categories: quantization-aware training (qat) and post-training quantization (ptq) @xcite . given the substantial overhead associated with training llms, ptq algorithms are predominantly employed to facilitate quantizing these models. ptq based on uniform quantization provides an effective approach for compressing full-precision or half-precision floating-point data into lower bit-width integer representations, such as 8-bit or fewer, thereby significantly reducing memory overhead. to further enhance quantization performance in llms, advanced techniques have been developed to address the challenges posed by anomalous values. activation-aware weight quantization optimizes the quantization process by considering the distribution of activations @xcite , while outlier-aware mixed-precision quantization dynamically adjusts bit-width allocation to better handle outliers @xcite . additionally, the incorporation of approximate second-order information improves quantization accuracy by capturing more precise gradient statistics @xcite . clustering-based non-uniform quantization offers another refinement by grouping similar values and applying tailored quantization levels, thereby mitigating the impact of irregular value distributions @xcite . these prevailing state-of-the-art quantization strategies demonstrate varying degrees of precision and performance efficiency of llm deployment. however, they often resort to a single, invariant quantization strategy throughout model inference, notwithstanding the dynamic nature of token generations. consequently, no single quantization model consistently attains optimal accuracy and speed across diverse workloads. to navigate this challenge, there arises a necessity to judiciously balance latency and precision across different inference stages. this requires a mechanism to dynamically switch between quantization strategies, thereby achieving peak inference speed with negligible accuracy loss. such a solution entails addressing the following three challenges: (1) analyzing the precision and performance of various quantization strategies; (2) determining precision switching points during inference, necessitating robust accuracy requirement modeling; considering the above challenges, the progressive mixed-precision decoding (pmpd) @xcite proposes a gradual decrease in precision needs as token generation advances during decoding. pmpd entails a gradual reduction in the bit-width of weight quantization to strike a balance between accuracy and efficiency demands. the method introduces two precision switching schemes: a prompt-agnostic static scheduler and a task-agnostic learned scheduler. however, it lacks a robust theoretical analysis of precision requirements, leading to heuristic-driven precision switching schedules that fall short of optimal performance. furthermore, the any-precision llm @xcite -based quantization scheme is unable to deliver end-to-end performance enhancements in gpu systems because of inefficient memory access patterns. in view of the defects of pmpd in precision management and memory access, in this paper, we propose a token-wise quantization precision switching framework, named flexquant. flexquant models precision demands during the decoding phase with fine granularity by leveraging model perplexity entropy, facilitating optimal dynamic precision switching. additionally, by integrating layer-wise mixed precision switching based on kullback-leibler (kl) divergence, flexquant offers fine-grained management of the model's effective quantization bit-width, striking an optimal balance between precision and performance. the main contributions of this work can be summarized as follows: ## conclusion in this work, we propose flexquant, a dynamic token-wise quantization framework designed to bridge the gap between the memory demands of llms and modern hardware capacity. flexquant uses token-level perplexity entropy to guide finegrained precision scheduling and applies kldivergence-based layer-wise optimization to ensure global consistency. this dual-level strategy enables adaptive, efficient quantization with minimal accuracy loss, making flexquant well-suited for realtime llm inference.