RQT : Hierarchical Residual Quantization for Multi-Model Compression (2025)

Abstract: Delta compression methods focus on efficiently serving multiple uniquely fine-tuned models, each tailored to specific tasks and user requirements. These approaches decompose a fine-tuned LLM into a base model and corresponding delta weights, which are compressed using low-rank or low-bit representations to reduce storage costs. However, their effectiveness is highly sensitive to the magnitude of the model deltas—a factor directly influenced by the scale of the training data. We propose the Residual Quantization Tree (RQT), a hierarchical quantization framework that automatically shares low-bit integer weights across similar fine-tuned models. The RQT construction employs a two-phase greedy algorithm: a bottom-up aggregation of models based on weight matrix similarity, and top-down residual quantization, in which each node optimizes the quantization parameters and then delegates residual errors to child nodes. We evaluate RQT on fine-tuned models across mathematics, coding, chatbot, and Chinese LLMs. The results show that RQT achieves an average accuracy degradation of approximately 3% (comparable to previous 4-bit post-training quantization) while maintaining an effective bitwidth of around 2 bits.

## introduction large language models (llms), such as gpt @xcite and llama @xcite , have demonstrated exceptional performance and are widely applied in various applications, including chatbots @xcite @xcite and coding assistants @xcite @xcite . these models achieve high accuracy in target domains through a two-stage process: pre-training on a large corpus of text data, followed by fine-tuning on task-specific datasets, such 4 3 2 bitwidth 10 20 30 40 50 60 70 average relative accuracy degradation (%) 3.45% gptq quarot omniquant bitdelta gpt-zip ## conclusion in this paper, we introduce the residual quantization tree (rqt), which mitigates the sensitivity of previous delta compression methods to the magnitude of delta values. we also present an efficient two-step greedy approach that separates model assignment from residual quantization. experimental results demonstrate the effectiveness of the rqt when compared to traditional ptq methods and delta-compression methods. even with an average bitwidth of approximately 2 bits, rqt achieves only a 3% average accuracy degradation on finetuned llama-2-7b models across a range of tasks.