Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression (2024)

Abstract: Key-value (KV) caching is an important technique to accelerate the inference of large language models (LLMs), but incurs significant memory overhead. To compress the size of KV cache, existing methods often compromise precision or require extra data for calibration, limiting their practicality in LLM deployment. In this paper, we introduce DecoQuant , a novel data-free low-bit quantization technique based on tensor decomposition methods, to effectively compress KV cache. Our core idea is to adjust the outlier distribution of the original matrix by performing tensor decomposition, so that the quantization difficulties are migrated from the matrix to decomposed local tensors. Specially, we find that outliers mainly concentrate on small local tensors, while large tensors tend to have a narrower value range. Based on this finding, we propose to apply low-bit quantization to the large tensor, while maintaining high-precision representation for the small tensor. Furthermore, we utilize the proposed quantization method to compress the KV cache of LLMs to accelerate the inference, and develop an efficient dequantization kernel tailored specifically for DecoQuant. Through extensive experiments, DecoQuant demonstrates remarkable efficiency gains, showcasing up to a 75% reduction in memory footprint while maintaining comparable generation quality.

## introduction large language models (llms) @xcite have made significant strides in advancing the progress of language intelligence. however, these large-sized models often incur higher inference latency, bringing significant challenges to practical deployment. therefore, it is urgent to reduce the running overhead of llms. to optimize the efficiency of llms during the inference process, a commonly used technique is key-value (kv) caching @xcite . in implementation, kv caching involves the storage of historical tokens associated with the attention key and value tensors of each layer, offering accelerated inference by trading increased memory consumption for a reduction in redundant calculations. however, applications of long-content generation, such as story generation and long demonstrations for in-context learning tasks, would lead to a significant increase in the size of the kv cache, resulting in unaffordable storage costs @xcite . in addition, managing a large cache often involves frequent i/o read and write operations, leading to considerable latency. the issue becomes even more severe when i/o operations need to span across multiple machines @xcite . therefore, we need to compress kv cache of large models to optimize the inference process. considering the above issues, considerable efforts have concentrated on kv cache compression to enhance inference efficiency. as a typical approach, recent work @xcite prunes tokens to keep the kv cache within a small size. this approach, while alleviating memory overhead, potentially leads to information loss in long text generation. furthermore, although post-quantization methods preserve all preceding text, low-bit quantization often results in substantial model performance degradation. this is primarily attributed to the common challenge of outlier problems in activation value quantization @xcite . additionally, current quantization techniques still rely on calibration or training @xcite to retain the model performance, thus imposing practical limitations in data-constrained settings (e.g., privacy data). this further highlights the need for a data-free approach to kv cache compression. to effectively quantize the kv cache (essentially activation values), we draw inspiration from smoothquant @xcite , which sug-gests that the issue of outliers can be transferred across multiple modules, by migrating the quantization difficulty to weights. however, unlike smoothquant, we take an improved approach by directly migrating the quantization difficulty by performing matrix decomposition on the activation values themselves, without comprising the precision of the weights. the underlying principle is that matrix decomposition can potentially adjust the outlier distribution of the original matrix @xcite , so that the decomposed local tensors or matrices are easier to quantize. to this end, in this paper, we propose an effective matrix decomposition based quantization method namely decoquant, to alleviate the quantization error due to outliers. our approach is developed based on an important empirical finding: when performing tensor decomposition (i.e., matrix product operator), the value range of the large local tensor (consisting of the major proportion of parameters) becomes narrower, indicating fewer outliers to be resolved in quantization. based on this finding, we propose a local tensor based quantization method, in which we apply low-bit quantization to the large tensor, while maintaining high-precision representation for the small tensor. in this way, we can achieve a lower quantization error when reconstructing the original matrix by multiplying all the local tensors. furthermore, we utilize the proposed quantization method to compress the kv cache of llms to accelerate the inference rate, and further develop an efficient dequantization kernel tailored specifically for decoquant. decoquant provides an effective quantization approach for llms, which can compress kv cache to accelerate the inference rate. it is featured by two major merits, namely (1) fully data-free by eliminating the need for complex calibration mechanisms and (2) highly flexible by supporting the quantization for weights only, activations only as well as both simultaneously. extensive experiments have demonstrated the effectiveness of the proposed approach in reducing the memory consumption of the kv cache and achieving competitive performance. with nearly lossless performance, we can achieve 4-bit kv cache quantization and 8-bit quantization for both weights and activations. ## preliminary in this section, we present the background for our approach about llm inference and quantization. ## methods in this section, we present an effective matrix decomposition based quantization method namely decoquant, to alleviate the quantization error due to outliers. we further utilize this method to quantize the kv cache for efficient inference of llms. ## experiments we mainly evaluate the decoquant on the language modeling task to compare it with other quantization approaches. then we explore its zero-shot generalization ability in open-ended document generation. finally, we quantitatively measure the effect of kv cache compression on system throughput. ## related work in this section, we present related works in three aspects as well as draw distinctions of our approach to existing literature. ## conclusion in this paper, we proposed decoquant, a new datafree quantization method designed specifically for kv cache compression, to improve data generation efficiency. by first decomposing the kv cache matrices into local tensors, our approach only quantized the large local tensor with the major proportion of parameters in low-bit precision while maintained the small tensor in 16-bit precision. this approach can mitigate the quantization difficulty from the original matrix to the small local tensor, which effectively reduces the quantization error in kv cache compression. during inference, we also developed an efficient dequantization technique based on the fused kernel tailored for dequantization of decoquant to accelerate the generation process. extensive experiments have demonstrated the effectiveness of the proposed approach in reducing the memory consumption of the kv cache and achieving competitive performance. for future work, we plan to explore the potential of leveraging decoquant for scenarios where communication overhead plays a dominant role in llm inference, specifically in the splitwise technique where prefilling and decoding phases are in different nodes.