| **KVQuant** is a methodology for efficient KV cache quantization that incorporates several innovations to acheive accurate low-precision quantization, | |
| thereby enabling efficient long context length inference. | |
| **TLDR:** KVQuant addresses the memory bottleneck with long context length inference by quantizing the KV cache to low precision. | |
| KVQuant achieves high accuracy with low-precision KV cache quantization by considering several consistent patterns observed in cached KV values across different LLMs, | |
| and by developing methods to exploit these patterns, including: | |
| - **Per-channel, Pre-RoPE** Key quantization to better match the outlier channels in Keys | |
| - Non-Uniform Quantization (**NUQ**) to better represent the non-uniform activations | |
| - **Dense-and-Sparse Quantization** to mitigate the impacts of numerical outliers on quantization difficulty | |
| - **Q-Norm** to mitigate distribution shift at ultra low precisions (eg. 2-bit) | |
| - **Attention-Sink Aware Quantization** to avoid quantization error with the first token, which is disproportionately sensitive to quantization error | |
| For more details please check out our [paper](https://arxiv.org/abs/2401.18079.pdf). | |
| ## Model description | |
| Quantizer file for running DBRX with 2-bit KV cache using KVQuant. | |
| * **Base Model:** [DBRX-Instruct](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm) | |
| * **Bitwidth:** 2-bit | |
| * **Sparsity Level:** 1% | |
| ## Links | |
| * **Paper**: [https://arxiv.org/abs/2401.18079.pdf](https://arxiv.org/abs/2401.18079.pdf) | |
| * **Code**: [https://github.com/SqueezeAILab/KVQuant](https://github.com/SqueezeAILab/KVQuant) | |
| --- | |
| license: mit | |
| --- |