KV Cache Compression Methods
Collection
Methods for compressing the KV cache in transformer inference. E8 lattice, KVarN, TurboQuant, KIVI. • 2 items • Updated
NexusQuant: E8 Lattice KV Cache Compression
Training-free KV cache compression for LLM inference. Uses E8 lattice vector quantization + Hadamard rotation. Calibration-free.
Headline
+0.276% wikitext PPL at 5.83x compression (Mistral-7B). NIAH retrieval preserved through 32K context. Validated on 9 architectures.
Head-to-head (Llama-3.1-8B-Instruct, 4K, n=30)
| Method | bpe | NIAH |
|---|---|---|
| FP16 | 16.0 | 29/30 |
| TurboQuant 2-bit | 2.125 | 0/30 |
| NexusQuant K2V2 | 2.0 | 30/30 |
Install
pip install nexusquant-kv
Usage
from nexusquant import compress_kv_cache
with compress_kv_cache(model, mode="quant_only", bits=2):
output = model.generate(input_ids, max_new_tokens=200)
Links
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support