KV-Cache-Compression-Lab: Systematic Analysis of KV-Cache Quantization for Efficient LLM Inference

Author: Samudhan S (@samudhan2003)
Portfolio: Part of 5-project Efficient Inference Series

Abstract

The Key-Value (KV) cache is a major memory bottleneck in autoregressive LLM inference, scaling linearly with sequence length and batch size. This lab presents a systematic empirical analysis of KV-cache compression techniques (FP16 baseline vs INT8 vs INT4) and their interaction with weight quantization, measuring the trade-off between VRAM reduction, latency improvement, and quality degradation on sub-1B scale models.

Motivation

During decoding, KV-cache memory often exceeds model weight memory:

• For Llama-2