Rotate, Clip, and Partition: Towards W 2 A 4 KV 4 Quantization by Integrating Rotation and Learnable Non-uniform Quantizer (2025)

Abstract: We propose Rotate, Clip, and Partition (RCP), a Quantization-Aware Training (QAT) approach that first realizes extreme compression of LLMs with W2A4KV4 (2-bit weight, 4-bit activation, and 4-bit KV-cache) configuration. RCP integrates recent rotation techniques with a novel non-uniform weight quantizer design by theoretically and empirically analyzing the impact of rotation on the non-uniformity of weight distribution. Our weight quantizer, Learnable Direct Partitioning (LDP), introduces learnable parameters to directly learn non-uniform intervals jointly with LLM weights. We also present a GPU kernel supporting GEMV on non-uniform W2A4 as proof of concept. Experiments show that RCP can compress LLaMA-2-7B to W2A4KV4 with a loss of only 2.84 WikiText2 PPL and 5.29 times reduced memory footprint. Furthermore, RCP can quantize challenging mobile-targeted LLaMA-3.2 models and domain-specific WizardCoder-7B and MetaMath-7B with no critical problems such as convergence failure and repetition. Code is available at https://github.com/songsm921/RCP .

## introduction large language models (llms) have made significant advancements, but their growing size and resource demands create challenges for deployment across data centers and mobile devices. to address these constraints, extensive research efforts have focused on improving quantization algorithms. notably, rotation-based post-training quantization (ptq) @xcite @xcite showed remarkable improvement on w4a4kv4 1 quantization, and stateof-the-art quantization-aware training (qat) (du et al., 2024; chen et al., 2024) made extreme weight quantization possible via careful design of datasets and training procedures. in this work, we propose rotate, clip, and partition (rcp), a rotation-based qat algorithm, to push the boundaries of extremely low-bit compression. based on empirical and theoretical analysis, we draw our main insight that rotating llm weights has two effects at once: eliminating outliers and increasing the non-uniformity of the weight distribution. the key component of rcp is learnable direct partitioning (ldp), which is a fully differentiable non-uniform weight quantizer working in three steps: 1) quantization range setup with learnable clipping parameters @xcite ; 2) non-uniform quantization via splitting the quantization range with learnable partitioning parameters; 3) non-uniform dequantization that maps the quantized weights to real-valued grids. rcp is the first to enable challenging w2a4kv4 and w3a4kv4 quantization on common llm models, significantly outperforming quarot @xcite and bitdistiller @xcite . especially, as we present in fig. 1 , rcp also works on small and mobile-targeted llama-3.2 models (ai@meta, 2024) that are harder to quantize. quarot @xcite proposed to apply random rotations while keeping the computational invariance suggested in @xcite . random rotation suppresses activation outliers and helps quantization, successfully achieving w4a4kv4 with minimal performance loss. as in fig. 2 , r 1 rotates each decoder layer's inputs and outputs, with its inverse (r t 1 ) fused into adjacent weights. r 2 and r 4 are required for online rotation of the mha and ffn intermediates, respectively. we factorize r 2 into two orthogonal matrices-r h for the v projection and r ′ h for the self-attention activation-and then apply r t 2 to the out-projection. finally, r 3 rotates q and k vectors after rope, enabling compression of the kv cache without altering attention outputs. ## motivation in this section, we confirm the difficulty of w2a4kv4 quantization by empirical evaluations and justify our key design of non-uniform weight quantizer (in section 4.3) through theoretical analysis on the effect of the rotation technique on weight distribution. ## methodology our proposition is a qat algorithm named rotate, clip, and partition (rcp) that combines the idea of random rotation with our novel learnable direct partitioning (ldp) quantizer. overall, rcp is a self-knowledge distillation @xcite algorithm that solves the following optimization problem: ## conclusion rcp enables weights to be quantized to extreme low-bit precision through learnable non-uniform quantization while harmonizing with rotation to optimize both activations and kv-cache to 4-bit. rcp has achieved the first w2a4kv4 configuration and implemented optimized kernels for inference, facilitating llm serving even in resource-constrained environments.