K ur T ail : Kurtosis-based LLM Quantization (2025)

Abstract: One challenge of quantizing a large language model (LLM) is the presence of outliers. Outliers often make uniform quantization schemes less effective, particularly in extreme cases such as 4-bit quantization. We introduce KurTail, a new post-training quantization (PTQ) scheme that leverages Kurtosis-based rotation to mitigate outliers in the activations of LLMs. Our method optimizes Kurtosis as a measure of tailedness. This approach enables the quantization of weights, activations, and the KV cache in 4 bits. We utilize layer-wise optimization, ensuring memory efficiency. KurTail outperforms existing quantization methods, offering a 13.3% boost in MMLU accuracy and a 15.5% boost in Wiki perplexity compared to QuaRot. It also outperforms SpinQuant with a 2.6% MMLU gain and reduces perplexity by 2.9%, all while reducing the training cost. For comparison, learning the rotation using SpinQuant for Llama3-70B requires at least four NVIDIA H100 80GB GPUs, whereas our method requires only a single GPU, making it more accessible.

## introduction large language models (llms) have advanced significantly in recent years, showcasing remarkable performance and capabilities. as these models grow in size and complexity, the computational cost required for their deployment and inference has increased dramatically. furthermore, with new inference time methods @xcite , enhancing inference speed (tokens per second) is increasingly important. this has shifted the focus toward accelerating model performance while reducing memory and computational requirements. an effective method to achieve this is posttraining quantization (ptq), which involves representing model weights and/or activations in lower * work done during an internship at axelera ai numerical precisions. ptq can significantly reduce the memory footprint and computational overhead and subsequently decrease latency and energy consumption, which are especially beneficial for inference on resource-constrained edge devices. serving a model involves two stages of prefilling and generation. during prefilling, the model processes the input prompt and stores the internal state, known as key-value (kv) caching. during generation, tokens are produced auto-regressively. quantizing each stage offers distinct advantages for improving inference efficiency. kv-cache quantization reduces memory requirements and accelerates data movement, which enhances the generation stage, particularly in scenarios involving long-context inference. weight quantization, on the other hand, reduces the memory footprint independently, and when it is combined with activation quantization, it also reduces the computational demands. however, activation quantization presents challenges due to large outliers in certain channels @xcite , which limits the effectiveness of uniform integer quantization as it destroys the dynamic range of the activations. while channel-wise quantization can effectively address this issue, the lack of hardware support makes it computationally expensive in practice. several methods have been proposed to address this challenge. @xcite and @xcite advocate for mixedprecision computation in which they store some of the channels in higher precision and less sensitive channels in lower precision to balance accuracy and efficiency. @xcite introduces channel-wise scaling into the layer normalization and the weights of linear layers. @xcite proposed random rotation which takes the advantage of the computational invariance framework @xcite to mitigate the outliers problem. we introduce kurtail -a novel approach to mitigating activation outliers by apply-ing learnable rotations 1 to the activations similar to spinquant @xcite . kurtail focuses on reducing the tail density of activations, captured by the kurtosis. unlike spinquant which requires expensive end-to-end training of the model's loss, we prove that layer-wise optimization of our kurtosis loss is equivalent to end-to-end training. we perform layer-wise inference to cache activations, and then optimize the rotations based on the cache independently. as a result, kurtail can be implemented in a significantly more memory-efficient manner. for instance, while spinquant requires at least four nvidia h100 80gb gpus to compute rotations for llama3-70b, kurtail achieves the same with just a single gpu. despite its lower computational requirements, kurtail outperforms existing methods in terms of perplexity and zero-shot reasoning tasks. kurtail outperforms existing quantization methods with a 13.3% increase in mmlu accuracy and a 15.5% decrease in wiki perplexity compared to @xcite . it also performs better than spinquant @xcite , achieving a 2.6% increase in mmlu accuracy and a 2.9% decrease in perplexity, all while reducing the cost of training the rotation. we also theoretically shed light on why rotations are preferable to arbitrary linear transformations. ## background post training quantization. previous work on post-training quantization fits into two main groups: weight-only quantization @xcite @xcite and weight-activation quantization @xcite @xcite . in weight only quantization, the weight are projected into a lower precision, such as 4 bits, 3 bits, or even less, and then de-quantized to higher precision before the actual computation, with all calculations still being done in high precision. several studies @xcite @xcite attempted to introduce quantization methods for both weight and activation. they showed that uniform quantizing is impractical for large language models since they suffer from large outliers. to address this issue, @xcite ; ashkboos et al. (2023); @xcite proposed a mixed-precision approach for handling outliers at higher precision. others @xcite pro1 we use rotation to refer to any orthogonal transformation. ## setup we developed kurtail using the hugging face library @xcite integrated with the pytorch framework @xcite and for evaluation we used eleutherai evaluation framework @xcite . for learning the transformation, we used 512 calibration samples for all models, except mixteral and llama 70b for which we use 256 calibration sample from the wikitext @xcite training set, each with a sequence length of 2048. for large models, we used less samples since they have more layers for which we can store the activations. for storing the activations we used layer-wise inference to reduce the gpu memory requirement. for optimizing the rotation, we use caley adam @xcite optimizer to find the rotation. for quantizing the activation, we used per-token dynamic symmetric quantization, where a single scale was applied to each row, and all values were clipped using a quantile of 0.98 in all experiments. for the kv-caches, we employed asymmetric quantization. for the weight quantization, we use roundto-nearest (rtn), and gptq @xcite , using per-column (or per-channel) symmetric quantization. for gptq quantization, we uses 128 calibration samples from the wikitext, each with a sequence length of 2048. learning the transformation and transforming llama3-70b with kurtail on an nvidia h100 gpu took around one hour which compare the spinquant it uses significantly less memory (4 a100 gpu and 2 hours). models. we evaluate kurtail on the llama-2 (touvron et al., 2023), llama-3 @xcite , phi-model (abdin et al., 2024) and qwen 2.5 @xcite family on both language generation and zero-shot tasks. we further also target the mixture of experts model mixtral @xcite . inference speed-up. kurtail's contribution focuses on a novel approach to learning the rotation and given the architectural similarity with spin-quant and quarot, we did not re-implement the low-level kernel for 4-bit matrix multiplication, as similar speedup results are expected. all results are based on simulated quantization; however, the real quantization will yield the same downstream performance. ## conclusion we introduced kurtail a novel technique for learning orthogonal transformations that rotate the activation distribution to address the outlier problem and overcomes the limitations of previous approaches. compared to quarot, which uses nonlearnable rotation, and spinquant, which requires substantial computational resources for learning rotations, kurtail offers a more efficient and robust solution by learning a rotation that minimizes the kurtosis of the representations. we further provide theoretical insights into why layer-wise optimization yields the same results as end-to-end training, and why orthogonal transformations are a suitable choice of matrix space for learning the transformation. finally, these results highlight kurtail 's ability to deliver efficiency and high performance across large-scale language models. limitations. in this work, we only focuses on dynamic per-token quantization for activations, which offers flexibility but does not fully exploit the potential of static tensor-wise quantization. static quantization, which precomputes scaling factors for improved efficiency, could further optimize inference speed and memory usage. however, it requires careful calibration, which we leave for future work.