HIGGS : Pushing the Limits of Large Language Model Quantization via the Linearity Theorem (2025)

Abstract: Quantizing large language models has become a standard way to reduce their memory and computational costs. Typically, existing methods focus on breaking down the problem into individual layer-wise sub-problems, and minimizing per-layer error, measured via various metrics. Yet, this approach currently lacks theoretical justification and the metrics employed may be sub-optimal. In this paper, we present a “linearity theorem” establishing a direct relationship between the layer-wise reconstruction error and the model perplexity increase due to quantization. This insight enables two novel applications: (1) a simple data-free LLM quantization method using Hadamard rotations and MSE-optimal grids, dubbed HIGGS, which outperforms all prior data-free approaches such as the extremely popular NF4 quantized format, and (2) an optimal solution to the problem of finding non-uniform per-layer quantization levels which match a given compression constraint, obtained by reduction to dynamic programming. On the practical side, we demonstrate improved accuracy-compression trade-offs on Llama-family models, advancing both data-free and non-uniform quantization for large language models.

## introduction quantization has become a standard technique for reducing the memory costs of large language models (llms), e.g. (dettmers et al., 2022; dettmers and zettlemoyer, 2022; frantar et al., 2022; lin et al., 2023; chee et al., 2023; tseng et al., 2024a; † equal contribution ⋄ massachusetts institute of technology ▽ institute of science and technology austria * king abdullah university of science and technology, saudi arabia @xcite . most existing highperformance approaches start from the natural strategy of quantizing layers one-at-a-time, while minimizing a given per-layer error function, such as perlayer quantization entropy @xcite or ℓ 1 -norm error @xcite . in this context, a reasonable question regards the relationship between the individual per-layer quantization error, measured in terms of, e.g., output mse, and the model's output error, measured in terms of, e.g., validation perplexity (ppl). while previous work observes that various layers can have vastly different "sensitivities" towards the model's output @xcite , it is currently not clear how these can be estimated. moreover, it is not clear what the correct error metric quantization techniques should be minimizing. ## background and related work post-training quantization @xcite of llms has become an ex-tremely active research area. here, we provide some background, focusing on the work closest to ours. the focus of early work in llm quantization has been on data-free methods @xcite @xcite using direct round-to-nearest (rtn) quantization over small weight groups. for example, given a group of g consecutive layer weights, viewed as a vector @xmath0, we define b-bit rtn as one key issue with this first wave of data-free rtn methods is that they tend to yield high accuracy loss below 8 bits per parameter. this can be addressed primarily in two ways: (1) by improving the rounding function in a data-aware way, and (2) by using more complex non-uniform grids. ## the linearity theorem this section provides an overview of the linearity theorem, which links the layer-wise l2 error induced by quantization, to the increase in model perplexity, providing a theoretical foundation for weight quantization methods. ## variable bitwidth quantization the second application of the linearity theorem is in dynamic quantization, i.e. choosing per-layer quantization bitwdiths that best reflect the "sensitivity" of different layers to quantization. here, we leverage the observation that uniform biwidth algorithm 3 error coefficient calibration input: calibration constants t1, . . . , tj ; pre-trained model w ⋆ = (w ⋆ 1 , . . . , w ⋆ l ) output: linear coefficients α1, . . . , αl for @xmath1, . . . , l do for @xmath2, . . . , j do ## conclusions we have presented a new result relating the perlayer quantization error with the model's global error, and have applied this result to two problems in llm quantization: accurate data-free quantization and optimal non-uniform compression. our approach leads to state-of-the-art performance for data-free quantization, and is compatible with efficient runtimes @xcite . remarkably, we observe that our approach is robust to being made completely datafree via random sampling; moreover, it appears to outperform popular calibration-based methods in the 3-4 bits/parameter range.