Achieving binary weight and activation for LLM s using Post-Training Quantization (2025)

Abstract: Quantizing large language models (LLMs) to 1-bit precision significantly reduces computational costs, but existing quantization techniques suffer from noticeable performance degradation when using weight and activation precisions below 4 bits (W4A4). In this paper, we propose a post-training quantization framework with W(1+1)A(1×4) configuration, where weights are quantized to 1 bit with an additional 1 bit for fine-grain grouping and activations are quantized to 1 bit with a 4-fold increase in the number of channels. For weight quantization, we propose utilizing Hessian-aware fine-grained grouping along with an EM-based quantization scheme. For activation quantization, we decompose INT4-quantized activations into a 4 × INT1 format equivalently and simultaneously smooth the scaling factors based on quantization errors, which further reduces the quantization errors in activations. Our method surpasses state-of-the-art (SOTA) LLM quantization baselines on W2A4 across multiple tasks, pushing the boundaries of existing LLM quantization methods toward fully binarized models. Code is available at https://github.com/JimmyCrave/LLM-PTQ-binarization.

## introduction the enormous computational and memory overhead limits the widespread adoption of largelanguage models (llms). model quantization methods @xcite is a major approach to alleviate these challenges. recent studies @xcite on post-training quantization techniques (ptq) can attain nearly lossless quantization under the w4a4 configuration. however, lower-bit quantization of llms remains challenging. existing methods @xcite still suffer from the performance collapse issue when the number of * corresponding author. figure 1 : the perplexity (↓) of gptq, quarot, atom, and our method on the llama1-7b model under various quantization bit-width settings reveals that gptq, quarot, and atom exhibit significant performance degradation when weights are quantized to 1 bit, whereas our method still demonstrates language generation capabilities close to those of the original model. most of the ptq llm quantization work adopts the round-to-nearest (rtn) method @xcite as the implementation scheme in practical operations, focusing primarily on how to preprocess the data for rtn quantization. for example, llm.int8 @xcite and atom @xcite handle outliers with high precision, effectively increasing the numerator of the scaling factor for some key values to reduce quantization error. another line of works, including smoothquant @xcite , quarot @xcite , and duquant @xcite smooth the distribution of data to be quantized via matrix rotation. although these works can achieve satisfactory 4 bit quantization, they struggle to further reduce the quantization bit-width, which implies that the simple rtn quantization method is not sufficient for very low-bit quantization with high performance. another line of works adopt vector quantiza-tion technique @xcite . these approaches quantized the model weights via grouping the weights and assigning the weights in the same group to the same value. those methods usually yield more accurate quantization models with smaller size, but cannot be speed-up the inference speed, as they have to dequantization the weight to recover float point format before computation. to address these issues, we propose a quantization framework, that can simultaneously achieve small model size, fast computation, and highquality inferece outputs. it consists of three parts: 1-bit weight quantization with another bit for finegrain grouping, and 1 × 4 bits activation quantization using 4 times 1 bit channel to represent 4 bit quantization. by this way, we are able to compute the inner loop vector product in pure binary operations drastically boosting the inference speed as well as reduce the model size. the main contributions of this work are as follows: ## related works considerable research efforts have been devoted to the low-resource computation of llms @xcite . existing works have achieved near-lossless quantization at w4a4 through methods such as mixed-precision quantization @xcite and outlier smoothing using rotation factors @xcite . however, these methods encounter difficulties when attempting to push towards even lower-bit quantization below 4 bits. recently, researchers start to seek binarization methods @xcite for llms. one line of works are based on quantization-aware training (qat) method. for example, bitnet b1.58 @xcite , bitnet a4.8 @xcite , onebit @xcite , and fbi-llm @xcite have designed 1-bit transformer architectures specifically for llms, replacing original linear layers with specific quantized version. these methods can reduce the quantization bit-width of weights to 1 bit and demonstrate significant advantages over baseline models. however, these qat methods require substantial computational resources, making it impractical for the quantization of llms. another line of works explored the possibility of binarizing llms within the ptq framework. billm @xcite and stbllm @xcite , inspired by the distribution characteristics of weight values and the hessian matrix, adopt binary residual approximations for significant weights and optimal partitioning for group binarization of non-significant weights, which pushes the quantization boundary of llm weights down to 1 bit. however, these works neglect the quantization of activation values, which obstructs the computational acceleration of the quantized model and ultimately results in relatively slow inference speed. most recently, people start to explore fully binarized models for both weight and activation. the works, abq-llm @xcite and qbb @xcite , propose the idea of decomposing a high-bit matrix into a set of binary matrices for accelerated computation. these methods hold promise for achieving int1 computational acceleration but are constrained by the limitations of quantization bit-width and performance, preventing further improvements. compared with state-of-the-art (sota) w4a4 quantization efforts, our w(1+1)a(1×4) quantized model can further reduce computational and memory overhead, which further pushes the boundaries of llm quantization and enriches the research on llm binarization. ## method in this section, we introduce our binarized weight and activation (bwa) approach, a paradigm de-figure 2 : binarized weight and activation attention signed to simultaneously accelerate inference speed and reduce memory footprint in large language models. we begin by outlining the architecture, and then describe an em-based method for computing binarized weights with two-level grouping structure, and conclude by discussing the strategies in these work and other relevant methods. ## experiments setup. we implemented our method on the py-torch @xcite framework, where all linear layer weights in the original model are quantized to 1+1 bit, and input activations of all linear layers are quantized to 1×4 bits. for weights, we adopt per-channel asymmetric quantization with a clipping ratio set to 1.0 across all experiments, utilizing the gptq quantization framework to compensate for quantization errors. for activations, we employ per-token asymmetric quantization with a clipping ratio of 1.0. to optimize performance, we use rtn for dynamic quantization of the activation matrix. for kv caches, we uniformly apply 4 bits quantization to store and load. the quantization group size is 128, and the number of outlier channels is 128 (approximately 3% of all channels). we use 128 random samples from the wikitext2 @xcite training set as the calibration dataset, with a sequence length of 2048. all experiments were conducted more than three times, and the average values were recorded. models and datasets. we apply our method to the open-source llama1 (7b, 13b) @xcite , llama2 (7b, 13b) @xcite , and vicuna (7b, 13b) @xcite models and evaluate their performance on language generation, commonsense qa, and language understanding tasks. the primary metric for language generation tasks is perplexity, assessed on datasets including wikitext2, ptb @xcite , and c4 @xcite . for commonsense qa tasks, the main metric is zero-shot accuracy, evaluated on datasets such as piqa @xcite , arc @xcite , boolq @xcite , hellaswag @xcite , and winogrande @xcite . except for the c4 dataset, where we randomly select 256 samples of length 2048 from the test set for evaluation, we utilize the entire test set portion of these datasets for our testing. for language understanding tasks, we mainly tested our method on the massive multitask language understanding (mmlu) @xcite benchmark. baseline. we compare our approach with stateof-the-art (sota) ptq methods for weights and activations. since few existing methods explore the w2a4 quantization setting, we implement w2a4 quantization for all compared methods to ensure fairness before evaluation. our main baselines include atom @xcite , quarot @xcite , and billm @xcite . atom and quarot are sota methods under the w4a4 quantization setting, while billm is the sota for the w(1+1)a16 quantization setting. ## limitations although our our method can achieve fully binarized computation to reduce computational overhead during inference, a trade-off must be made regarding the accuracy of the quantized model. consequently, we need to utilize additional bits to store the information of the quantized matrices, with the actual storage bits for weights and activations equivalent to 2 bits and 4 bits, respectively. this implies that our model has not been compressed to the theoretical extreme of a boolean model, leaving room for further improvement. meanwhile, although the performance of our our method across various evaluation tasks is close to that of the prequantized model, it is not truly lossless quantization. this loss indicates that the quantized model has not fully restored the representational capacity of the original model. in the future, we consider employing methods that integrate quantization-aware training to further enhance the efficiency and performance of our method. we use ⊙ to denote element-wise multiplication of matrices and a fraction to represent element-wise division. x fp16 stands for the original value matrix of the activations before quantization. the value of i ranges from 0 to 3. based on the equation 11, we minimize the first-order overall quantization error e to zero while preserving the distribution of quantized values in the original low-bit setting, thereby further enhancing the performance of the quantized model.