B it D istiller: Unleashing the Potential of Sub-4-Bit LLM s via Self-Distillation (2024)

Abstract: The upscaling of Large Language Models (LLMs) has yielded impressive advances in natural language processing, yet it also poses significant deployment challenges. Weight quantization has emerged as a widely embraced solution to reduce memory and computational demands. This paper introduces BitDistiller, a framework that synergizes Quantization-Aware Training (QAT) with Knowledge Distillation (KD) to boost the performance of LLMs at ultra-low precisions (sub-4-bit). Specifically, BitDistiller first incorporates a tailored asymmetric quantization and clipping technique to maximally preserve the fidelity of quantized weights, and then proposes a novel Confidence-Aware Kullback-Leibler Divergence (CAKLD) objective, which is employed in a self-distillation manner to enable faster convergence and superior model performance. Empirical evaluations demonstrate that BitDistiller significantly surpasses existing methods in both 3-bit and 2-bit configurations on general language understanding and complex reasoning benchmarks. Notably, BitDistiller is shown to be more cost-effective, demanding fewer data and training resources. The code is available at https://github.com/DD-DuDa/BitDistiller.

## introduction scaling up model sizes has been pivotal to the success of large language models (llms), yielding unprecedented performance across diverse natural language processing tasks @xcite @xcite . however, such escalating model size poses significant challenges in deployment, particularly on resourceconstrained devices, due to the substantial memory footprint and computational requirements. weight quantization has emerged as a popular strategy to enhance the efficiency and accessibility of llms by reducing model size with minimal performance loss @xcite . in practice, 2 . in this work, we present bitdistiller, a novel framework that synergizes qat with knowledge distillation (kd) to significantly boost the performance of sub-4-bit quantized llms. to minimize quantization error, bitdistiller employs a tailored asymmetric quantization and clipping strategy to maintain the capabilities of the full-precision model as much as possible, particularly at ultra-low-bit levels. for efficient and effective low-bit representation learning, bitdistiller leverages a simple yet effective self-distillation approach, wherein the full-precision model acts as its own teacher to refine the low-bit student model. notably, bitdistiller innovates with a confidence-aware kullback-leibler divergence (cakld) objective that optimizes knowledge transferring efficacy, enabling faster convergence and enhanced model performance. our empirical evaluations, conducted on a diverse suite of general language understanding and complex reasoning tasks including mathematics and coding, demonstrate that bitdistiller significantly outperforms existing ptq and qat methods in the realm of sub-4-bit quantization. as illustrated in figure 1 , bitdistiller achieves the most favorable scaling law in both 3-bit and 2-bit configurations on the code reasoning benchmark. moreover, bitdistiller is demonstrated to be more costeffective, requiring less training data and fewer training resources, thereby marking a significant advancement toward deploying robust large language models on resource-constrained devices. ## experiments we evaluate bitdistiller on the llama-2 @xcite families and domain-specific llms with sub-4-bit quantization. we have set up comparative experiments to demonstrate the proficiency of our method against existing ptq and qat methods. our findings illustrate that bitdistiller substantially enhances both the general language performance and the accuracy of reasoning tasks. ## conclusion bitdistiller leverages qat with self-distillation to boost sub-4-bit llm performance. the asymmetric quantization and clipping strategies, coupled with the innovative cakld objective, facilitate faster learning and superior performance. bitdistiller outperforms existing ptq and qat methods, achieving notable improvements in 3/2-bit settings across diverse language and reasoning tasks. moreover, bitdistiller is more cost-efficient with fewer data and training resources required.