Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models (2023)

Abstract: We investigate the effects of post-training quantization and quantization-aware training on the generalization of Transformer language models. We present a new method called self-distilled quantization (SDQ) that minimizes accumulative quantization errors and outperforms baselines. We apply SDQ to multilingual models XLM-R _{\text{Base}} and InfoXLM _{\text{Base}} and demonstrate that both models can be reduced from 32-bit floating point weights to 8-bit integer weights while maintaining a high level of performance on the XGLUE benchmark. Our results also highlight the challenges of quantizing multilingual models, which must generalize to languages they were not fine-tuned on.

## introduction a main aim of neural network quantization is to reduce the size and computational demands of a model while maintaining its performance. there are two main approaches: quantization-aware training (qat) @xcite @xcite @xcite and post-training quantization (ptq) @xcite @xcite . both of these approaches have limitations in terms of dealing with accumulative quantization errors that are propogated within the layers of a neural network during the forward pass @xcite . to address this issue, we propose a method called self-distilled quantization (sdq) that combines selfattention and output distillation with quantization to compress large language models. sdq involves injecting quantization noise into the student network during training and distilling knowledge from a fine-tuned teacher network from both its final output and outputs of intermediate self-attention layers. by distilling knowledge of the self-attention layers, as depicted in figure 1 , we further reduce the compounding effect of quantization errors in ## methodology we begin by defining a dataset d := {(x i , y i )} d @xmath0, y i ), where each x i := (x 1 , . . . , x n ) and x @xmath1. for structured prediction y @xmath2, 1} n ×dy and for single and pairwise sentence classification, y @xmath3, 1} dy , where d y is the number of classes. let y @xmath4:= {w l , b l } l @xmath5. the loss function for standard classification fine-tuning is defined as the cross-entropy loss ℓ ce (y s , y). self-distilled quantization for self-distilled quantization, we also require a fine-tuned teacher network f θ , that has been tuned from the pretrained state f θ , to retrieve the soft teacher labels y t := f θ (x), where y @xmath6. the soft label y t can be more informative than the one-hot targets y used for standard classification as they implicitly approximate pairwise class similarities through logit probabilities. the kullbeck-leibler divergence (kld) ℓ kld is then used with the main task cross-entropy loss ℓ ce to express ℓ sdq kld as shown in equation 2 , ## empirical results we begin by referring the reader to the supplementary material for the experimental setup in subsection a.2 and subsection a.3. before discussing the main results on xglue, we first analyse the mean absolute quantization error and the frobenius norm of the elementwise difference in selfattention blocks between an int-8 dynamically quantized infoxlm base and an unquantized fp-32 infoxlm base in figure 2 . we see in figure 2a that the output layer contains the largest mean absolute error across each layer and highest error variance. in contrast, the query, key and value (qkv) parameters have much smaller error. however, since most of the parameters are found in the qkv layers, the sum of the quantization error is larger, as seen in figure 2b . this motivates us to focus on the output of the self-attention block when minimizing quantization errors with our proposed loss in equation 2 as the mean error is higher near the output as it accumulates errors from previous layers in the block. this is also reflected in the parameter distribution of each layer type across all layers in figure 3 , where the x-axis is the mean absolute quantization error and the y-axis is the layer indices. we see the quantization noise is more apparent on the output layer as the gaussian distrbutions are non-smooth and have clear jitter effect. ## conclusion in this paper we proposed an attention-based distillation that minimizes accumulative quantization errors in fine-tuned masked language models. we identified that most of the quantization errors accumulate at the output of self-attention blocks and the parameter distribution of the output layer is effected more by quantization noise. the proposed distillation loss outperforms baseline distillation without the attention loss and the resulting int-8 models are within 1 understanding score points on the xglue benchmark with real quantization post-training. moreover, fine-tuning the teacher network with quantization-aware training can further improve student network performance on some of the tasks. further compression can be achieved up to 4-bit and 2-bit weights but performance steeply degrades as the network capacity is drastically reduced coupled with the models having to generalize to multiple languages it was not trained on.