Quadapter: Adapter for GPT -2 Quantization (2022)

Abstract: Transformer language models such as GPT-2 are difficult to quantize because of outliers in the activations leading to a large quantization error. To adapt to the error, one must use quantization-aware training, which entails a fine-tuning process based on the dataset and the training pipeline identical to those for the original model. Pretrained language models, however, often do not grant access to their datasets and training pipelines, forcing us to rely on arbitrary ones for fine-tuning. In that case, it is observed that quantization-aware training overfits the model to the fine-tuning data. To this end introduced is a quantization adapter (Quadapter), a small set of parameters that are learned to make activations quantization-friendly by scaling them channel-wise.For quantization without overfitting, we introduce a quantization adapter (Quadapter), a small set of parameters that are learned to make activations quantization-friendly by scaling them channel-wise. It keeps the model parameters unchanged. By applying our method to the challenging task of quantizing GPT-2, we demonstrate that it effectively prevents the overfitting and improves the quantization performance.

## introduction quantizing a transformer model is not a simple matter due to numerous channel-dependent outliers in activations @xcite . they lead to a large quantization error @xcite , and we observe that the problem is worse in the decoder-only transformers like gpt-2. one solution to the difficulty is quantization-aware training (qat), an approach that fine-tunes the model parameters in response to the numerical error arising from quantization. post-training quantization (ptq) -a counterpart of qat that performs quantization without modifying model parameters -is not powerful enough to cope with the outliers. while qat is effective, it requires the dataset and the training pipeline, and the problem is that they are often inaccessible when dealing with the original pretrained model without any downstream task. one then cannot but use arbitrary fine-tuning data for qat. however, the fine-tuning returns worse accuracies for distributions unseen during training (outof-distribution with regard to fine-tuning; f-ood) despite improving for the training distribution (indistribution with regard to fine-tuning; f-id) @xcite . this is consistent with our observation that qat overfits the model to the finetuning data as in figure 1 . the resulting quantized model therefore has its generality impaired. this violates the premise of a general-purpose language model, which must operate well across various texts of the target language. our hypothesis is that qat incurs the overfitting because it changes all the parameters of the model. this difficulty is much like the research topic of continual learning, where it is important that a model should not forget its past capability when transferring to a new task @xcite . adapter is a strategy to adapt to a new distribution by training only a small number of parameters. it is a popular method to lessen the catastrophic forgetting. we borrow this concept to propose quadapter, a lightweight module to adapt to the quantization error on behalf of the intact original model. the contribution of this work is that we successfully quantize gpt-2, overcoming the large inter-channel variance and the qat overfitting issue with quadapter. to the best of our knowledge, this is the first work to quantize both weights and activations of gpt-2 without the complete training pipeline. ## methods quadapter is simply a set of learnable parameters. on the other hand, the quadapter block represents the actual working mechanism of quadapter, involving two consecutive layers of linear relations, their quantizers, and their associated quadapter instance. the effectiveness of quadapter comes from the interaction amongst the involved components, and from the two-phase training procedure. ## limitations one limitation of quadapter is that it requires two consecutive layers of linear relations. in other words, it can be a mediator only for convolution layers, linear layers, or normalization layers (when followed by a linear or convolution layer), but not if residual connections or nonlinear activation functions intervene. ## conclusions we identify two challenges in quantizing autoregressive transformer language models: the overfitting issue of qat and the inter-channel variance in activations. through experiments, we demonstrate that quadapter not only mitigates the two problems but also serves as an effective ptq technique.