Compression of Generative Pre-trained Language Models via Quantization (2022)

Abstract: The increasing size of generative Pre-trained Language Models (PLMs) have greatly increased the demand for model compression. Despite various methods to compress BERT or its variants, there are few attempts to compress generative PLMs, and the underlying difficulty remains unclear. In this paper, we compress generative PLMs by quantization. We find that previous quantization methods fail on generative tasks due to the homogeneous word embeddings caused by reduced capacity and the varied distribution of weights. Correspondingly, we propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different modules. Empirical results on various tasks show that our proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. With comparable performance with the full-precision models, we achieve 14.4x and 13.4x compression rate on GPT-2 and BART, respectively.

## introduction transformer-based generative pre-trained language models (plms) show strong abilities of multitask and few-shot learning, and achieve remarkable performances on various tasks @xcite @xcite @xcite . however, they are usually expensive in terms of both computation and memory due to a large number of parameters, and the token-by-token generation process. many methods have been proposed to compress plms, but mostly focus on understanding tasks like sentence classification with bert @xcite @xcite @xcite . recent works try to compress gpt-2 using tensor decomposition @xcite , and knowledge distillation @xcite , but the compression ratio achieved is much smaller than that of bert. yet the underlying difficulty remains unclear. in this paper, we firstly explore compressing generative plms by quantizing the parameters from full-precision to lower bits. we find that directly applying previous quantization methods designed for bert or computer vision tasks to generative plms lead to poor performance. figure 1 shows that the performance drops sharply as the weight bit-width decreases. to investigate the difficulty of quantizing generative plms, we find that the learned embeddings tend to be homogeneous and hard to distinguish due to the reduced capacity caused by quantization, while the weight distributions also vary significantly across different modules and different transformer layers. these problems are further magnified due to the nature of sequential left-to-right prediction of generative plms, as the quantization error will accumulate across time. to alleviate the above problems, we propose a token-level contrastive distillation to contrast on tokens and make the word embedding distinguishable. besides, we propose a module-wise dynamic scaling for the quantizer to better adapt to different modules. empirical results on language modeling, next utterance prediction and summarization show that compared to the full-precision baseline, our quantized gpt and bart (abbreviated as quant-gpt and quantbart) achieve comparable performance for 8/4-bit weight, and have only a slight drop for 2-bit weight, while being over 13× smaller. ## difficulty of qunatizing generative pre-trained language models in this section, we show that it is challenging to train a low-bit generative pre-trained model with conventional quantization approaches directly. before diving into details, we first review the necessary backgrounds of quantization. ## proposed method based on the observations in section 2.2, we propose a quantization method which utilizes tokenlevel contrastive distillation to make the word embedding distinguishable (section 3.1) and a modulewise dynamic scaling adjustment to learn better clipping factors (section 3.2). ## conclusion this paper studies low-bit quantization of generative plms. we find that the difficulty of quantizing generative plms lies in homogeneous word embedding and varied distribution of weights. to alleviate the two problems, we propose token-level contrastive learning to learn more distinguishable token emebeddings, as well as a module-dependent dynamic scaling for more accurate quantization. extensive experiments on language modeling, next utterance prediction and abstractive summarization demonstrate the efficacy of our proposed method. we hope our work sheds a light on the compression of generative plms in future exploration.