Kronecker Decomposition for GPT Compression (2022)

Abstract: GPT is an auto-regressive Transformer-based pre-trained language model which has attracted a lot of attention in the natural language processing (NLP) domain. The success of GPT is mostly attributed to its pre-training on huge amount of data and its large number of parameters. Despite the superior performance of GPT, this overparameterized nature of GPT can be very prohibitive for deploying this model on devices with limited computational power or memory. This problem can be mitigated using model compression techniques; however, compressing GPT models has not been investigated much in the literature. In this work, we use Kronecker decomposition to compress the linear mappings of the GPT-2 model. Our Kronecker GPT-2 model (KnGPT2) is initialized based on the Kronecker decomposed version of the GPT-2 model and then is undergone a very light pre- training on only a small portion of the training data with intermediate layer knowledge distillation (ILKD). Finally, our KnGPT2 is fine-tuned on downstream tasks using ILKD as well. We evaluate our model on both language modeling and General Language Understanding Evaluation benchmark tasks and show that with more efficient pre-training and similar number of parameters, our KnGPT2 outperforms the existing DistilGPT2 model significantly.

## introduction recently, development and deployment of pretrained language models (plms) has improved the performance of nlp models significantly @xcite @xcite @xcite . plms are mostly transformer-based models, which are pre-trained on enormous unlabeled data. although transformer-based plms are powerful in performance, their huge size is a barrier for efficient training or inference of these models on lower capacity devices with memory, computation and energy constraints. therefore, there has been a growing volume of literature focused on developing frameworks for compressing these large plms. like other deep learning models, the main directions of model compression for plms are using following methods in isolation or combination: low-bit quantization @xcite , pruning @xcite , knowledge distillation (kd) @xcite and matrix decomposition @xcite . plms can be divided into encoder-based and auto-regressive models such as the bert @xcite and @xcite family respectively. although the size of bert family models is usually smaller than the gpt family, compressing the bert family has been investigated much more in the literature (e.g. distilbert @xcite , tinybert @xcite , mobilebert @xcite , alp-kd @xcite , mate-kd @xcite , annealing-kd @xcite and bertquant @xcite ). on the other hand, to the best of our knowledge, the gpt family has barely a handful of compressed models @xcite , among them the distilgpt2 foot_0 model has attracted wide attention in the literature. the distilgpt2 model is heavily pre-trained for 3 epochs on the large openwebtext dataset foot_1 . moreover, it is evident in the literature that the gpt model cannot compete with bert on natural language understanding (nlu) tasks @xcite . therefore, developing an efficient compressed gpt model with comparable nlu performance is still an open problem. in this paper, we use kronecker decomposition, which has been recently used for @xcite , for compression of the gpt-2 model (we refer to our model as kngpt2 in this paper). we use kronecker decomposition to represent the weight matrices of linear layers in gpt-2 by smaller matrices which can reduce the size and computation overhead. we use kronecker decomposition to compress the embedding and transformer layers of gpt-2. for transformer layers, the linear layers of multi-head attention (mha) and the feed-forward network (ffn) blocks of transformer layers are replaced with their kronecker decomposition. kronecker decomposition leads to reduction in expressiveness of the model. we use a very light pre-training with intermediate layer knowledge distillation (ilkd) to address this issue, which improves the performance of the compressed model significantly. it is worth mentioning that for our pre-training, we use 1/10 th of the distilgpt2's pretraining data (i.e. openwebtext) only for 1 epoch (instead of 3 epochs in distilgpt2). furthermore, in this paper, our framework is applied to gpt-2 but it can be easily exploited to compress other models as well. to summarize contributions of this paper, we mention the following points: distilgpt2 foot_2 is one of the most successful and well-known compressed versions of gpt-2 which is considered as a baseline in this paper. distil-gpt2 has 82m parameters compared to 124m parameters for gpt-2 small and is trained using kd on openwebtextcorpus which is a reproduction of openai's webtext dataset. ## experiments we evaluated our proposed algorithm, kngpt2, on language modeling and text classification. for language modeling we use the wikitext-103 @xcite dataset.for classification we use seven of the classification tasks of the general language understanding evaluation (glue) benchmark @xcite . these datasets can be broadly divided into 3 families of problems. single set tasks which include linguistic acceptability (cola) and sentiment analysis (sst-2), similarity and paraphrasing tasks (mrpc and qqp), and inference tasks which include natural language inference (mnli and rte) and question answering (qnli). ## conclusion in this paper, we compressed gpt-2 by compressing linear layers of a gpt model using kronecker decomposition. our model is pre-trained on a relatively small (10 times smaller than the dataset used for baseline) dataset which makes the pre-training much faster. our proposed model significantly outperformed the baseline on the glue benchmark. using kd can help to further reduce the perfor-mance drop of the compressed model. using kronecker decomposition on larger gpt models and for higher compression factors are two interesting future directions.