Boost Transformer-based Language Models with GPU -Friendly Sparsity and Quantization (2023)

Abstract: Along with the performance improvement in NLP domain, the sizes of transformer-based language models (TLM) are also dramatically increased. Some prior works intend to compress TLM models into more compact forms, but do not fully consider the hardware characters may not support the efficient execution for these forms, leading to the deployment of TLM on hardware with noticeable acceleration is still challenging. This paper thoroughly designs a compression scheme named GPUSQ-TLM to maximally utilize the GPU-friendly 2:4 fine-grained structured sparsity and quantization characters. Especially, a dense TLM model is first pruned to meet the GPU’s acceleration constraint of sparse patterns with FP16 type, then it is further quantized into a fixed-point one by quantization-aware training, to provide an extra speedup for integer tensors on GPU. A mixed-strategy knowledge distillation of labels, logits and feature maps is used for best accuracy compensation during pruning and quantization process. Experiment results show GPUSQ-TLM scheme achieves state-of-the-art compression on TLM model of various encoder and decoder blocks with negligible accuracy degradation on SQuAD, GLUE, CNN-DM & XSum and WikiText benchmarking tasks. Moreover, GPUSQ-TLM can boost actual deployment performance by up to 4.08-4.25x latency and 6.18-6.79x throughput on A100 GPU.

## introduction equipped with the attention mechanism and architecture @xcite , the transformerbased language models (tlm) are proficient in handling long-range dependencies of the sequence inputs. the subsequent studies showed that transformer-based pre-trained language models @xcite could achieve state-of-the-art performances on various natural language processing (nlp) (wolf * tao chen and zhongxue gan are corresponding authors. et al., 2019) benchmarks, including question answering @xcite , paraphrase detection @xcite , sentiment analysis @xcite , natural language inference @xcite , and text classification @xcite , etc. meanwhile, the transformer-based structure also expanded its success to other disciplines like computer vision @xcite , music @xcite , chemistry @xcite , life sciences @xcite , and pharmaceutics @xcite . along with the performance improvement, transformer-based language models' scales are also dramatically increased. bert model @xcite sets the milestone for pre-trained language models with transformer encoder as its backbone. it has 340 million parameters for the large version. generative pre-trained transformer (gpt) series are dedicated to scaling pre-trained transformer decoder architecture and proved that a large-scale pretrained language model could achieve impressive few-shot performance with diverse downstream tasks. the parameters scale increases from 110 million of gpt-1 @xcite to 1.5 billion of gpt-2 @xcite and finally boosts to 175 billion of gpt-3 @xcite . megatron-lm model @xcite is with 1 trillion parameters trained on 3072 gpus. switch transformers @xcite further increase the scale of pre-trained language models to 1.6 trillion parameters with a mixture of experts (moe) style. a detailed comparison of state-of-theart transformer-based models can refer to figure 1 . model compression techniques by transferring the large-scale tlm models to a lightweight version can benefit more efficient computation with less memory and energy consumption. there are some previous studies to compress the tlm models to compact forms. however, there are some main drawbacks in these prior arts: our main contributions include: ## experiments for the experiments in this paper, we choose py-torch @xcite with version 1.12.0 as the framework to implement all algorithms. the results of the dense model training, sparse and qat compression experiments, and the acceleration performance are obtained with a100 (nvidia, 2020) gpu clusters. all the reference algorithms use the default data type provided in public repositories.