P re Q uant: A Task-agnostic Quantization Approach for Pre-trained Language Models (2023)

Abstract: While transformer-based pre-trained language models (PLMs) have dominated a number of NLP applications, these models are heavy to deploy and expensive to use. Therefore, effectively compressing large-scale PLMs becomes an increasingly important problem. Quantization, which represents high-precision tensors with low-bit fix-point format, is a viable solution. However, most existing quantization methods are task-specific, requiring customized training and quantization with a large number of trainable parameters on each individual task. Inspired by the observation that the over-parameterization nature of PLMs makes it possible to freeze most of the parameters during the fine-tuning stage, in this work, we propose a novel “quantize before fine-tuning” framework, PreQuant, that differs from both quantization-aware training and post-training quantization. {pasted macro ‘OUR’} is compatible with various quantization strategies, with outlier-aware parameter-efficient fine-tuning incorporated to correct the induced quantization error. We demonstrate the effectiveness of PreQuant on the GLUE benchmark using BERT, RoBERTa, and T5. We also provide an empirical investigation into the workflow of PreQuant, which sheds light on its efficacy.

## introduction pre-trained language models (plms) have shown superior performance in various nlp applications. despite their impressive success, these transformerbased models typically contain hundreds of millions of parameters. massive model scale is becoming an increasing burden, preventing researchers from making full use of large-scale plms. according to a recent study, only 0.5% to 4% of research papers published at the recent five nlp conferences tend to adopt large plms (plms with over a billion parameters) @xcite . this suggests that the inefficiency of deploying large plms is hampering the development of nlp research. therefore, compressing plms becomes an urgent and important problem. various model compression methods have been proposed, such as knowledge distillation @xcite @xcite , weight sharing @xcite , network pruning @xcite @xcite , and quantization @xcite @xcite . among these compression methods, quantization is a promising solution. the core idea of quantization is to use low bit precision to store weight and activation tensors, and use fixed-point arithmetic to speed up inference. there are some prior works on quantizing plms covering different strategies and granularities. however, these quantization methods generally neglect the characteristics of plms -the distinction between fine-tuning a model and training a model from scratch -but treat quantizing plms no different from quantizing regular neural networks. in other words, these methods are task-specific, which design customized quantization for plms. there are two main limitations: first, these task-specific methods need to conduct both quantization and fine-tuning for each downstream task, with the quantization being applied either during or after the fine-tuning stage, which is inefficient; second, the number of trainable parameters are still very large during fine-tuning, which is computational expensive. in this work, we consider the quantization pipeline specially for the pre-training scenario. our motivation starts from the distinction between "training from scratch" and "pre-training then finetuning". unlike the weights from random initialization, the weights of the pre-trained model already contain rich information during pre-training. to utilize such information in a more efficient manner, we propose to directly quantize the pre-trained model in a task-agnostic way to obtain a "prequantized" model before fine-tuning. we then introduce a parameter-efficient fine-tuning and show that fine-tuning could be finished with minimal weight updates. in particular, we freeze most of the quantized weights in the "pre-quantized" model, and only fine-tune a very small subset of its model parameters in the fine-tuning process. through an extensive set of explorations and experiments, we demonstrate the feasibility and advantages of the "quantizing the plm first, then fine-tuning" pipeline, which we name as prequant. the main contributions are summarized as follows: ## preliminary a number of works have been employing various quantization techniques on the field of pre-trained language models. existing quantization methods can be categorized into two prominent branches: quantization-aware training and post-training quantization. ## conclusions as the scale of pre-trained language models increases, model compression becomes a prerequisite prior to model deployment in resource-limited scenarios. quantization is an effective and promising technique to compress large plms. existing quantization methods including ptq and qat perform quantizations either during or after task-specific fine-tuning process. since these approaches are highly task-specific, it's hard to transfer them to different tasks with low cost. in this paper, we propose a "quantizing the plm first, then finetuning" framework, prequant, which includes a task-agnostic quantization stage and an outlieraware parameter-efficient fine-tuning stage. we compress widely used plms with prequant, including bert, roberta and t5 variants. the experimental results on the glue benchmark are reported to demonstrate the effectiveness of pre-quant. we also reveal that prequant is more flexible and efficient than its competitive counterparts. an elaborate empirical study is conducted on the workflow of prequant, we hope the findings could shed some light on the quantization research of plms.