SAMP : A Model Inference Toolkit of Post-Training Quantization for Text Processing via Self-Adaptive Mixed-Precision (2023)

Abstract: The latest industrial inference engines, such as FasterTransformer and TurboTransformers, have verified that half-precision floating point (FP16) and 8-bit integer (INT8) quantization can greatly improve model inference speed. However, the existing INT8 quantization methods are too complicated, and improper usage will lead to model performance damage greatly. In this paper, we develop a toolkit for users to easily quantize their models for inference, in which Self-Adaptive Mixed-Precision (SAMP) is proposed to automatically control quantization rate by a mixed-precision architecture to balance model accuracy and efficiency. Experimental results show that our SAMP toolkit has a higher speedup than PyTorch and FasterTransformer while ensuring the required accuracy. In addition, SAMP is based on a modular design, decoupling the tokenizer, embedding, encoder and target layers, which allows users to handle various downstream tasks and can be seamlessly integrated into PyTorch.

## introduction text understanding is one of the basic tasks in the field of natural language processing (nlp), including information retrieval, dialogue system, sentiment recognition, summarization, language model, etc. transformer-based models @xcite have achieved state-of-the-art in many downstream tasks, such as bert @xcite , xlnet @xcite , google t5 @xcite , etc. in some large industrial systems, training frameworks (e.g. tensorflow @xcite or pytorch @xcite ) are not good options to deploy models due to the lack of high gpu occupation considerations and good memory management of them during the inference phase @xcite . conventional inference acceleration tools for deep learning models such as nvidia tensorrt @xcite , turbotransformers @xcite and lightseq @xcite are primarily designed for fixed-size or variable-length inputs. these tools' optimization concepts mainly take into account memory management, operation fusion, or other data pruning techniques in the online computing systems, mostly single-precision calculation (only floating-point is used). so the acceleration performance is limited. on this basis, fastertransformer developed by nvidia performs fixed-point acceleration on transformer models (using fully-quantization in all transformer layers), and has achieved an excellent speedup compared with floating-point. however, this method of fullyquantized in all transformer layers makes it difficult for int8-quantization inference results to achieve the accuracy of floating-point calculations, resulting in a large loss of calculation accuracy in specific tasks, making it difficult to be widely used. on the other hand, we find that the kernel-fusion policy in fastertransformer int8-quantization implementation can still be optimized. to solve these problems, we propose an inference toolkit samp, which contains a self-adaptive mixed-precision encoder and a series of advanced fusion strategies. objectively, the mixed-precision calculation of floating-point and fixed-point can obtain better calculation accuracy than fully-fixedpoint calculation. self-adaptive mixed-precision encoder can find an optimal combination of mixedprecision among a large number of general matrix multiplication (gemm) operations and transformer layers, which can align the performance of model inference most closely with user needs (calculation accuracy and inference latency). advanced fusion strategies make fusion improvements for embedding kernels and quantizationrelated operations respectively, reducing cuda kernel calls by half. moreover, samp is an endto-end toolkit implemented by c++ programming language (from tokenizer to embedding, encoder, and downstream tasks), which has excellent inference speed and reduces the threshold of industrial application. ## conclusion in this paper, we introduce a new inference toolkit samp for nlp models. the main contribution of samp is to solve the problem of serious performance loss of the existing quantization inference tools in the industrial application of text understanding. and it also pioneers the application of quantization inference to various downstream tasks through a wide variety of task-type coverage. samp is light-weight, flexible, and user-friendly. at present, it has been widely used in our business, which greatly saves the deployment cost of industrial applications. in the future work, we will focus on optimizing the quantization effect of gemms in mha, and explore fixed-point acceleration methods with lower bit width than 8-bit integer, and introduce samp to more models.