Zero-Shot Dynamic Quantization for Transformer Inference (2022)

Abstract: We introduce a novel run-time method for significantly reducing the accuracy loss associated with quantizing BERT-like models to 8-bit integers. Existing methods for quantizing models either modify the training procedure, or they require an additional calibration step to adjust parameters that also requires a selected held-out dataset. Our method permits taking advantage of quantization without the need for these adjustments. We present results on several NLP tasks demonstrating the usefulness of this technique.

## introduction transformer-based neural networks (nn) such as bert @xcite , roberta @xcite and xlm-r @xcite , pretrained on large amounts of data, have led to stateof-the-art (sota) results on many nlp tasks such as machine translation @xcite ), text classification @xcite ) and question answering @xcite ). however, run-time inference of such large models is very costly due to their large computational requirements. in addition, deploying these models on smaller footprint mobile devices @xcite or cost-effective @xcite cpu based machines require aggressive optimization techniques for both speed and network size. one popular speed optimization technique is nn quantization @xcite @xcite , where network weights and activations are transformed from 32-bit floating-point representations to integers (typically 8-bit). running inference using integer operations has two key advantages. first, the model size footprint is considerably reduced e.g. 8-bit quantization shrinks models by a factor of four. second, inference throughput is significantly increased by using more efficient integer-based "single instruction multiple data" (simd) @xcite instructions while improving memory bandwidth utilization, which is typically a bottleneck limiting computational throughput for nns @xcite . fundamentally, quantization leads to a quantitative loss of information due to the lowered numerical precision. as a result, applying integer quantization directly to nn models leads to considerable drop in accuracy @xcite . however, by carefully adjusting the quantization parameters such as the clipping thresholds, the accuracy loss can be significantly reduced, if not eliminated. the majority of quantization research @xcite involve a mix of quantization-aware training (qat) and post-training calibration techniques with varying complexities to resolve the quantization performance gap. several works @xcite @xcite @xcite @xcite detail techniques for qat as well as approaches where the quantization parameters are optimized using statistics gathered during training. while these approaches typically close the gap in the quantized model accuracy, they requires access to the training pipeline as well as the training data. in addition, these methods are not applicable to black-box models where both training procedures and data are not available. also, these methods may be affected by training instabilities, increasing the complexity of the training regimes as described in @xcite . post-training approaches such as @xcite require calibration techniques on selected datasets. for example, in @xcite between the unquantized and quantized activations on each layer was used to tune the quantization clipping thresholds. special care needs to be taken when selecting a calibration dataset; as it needs to be diverse enough but yet task specific. in certain cases this leads to low accuracy, or even unpredictable behaviour, if the run-time input deviates from the calibration dataset. two methods that share our high-level goals of eliminating the need for training datasets are introduced in @xcite . these methods are implemented with cnn-based @xcite networks, and are used for image classification and object detection tasks. @xcite reduces the quantization error by rescaling the weights of consecutive cnn layers while taking advantage of the equivariance property of the piece-wise linear relu function. @xcite , on the other hand, tunes the quantization parameters using synthetic data generated utilizing mean and variance statistics obtained from the batch normalization layers of the model itself. while both methods are applicable for mainly cnn-based networks, our algorithm is considerably simpler to implement and targets transformers @xcite ; particularly sota nlp networks with bert-like @xcite pre-trained representations. in this work, we present a method that utilizes the interquartile range (iqr) @xcite , which is a measure of statistical dispersion, to clip the activations dynamically during inference time. our method ensures that at least 75% of the token-wise extreme activations are not modified, while leaving the remaining 25% to be statistically modified as outliers, leading to a robust behaviour while considerably improving quantization accuracy. our method works for any transformer-based "trained" model and does not require any form of training or calibration. overall, our contributions can be summarized as follows: ## experimental setup our run-time inference engine, implemented in c++, supports both fp32 and optimized 8-bit integer quantized inference (i8). we quantize model weights at load-time and dynamically quantize activations at run-time. the tm-iqr technique is a straightforward modification with a negligible impact on inference speed, as shown in ## results since our method does not modify the training pipeline or tune the quantization parameters on training sets, we compare our results directly to the fp32 numbers. we are not expecting our method to outperform fp32 but rather to reduce the negative effect of quantization while keeping its speed as well as simplifying the model deployment process. ## conclusion we show that bert-like models can be quantized to 8-bit integers with good accuracy without the need to modify training procedures or add extra data sets for parameter calibration. we present a robust statistically-based algorithm that dynamically adjusts the quantization clipping to maintain reasonable accuracy. our empirical results demonstrate the effectiveness of our method on a number of nlp monolingual and multilingual tasks, trained on both base and large size bert-like models.