F ast F ormers: Highly Efficient Transformer Models for Natural Language Understanding (2020)

Abstract: Transformer-based models are the state-of-the-art for Natural Language Understanding (NLU) applications. Models are getting bigger and better on various tasks. However, Transformer models remain computationally challenging since they are not efficient at inference-time compared to traditional approaches. In this paper, we present FastFormers, a set of recipes to achieve efficient inference-time performance for Transformer-based models on various NLU tasks. We show how carefully utilizing knowledge distillation, structured pruning and numerical optimization can lead to drastic improvements on inference efficiency. We provide effective recipes that can guide practitioners to choose the best settings for various NLU tasks and pretrained models. Applying the proposed recipes to the SuperGLUE benchmark, we achieve from 9.8x up to 233.9x speed-up compared to out-of-the-box models on CPU. On GPU, we also achieve up to 12.4x speed-up with the presented methods. We show that FastFormers can drastically reduce cost of serving 100 million requests from 4,223 USD to just 18 USD on an Azure F16s_v2 instance. This translates to a sustainable runtime by reducing energy consumption 6.9x - 125.8x according to the metrics used in the SustaiNLP 2020 shared task.

## introduction since bert @xcite has been introduced, transformer @xcite based pretrained language models have dominated the natural language understanding (nlu) field. transformer models have provided unprecedented accuracy improvement compared to traditional models @xcite . however, the models' computational cost at inference time is prohibitively challenging to be widely 1 https://docs.microsoft.com/en-us/ azure/virtual-machines/fsv2-series adopted in real world production scenarios which requires low latency, fast inference and low serving costs. in this work, we present fastformers, a set of methods and recipes that provides highly efficient inference for transformer models which enables deployment in large scale production scenarios. we specifically focus on the inference time efficiency since it mostly dominates the cost of production deployment. mainly, we utilize three methods: knowledge distillation, structured pruning and model quantization. first, we investigate the efficacy of various knowledge distillation techniques to significantly reduce the size of the models with respect to the depth and hidden state sizes while preserving the accuracy. second, we explore structured pruning that further reduces the size of the models by reducing the number of self-attention heads and the number of intermediate hidden states in the feedforward layers to achieve more efficiency while trying to preserve the accuracy as well. finally, we explore model quantization which enables faster model executions by optimally utilizing hardware acceleration capabilities. on cpu, 8-bit integer quantization method is applied to utilize the most efficient cpu instructions available: vector neural network instructions (vnni). on gpu, all the model parameters are converted into 16-bit floating point data type to maximally utilize efficient tensor cores. furthermore, computational graph optimizations which fuse multiple graph nodes are performed by utilizing onnxruntime foot_0 library. in addition, we explore optimal settings of allocating cpu and gpu resources to achieve better utilization. the proposed methods are optimized and evaluated on both cpus and gpus which are the most commonly available hardware platforms. per-formance evaluations are conducted on super-glue @xcite which is one of the general purpose open domain nlu benchmarks. for the efficiency measurement, wall clock times and energy efficiency are measured while performing inferences on the test sets of boolq, cb, copa, multirc, record, rte and wic tasks from superglue. the energy efficiency is measured by an open source python library called experiment-impact-tracker proposed in @xcite . to make sure the optimized models preserve similar accuracy, the accuracy of all the models is measured together. the contributions of this paper are presenting a set of recipes for efficient inference of transformer nlu models, analyzing the effect of various optimization techniques and finally making the code publicly available foot_1 to facilitate utilizing fastformers for efficient inference of transformers models. the rest of the paper is organized as follows: section 2 presents knowledge distillation techniques, section 3 presents structured pruning techniques, section 4 discusses model quantization approaches, section 5 presents runtime optimization techniques, section 6 presents results on various tasks and finally section 7 concludes the findings and future directions. ## knowledge distillation knowledge distillation @xcite is a well known model compression technique where the large model is used as a teacher for a smaller student model. the knowledge distillation is the process of training the student model to mimic the behaviour of the larger teacher model. knowledge distillation has been shown to improve the efficiency of the transformer-based architectures for the nlu tasks @xcite as well as natural language generation tasks such as machine translation @xcite . ## structured pruning there has been a significant amount of research on model's weights pruning approaches inspired by the lottery ticket hypothesis @xcite . furthermore, there are various papers published to apply this pruning strategy to transformer models including: @xcite @xcite . most of such approaches focused on random pruning to reduce the number of parameters following the lottery ticket 7 https://sites.google.com/view/ sustainlp2020/shared-task?@xmath0. while this can reduce the size of the model on the computer storage, it may not improve the inference performance since it is not focusing on better utilization of the computing resources. since our main focus in fastformers is to improve inference efficiency, randomly pruning a subset of the model's parameters may not improve performance. in this work, we focus on structured pruning which directly reduces the computation requirements. voita et al. (2019 portant hidden states in the feed-forward layers. we use a first order method for computing the importance score, which utilizes the first order gradient information proposed by @xcite al. (2019); sanh et al. (2020); hou et al. (2020 ) instead of using magnitude based pruning @xcite . before doing the importance score computation, we add a mask variable to each attention head for the gradient computation of the heads. next, we run forward and backward passes of the model on the entire validation data set, then the absolute values of the gradients are accumulated. these accumulated values are used as importance scores which we use to sort the importance of the heads and the intermediate hidden states. based on the target model size, we select a given number of top heads and top hidden states from the network. once the sorting and selection steps are done, we re-group and reconnect the remaining heads and hidden states which result in a smaller sized model. when heads and hidden states are pruned, we use the same pruning ratio across different layers. this enables further optimizations to work seamlessly with the pruned models. in our experiments, we observed that the pruned model can get better accuracy when it goes through another round of knowledge distillation; this has also been noted in @xcite . therefore, we do one more knowledge distillation by using the nonpruned model as a teacher model and the pruned model as an initializer of student model. ## low precision inference after knowledge distillation and structured pruning compression, models can benefit form more efficient numerical computations by quantizing the model parameters on cpu and gpu @xcite . it has been shown that the accuracy of transformer models doesn't get severely compromised when utilizing 8-bit or 16-bit lower precision arithmetic @xcite @xcite @xcite @xcite . modern cpus and gpu accelerators are capable of computing those lower numerical arithmetic efficiently. for example, cascade lake cpus have a special 8-bit vector instruction set called avx (advanced vector extensions)-512-vnni and v100 gpus can utilize its efficient tensor cores with 16-bit floating point data. ## runtime optimization on top of the structural and numerical optimizations applied, we can utilize various ways to further optimize the computations. in particular, we focus on multi-processing optimizations and computational graph optimization. multi-processing optimization: the evaluation machine for the sustainlp 2020 shared task has two cascade lake 6248 cpus with 40 physical cores. the default execution engines for hugging-face's transformers including pytorch foot_8 and ten-sorflow foot_9 usually use all available cpu cores for a single operator. this is not the optimal way of utilizing available cpu cores for several reasons. the operators in compressed transformer architectures are not big enough to fully utilize the parallelism of 40 cpu cores. therefore, the overheads of parallelizing the operation significantly overshadow the actual gains from the parallelism. another important factor is that the parallelization to all cores reduces the cache locality and degrades the overall efficiency of cpu utilization. therefore, we implement a multiple instance inference by leveraging multiprocessing module of python foot_10 programming language. it is preferable to use multi-threading instead of multi-processing to avoid additional copy of program memory, but python's multi-threading cannot really utilize multiple threads due to the global interpreter lock (gil) foot_11 . even with this limitation, multi-instance inference with multiprocessing brings much more efficient computation. to maximize the cache locality, each inference instance is pinned to specific physical cores using linux' taskset command. utilizing hyper-threading harms the cache utilization, so we always keep the total number of utilized threads (the number of threads per one inference instance multiplied by the number of instances) not exceeding the number of physical cores in the machine. the optimal number of multiple processes for the best efficiency varies by the model, hardware settings and the data set. we conduct experiment with all target tasks and investigate the best setting for each task. computational graph optimizations: computational graph optimization can further improve the efficiency of the neural network inference by pruning unused graph nodes and fusing multiple operations together. in particular, graph node fusion reduces additional memory allocation and copy which potentially degrade the efficiency. also, graph node fusion potentially improves parallelism by increasing the size of individual operators. we replace gaussian error linear units (gelu) with rectified linear units (relu) for the computational efficiency while model is distilled without losing any accuracy. we fuse relu and bias addition operations followed by matrix multiplications into fbgemm's fused post processing operation. moreover, we utilize multi-head attention node fusion provided by onnxruntime. we use a customized onnxruntime based on v1.3.1. ## conclusion in this paper, we have introduced fastformers, which achieves efficient inference-time performance for transformer-based models on various nlu tasks. we showed that utilizing knowledge distillation, structured pruning and numerical optimization can lead to drastic improvements on inference efficiency. we showed that the improvements can be up to 200x speed-up and results in more than 200x inference cost saving with 22x energy saving. we open source fastformers for the community hoping it can drive more sustainable optimizations for transformers models. for future directions, some other methods such as early exiting @xcite @xcite and linear time complexity self-attention models @xcite could be added to the proposed recipes and possibly improve the efficiency further.