D ee BERT : Dynamic Early Exiting for Accelerating BERT Inference (2020)

Abstract: Large-scale pre-trained language models such as BERT have brought significant improvements to NLP applications. However, they are also notorious for being slow in inference, which makes them difficult to deploy in real-time applications. We propose a simple but effective method, DeeBERT, to accelerate BERT inference. Our approach allows samples to exit earlier without passing through the entire model. Experiments show that DeeBERT is able to save up to ~40% inference time with minimal degradation in model quality. Further analyses show different behaviors in the BERT transformer layers and also reveal their redundancy. Our work provides new ideas to efficiently apply deep transformer-based models to downstream tasks. Code is available at https://github.com/castorini/DeeBERT .

## introduction large-scale pre-trained language models such as elmo @xcite , gpt @xcite , bert @xcite , xlnet @xcite , and roberta @xcite have brought significant improvements to natural language processing (nlp) applications. despite their power, they are notorious for being enormous in size and slow in both training and inference. their long inference latencies present challenges to deployment in real-time applications and hardwareconstrained edge devices such as mobile phones and smart watches. to accelerate inference for bert, we propose deebert: dynamic early exiting for bert. the inspiration comes from a well-known observation in the computer vision community: in deep convolutional neural networks, higher layers typically produce more detailed and finer-grained features @xcite . therefore, we hypothesize that, for bert, features provided by the intermediate transformer layers may suffice to classify some input samples. deebert accelerates bert inference by inserting extra classification layers (which we refer to as off-ramps) between each transformer layer of bert (figure 1 ). all transformer layers and offramps are jointly fine-tuned on a given downstream dataset. at inference time, after a sample goes through a transformer layer, it is passed to the following off-ramp. if the off-ramp is confident of the prediction, the result is returned; otherwise, the sample is sent to the next transformer layer. in this paper, we conduct experiments on bert and roberta with six glue datasets, showing that deebert is capable of accelerating model inference by up to ∼40% with minimal model quality degradation on downstream tasks. further analyses reveal interesting patterns in the models' transformer layers, as well as redundancy in both bert and roberta. ## related work bert and roberta are large-scale pre-trained language models based on transformers @xcite . despite their groundbreaking power, there have been many papers trying to examine and exploit their over-parameterization. @xcite and @xcite analyze redundancy in attention heads. q-bert @xcite uses quantization to compress bert, and layer-drop @xcite uses group regularization to enable structured pruning at inference time. on the knowledge distillation side, tinybert @xcite and distilbert @xcite both distill bert into a smaller transformer-based model, and @xcite distill bert into even smaller non-transformer-based models. our work is inspired by @xcite al. (2010), teerapittayanon et al. (2017), and huang et al. (2018) , but mainly differs from previous work in that we focus on improving model efficiency with minimal quality degradation. ## early exit for bert inference deebert modifies fine-tuning and inference of bert models, leaving pre-training unchanged. it adds one off-ramp for each transformer layer. an inference sample can exit earlier at an off-ramp, without going through the rest of the transformer layers. the last off-ramp is the classification layer of the original bert model. ## conclusions and future work we propose deebert, an effective method that exploits redundancy in bert models to achieve better quality-efficiency trade-offs. experiments demonstrate its ability to accelerate bert's and roberta's inference by up to ∼40%, and also reveal interesting patterns of different transformer layers in bert models. there are a few interesting questions left unanswered in this paper, which would provide interesting future research directions: (1) deebert's training method, while maintaining good quality in the last off-ramp, reduces model capacity available for intermediate off-ramps; it would be important to look for a method that achieves a better balance between all off-ramps. (2) the reasons why some transformer layers appear redundant foot_0 and why dee-bert considers some samples easier than others remain unknown; it would be interesting to further explore relationships between pre-training and layer redundancy, sample complexity and exit layer, and related characteristics.