PCEE - BERT : Accelerating BERT Inference via Patient and Confident Early Exiting (2022)

Abstract: BERT and other pretrained language models (PLMs) are ubiquitous in modern NLP. Even though PLMs are the state-of-the-art (SOTA) models for almost every NLP task (CITATION), the significant latency during inference prohibits wider industrial usage. In this work, we propose Patient and Confident Early Exiting BERT (PCEE-BERT), an off-the-shelf sample-dependent early exiting method that can work with different PLMs and can also work along with popular model compression methods. With a multi-exit BERT as the backbone model, PCEE-BERT will make the early exiting decision if enough numbers (patience parameter) of consecutive intermediate layers are confident about their predictions. The entropy value measures the confidence level of an intermediate layer’s prediction. Experiments on the GLUE benchmark demonstrate that our method outperforms previous SOTA early exiting methods. Ablation studies show that: (a) our method performs consistently well on other PLMs, such as ALBERT and TinyBERT; (b) PCEE-BERT can achieve different speed-up ratios by adjusting the patience parameter and the confidence threshold. The code for PCEE-BERT can be found at https://github.com/michael-wzhu/PCEE-BERT .

## introduction since bert @xcite , the pre-trained language models (plms) have become the default state-of-the-art (sota) models for natural language processing (nlp). the recent years have witnessed the rise of many plms, such as gpt @xcite , xlnet @xcite , albert @xcite , and so forth. these bert-style models achieved considerable improvements in many natural language processing (nlp) tasks by pre-training on the unlabeled corpus and fine-tuning on labeled tasks, such as text classifi-cation, natural language inference (nli) and sequence labeling. despite their excellent performances, there are two issues for plms. first, previous studies show that plms such as bert suffer from the over-thinking problem. @xcite shows that in the sentence classification task, bert's last few layers may be too deep for some samples. for a sentence classification task, if we insert a classifier on a certain intermediate layer and drop the deeper layers, these intermediate layers may outperform the last layer. the second drawback of plms is their high latency. sentence classification (cls) tasks play a central role in many application scenarios, such as dialogue systems, document analysis, content recommendation, etc. however, these applications are time-sensitive. for example, if a task-oriented dialogue (tod) system takes a lot of time to respond, users will no doubt stop using this system. user experience studies show that a response has to be made in between 0-100 ms. thus, a cls mod- @xcite and our pcee-bert, a novel early exiting method that combines the score-based early exiting with the patience-based early exiting. there exists a branch of literature focusing on making plms' inference more efficient via network pruning @xcite @xcite , knowledge distillation @xcite @xcite , weight quantization @xcite @xcite and adaptive inference @xcite @xcite . adaptive inference has drawn much attention. the idea of adaptive inference is to deal with simple examples with only shallow layers of bert and process more difficult queries with deeper layers, thus significantly speeding up the inference time on average while maintaining high accuracy. the speed-up ratio can be easily controlled with certain hyper-parameters to handle significant changes in query traffic without re-deploying the model services or maintaining a group of models. early exiting is one of the most important adaptive inference methods @xcite . as depicted in figure 2 (b), it implements adaptive inference by installing an early exit, i.e., an intermediate prediction layer, at each layer of bert and early exiting "easy" samples to speed up inference. at the training stage, all the exits are jointly optimized with bert's parameters. at the inference stage, there are two different settings. first, in budgeted exiting mode, the model makes a prediction with a fixed exit for all queries. this mode deals with heavy traffic by assigning a shallower exit for prediction. the other one is dynamic exiting mode. that is, some strategies for early exiting are designed to decide whether to exit at each layer given the currently obtained predictions (from previous and current layers) @xcite @xcite . in this mode, different samples can exit at different depths. there are mainly three early exiting strategies for bert dynamic exiting. the first one is scorebased early exiting. branchynet @xcite , fastbert @xcite , and dee-bert @xcite calculated the entropy of the prediction probability distribution as an estimation for the confidence of exiting classifiers to enable dynamic early exiting. shallow-deep nets @xcite and righttool @xcite leveraged the maximum of the predicted distribution as the exiting signal. the second type is the learned exiting @xcite . in this type of work, an early exiting signal is generated by a learnable module in the neural network. for example, berxit @xcite install a fully connected layer right after each transformer block of bert to output a score that is used to decide whether the bert should stop inference and exit early. the third type is patience-based early exiting, which relies on cross-layer comparison to formulate the exiting signal. pabee @xcite propose a dynamic exiting strategy analogous to early stopping model training. that is, if the exits' predictions remain unchanged for a pre-defined number of times (patience), the model will stop inference and exit early. pabee achieves sotas results for bert early exiting. despite its state-of-the-art performances during early exiting, pabee is inflexible in adjusting the speedup ratios. on a given task, once the multiexit bert is fine-tuned and the patience parameter is fixed, pabee can only achieve a fixed average speedup ratio. thus, pabee can not achieve speedup ratios of certain values. this drawback makes pabee inconvenient to use in real industrial scenarios. thus, it is of great importance to come up with a method that can flexibly adjust its speedup ratios and performs comparable to or better than pabee. in this work, we propose patiently confidently early exiting bert (pcee-bert), a novel early exiting method that combines the advantage of score-based methods and the patience based early exiting method. a multi-exit bert is adopted as the backbone model, and an intermediate classifier (i.e., an exit) is installed right after each transformer block. pcee-bert will early exit if there are enough numbers (i.e., the patience parameter) of consecutive exits being confident for their predicted distributions. we mainly use entropy as the confidence measure. intuitively, our method requires patience and confidence. it will not rush into an early exiting if we only see a couple of intermediate layers being confident. in addition, it allows the next layer to modify the predictions. in this way, our pcee-bert can exit with higher accuracy while maintaining flexibility. extensive experiments are conducted on the glue benchmark @xcite . the results show that our method outperforms the previous sota early exiting methods, especially in cases where the speedup ratio is large. in addition, one can adjust the patience and confidence threshold so that pcee-bert can arrive at different speedup ratios. a series of ablation studies are conducted, resulting in the following observations: the rest of the paper is organized as follows. first, we introduce the preliminaries for multi-exit bert and early exiting. second, we elaborate on our pcee-bert method. third, we conduct experiments on the glue benchmark and conduct a series of ablation studies. finally, we conclude with possible future works. ## preliminaries in this section, we introduce the necessary background for bert early exiting. throughout this work, we consider the case of multi-class classification with samples {(x, y), @xmath0, @xmath1, @xmath2, 2, ..., n }, e.g., sentences, and the number of classes is k. ## conclusion in this work, we propose pcee-bert, a novel efficient inference method that can yield a better performance-speed trade-off than the existing early exiting methods. pcee-bert adopts bert as the backbone model and makes the exiting decision if there are enough intermediate layers to make confident predictions. the confidence level is measured by the entropy of the predicted distributions. experiments on the glue benchmark demonstrate that our method outperforms the previous sota early exiting methods, especially when the speedup ratio is large. in addition, pcee-bert can achieve different speedup ratios by adjusting the patience parameter and the confidence threshold, which makes it more flexible in industrial usage. ablation studies show that: