C ascade BERT : Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade (2021)

Abstract: Dynamic early exiting aims to accelerate the inference of pre-trained language models (PLMs) by emitting predictions in internal layers without passing through the entire model. In this paper, we empirically analyze the working mechanism of dynamic early exiting and find that it faces a performance bottleneck under high speed-up ratios. On one hand, the PLMs’ representations in shallow layers lack high-level semantic information and thus are not sufficient for accurate predictions. On the other hand, the exiting decisions made by internal classifiers are unreliable, leading to wrongly emitted early predictions. We instead propose a new framework for accelerating the inference of PLMs, CascadeBERT, which dynamically selects proper-sized and complete models in a cascading manner, providing comprehensive representations for predictions. We further devise a difficulty-aware objective, encouraging the model to output the class probability that reflects the real difficulty of each instance for a more reliable cascading mechanism. Experimental results show that CascadeBERT can achieve an overall 15% improvement under 4x speed-up compared with existing dynamic early exiting methods on six classification tasks, yielding more calibrated and accurate predictions.

## introduction large-scale pre-trained language models (plms), e.g., bert and roberta, have demonstrated superior performance on various natural language understanding tasks @xcite . while the increased model size brings more promising results, the long inference time hinders the deployment of plms in real-time applications. researchers have recently exploited various kinds of approaches for accelerating the inference of plms, 1 our code is available at https://github.com/ lancopku/cascadebert @xmath0: classifier : bert layer deebert @xmath1.992 .001 .007 ✓ 2 0 1 .991 .004 .005 ✓ 2 0 1 .250 .504 .246 ✗ ⋯ ⋯ premise: also, the final rule is not intended to have any retroactive effect and administrative procedures must be exhausted prior to any judicial challenge to the provisions of the rule. hypothesis: the final rule isn't meant to have a retroactive effect. 0: neutral ✗ 1: entailment ✓ 2: contradiction ✗ figure 1 : an easy instance with a large word overlap (colored in orange) between the premise and the hypothesis from the mnli dataset. the classifiers in shallow layers of a dynamic early exiting model cannot predict correctly, while bert-complete @xcite , a small bert pre-trained from scratch with the same size can make a correct and confident prediction. a mainstream method for instance-level speed-up is dynamic early exiting, which emits predictions based on intermediate classifiers (or off-ramps) of internal layers when the predictions are confident enough @xcite @xcite . in this paper, we focus on dynamic early exiting, as it can be utilized to accelerate inference and reduce the potential risk of the overthinking problem @xcite . such a paradigm is intuitive and simple, while faces a performance bottleneck under high speed-up ratios, i.e., the task performance is poor when most examples are exited in early layers. we conduct probing experiments to investigate the mechanism of dynamic exiting, and find that the poor performance is due to the following two reasons: (1) the shallow representations lack high-level semantic information, and are thus not sufficient for accurate predictions. as plms like bert exhibit a hierarchy of representations, e.g., shallow layers extract low-level features like lexical/syntactic information while deep layers capture semantic-level relations @xcite , we argue that the high-level semantic inference ability is usually required even for easy instances. as shown in figure 1 , the classifier of the second layer in a representative early exiting model deebert @xcite cannot predict correctly even for an easy instance with a large word overlap. on the contrary, bert-complete, a shallow 2-layer model pre-trained from scratch @xcite that is thus capable of extracting semantic-level features, can make confident and correct predictions like that in deep layers of deebert. (2) the intermediate classifiers in the early exiting models cannot provide reliable exiting decisions. we design a metric to examine the ability of models to distinguish difficult instances from easy ones, which can reflect the quality of exiting decisions. we find that the predictions of internal classifiers cannot faithfully reflect the instance difficulty, resulting in wrongly emitted results and thus hindering the efficiency of early exiting. to remedy those drawbacks, we instead extend the dynamic early exiting idea to a model cascade, and propose cascadebert, which conducts inference based on a series of complete models in a cascading manner with a dynamic stopping mechanism. specifically, given an instance for inference, instead of directly exiting in the middle layers of a single model, the framework progressively checks if the instance can be solved by the current plm from the smallest to the largest one, and emits the prediction once the plm is confident about the prediction. furthermore, we propose a difficultyaware regularization to calibrate the plms' predictions according to the instance difficulty, making them reflect the real difficulty of each instance. therefore, the predictions can be utilized as a good indicator for the early stopping in inference. ex-perimental results on six classification tasks in the glue benchmark demonstrate that our model can obtain a much better task performance than previous dynamic early exiting baselines under high speed-up ratios. further analysis demonstrates that the proposed difficulty-aware objective can calibrate the model predictions, and proves the effectiveness and the generalizability of cascadebert. ## investigations into early exiting dynamic early exiting aims to speed-up the inference of plms by emitting predictions based on internal classifiers. for each instance, if the internal classifier's prediction based on the current layer representation of the instance is confident enough, e.g., the maximum class probability exceeds a threshold @xcite , then the prediction is emitted without passing through the entire model. however, whether the internal representations could provide sufficient information for accurate predictions and whether the intermediate classifiers can be utilized for making accurate exiting decisions still remain unclear. in this section, we investigate the working mechanism of dynamic early exiting by exploring these two questions. ## methodology to remedy the drawbacks of conducting dynamic exiting in a single model, we extend the idea to a model cascade, by proposing cascadebert, that utilizes a suite of complete plms with different number of layers for acceleration in a cascading manner, and further devise a difficulty-aware calibration regularization to inform the model of instance difficulty. ## experiments we evaluate our method on the classification tasks in the glue benchmark @xcite with bert @xcite . we first give a brief introduction of the dataset used and the experimental setting, followed by the description of baseline models for comprehensive evaluation. the results and analysis of the experiments are presented last. ## related work model-level compression includes knowledge distillation (kd), pruning and quantization. kd focuses on transferring the knowledge from a large teacher model to a compact student model @xcite . sanh et al. (2019) propose distil-bert and sun et al. (2019) enhance kd by aligning the internal representations of the student and the teacher model. besides, jiao et al. (2020) propose tinybert via a two-stage kd on augmented data. pruning methods deactivate the unimportant structures in the model like attention heads @xcite and layers @xcite . quantization methods target at using fewer physical bits to efficiently represent the model @xcite @xcite . we do not compare pruning and quantization methods since these techniques are orthogonal to our framework. instance-level speed-up accelerates the inference via adapting the computation according to the instance complexity @xcite . a representative framework is dynamic early exiting, which has been verified in natural language understanding @xcite @xcite @xcite , sequence labeling @xcite , question answering @xcite and document ranking @xcite . in this paper, we probes the work mechanism of dynamic early exiting, and find that it faces a serious performance bottleneck under high speed-up ratios. to remedy this, we generalize the idea to a model cascade and prove it is effectiveness even under high speed-up ratios for various natural language understanding tasks. concurrently with our work, enomoto and eda (2021) adopt the similar idea and achieve better inference efficiency on image classification tasks. ## conclusion in this paper, we point out that current dynamic early exiting framework faces a performance bottleneck under high speed-up ratios, due to insufficient shallow layer representations and poor exiting decisions of the internal classifiers. to remedy this, we propose cascadebert, a model cascade framework with difficulty-aware regularization for accelerating the inference of plms. experimental results demonstrate that our proposal achieves substantial improvements over previous dynamic exiting methods. further analysis validates that the framework is generalizable and produces more calibrated results.