Accelerating Inference for Pretrained Language Models by Unified Multi-Perspective Early Exiting (2022)

Abstract: Conditional computation algorithms, such as the early exiting (EE) algorithm, can be applied to accelerate the inference of pretrained language models (PLMs) while maintaining competitive performance on resource-constrained devices. However, this approach is only applied to the vertical architecture to decide which layers should be used for inference. Conversely, the operation of the horizontal perspective is ignored, and the determination of which tokens in each layer should participate in the computation fails, leading to a high redundancy for adaptive inference. To address this limitation, a unified horizontal and vertical multi-perspective early exiting (MPEE) framework is proposed in this study to accelerate the inference of transformer-based models. Specifically, the vertical architecture uses recycling EE classifier memory and weighted self-distillation to enhance the performance of the EE classifiers. Then, the horizontal perspective uses recycling class attention memory to emphasize the informative tokens. Conversely, the tokens with less information are truncated by weighted fusion and isolated from the following computation. Based on this, both horizontal and vertical EE are unified to obtain a better tradeoff between performance and efficiency. Extensive experimental results show that MPEE can achieve higher acceleration inference with competent performance than existing competitive methods.

## introduction pretrained language models (plms) @xcite @xcite have shown promising performance in many natural language processing tasks. the success of plms incurs computational consumption and long inference latency, which prevents these models from being deployed on resource-constrained devices, such as edge devices, or in time-sensitive scenarios. to address these issues, model compression techniques @xcite , including knowledge distillation @xcite , pruning @xcite and quantization @xcite , have been applied to plms to accelerate inference. however, these methods permanently remove some components of the model, leading to an inevitable decline in performance. additionally, the complexity of these models cannot be adjusted based on the requirements of different hardware since the compression is implemented before the deployment. conversely, several studies have suggested the use of conditional computation algorithms, such as the early exiting (ee) algorithm @xcite , in which each input sample uses a different part of the model so that the computation or latency is reduced on average. based on this, an ee classifier is added between transformer layers. the shallow layer allows the easy samples to exit earlier without the need to perform computations up to the final layer. computations related to deeper layers should only be reserved for hard samples. there are two main limitations of existing ee methods. for existing ee methods, there has been a focus on the exploration of accelerated inference for easy samples. however, for hard samples, acceleration inference is not achieved and still needs to be performed through the last layer. furthermore, the computation consumption is quadratically proportional to the input sequence length. as a result, the average acceleration inference is limited. in addition, the existing early exiting approach is only performed on the vertical perspective, and the operations of the horizon perspective are ignored. that is, each transformer encoder layer is regarded as a basic unit. the model determines how many transformer encoder layers are needed for inference but ignores which tokens in each layer are required to participate in the computation. recent studies @xcite have shown that sequence tokens from the horizontal perspective have high redundancy. to further verify the redundant information from the horizontal perspective, we use the cosine distance to calculate the similarity between tokens in each layer, as shown in figure 1 . the token similarity in the shallow layer is low, indicating that the shallow layer is not fully encoded. the horizontal perspectives of deeper layers, especially the last layer, show high redundancy. this suggests that the previous ee approaches ignored the possibility of reducing the horizontal perspective redundancy to accelerate inference. therefore, these ee approaches yield a suboptimal accelerated inference. to address this issue, an intuitive idea is to reduce the length of the sequence token as the layers grow. tr-bert @xcite uses a reinforcement learning (rl) method to select the tokens that need to be dynamically reduced. however, additional training is needed, which increases the cost of training. in addition, changes to the device platform require retraining of these components, which may greatly limit the application scenarios. in this paper, we propose a unified horizontal and vertical multi-perspective early exiting framework to reduce the computation and latency for fast inference of transformer-based models. this framework contains layer-wise ee for the vertical perspective and sequential token-wise ee for the horizontal perspective. in sequential token-wise ee, different tokens are forced to exit at different layers to reduce computation by emphasizing the informative tokens to the downstream task. conversely, the tokens with less information are truncated, which isolates them from the following computation. to measure the importance of tokens, we use class attention to learn the amount of prediction information. due to the instability of the class attention in the shallow layer, recycling class attention memory is used to enhance the reuse of information across layers and to better identify informative tokens. the tokens with less information do not completely exit. instead, they take a weighted fusion into an ee fusion token for subsequent calculations so that additional parameters and computations are not introduced. for the vertical perspective ee, we introduce ee classifiers in the middle of two transformer layers. samples exit early at the shallow layer when the confidence level is higher than the threshold and skip computation in other layers. this allows the input samples to be predicted with a shallow ee classifier rather than a deep ee classifier. since the shallow ee classifier is weakly expressive, we introduce weighted ensemble self-distillation and recycling ee classifier memory to enhance the performance of the shallow ee classifier. thus, a performance guarantee is provided at high acceleration inferences. the unified multi-perspective early exiting framework has horizontal sparsity and vertical sparsity. based on extensive experiments, it is shown that the proposed model reduces computation and improves acceleration inference while maintaining high performance. the rest of the paper is organized as follows. in section 2, the preliminaries of plms are introduced. in section 3, a detailed description of the proposed methods is provided. the empirical experiments are reported and analyzed in section 4. section 5 briefly introduces the previous studies on acceleration inference for plms. conclusions are finally drawn in section 6. ## preliminaries pretrained language model. a plm consists of l transformer encoders @xcite . the sequence token @xmath0, x 1 , x 2 , ..., x n ] of the text processed by the tokenizer is input into the plm. n is the number of sequence tokens (without the [cls] token), and the [cls] token denote the global text information and final classification. the transformer contains multi-head self-attention (mhsa) and feed-forward network (ffn) modules. the transformer coding process of each layer is defined as cls token recycling ee classifier memory early exiting classifier1 early exiting classifier2 early exiting classifieri early exiting classifierl recycling ee classifier memory recycling ee classifier memory ... ... ... label multi-head self-attention feed forward network class attention score2 0.21 0.64 0.81 0.59 0.16 0.68 0.41 0.16 0.41 0.21 informative tokens ee fusion token sequence token-wise ee class attention memory class attention1 class attention3 cls token recycling ee classifier memory … … … transformer layer 1 … … … transformer layer 2 … … … transformer layer i … … … transformer layer l inputs ( ) l kd ( ) l ce ( ) i kd ( ) i ce (2) kd (2) ce (1) kd (1) ce figure 2: overall architecture of the unified multi-perspective early exiting framework for accelerating. ## multi-perspective early exiting figure 2 shows an overview of the proposed unified multi-perspective early exiting framework, which consists of layer-wise and sequential tokenwise early exiting. layer-wise and sequential token-wise early exiting are orthogonal methods that exit unnecessary computations early by the layer-wise and token-wise of the model, respectively, to achieve faster inference. therefore, we combine these two different perspectives to achieve accelerated inference and minimize computational cost. ## related work model compression. knowledge distillation @xcite refers to the training of smaller student models using the knowledge supervision of pretrained larger teacher models. the student model uses fewer layers to learn knowledge from the teacher's hidden units and logits, e.g., distilbert @xcite and bert-pkd @xcite . pruning @xcite refers to the removal of less important weights or computational units. @xcite analyzed multi-head selfattention importance and removed it. @xcite discarded unimportant layers in the finetuning process. power-bert @xcite progressively reduces the sequence length and accelerates the bert model. however, ignoring the deleted token leads to information loss and thus reduces the performance of the model. @xcite proposed using reinforcement learning to select unimportant tokens to reduce the sequence length. this approach requires additional strategies to converge the model. quantification (shen et al., 2020) is the process of reducing the number of bits needed to represent the model weights. @xcite proposed to quantize the activation function and the weight parameters simultaneously to reduce quantization errors. early exiting. the early exiting approach refers to allowing different samples to exit early in different layers depending on the properties of the input samples. deebert @xcite performs early prediction by introducing multiple ee classifiers and exits early if the confidence is greater than the threshold, and conversely passes to the next layer to continue the computation. fastbert @xcite uses self-distillation to train ee classifiers. pabee @xcite achieves early exit inference when ee classifier predictions are held continuously constant. ## conclusions in this paper, we propose a unified multiperspective early exiting framework that significantly reduces the computation cost and improves acceleration inference within a small performance loss. the multi-perspective early exit framework includes horizontal and vertical perspectives early exiting. it has horizontal sparsity and vertical sparsity for faster inference. extensive experimental results show that compared to previous approaches, the proposed model provides a better tradeoff between model performance and inference efficiency. future work attempt to extend the proposed approach to vision and language pretrained models, taking into account the properties of different modalities.