A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation (2022)

Abstract: Early exiting allows instances to exit at different layers according to the estimation of difficulty. Previous works usually adopt heuristic metrics such as the entropy of internal outputs to measure instance difficulty, which suffers from generalization and threshold-tuning. In contrast, learning to exit, or learning to predict instance difficulty is a more appealing way. Though some effort has been devoted to employing such “learn-to-exit” modules, it is still unknown whether and how well the instance difficulty can be learned. As a response, we first conduct experiments on the learnability of instance difficulty, which demonstrates that modern neural models perform poorly on predicting instance difficulty. Based on this observation, we propose a simple-yet-effective Hash-based Early Exiting approach HashEE) that replaces the learn-to-exit modules with hash functions to assign each token to a fixed exiting layer. Different from previous methods, HashEE requires no internal classifiers nor extra parameters, and therefore is more efficient. HashEE can be used in various tasks (including language understanding and generation) and model architectures such as seq2seq models. Experimental results on classification, regression, and generation tasks demonstrate that HashEE can achieve higher performance with fewer FLOPs and inference time compared with previous state-of-the-art early exiting methods.

## introduction early exiting is a widely used technique to accelerate inference of deep neural networks. with the rising of pre-trained language models (plms) @xcite @xcite @xcite @xcite , early exiting is drawing increasing attention in the nlp community. at its core, early exiting allows simple instances to exit early while allowing hard instances to exit late. thus, how to measure instance difficulty is a crucial problem. most existing early exiting methods attach multiple internal classifiers to the plm and adopt some heuristic metrics, such as entropy @xcite or maximum softmax score @xcite of internal outputs, to measure instance difficulty. however, these methods can not easily generalize to new tasks. on the one hand, these metrics are not accessible on some tasks such as regression. on the other hand, in order for these methods to perform well, one usually needs to fine-tune the threshold, which varies widely across different tasks and datasets. another way to measure instance difficulty is to directly learn it. recent studies @xcite that use the idea of "learn-toexit" have achieved promising results. they jointly train a neural model to predict for each instance the exiting layer. at their core, the learn-to-exit module is to estimate the difficulty for each instance. compared with previous heuristically designed metrics for difficulty, learn-to-exit is task-agnostic and does not require threshold-tuning, therefore is a more promising way. despite their success, it is still unknown whether or how well the instance difficulty can be learned. as a response, in this work, we construct datasets for two kinds of instance difficulty: (a) humandefined difficulty, and (b) model-defined difficulty. the dataset for human-defined difficulty has two labels, 0 for instances that can be annotated by human and 1 for instances that cannot. for modeldefined difficulty, we train a multi-exit bert @xcite , which is attached with an internal classifier at each layer, on a sentence-level classification task, snli @xcite , and a token-level classification task, ontonotes ner @xcite . the trained multi-exit berts are then used to annotate for each development instance whether it can be correctly predicted by each internal classifier. thus, our constructed sentence-level and token-level model-defined diffi-culty datasets are multi-label classification datasets. experimental results demonstrate that, modern neural networks perform poorly on predicting instance difficulty. this observation is consistent with previous work @xcite on estimating instance difficulty for curriculum learning. given that instance difficulty is hard to be predicted, then what works in the learn-to-exit modules? we hypothesis that the consistency between training and inference may play an important role. that is, for a training instance x i that is predicted to exit at layer l, an inference instance x j that is similar with x i should be predicted to exit at layer l, too. since neural networks are usually smooth functions @xcite , this consistency can be easily satisfied by neural learn-to-exit modules. if this hypothesis holds, we can replace the learn-to-exit module with a simple hash function. in particular, we use hash functions to assign each token to a fixed exiting layer. this hash-based early exiting method is named hashee. compared with previous methods that use heuristic metrics for difficulty or jointly learn to exit, hashee offers several advantages: (a) hashee requires no internal classifiers nor extra parameters, which are necessary in previous work. (b) hashee can perform token-level early exiting without supervision, therefore can be widely used on various tasks including language understanding and generation. (c) the speed-up ratio can be easily tuned by modifying the hash function. (d) hashee can significantly accelerate model inference on a per-batch basis instead of per-instance basis as in previous work @xcite @xcite . we conduct experiments on classification, regression, and generation tasks. experimental results on elue @xcite demonstrate that hashee, despite its simplicity, can achieve higher performance with fewer flops and inference time than previous state-of-the-art methods on various tasks. besides, our experiments on several text summarization datasets show that hashee can reduce ∼50% flops of bart @xcite and cpt @xcite while maintaining 97% rouge-1 score. foot_0 2 can instance difficulty be learned? in this section, we examine whether or to what extent instance difficulty can be learned. in particular, we manage to evaluate how well a neural network that trained on some data with difficulty annotation can generalize to unseen data. here we consider two kinds of difficulty: human-defined difficulty and model-defined difficulty. ## related work large-scale pre-trained language models (plms) have achieved great success in recent years. despite their power, the inference is time-consuming, which hinders their deployment in low-latency scenarios. to accelerate plm inference, there are currently two streams of work: (1) compressing a cumbersome plm through knowledge distillation @xcite @xcite , model pruning @xcite , quantization @xcite , module replacing @xcite , etc. (2) selectively activating parts of the model conditioned on the input, such as universal transformer (dehghani et al., 2019), fastbert @xcite , deebert @xcite , pabee @xcite , leebert @xcite , cascadebert @xcite , elasticbert @xcite and other similar methods @xcite @xcite @xcite . different from these methods, our proposed hashee requires no internal classifiers (which imply extra parameters) and supervision, and therefore can be widely used in a variety of tasks and model architectures. ## conclusion we first empirically study the learnability of instance difficulty, which is a crucial problem in early exiting. based on the observation that modern neural models perform poorly on estimating instance difficulty, we propose a hash-based early exiting approach, named hashee, that removes the learning process and only sticks to the consistency between training and inference. our experiments on classification, regression, and generation tasks show that hashee can achieve state-of-the-art performance with fewer computation and inference time. here we also explore other configurations. for each configuration, we assign tokens to b buckets, corresponding to exiting layers {1 + 12b/b} b-1 @xmath0. for instance, if we have 12 layers and 3 buckets, the 3 buckets correspond to the {1, 5, 9} layers. overall results are given in