Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings (2023)

Abstract: Adaptive inference is a simple method for reducing inference costs. The method works by maintaining multiple classifiers of different capacities, and allocating resources to each test instance according to its difficulty. In this work, we compare the two main approaches for adaptive inference, Early-Exit and Multi-Model, when training data is limited. First, we observe that for models with the same architecture and size, individual Multi-Model classifiers outperform their Early-Exit counterparts by an average of 2.3%. We show that this gap is caused by Early-Exit classifiers sharing model parameters during training, resulting in conflicting gradient updates of model weights. We find that despite this gap, Early-Exit still provides a better speed-accuracy trade-off due to the overhead of the Multi-Model approach. To address these issues, we propose SWEET (Separating Weights for Early-Exit Transformers) an Early-Exit fine-tuning method that assigns each classifier its own set of unique model weights, not updated by other classifiers. We compare SWEET’s speed-accuracy curve to standard Early-Exit and Multi-Model baselines and find that it outperforms both methods at fast speeds while maintaining comparable scores to Early- Exit at slow speeds. Moreover, SWEET individual classifiers outperform Early-Exit ones by 1.1% on average. SWEET enjoys the benefits of both methods, paving the way for further reduction of inference costs in NLP.

## introduction pre-trained transformer-based language models such as bert @xcite , deberta @xcite , and gpt3 @xcite have become the go-to tool in nlp . although powerful, the growing size of these models has been a major drawback (thompson et al., 2020; schwartz et al., 1 separating weights in early exit transformers. in both methods, multiple classifiers of increasing sizes are run serially, until a confident prediction is made. in early-exit (left), a single model with multiple classifiers is used, such that early computations are reused by later classifiers. in multi-model (right), a sequence of independent models is used, allowing each classifier to decouple its parameters from other classifiers. 2020a), making them costly to run. various attempts to reduce inference cost have been proposed, including distillation @xcite , pruning @xcite and quantization @xcite . this work focuses on adaptive inference @xcite , a recent approach in which the variability of sample difficulty is leveraged toward a smarter allocation of computational resources. an appealing property of adaptive inference is that it enables dynamic control of the speed-accuracy trade-off. there are two main approaches to adaptive inference, both using a set of classifiers of different sizes. in early-exit @xcite , multiple classification heads are added to the same model at different layers, allowing for early exit during inference (fig. 1 , left). another approach (henceforth multi-model) is to apply multiple independent classifiers of varying capacities serially until a prediction is made @xcite . these approaches have complementary benefits; multi-model allows for easier batching at inference time, and potentially larger savings due to using very efficient models @xcite . early-exit on the other hand is more memory efficient, faster to train, and enables re-use of early computation if an early exit was not taken. in this work, we compare the speed-accuracy behavior of the two approaches when training data is limited. 3 we first observe that early-exit model weights are updated by multiple conflicting gradient signals throughout the training process (fig. 2 , left). we show that this leads to a decrease in performance of individual early-exit classifiers compared to multi-model ones (2.3% gap on average). we find that this gap is higher for the earliest classifiers (5.2% on average) than for later ones (1.4%). we also find that while each multi-model classifier outperforms its early-exit counterpart, it does not translate to an overall better speed-accuracy trade-off. instead, we find that each method dominates performance in a different region: multi-model outperforms early-exit at fast inference speeds, while early-exit is better at slow speeds. multi-model downgraded scores at slow speeds are likely caused by the overhead of running models sequentially to predict hard samples. inspired by our findings, we present sweet, 4 an early-exit method for bridging the performance gap between standard early-exit and multi-model. in sweet, each early-exit classifier only updates the parameters of layers preceding it up to the previous classifier. this way, each model parameter is updated by a single classifier, thus avoiding conflicting gradients during the training of early-exit models (fig. 2 , right ). we experiment with two established pretrained models: bert @xcite and deberta @xcite . we fine-tune early-exit models using sweet on seven text classification tasks from glue @xcite and compare them to early-exit and multi-model baselines. the speed-accuracy curve of sweet dominates both baselines at fast speeds for 21 out of 28 experiments conducted. as for individual clas3 reducing inference costs is particularly helpful when computational resources are limited. such conditions are often paired with restricted access to labeled data or a low budget for data annotation. to evaluate the effectiveness of adaptive inference methods in such scenarios, we limit training data size to a few thousand examples in all our experiments. ## background: adaptive inference adaptive inference aims to reduce inference costs of deep neural nets by matching model and sample complexities. sample difficulty usually varies in real-world data, meaning not all instances require processing by the most powerful classifier. therefore, we can allocate fewer resources to easier instances, and reduce the average inference cost, potentially at the cost of performance degradation. several exit strategies have been developed to control the speed-accuracy trade-off of a model by deciding when to make an early prediction and halt computation @xcite @xcite @xcite . in this work, we mainly experiment with confidence-based exiting @xcite , in which computation halts if the softmax probability assigned to a given label exceeds a predetermined threshold. dynamic control of model's inference speed is gained through setting different threshold values. there are two main adaptive inference approaches, early-exit and multi-model, which we describe below. early-exit early-exit models are deep neural nets with multiple output points, following intermediate layers. in this work, we focus on early-exit implementation as presented in @xcite . during fine-tuning, foot_1 instances are passed through the model, and a loss is calculated based on predictions made by all classifiers. this leads to some model weights being updated by gradient signals from multiple classifiers (fig. 2 , left). at inference time, an instance is passed through the model and its label is predicted sequentially until a decision to exit early is taken, and computation is halted. an appealing property of early-exit is that it allows for efficient re-use of previous computation from lower layers in higher classifiers. however, this means that some model parameters are updated by multiple classifiers, which may lead to sub-optimal performance of individual classifiers due to conflicting gradients. in this work, we study the effect of this property on early-exit models. multi-model in multi-model adaptive inference, a set of independent models of increasing capacity are fine-tuned separately for the same task. at inference time, the models are used sequentially from smallest to largest until a prediction meets some predetermined criterion or until the largest model has been used. this method is more robust than early-exit, being easier to extend and enabling the use of different architectures. additionally, multi-model potentially allows for further computational savings by using models smaller than the smallest early-exit model (a single layer of the backbone model, @xcite . however, it may add overhead to the prediction time of hard instances, as those pass through multiple models with early computations being discarded, bringing the total runtime to exceed that of using the largest model. ## sweet: separating weights for early-exit transformers based on our findings, we aim to design an early-exit method that takes advantage of the benefits of both multi-model and early-exit approaches: making the lower early-exit classifiers as accurate as the multi-model ones, without the additional overhead of the multi-model approach. ## further analysis we turn to evaluate the robustness of our approach. we start with evaluating sweet using a different exit strategy. we then test how our results generalize when using varying amounts of training data. a different exit strategy: learning to exit our experiments so far have used a confidence-based exit strategy (section 2). here we consider a different strategy-learning to exit @xcite , in which a separate module is trained for each classifier, learning whether the classifier's prediction should be trusted, and an exit should be made. this method also allows for a natural extension to regression problems, unlike confidence-based methods that are limited to classification tasks. we use bert {base, large} as a backbone model and fine-tune in the same procedure described in section 3.3. our results (fig. 7 ) reveal that the methods behave similarly to the confidencebased setup; sweet outperforms both baselines at the early stage of the curve (fast speeds), while performing on par with early-exit at slow speeds. we also measure the performance of individual exit layers, as in section 3.2, for models fine-tuned using learning-to-exit. our results ( varying data sizes our experiments so far have focused on low-resource setups (a training set of several thousand examples). we now evaluate how sweet performs as we vary the amount of training data. we fine-tune a bert base model on different portions of the mnli dataset using the same experimental setup as in section 3.2. our results, shown in fig. 8 , indicate that sweet is mostly beneficial when training data is limited to a few thousand examples, but still somewhat useful even with as many as 60k training instances. nonetheless, the effects of conflicting gradients on the earliest early-exit classifier tend to disappear when training on the full dataset (400k instances), making it as good as the smallest multi-model classifier. moreover, in that setup, the use of sweet seems to substan- ## related work several approaches have been proposed for reducing the inference time of large models @xcite . these include knowledge distillation @xcite , in which the knowledge of a large teacher model is transferred to a smaller student model; pruning @xcite , which removes unnecessary model parameters; and weight quantization @xcite , which reduces the floating point precision of model weights. see @xcite , for a recent survey. adaptive inference methods studied in this work are an orthogonal approach, which can be used in combination with these methods @xcite . the use of adaptive inference in deep neural networks has been extensively studied, with successful implementations in various types of networks including recurrent neural networks @xcite and convolutional neural networks @xcite ). in the context of this work, it has also been applied to existing backbone pre-trained language models: xin et al. (2020) implemented early exit strategies on top of classification tasks. @xcite introduced a patience-based exiting, requiring sequential classifiers to agree on a label for a prediction to be made. @xcite extended the use of early-exit in transformers to regression tasks, as well as addressed the issue of reduced performance of the final classifier through the use of an alter-nating training algorithm. schuster et al. (2021 multi-model approaches to adaptive inference have been proposed for vision tasks @xcite as well as for nlp @xcite . @xcite introduced a two-tier design with a highly efficient model serving as the first model and a powerful model serving as the second, enabling the possibility of achieving extremely fast inference speed. finally, the concept of conflicting gradients has been mainly studied in the context of multi-task learning, where a single model is faced with solving different tasks @xcite . to the best of our knowledge, no previous work examined this in the context of early-exit. ## conclusion in this work, we analyzed the performance of two common adaptive inference methods-early-exit and multi-model. we found evidence that model weights are updated by conflicting gradients in the training process of early-exit models, causing classifiers to perform at a sub-optimal level. despite this, we showed that regarding the entire speed-accuracy curve, early-exit is still favorable to multi-model due to the overhead of using independent model runs in a multi-model setup. to address these findings, we proposed sweet, a novel early-exit method, which avoids conflicting gradients by allocating each early-exit classifier a subset of model weights which are updated solely by it. we found that for early-exit models trained with sweet, early classifiers perform better than those of standard early-exit, but later classifiers of sweet are not as good. these measures lead to sweet outperforming both early-exit and multi-model in fast speeds, with slightly worse results than early-exit at slow speeds. overall, our results demonstrate that early-exit models can benefit from fine-tuning algorithms that are tailored to their architecture, and that sweet is a promising approach for improving the speedaccuracy trade-off of early-exit models in the context of adaptive inference.