The Right Tool for the Job: Matching Model and Instance Complexities (2020)

Abstract: As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. To better respect a given inference budget, we propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) “exit” from neural network calculations for simple instances, and late (and accurate) exit for hard instances. To achieve this, we add classifiers to different layers of BERT and use their calibrated confidence scores to make early exit decisions. We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks. Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing models which are up to five times faster than the state of the art, while preserving their accuracy. Our method also requires almost no additional training resources (in either time or parameters) compared to the baseline BERT model. Finally, our method alleviates the need for costly retraining of multiple models at different levels of efficiency; we allow users to control the inference speed/accuracy tradeoff using a single trained model, by setting a single variable at inference time. We publicly release our code.

## introduction the large increase in the size of artificial intelligence models often increases production costs @xcite , and can also limit adoption on real-time devices. compared to training, which is a one-time large investment, inference costs are incurred for every instance in production, and can thus add up if high enough, the model takes an early exit, avoiding the computation associated with successive (higher) layers (grayed out). otherwise, the model continues to the next layer/classifier. we apply our method to the bert-large model, modifying its fine-tuning procedure by adding multiple output layers to some of its original ℓ = 24 layers. 4 a classifier at the kth layer, is more efficient, though (presumably) less accurate than a classifier at a later ℓth layer (where ℓ > k). at inference time, we run each instance on these classifiers in increasing order of depth. for each classification decision, we use its confidence as an inferencestopping criterion, continuing to the next, larger classifier only if the current classifier is not confident enough in its prediction. since confidence scores play an important role, we use calibration techniques to make them more reliable. associating classifiers with different layers of the same network allows us to reuse the computation performed by the simple classifiers for the complex ones. see figure 1 for an illustration. we experiment with three text classification benchmarks and two natural language inference (nli) benchmarks. we consider each of our classifiers with different bert layers as individual baselines. we find that using our method leads to a consistently better speed/accuracy tradeoff in almost all cases. in particular, in some cases, we obtain similar performance while being as much as five times faster than our strongest baseline (the original bert-large mode with a single classification layer after the last layer). our approach, while allowing substantially faster inference compared to the standard bertlarge model, is neither slower to fine-tune nor significantly larger in terms of parameters, requiring less than 0.005% additional parameters. moreover, our method is quite flexible: unlike other approaches for inference speed-up such as model distillation or pruning, which require training a different model for each point along the speed/accuracy curve, our method only requires training a single model, and by setting a single variable at inference time-the confidence threshold-supports each point along that curve. finally, our method is orthogonal to compression methods such as model distillation @xcite . our experiments with a distilled version of bert @xcite show that our method further improves the speed/accuracy curve on top of that model. we publicly release our code. foot_3 ## premise: models vary in size, examples vary in complexity our goal in this paper is to make model inference more efficient. our premise relies on two general observations: first, as nlp models become bigger (e.g., in number of parameters), they become both better (in terms of downstream task accuracy), and slower to run. this trend is consistently observed, most notably in recent contextual representations work that compares different variants of the same model @xcite @xcite . second, inputs are not equally difficult. for example, instances differ in length and wealth of linguistic phenomena, which affects the amount of processing required to analyze them. consider the examples below for the task of sentiment analysis: sentence 1 is short and simple to process. in contrast, sentence 2 is long, contains misleading positive phrases ("award-winning writer"), and uses figurative speech ("the plot was written by a 12 year-old"). as a result, it is potentially harder to process. 6 this work leverages these two observations by introducing a method to speed-up inference by matching simple instances with small models, and complex instances with large models. ## experiments to test our approach, we experiment with three text classification and two natural language inference (nli) tasks in english. nli is a pairwise sentence classification task, where the goal is to predict whether a hypothesis sentence entails, contradicts or is neutral to a premise sentence @xcite . below we describe our datasets, our baselines, and our experimental setup. datasets for text classification, we experiment with the ag news topic identification dataset @xcite and two sentiment analysis datasets: imdb @xcite and the binary stanford sentiment treebank (sst; @xcite ). 8 for nli, we experiment with the snli @xcite and multinli (mnli; @xcite datasets. we use the standard train-development-test splits for all datasets except for mnli, for which there is no public test set. as mnli contains two validation sets (matched and mismatched), we use the matched validation set as our validation set and the mismatched validation set as our test set. see baselines we use two types of baselines: running bert-large in the standard way, with a single output layer on top of the last layer, and three efficient baselines of increasing size (figure 2 ). each is a fine-tuned bert model with a single output layer after some intermediate layer. importantly, these baselines offer a speed/accuracy tradeoff, but not within a single model like our approach. as all baselines have a single output layer, they all have a single loss term, such that bert layers 1, . . . , k only focus on a single classification layer, rather than multiple ones as in our approach. as with our model, the single output layer in each of 8 for sst, we only used full sentences, not phrases. as an upper bound to our approach, we consider a variant of our model that uses the exact amount of computation required to solve a given instance. it does so by replacing the confidence-based earlyexit decision function with an oracle that returns the fastest classifier that is able to solve that instance, or the fastest classifier for instances that are not correctly solved by any of the classifiers. ## results a better speed/accuracy tradeoff. figure 3 presents our test results. 10 the blue line shows our model, where each point corresponds to an increasingly large confidence threshold. the leftmost (rightmost) point is threshold 0 (1), with x-value showing the fraction of processing time relative to the standard baseline. our first observation is that our efficient baselines constitute a fast alternative to the standard bert-large model. on ag, a classifier trained on layer 12 of bert-large is 40% faster and within 0.5% of the standard model. on snli and imdb a similar speedup results in 2% loss in performance. most notably, our approach presents a similar or better tradeoff in almost all cases. our model is within 0.5% of the standard model while being 40% (imdb) and 80% (ag) faster. for sst, our curve is strictly above two of the efficient baselines, while being below the standard one. in the two nli datasets, our curve is slightly above the curve for the medium budgets, and below it for lower ones. finally, the results of the oracle baseline indicate the further potential of our approach: in all cases, the oracle outperforms the original baseline by 1.8% (ag) to 6.9% (mnli), while being 4-6 times faster. these results motivate further exploration of better early-exit criteria (see section 6). they also highlight the diversity of the different classifiers. one might expect that the set of correct predictions by the smaller classifiers will be contained in the corresponding sets of the larger classifiers. the large differences between the original baseline and our oracle indicate that this is not the case, and motivate future research on efficient ensemble methods which reuse much of the computation across different models. extreme case analysis our results hint that combining the loss terms of each of our classifiers hurts their performance compared to our baselines, which use a single loss term. for the leftmost point in our graphs-always selecting the most efficient classifier-we observe a substantial drop in performance compared to the corresponding most efficient baseline, especially for the nli datasets. for our rightmost point (always selecting the most accurate classifier), we observe a smaller drop, mostly in sst and mnli, compared to the corresponding baseline, but also slower runtime, probably due to the overhead of running the earlier classifiers. these trends further highlight the potential of our method, which is able to outperform the baseline speed-accuracy curves despite the weaker starting point. it also suggests ways to further improve our method by studying more sophisticated methods to combine the loss functions of our classifiers, and encourage them to be as precise as our baselines. we defer this to future work. ## a criterion for "difficulty" our approach is motivated by the inherent variance in the level of complexity of text instances, and leverages this variance to obtain a better dataset length consistency ag -0.13 0.37 imdb -0.17 0.47 sst -0.19 0.36 snli -0.08 0.44 mnli -0.13 0.39 our method also automatically identifies instances on which smaller models are highly confident in their predictions. here we analyze our data using other definitions of difficulty. perhaps surprisingly, we find that the various definitions are not strongly correlated with ours. the results we observe below, combined with the performance of our oracle baseline (section 5), motivate further study on more advanced methods for early exiting, which could potentially yield even larger computational gains. shorter is easier? we first consider the length of instances: is our model more confident in its decisions on short documents compared to longer ones? to address this we compute spearman's ρ correlation between the confidence level of our most efficient classifier and the document's length. the results in consistency and difficulty our next criterion for "difficulty" is the consistency of model predictions. @xcite proposed a notion of "unforgettable" training instances, which once the model has predicted correctly, it never predicts incorrectly for the remainder of training iterations. such instances can be thought of as "easy" or memorable examples. similarly, @xcite defined test instances as "predictable" if multiple simple models predict them correctly. inspired by these works, we define the criterion of consistency: whether all classifiers in our model agree on the prediction of a given instance, regardless of whether it is correct or not. comparison with hypothesis-only criteria @xcite and @xcite showed that some nli instances can be solved by only looking at the hypothesis-these were artifacts of the annotation process. they argued that such instances are "easier" for machines, compared to those which required access to the full input, which they considered "harder." inter-annotator consensus both nli datasets include labels from five different annotators. we treat the inter-annotator consensus (iac) as another measure of difficulty: the higher the consensus is, the easier the instance. we compute iac for each example as the fraction of annotators who agreed on the majority label, hence this number ranges from 0.6 to 1.0 for five annotators. the correlation with our most efficient classifiers is rather weak, only 0.08 and 0.14. surprisingly, as we move to larger models, the correlation increases, up to 0.32 for the most accurate classifiers. this indicates that the two measures perhaps capture a different notion of difficulty. confidence across labels figure 5 shows the proportion of instances in our validation set that are predicted with high confidence by our calibrated model (90% threshold) for each dataset, label, and model size. we first note that across all datasets, and almost all model sizes, different labels are not predicted with the same level of confidence. for instance, for ag, the layer 0 model predicts the tech label with 87.8% average confidence, compared to 96.8% for the sports label. moreover, in accordance with the overall performance, across almost all datasets and model sizes, the confidence levels increase as the models get bigger in size. finally, in some cases, as we move towards larger models, the gaps in confidence close (e.g., imdb and sst), although the relative ordering hardly ever changes. two potential explanations come up when observing these results; either some labels are easier to predict than others (and thus the models are more confident when predicting them), or the models are biased towards some classes compared to others. to help differentiate between these two hypotheses, we plot in figure 6 the average confidence level and the average f 1 score of the most efficient classifier across labels and datasets. the plot indicates that both hypotheses are correct to some degree. some labels, such as sports for ag and positive for imdb, are both predicted with high confidence, and solved with high accuracy. in contrast, our model is overconfident in its prediction of some labels (business for ag, positive for sst), and underconfident in others (tech for ag, entailment for mnli). these findings might indicate that while our method is designed to be globally calibrated, it is not necessarily calibrated for each label individually. such observations relate to existing concerns regarding fairness when using calibrated classifiers @xcite . ## related work methods for making inference more efficient have received considerable attention in nlp over the years @xcite . as the field has converged on deep neural architecture solutions, most efforts focus on making models smaller (in terms of model parameters) in order to save space as well as potentially speed up inference. in model distillation @xcite a smaller model (the student) is trained to mimic the behavior or structure of the original, larger model (the teacher). the result is typically a student that is as accurate as the teacher, but smaller and faster @xcite @xcite . @xcite removes some of the weights in the network, resulting in a smaller, potentially faster network. the basic pruning approach removes individual weights from the network @xcite . more sophisticated approaches induce structured sparsity, which removes full blocks @xcite @xcite . @xcite and @xcite pruned deep models by applying dropout to different layers, which allows dynamic control of high confidence by the model is sometimes explained by "easy" classes that are predicted with high f 1 (e.g., sports in ag). other cases might stem from biases of the model which make it overconfident despite the label being harder than other labels (e.g., positive in sst). the speed/accuracy tradeoff of the model without retraining. our method also allows for controlling this tradeoff with a single training pass, and yields computational savings in an orthogonal manner: by making early exit decisions. quantization is another popular method to decrease model size, which reduces the numerical precision of the model's weights, and therefore both speeds up numerical operations and reduces model size @xcite @xcite . some works introduced methods to allocate fewer resources to certain parts of the input (e.g., certain words), thereby potentially reducing training and/or inference time @xcite . our method also puts less resources into some of the input, but does so at the document level rather than for individual tokens. a few concurrent works have explored similar ideas for dynamic early exits in the transformer model. elbayad et al. (2020) and dabre et al. (2020) ## conclusion we presented a method that improves the speed/accuracy tradeoff for inference using pretrained language models. our method makes early exits for simple instances that require less processing, and thereby avoids running many of the layers of the model. experiments with bert-large on five text classification and nli datasets yield substantially faster inference compared to the standard approach, up to 80% faster while maintaining similar performance. our approach requires neither additional training time nor significant number of additional parameters compared to the standard approach. it also allows for controlling the speed/accuracy tradeoff using a single model, without retraining it for any point along the curve.