Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems (2022)

Abstract: Do all instances need inference through the big models for a correct prediction? Perhaps not; some instances are easy and can be answered correctly by even small capacity models. This provides opportunities for improving the computational efficiency of systems. In this work, we present an explorative study on ‘model cascading’, a simple technique that utilizes a collection of models of varying capacities to accurately yet efficiently output predictions. Through comprehensive experiments in multiple task settings that differ in the number of models available for cascading (K value), we show that cascading improves both the computational efficiency and the prediction accuracy. For instance, in K=3 setting, cascading saves up to 88.93% computation cost and consistently achieves superior prediction accuracy with an improvement of up to 2.18%. We also study the impact of introducing additional models in the cascade and show that it further increases the efficiency improvements. Finally, we hope that our work will facilitate development of efficient NLP systems making their widespread adoption in real-world applications possible.

## introduction pre-trained language models such as roberta @xcite , electra @xcite , and t5 @xcite have achieved remarkable performance on numerous natural language processing benchmarks @xcite @xcite . however, these models have a large number of parameters which makes them slow and computationally expensive; for instance, t5-11b requires ∼87 × 10 11 floating point operations (flops) for an inference. this limits their widespread adoption in real-world applications that prefer computationally efficient systems in order to achieve low response times. the above concern has recently received considerable attention from the nlp community leading figure 1 : illustrating a cascading approach with three models (mini, med, and base) arranged in increasing order of capacity. an input is first passed through the smallest model (mini) which fails to predict with sufficient confidence. therefore, it is then inferred using a bigger model (med) that satisfies the confidence constraints and the system outputs its prediction ('contradiction' as dog has four legs). thus, by avoiding inference through large/expensive models, the system saves computation cost without sacrificing the accuracy. to development of several techniques, such as (1) network pruning that progressively removes model weights from a big network @xcite , (2) early exiting that allows multiple exit paths in a model @xcite , (3) adaptive inference that adjusts model size by adaptively selecting its width and depth @xcite , (4) knowledge distillation that transfers 'dark-knowledge' from a large teacher model to a shallow student model @xcite , and (5) input reduction that eliminates less contributing tokens from the input text to speed up inference @xcite . these methods typically require architectural modifications, network manipulation, saliency quantification, or even complex training procedures. moreover, computational efficiency in these methods often comes with a compromise on accuracy. in contrast, model cascading, a simple technique that utilizes a collection of models of varying capacities to accurately yet efficiently output predictions has remained underexplored. in this work, we address the above limitation by first providing mathematical formulation of model cascading and then exploring several approaches to do it. in its problem setup, a collection of models of different capacities (and hence performances) are provided and the system needs to output its prediction by leveraging one or more models. on one extreme, the system can use only the smallest model and on the other extreme, it can use all the available models (ensembling). the former system would be highly efficient but usually poor in performance while the latter system would be fairly accurate but expensive in computation. model cascading strives to get the best of both worlds by allowing the system to efficiently utilize the available models while achieving high prediction accuracy. this is in line with the 'efficiency nlp' @xcite policy document put up by the acl community. consider the case of commitmentbank (de marneffe et al., 2019) dataset on which bert-medium model having just 41.7m parameters achieves 75% accuracy and a bigger model bert-base having 110m parameters achieves 82% accuracy. from this, it is clear that the performance of the bigger model can be matched by inferring a large number of instances using the smaller model and only a few instances using the bigger model. thus, by carefully deciding when to use bigger/more expensive models, the computational efficiency of nlp systems can be improved. so, how should we decide which model(s) to use for a given test instance? figure 1 illustrates an approach to achieve this; it infers an instance sequentially through models (ordered in increasing order of capacity) and uses a threshold over the maximum softmax probability (maxprob) to decide whether to output the prediction or pass it to the next model in sequence. the intuition behind this approach is that maxprob shows a positive correlation with predictive correctness. thus, instances that are predicted with high maxprob get answered at early stages as their predictions are likely to be correct and the remaining ones get passed to the larger models. hence, by avoiding inference through large and expensive models (primarily for easy instances), cascading makes the system computationally efficient while maintaining high prediction performance. we describe several such cascading methods in section 3.2. furthermore, cascading allows custom computation costs as different number of models can be used for inference. we compute accuracies for a range of costs and plot an accuracy-cost curve. then, we calculate its area (auc) to quantify the efficacy of the cascading method. larger the auc value, the better the method is as it implies higher accuracy on average across computation costs. we conduct comprehensive experiments with 10 diverse nlu datasets in multiple task settings that differ in the number of models available for cascading (k value from section 3). we first demonstrate that cascading achieves considerable improvement in computational efficiency. for example, in case of qqp dataset, cascading system achieves 88.93% computation improvement over the largest model (m 3 ) in @xmath0.e. it requires just 11.07% of the computation cost of model m 3 to attain equal accuracy. then, we show that cascading also achieves improvement in prediction accuracy. for example, on cb dataset, the cascading system achieves 2.18% accuracy improvement over m 3 in the @xmath1. similar improvements are observed in settings with different values of k. lastly, we show that introducing additional model in the cascade further increases the efficiency benefits. in summary, our contributions and findings are: 1. model cascading: we provide mathematical formulation of model cascading, explore several methods, and systematically study its benefits. 2. cascading improves efficiency: using accuracy-cost curves, we show that cascading systems require much lesser computation cost to attain accuracies equal to that of big models. 3. cascading improves accuracy: we show that cascading systems consistently achieve superior prediction performance than even the largest model available in the task setting. 4. comparison of cascading methods: we compare performance of our proposed cascading methods and find that dtu (3.2) outperforms all others by achieving the highest auc of accuracy-cost curves on average. we note that model cascading is trivially easy to implement, can be applied to a variety of problems, and can have good practical values. ## related work in recent times, several techniques have been developed to improve the efficiency of nlp systems, such as network pruning @xcite @xcite , quantization @xcite @xcite , knowledge distillation @xcite @xcite , and input reduction @xcite . our work is more closely related to dynamic inference @xcite and adaptive model size @xcite @xcite . @xcite proposed dynamic early exiting for bert (deebert) that speeds up bert inference by inserting extra classification layers between each transformer layer. it allows an instance to choose conditional exit from multiple exit paths. all the weights (including newly introduced classification layers) are jointly learnt during training. @xcite proposed progressive wordvector elimination (power-bert) that reduces intermediate vectors computed along the encoder pipeline. they eliminate vectors based on significance computed using self-attention mechanism. @xcite extended power-bert to length-adaptive transformer which adaptively determines the sequence length at each layer. @xcite proposed a dynamic bert model (dynabert) that adjusts the size of the model by selecting adaptive width and depth. they first train a width-adaptive bert and then distill knowledge from full-size models to small sub-models. lastly, cascading has been studied in machine learning and vision with approaches such as haarcascade @xcite but is underexplored in nlp @xcite . we further note that cascading is non-trivially different from 'ensembling' as ensembling always uses all the available models instead of carefully selecting one or more models for inference. our work is different from existing methods in the following aspects: (1) existing methods typically require architectural changes, network manipulation, saliency quantification, knowledge distillation, or complex training procedures. in contrast, cascading is a simple technique that is easy to implement and does not require such modifications, (2) the computational efficiency in existing methods often comes with a compromise on accuracy. contrary to this, we show that model cascading surpasses the accuracy of even the largest models, (3) existing methods typically require training a separate model for each computation budget; on the other hand, a single cascading system can be adjusted to meet all the computation constraints. (4) finally, cascading does not require an instance to be passed sequentially through the model layers; approaches such as routing (section 3) allow passing it directly to a suitable model. ## model cascading we define model cascading as follows: given a collection of models of varying capacities, the system needs to leverage one or more models in a computationally efficient way to output accurate predictions. as previously mentioned, a system using only the smallest model would be highly efficient but poor in accuracy and a system using all the available models would be fairly accurate but expensive in computation. the goal of cascading is to achieve high prediction accuracy while efficiently leveraging the available models. the remainder of this section is organized as follows: we provide mathematical formulation of cascading in 3.1 and describe its various approaches in 3.2. ## conclusion and discussion we systematically explored model cascading and proposed several methods for it. through comprehensive experiments with 10 diverse nlu datasets, we demonstrated that cascading improves both the computational efficiency and the prediction accuracy. we also studied the impact of introducing another model in the collection and showed that it further improves the computational efficiency of the cascading system. selecting optimal operating threshold: the selection of confidence threshold for models in the cascade is dependent on the computation budget of the system. a low-budget system can select low threshold for the low-cost models (so that lowcost models answer majority of the questions leading to less computation cost) and similarly, highbudget systems can afford to select high thresholds to achieve higher accuracy. in order to select thresholds in an application-independent manner, the ml's standard practice of using the validation data to tune the hyperparameters can be used. outlier/ood detection techniques: outlier/ood detection techniques such as @xcite @xcite can also be explored to decide which instance to pass to the bigger models in the cascade. including linear models in the cascade: this idea can be extended to include non-transformer based less expensive models like linear models or lstm based models. since the computation cost of these models is significantly lower than the transformer based models and yet they achieve nontrivial predictive performance, a cascading system with these models could achieve even more improvement in computational efficiency. we plan to explore this aspect in the future work.