W ISDOM OF C OMMITTEES : A N O VERLOOKED A PPROACH T O F ASTER AND M ORE A CCURATE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Committee-based models (ensembles or cascades) construct models by combining existing pre-trained ones.
While ensembles and cascades are well-known techniques that were proposed before deep learning, they are not considered a core building block of deep model architectures and are rarely compared to in recent literature on developing efﬁcient models.
In this work, we go back to basics and conduct a comprehensive analysis of the efﬁciency of committee-based models.
We ﬁnd that even the most simplistic method for building committees from existing, independently trained networks can match or exceed the accuracy of stateof-the-art models while being drastically more efﬁcient.
These simple committeebased models also outperform sophisticated neural architecture search methods (e.g., BigNAS).
These ﬁndings hold true for several tasks, including image classiﬁcation, video classiﬁcation, and semantic segmentation, and various architecture families, such as ViT, EfﬁcientNet, ResNet, MobileNetV2, and X3D.
For example, an EfﬁcientNet cascade can achieve a 5.4x speedup over B7 and a ViT-based cascade can achieve a 2.3x speedup over ViT-L-384 while being equally accurate.
I NTRODUCTION
[Figure]
Optimizing the efﬁciency of neural networks is important for real-world applications as they can only use limited computational resources and often have requirements on response time.
There has been considerable work in this direction (Howard et al., 2017; Zhang et al., 2018; Tan & Le, 2019), but they mostly focus on designing novel network architectures that achieves a favorable speed-accuracy trade-off.
Here, we do not present any novel method or architecture design.
Instead, we focus on analyzing the accuracy and efﬁciency of a simple paradigm: committeebased models.
We use the term “committee” to refer to model ensembles or cascades, which indicates that they are built using multiple independent models.
Committee-based models have been extensively studied and used before deep learning (Breiman, 1996; Schapire, 1990; Freund & Schapire, 1997;
Viola & Jones, 2001).
However, when comparing the efﬁciency of deep models, committeebased models are rarely considered in recent work (Howard et al., 2017; Zhang et al., 2018; Tan & Le, 2019).
There still lacks a systematic understanding of their efﬁciency in comparison with single models – models that only use one network.
Such an understanding is informative for both researchers to push the frontier of efﬁcient models and practitioners to select model designs in real-world applications.
To ﬁll this knowledge gap, we conduct a comprehensive analysis of the efﬁciency of committeebased models.
To highlight the practical beneﬁt of committee-based models, we intentionally choose the simplest possible method, which directly uses off-the-shelf, independently pre-trained models to build ensembles or cascades.
We ensemble multiple pre-trained models via a simple average over their predictions (Sec. 3).
For cascades, we sequentially apply each model and use a simple heuristic ( e.g. , maximum probability in the prediction) to determine when to exit from the cascade (Sec. 4).
We show that even this method already outperforms state-of-the-art architectures found by costly neural architecture search (NAS) methods.
Note that this method works with off-the-shelf models and does not use specialized techniques.
For example, it differs from Boosting (Schapire, 1990) where each new model is conditioned on previous ones, and does not require the weight generation mechanism in previous efﬁcient ensemble methods (Wen et al., 2020).
This method does not require the training of an early exit policy (Bolukbasi et al., 2017; Guan et al., 2018) or the specially designed multi-scale architecture (Huang et al., 2018) in previous work on building cascades.
To be clear, the contribution of this paper is not in the invention of model ensembles and cascades, as they have been known for decades, and is not in a new proposed method to build them.
Instead, it is in a thorough evaluation and comparison of committee-based models to commonly used model architectures.
Our analysis shows that committee-based models provide a simple complementary paradigm to achieve superior efﬁciency without tuning the architecture.
One can often improve accuracy while reducing inference and training cost by building committees out of existing networks.
Our ﬁndings generalize to a wide variety of tasks, including image classiﬁcation, video classiﬁcation, and semantic segmentation, and hold true for various architecture families: ViT (Dosovitskiy et al., 2021), EfﬁcientNet (Tan & Le, 2019), ResNet (He et al., 2016), MobileNetV2 (Sandler et al., 2018), and X3D (Feichtenhofer, 2020).
We summarize our ﬁndings as follows:
• Ensembles are more cost-effective than a single model in the large computation regime (Sec. 3).
For example, an ensemble of two separately trained EfﬁcientNet-B5 models matches B7 accuracy, a state-of-the-art
ImageNet model, while having almost 50% less FLOPs (20.5B vs. 37B)
• Cascades outperform single models in all computation regimes (Sec. 4&5).
Our cascade matches B7 accuracy while using on average 5.4x fewer FLOPs.
We can also achieve a 2.3x speedup over ViT-L-384, a Transformer-based architecture, while matching its accuracy on ImageNet.
• We further show that (1) the efﬁciency of cascades is evident in both FLOPs and on-device latency and throughput (Sec. 5.1); (2) cascades can provide a guarantee on worst-case FLOPs (Sec. 5.2); (3) one can build self-cascades using a single model with multiple inference resolutions to achieve a signiﬁcant speedup (Sec. 6).
• Committee-based models are applicable beyond image classiﬁcation (Sec. 7) and outperform single models on the task of video classiﬁcation and semantic segmentation.
Our cascade outperforms X3D-XL by 1.2% on Kinetics-600 (Carreira et al., 2018) while using fewer FLOPs.
R ELATED W ORK
Efﬁcient Neural Networks.
There has been signiﬁcant progress in designing efﬁcient neural networks.
In early work, most efﬁcient networks, such as MobileNet (Howard et al., 2017; Sandler et al., 2018) and ShufﬂeNet (Howard et al., 2019), were manually designed.
Recent work started to use neural architectures search (NAS) to automatically learn efﬁcient network designs (Zoph et al., 2018; Cao et al., 2019; Tan et al., 2019; Tan & Le, 2019; Chaudhuri et al., 2020).
They mostly fcous on improving the efﬁciency of single models by designing better architectures, while we explore committee-based models without tuning the architecture.
Ensembles.
Ensemble learning has been well studied in machine learning and there have been many seminal works, such as Bagging (Breiman, 1996), Boosting (Schapire, 1990), and AdaBoost (Freund & Schapire, 1997).
Ensembles of neural networks have been used for many tasks, such as image classiﬁcation (Szegedy et al., 2015; Huang et al., 2017a), machine translation (Wen et al., 2020), active learning (Beluch et al., 2018), and out-of-distribution robustness (Lakshminarayanan et al., 2017; Fort et al., 2019; Wenzel et al., 2020).
But the efﬁciency of model ensembles has rarely been systematically investigated.
Recent work indicated that ensembles can be more efﬁcient than single models for image classiﬁcation (Kondratyuk et al., 2020; Lobacheva et al., 2020).
Our work further substantiates this claim through the analysis of modern architectures on large-scale benchmarks.
Cascades.
A large family of works have explored using cascades to speed up certain tasks.
For example, the seminal work from Viola & Jones (2001) built a cascade of increasingly complex classiﬁers to speed up face detection.
Cascades have also been explored in the context of deep neural networks.
Bolukbasi et al.
(2017) reduced the average test-time cost by learning a policy to allow easy examples to early exit from a network.
A similar idea was also explored by Guan et al.
Huang et al.
(2018) proposed a specially designed architecture Multi-Scale DenseNet to better incorporate early exits into neural networks.
Given a pool of models, Streeter (2018) presented an approximation algorithm to produce a cascade that can preserve accuracy while reducing FLOPs and demonstrated improvement over state-of-the-art NAS-based models on ImageNet.
Different from previous work that primarily focuses on developing new methods to build cascades, we show that even the most straightforward method can already provide a signiﬁcant speedup without training an early exit policy (Bolukbasi et al., 2017; Guan et al., 2018) or designing a specialized multi-scale architecture (Huang et al., 2018).
Dynamic Neural Networks.
Dynamic neural networks allocate computational resources based on the input example, i.e. , spending more computation on hard examples and less on easy ones (Han et al., 2021).
For example, Shazeer et al.
(2017) trained a gating network to determine what parts in a high-capacity model should be used for each example.
Recent work (Wu et al., 2018; Veit &
Belongie, 2018; Wang et al., 2018) explored learning a policy to dynamically select layers or blocks to execute in ResNet based on the input image.
Our analysis shows that cascades of pre-trained networks are actually a strong baseline for dynamic neural networks.
E NSEMBLES ARE A CCURATE , E FFICIENT , AND F AST TO T RAIN
Model ensembles are useful for improving accuracy, but the usage of multiple models also introduces extra computational cost.
When the total computation is ﬁxed, which one will give a higher accuracy: single models or ensembles?
The answer is important for real-world applications but this question has rarely been systematically studied on modern architectures and large-scale benchmarks.
We investigate this question on ImageNet (Russakovsky et al., 2015) with three architecture families:
EfﬁcientNet (Tan & Le, 2019), ResNet (He et al., 2016), and MobileNetV2 (Sandler et al., 2018).
Each architecture family contains a series of networks with different levels of accuracy and computational cost.
Within each family, we train a pool of models, compute the ensemble of different combinations of models, and compare these ensembles with the single models in the family.
We denote an ensemble of n image classiﬁcation models by { M 1 , . . .
, M n } , where M i is the i th model.
Given an image x , α i =
M i ( x ) is a vector representing the logits for each class.
To ensemble the n models, we compute the mean of logits 1 α ens = 1 n (cid:80) i
α i and predicts the class for image x by applying argmax to α ens .
The total computation of the ensemble is FLOPs ens
= (cid:80) i FLOPs( M i ) , where FLOPs( · ) gives the FLOPs of a model.
We show the top-1 accuracy on ImageNet and FLOPs of single models and ensembles in Figure 2.
Since there are many possible combinations of models to ensemble, we only show those Pareto optimal ensembles in the ﬁgure.
We see that ensembles are more cost-effective than large single models, e.g. , EfﬁcientNet-B5/B6/B7 and ResNet-152/200.
But in the small computation regime, single models outperform ensembles.
For example, the ensemble of 2 B5 matches B7 accuracy while using about 50% less FLOPs.
However, ensembles use more FLOPs than MobileNetV2 when they have a similar accuracy.
A possible explanation of why model ensembles are more powerful at large computation than at small computation comes from the perspective of bias-variance tradeoff.
Large models usually have small bias but large variance, where the variance term dominates the test error.
Therefore, ensembles are beneﬁcial at large computation as they can reduce the variance in prediction (Breiman, 1996).
For small models, the bias term dominates the test error.
Ensembles can reduce the variance, but this cannot compensate the fact that the bias of small models is large.
Therefore, ensembles are less powerful at small computation.
[Figure]
Our analysis indicates that instead of using a large model, one should use an ensemble of multiple relatively smaller models, which would give similar performance but with fewer FLOPs.
In practice, model ensembles can be easily parallelized ( e.g. , using multiple accelerators), which may provide further speedup for inference.
Moreover, often the total training cost of an ensemble is much lower than that of an equally accurate single model (see appendix for more details).
F ROM E NSEMBLES TO C ASCADES
In the above we have identiﬁed the scenarios where ensembles outperform or underperform single models.
Speciﬁcally, ensembles are not an ideal choice when only a small amount of computation is allowed.
In this section, we show that by simply converting an ensemble to a cascade, one can signiﬁcantly reduce the computation and outperform single models in all computation regimes.
Algorithm
Input : Models { M i } .
Thresholds { t i } .
Test image x . for k = 1 , 2 , . .
. , n do α cas = 1 k (cid:80) ki
=1 α i = 1 k (cid:80) ki
=1 M i ( x )
FLOPs cas =
(cid:80) ki =1 FLOPs( M i )
Early exit if conﬁdence score g ( α cas ) ≥ t k end for
Return α cas and FLOPs cas
Applying an ensemble is wasteful for easy examples where a subset of models will give the correct answer.
Cascades save computation via early exit - potentially stopping and outputting an answer before all models are used.
The total computation can be substantially reduced if we accurately determine when to exit from cascades.
For this purpose, we need a function to measure how likely a prediction is correct.
This function is termed conﬁdence (more details in Sec. 4.1).
A formal procedure of cascades is provided in
Algorithm 1.
Note that our cascades also average the predictions of the models having been used so far.
So for examples where all models are used, the cascade effectively becomes an ensemble.
4. C ONFIDENCE F UNCTION
Let g ( · ): R N → R be the conﬁdence function, which maps maps predicted logits α to a conﬁdence score. The higher g ( α ) is, the more likely the prediction
α is correct.
Previous work (Huang et al., 2017b; Streeter, 2018) tried several simple metrics to indicate the prediction conﬁdence, such as the the maximum probability in the predicted distribution, the gap between the top-2 logits or probabilities, and the (negative) entropy of the distribution.
As shown in Figure 3, all the metrics demonstrate reasonably good performance on measuring the conﬁdence of a prediction, i.e. , estimating how likely a prediction is correct (see appendix for more details and results).
In the following experiments, we adopt the maximum probability metric, i.e. , g ( α ) = max(softmax( α )) 2 .
[Figure]
[Figure]
For a cascade of n models { M i } , we also need ( n − 1) thresholds { t i } on the conﬁdence score, where we use t i to decide whether a prediction is conﬁdent enough to exit after applying model M i (see Algorithm 1).
As we deﬁne g ( · ) as the maximum probability, t i is in [0 , 1] .
A smaller t i indicates more images will be passed to the next model
M i +1 .
A cascade will reduce to an ensemble if all the thresholds { t i } are set to 1.
t n is unneeded, since the cascade will stop after applying the last model M n , no matter how conﬁdent the prediction is.
We can ﬂexibly control the trade-off between the computation and accuracy of a cascade through thresholds { t i } .
To understand how the thresholds inﬂuence a cascade, we visualize several 2-model cascades in Figure 4.
For each cascade, we sweep t 1 from 0 and 1 and plot the results.
Note that all the curves in Figure 4 have a plateau, indicating that we can signiﬁcantly reduce the average FLOPs without hurting the accuracy if t 1 is properly chosen.
We select the thresholds { t i } on held-out validation images according to the target FLOPs or validation accuracy.
In practice, we ﬁnd such thresholds via grid search.
Note that the thresholds are determined after all models are trained.
We only need the logits of validation images to determine { t i } , so computing the cascade performance for a speciﬁc choice of thresholds is fast, which makes grid search computationally possible.
4. C ONVERTING E NSEMBLES TO C ASCADES
For each ensemble in Figure 2, we convert it to a cascade that uses the same set of models.
During conversion, we set the conﬁdence thresholds such that the cascade performs similar to the ensemble while the FLOPs are minimized.
By design in cascades some inputs incur more FLOPs than others.
So we report the average FLOPs computed over all images in the test set.
We see that cascades consistently use less computation than the original ensembles and outperform single models in all computation regimes and for all architecture families.
Taking 2 EfﬁcientNet-B as an example (see Figure 2a), the ensemble initially obtains a similar accuracy to B3 but uses more
FLOPs.
After converting this ensemble to a cascade, we successfully reduce the average FLOPs to 1.3B (1.4x speedup over B3) and still achieve B3 accuracy.
Cascades also outperform small
MobileNetV2 models in Figure 2c.
M ODEL S ELECTION FOR B UILDING C ASCADES
The cascades in Figure 2 do not optimize the choice of models and directly use the set of models in the original ensembles.
For best performance, we show that one can design cascades to match a speciﬁc target FLOPs or accuracy by selecting models to be used in the cascade.
Let M be the set of available models, e.g. , models in the EfﬁcientNet family.
Given a target FLOPs β , we select n models M =
{ M i ∈ M} and conﬁdence thresholds T
= { t i } by solving the following problem:
[Equation]
[Table]
[Table] where C ( M, T ) is the cascade of models { M i } with thresholds { t i } , Accuracy( · ) gives the validation accuracy of a cascade [Table] and FLOPs( · ) gives the average FLOPs.
Similarly, we can also build a cascade to match a target validation accuracy γ by minimizing FLOPs ( C ( M, T )) with the constraint Accuracy ( C ( M, T )) ≥ γ .
Note that this optimization is done after all models in M were independently trained.
The difﬁculty of this optimization depends on the size of M and the number of models in the cascade n .
The problem will be challenging if |M| or n is large.
In our case, |M| and n are not prohibitive, e.g. , |M| = 8 and n ≤ 4 for EfﬁcientNet family.
We are therefore able to solve the optimization problem with exhaustive search.
See appendix for more details.
5. T ARGETING FOR A S PECIFIC FLOP S OR A CCURACY
For each single EfﬁcientNet, ResNet or MobileNetV2, we search for a cascade to match its FLOPs (Middle in Table 1) or its accuracy (Right in Table 1).
Notably, other than convolutional networks, we also consider a Transformer-based architecture – ViT (Dosovitskiy et al., 2021).
We build a cascade of ViT-Base and ViT-Large to match the computation or accuracy of ViT-Large (Table 2).
For ViT, we measure the speedup in throughput (more details on throughput below).
When building cascades, we consider all networks in the same family as the set of available models.
The same model type is allowed to be used for multiple times in a cascade but they will be different models trained separately.
For ImageNet experiments, the search is conducted on a small set of held-out training images and cascades are evaluated on the original validation set.
We provide more experimental details in appendix.
Results in Table 1&2 further substantiate our ﬁnding that cascades are more efﬁcient than single models in all computation regimes.
For small models, we can outperform MobileNetV2-1.0@ Compared with Efﬁcient-B6 or B7, our cascade achieves a 3.8x or 5.5x reduction in latency respectively. [Table]
[Table] by 1.4% using equivalent FLOPs. For large models, we can obtain 2.3x speedup over ViT-Land 5.4x over EfﬁcientNet-B7 while matching their accuracy.
To better understand how a cascade works [Table] we compute the percentage of images that exit from the cascade at each stage.
The cascade above that matches B7 accuracy contains four models: [B3, B5,
B5, B5].
In this cascade, 67.3% images only consume the cost of B3 and only 5.5% images use all four models.
This saves a large amount of computation compared with using B7 for all the images.
On-device Latency and Throughput.
In the above, we mostly use average FLOPs to measure the computational cost.
We now report the latency and throughput of cascades on TPUv3 in Table 3&to conﬁrm that the reduction in FLOPs can translate to the real speedup on hardware.
Cascades are useful for online processing with a ﬁxed batch size 1.
Using batch size 1 is sub-optimal for hardware, but it still happens in real-world applications, e.g. , mobile phone cameras processing a single image (Wadhwa et al., 2018) or servers that need to rapidly return a result without waiting for enough queries to form a batch.
Table 3 shows the average latency of cascades on TPUv3 with batch size 1.
Cascades are up to 5.5x faster than single models with comparable accuracy.
Cascades are also useful for ofﬂine data processing, where work can be batched to fully utilize the hardware.
We can apply the ﬁrst model in the cascade to all examples, and then select only a subset of examples to apply the second model and so forth.
Table 4 reports the throughput (images processed per second) of cascades on TPUv3 via batch processing.
Cascades have signiﬁcantly higher throughput than comparable models.
We provide more results in appendix.
Cascades outperform novel architectures found by costly NAS methods.
[Table]
Comparison with NAS.
We also compare with state-of-the-art NAS methods, e.g. , BigNAS (Yu et al., 2020), OFA (Cai et al., 2020) and Cream (Peng et al., 2020), which can ﬁnd architectures better than EfﬁcientNet.
But as shown in Table 5, a simple cascade of EfﬁcientNet without tuning the architecture already outperforms these sophisticated NAS methods.
The strong performance and simplicity of cascades should motivate future research to include them as a strong baseline when proposing novel architectures. [Table] [Table] [Table]
5. G UARANTEE ON W ORST - CASE FLOP S
Up until now we have been measuring the computation of a cascade using the average FLOPs across all images.
But for some images, it is possible that all the models in the cascade need to be applied.
In this case [Table] the average FLOPs cannot fully indicate the computational cost of a cascade.
Therefore, we now consider worst-case FLOPs of a cascade, the sum of FLOPs of all models in the cascade.
For the three cascades in Table 1 that match the accuracy of B5, B6 or B
[Table]
we report their worst-case
FLOPs in Table 6.
We notice that the cascade for B7 uses fewer FLOPs than B7 in the worst-case scenario
[Table]
but the cascades for B5 and B6 have higher worst-case FLOPs than the network they compare to (see ‘w/o’ in Table 6)
[Table]
We can easily ﬁnd cascades with a guarantee on worst-case FLOPs by adding one more constraint on the choice of models: (cid:80) i FLOPs( M i ) ≤ β worst , where β worst is the upper bound on the worstcase FLOPs of the cascade.
With the new condition, we re-select models in the cascades to match of accuracy of B5 or B6.
As shown in Table 6, compared with single models, the new cascades achieve a signiﬁcant speedup in average-case FLOPs and also ensure its worst-case FLOPs are smaller.
The new cascades with the guarantee on worst-case FLOPs are useful for applications with strict requirement on response time.
S ELF - CASCADES
Cascades typically contain multiple models.
This requires training multiple models and combining them after training.
What about when only one model is available?
We demonstrate that one can convert a single model into a cascade by passing the same input image at different resolutions to the model.
Here, we leverage the fact that resizing an image to a higher resolution than the model is trained on often yields a higher accuracy (Touvron et al., 2019) at the cost of more computation.
We call such cascades as “self-cascades” since these cascade only contain the model itself.
Given a model M , we build a 2-model cascade, where the ﬁrst model is applying M at resolution r 1 and the second model is applying M at a higher resolution r 2 ( r 2 > r 1 ) .
We build self-cascades using EfﬁcientNet models.
Since each EfﬁcientNet is deﬁned with a speciﬁc resolution ( e.g. ,for B1), we set r 1 to its original resolution and set r 2 to a higher resolution.
We set the conﬁdence threshold such that the self-cascade matches the accuracy of a single model.
Table 7 shows that self-cascades easily outperform single models, i.e. , obtaining a similar accuracy with fewer FLOPs.
Table 7 also suggests that if we want to obtain B7 accuracy, we can train a B model and then build a self-cascade, which not only uses much fewer FLOPs during inference, but also takes much shorter time to train.
Self-cascades provide a way to convert one single model to a cascade which will be more efﬁcient than the original single model.
The conversion is almost free and does not require training any additional models.
They are useful when one does not have resources to train additional models or the training data is unavailable ( e.g. , the model is downloaded).
A PPLICABILITY BEYOND I MAGE C LASSIFICATION
We now demonstrate that the beneﬁt of cascades generalizes beyond image classiﬁcation.
7. V IDEO C LASSIFICATION
Similar to image classiﬁcation, a video classiﬁcation model outputs a vector of logits over possible classes.
We use the same procedure as above to build cascades of video classiﬁcation models.
[Table]
We consider the X3D (Feichtenhofer, 2020) architecture family for video classiﬁcation, which is the state-of-the-art in terms of both the accuracy and efﬁciency.
The X3D family contains a series of models of different sizes.
Speciﬁcally, we build cascades of X3D models to match the FLOPs or accuracy of X3D-M, X3D-L or X3D-XL on Kinetics- [Table] (Carreira
[Table]
al
[Table]
The results are summarized in Table 8, where cascades signiﬁcantly outperform the original X3D models. Following X3D Feichtenhofer (2020), ‘ × 30’ in Table 8 means we sample 30 clips from each input video during evaluation (see appendix for more details). Our cascade outperforms X3DXL, a state-of-the-art video classiﬁcation model, by 1.2% while using fewer average FLOPs.
Our cascade can also match the accuracy of X3D-XL with 3.7x fewer average FLOPs.
7. S EMANTIC S EGMENTATION
[Table]
In semantic segmentation, models predict a vector of logits for each pixel in the image.
This differs from image classiﬁcation, where the model makes a single prediction for the entire image.
We therefore revisit the conﬁdence function deﬁnition to handle such dense prediction tasks.
Similar to before, we use the maximum probability to measure the conﬁdence of the prediction for a single pixel p , i.e. , g ( α p ) =
max(softmax( α p )) , where α p is the predicted logits for pixel p .
Next, we need a function g dense ( · ) to rate the conﬁdence of the dense prediction for an image, so that we can decide whether to apply the next model to this image based on this conﬁdence score.
For this purpose, we deﬁne g dense ( · ) as the average conﬁdence score of all the pixels in the image: g dense ( R ) = 1 | R | (cid:80) p ∈ R g ( α p ) , where R represents the input image.
In a cascade of segmentation models, we decide whether to pass an image R to the next model based on g dense ( · ) .
Since the difﬁculty to label different parts in one image varies signiﬁcantly, e.g. , roads are easier to segment than trafﬁc lights, making a single decision for the entire image can be inaccurate and leads to a waste of computation.
Therefore, in practice, we divide an image into grids and decide whether to pass each grid to the next model separately.
We conduct experiments on Cityscapes (Cordts et al., 2016) and use mean IoU (mIoU) as the metric.
We build a cascade of DeepLabv3-ResNet-50 and DeepLabv3-ResNet-101 (Chen et al., 2017) and report the reults in Table 9.
s is the size of the grid.
The full image resolution is 1024 × 2048, so s = 512 means the image is divided into 8 grids.
If we operate on the full image level (‘full’), the cascade will use more FLOPs than ResNet-101.
But if operating on the grid level, the cascade can successfully reduce the computation without hurting the performance.
For example, the smaller grid size (‘ s = 128 ’) yields 1.3x reduction in FLOPs while matching the mIoU of ResNet-101.
C ONCLUSION
We show that committee-based models, i.e. , model ensembles or cascades, provide a simple complementary paradigm to obtain efﬁcient models without tuning the architecture.
Notably, cascades can match or exceed the accuracy of state-of-the-art models on a variety of tasks while being drastically more efﬁcient.
Moreover, the speedup of model cascades is evident in both FLOPs and on-device latency and throughput.
The fact that these simple committee-based models outperform sophisticated
NAS methods, as well as manually designed architectures, should motivate future research to include them as strong baselines whenever presenting a new architecture.
For practitioners, committee-based models outline a simple procedure to improve accuracy while maintaining efﬁciency that only needs off-the-shelf models.
A O N - DEVICE L ATENCY AND T HROUGHPUT
We report the on-device latency and throughput of cascades to conﬁrm that the reduction in FLOPs can translate to the real speedup on hardware.
The throughput or latency of a model is highly dependent on the batch size.
So we consider two scenarios: (1) online processing, where we use a ﬁxed batch size 1, and (2) ofﬂine processing, where we can batch the examples.
Online Processing.
Cascades are useful for online processing with a ﬁxed batch size 1.
Using batch size 1 is sub-optimal for the utilization of accelerators like GPU or TPU but it still happens in some real-world applications, e g. , mobile phone cameras processing a single image (Wadhwa et al., 2018) or servers that need to rapidly return a result without waiting for enough queries to form a batch.
We report the average latency of cascades on TPUv3 with batch size 1 in Table 10.
Cascades achieve a similar accuracy to single models but use a much smaller average latency to return the prediction.
Ofﬂine Processing.
Cascades are also useful for ofﬂine processing of large-scale data.
For example, when processing all frames in a large video dataset, we can ﬁrst apply the ﬁrst model in the cascade to all frames, and then select a subset of frames based on the prediction conﬁdence to apply following models in the cascade.
In this way all the processing can be batched to fully utilize the accelerators.
We report the throughput of cascades on TPUv3 in Table 11, which is measured as the number of images processed per second.
We use batch size 16 when running models on TPUv3 for the case of ofﬂine processing.
As shown in Table 11, cascades achieve a much higher throughput than single models while being equally accurate.
For clariﬁcation, only the throughput of ViT models in Tableis measured on RTX 3090 while the throughput for other cases is measured on TPUv3.
B T RAINING D ETAILS OF M ODELS ON I MAGE N ET
For our experiments on ImageNet, we consider 4 architecture families: EfﬁcientNet (Tan & Le, 2019), ResNet (He et al., 2016), MobileNetV2 (Sandler et al., 2018), and ViT (Dosovitskiy et al., 2021).
All the single models used in our experiments are independently trained with their original training procedure.
We do not change the training schedule or any other hyper-parameters.
We train EfﬁcientNet models with the ofﬁcial open-source implementation 3 provided by the authors.
Readers can refer to their implementation for more training details of EfﬁcientNet.
We do not train
MobileNetV2 and ViT oursvelves.
For MobileNetV2, we download the pre-trained checkpoints from its ofﬁcial open-source implementation 4 provided by the authors.
For ViT, we directly use the pre-trained checkpoints provided by the Hugging Face Team 5 .
[Table]
[Table]
C E FFICIENCY OF E NSEMBLES C. T RAINING T IME OF E NSEMBLES
In Sec. 3, we show that ensembles match the accuracy of large single models with fewer inference
FLOPs.
We now show that the total training cost of an ensemble can also be lower than an equally accurate single model. [Table]
[Table] We show the training time of single EfﬁcinetNet models in Table 12.
We use 32 TPUv3 cores to train B0 to B5, and 128 TPUv3 cores to train B6 or B7.
All the models are trained with the public ofﬁcial implementation of EfﬁcientNet.
We choose the ensemble that matches the accuracy of B or B7 and compute the total training time of the ensemble based on Table 12.
As shown in Table 13, the ensemble of 2 B5 can match the accuracy of B7 while being faster in both training and inference.
C. E XPERIMENTAL D ETAILS
When analyzing the efﬁciency of ensembles, we conduct experiments on three architecture families:
EfﬁcientNet (Tan & Le, 2019), ResNet (He et al., 2016) and MobileNetV2 (Sandler et al., 2018).
The
EfﬁcientNet family consists of eight networks: EfﬁcientNet-B0 to B7.
We consider four networks in the ResNet family: ResNet-50/101/152/200.
For MobileNetV2, we consider MobileNetV20.75@160, 1.0@160, 1.0@192, 1.0@224, and 1.4@224.
Each MobileNetV2 network is represented in the form of w @ r , where w is the width multiplier and r is the image resolution.
For EfﬁcientNet, we consider ensembles of two to four models of either the same or different architectures.
When the same architecture is used multiple times in an ensemble, e.g. , B5 and B5, they are different models trained separately from different random initializations.
The FLOPs range of ResNet or MobileNetV2 models is relatively narrow compared with EfﬁcientNet, so we only consider ensembles of two models for ResNet and MobileNetV2.
D F ROM E NSEMBLES TO C ASCADES D. C ONFIDENCE F UNCTION
The higher the conﬁdence score g ( α ) is, the more likely the prediction given by α is correct.
In Sec 4.1, we compare different choices for the conﬁdence function in Figure 3.
For a speciﬁc conﬁdence function, we select the top- k % images with highest conﬁdence scores.
Then we compute the classiﬁcation accuracy within the selected images.
If a higher conﬁdence score indicates that the prediction is more likely to be correct, the accuracy should drop as as k increases.
Figure 3 is generated with a EfﬁcientNet-B0 model trained on ImageNet, where we sweep k from 0 to 100 and compute the accuracy within the selected top- k % images from the ImageNet validation set.
When k = 100 , all the images are selected so the accuracy is exactly the accuracy of
EfﬁcientNet-B0 (77.1%). The ‘Upper Bound’ curve represents the best possible performance for the metric. It has 100% accuracy when k ≤ 77 . 1 , i.e. , all the selected images are correctly classiﬁed.
The accuracy starts to drop when k becomes larger, since some misclassiﬁed images are inevitably chosen. We observe all the metrics demonstrate reasonably good performance in estimating how likely a prediction is correct, where the entropy performs slightly worse than other metrics.
[Figure]
[Table]
We also compare the performance of the cascade of ViT-B-224 and ViT-L-224 on ImageNet with different conﬁdence functions in Table 14. For each conﬁdence function, we set the threshold such that the cascade has a similar throughput when using different conﬁdence functions ( ∼ 409 images per second). We observe that the cascade achieves a similar accuracy with different conﬁdence functions.
D. C ALIBRATION OF M ODELS
As a side observation, when analyzing the conﬁdence function, we notice that models in our experiments are often slightly underconﬁdent, i.e. , the conﬁdence p is slightly lower than the actual accuracy of images whose prediction conﬁdence is p (see the ‘Original’ curve in Figure 5).
This observation contradicts the common belief that deep neural networks tend to be overconﬁdent (Guo et al., 2017).
Here, the conﬁdence of a prediction is deﬁned as the probability associated with the predicted class (Guo et al., 2017), which is equivalent to the maximum probability in the predicted distribution.
A model is considered as calibrated if the conﬁdence of its prediction correctly estimates the true correctness likelihood (neither overconﬁdent nor underconﬁdent).
The calibration of models can inﬂuence the ensemble performance when we ensemble models via simple averaging.
If models in an ensemble are poorly calibrated, overconﬁdent models may dominate the prediction, making those underconﬁdent models useless in the ensemble.
For models in our experiments, we also have tried calibrating them before computing the ensemble performance.
To calibrate a model, we use Platt scaling via a learned monotonic calibration network.
As shown in the ‘Calibration’ curve in Figure 5, the calibration improves the connection between the prediction conﬁdence and accuracy, i.e. , the conﬁdence after calibration almost equals to the actual accuracy.
But we also notice that calibrating models only has a small inﬂuence on the ﬁnal ensemble performance in our experiments, which might be because these models are just slightly underconﬁdent before calibration.
Therefore, we do not calibrate any model when computing the ensemble in all our experiments.
D. D ETERMINE THE T HRESHOLDS
Given the n models { M i } in a cascade, we also need ( n − 1) thresholds { t i } on the conﬁdence score.
We can ﬂexibly control the trade-off between the computation and accuracy of a cascade through thresholds { t i } .
We determine the thresholds { t i } based on the target FLOPs or accuracy on validation images.
In practice, we ﬁnd such thresholds via grid search, i.e. , enumerating all possible combinations for { t i } .
Note that the thresholds are determined after all models are trained.
We only need the logits of validation images to determine { t i } , so computing the cascade performance for a speciﬁc choice of thresholds is fast, which makes grid search computationally possible.
As t i is a real number, we make sure two trials of t i are sufﬁciently different by only considering the percentiles of conﬁdence scores as possible values.
When n > 2 , there might be multiple choices of { t i } that can give the target FLOPs or accuracy.
In that case, we choose { t i } that gives the higher accuracy or fewer FLOPs.
Many choices for { t i } can be easily ruled out as the FLOPs or accuracy of a cascade changes monotonically with respect to any threshold t i .
In practice, we often want the accuracy of a cascade to match the accuracy of a single model.
To do that, we determine the thresholds such that the cascade matches the accuracy of the single model on validation images.
Such thresholds usually enable the cascade to have a similar test accuracy to the single model.
For ImageNet, we randomly select ∼ 25k training images and exclude them from training.
We use these held-out training images to determine the conﬁdence thresholds.
The ﬁnal accuracy is computed on the original ImageNet validation set.
E M ODEL S ELECTION FOR B UILDING C ASCADES E. T ARGETING FOR A S PECIFIC FLOP S OR A CCURACY
We can build cascades to match a speciﬁc FLOPs or accuracy by optimizing the choice of models and conﬁdence thresholds, e.g. , solving Eq. 1 when targeting for a speciﬁc FLOPs.
Note that this optimization is done after all models in M are trained.
The optimization complexity is exponential in |M| and n , and the problem will be challenging if |M| and n are large.
In our experiments, |M| and n are not prohibitive.
Therefore, we solve the optimization problem with exhaustive search.
One can also use more efﬁcient procedures such as the algorithm described in (Streeter, 2018). [Table]
Concretely, for EfﬁcientNet, |M| = 8 and n ≤ 4 ; for ResNet, |M| = 4 and n = 2 ; for MobileNetV2, |M| = 5 and n = 2 .
The networks in each family are the same as listed in Sec.
C.2.
We do not consider cascades of more than four EfﬁcientNet models because we observe diminishing returns as n increases.
Same as ensembles in Sec C.2, we only consider cascades of two ResNet or
MobileNetV2 models due to their relatively narrow FLOPs range.
We allow the same network to be used by multiple times in one cascade but they will be different models trained separately.
Taking EfﬁcientNet for an example, we have in total 4672 ( = 8 4 +8 3 +8 2 ) possible combinations of models.
In practice, we ﬁrst train each EfﬁcientNet model separately for four times and pre-compute their predicted logits.
Then for each possible combination of models, we load the logits of models and determine the thresholds according to the target FLOPs or accuracy.
Finally, we choose the best cascade among all possible combinations.
Similar as above, we choose models and thresholds on held-out training images for ImageNet experiments.
No images from the
ImageNet validation set are used when we select models for a cascade.
E. C ASCADES CAN BE S CALED U P
One appealing property of single models is that they can be easily scaled up or down based on the available computational resources one has.
We show that such property is also applicable to cascades, i.e. , we can scale up a base cascade to respect different FLOPs constraints.
This avoids the model selection procedure when designing cascades for different FLOPs, which is required for cascades in Table 1.
Speciﬁcally, we build a 3-model cascade to match the FLOPs of EfﬁcientNet-B0.
We call this cascade C0 (see below for details of building C0).
Then, simply by scaling up the architectures in
C0, we obtain a family of cascades C0 to C7 that have increasing FLOPs and accuracy.
The models in C0 are from the EfﬁcientNet family.
The results of C0 to C7 in Table 15 show that simply scaling up C0 gives us a family of cascades that consistently outperform single models in all computation regimes.
This ﬁnding enhances the practical usefulness of cascades as one can select cascades from this family based on available resources, without worrying about what models should be used in the cascade.
Details of building C0.
The networks in EfﬁcientNet family are obtained by scaling up the depth, width and resolution of B0.
The scaling factors for depth, width and resolution are deﬁned as d = α φ , w = β φ and r = γ φ , respectively, where α
= 1 . 2 , β = 1 . 1 and γ = 1 . 15 , as suggested in
Tan et al.
(Tan & Le, 2019).
One can control the network size by changing φ .
For example, φ =gives B0, φ = 1 gives B2, and φ = 7 gives B7.
We build a 3-model cascade C0 to match the FLOPs of EfﬁcientNet-B0 by solving Eq.
1 on held-out training images from ImageNet.
When building C0, we consider 13 networks from EfﬁcientNet family.
As we want C0 to use similar FLOPs to B0, we make sure the 13 networks include both [Table] networks smaller than B0 and networks larger than B0.
Their φ are set to -4.0, -3.0, -2.0, -1.0, 0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.50, 1.75, 2.0, respectively.
The φ of the three models in C0 are -2.0, 0.0 and 0.75.
Then simply scaling up the architectures in C0, i.e. , increasing the φ of each model in C0, gives us a family of cascades C0 to C7 that have increasing FLOPs and accuracy.
The thresholds in C0 to C7 are determined such that their FLOPs are similar to B0 to B7.
E. E XIT R ATIOS
To better understand how a cascade works, we compute the exit ratio of the cascade, i.e. , the percentage of images that exit from the cascade at each stage.
Speciﬁcally, we choose the cascades in
Table 1&4 that match the accuracy of B1 to B7 and report their exit ratios in Table 16.
For all the cascades in Table 16, most images only consume the cost of the ﬁrst model in the cascade and only a few images have to use all the models.
This shows that cascades are able to allocate fewer resources to easy images and explains the speedup of cascades over single models.
E. N UMBER OF M ODELS IN C ASCADES
We study the inﬂuence of the number of models in cascades on the performance.
Concretely, we consider the EfﬁcientNet family and follow the same experimental setup as in Sec.
E.1.
We sweep the target FLOPs from 1 to 40 and ﬁnd cascades of 2, 3 or 4 models.
As shown in Figure 6, the performance of cascades keeps improving as the number of models increases.
We see a big gap between 2-model cascades and 3-model cascades, but increasing the number of models from 3 todemonstrates a diminishing return. [Figure] EfficientNet [Figure]
F A PPLICABILITY BEYOND I MAGE C LASSIFICATION F. V IDEO C LASSIFICATION
For video classiﬁcation, we conduct experiments on Kinetics-600 (Carreira et al., 2018).
Following
X3D (Feichtenhofer, 2020), we sample 30 clips from each input video when evaluating X3D models on Kinetics-600.
The 30 clips are the combination of 10 uniformly sampled temporal crops andspatial crops.
The ﬁnal prediction is the mean of all individual predictions.
F. S EMANTIC S EGMENTATION
Conﬁdence Function.
We notice that many pixels are unlabeled in semantic segmentation datasets, e.g. , Cityscapes (Cordts et al., 2016), and are ignored during training and evaluation.
These unlabeled pixels may introduce noise when we average the conﬁdence score of all the pixels.
To ﬁlter out unlabeled pixels in the image, we only consider pixels whose conﬁdence is higher than a preset threshold t unlab .
So we update the deﬁnition of g dense ( · ) as follows:
g dense ( R ) = 1 | R (cid:48) | (cid:80) p ∈ R (cid:48) g ( α p ) , where R (cid:48)
= { p | g ( α p ) > t unlab , p ∈ R } .
Experimental Details.
We conduct experiments on the Cityscapes (Cordts et al., 2016) dataset, where the full image resolution is 1024 × 2048.
We train DeepLabv3 (Chen et al., 2017) models on the train set of Cityscapes and report the mean IoU (mIoU) over classes on the validation set.
The threshold t unlab to ﬁlter out unlabeled pixels is set to 0.5.
For DeepLabv3-ResNet-50 or DeepLabv3-ResNet-101, we follow the original architecture of ResNet-50 or ResNet-101, except that the ﬁrst 7x7 convolution is changed to three 3x3 convolutions (see resnet_v1_beta in the ofﬁcial DeepLab imeplmentation 6 ).