A NYTIME N EURAL N ETWORK : A V ERSATILE T RADEOFF B ETWEEN C OMPUTATION AND A CCURACY
Anonymous authors
Paper under double-blind review
A BSTRACT
Anytime predictors ﬁrst produce crude results quickly, and then continuously reﬁne them until the test-time computational budget is depleted.
Such predictors are used in real-time vision systems and streaming-data processing to efﬁciently utilize varying test-time budgets, and to reduce average prediction cost via earlyexits.
However, anytime prediction algorithms have difﬁculties utilizing the accurate predictions of deep neural networks (DNNs), because DNNs are often computationally expensive without competitive intermediate results.
In this work, we propose to add auxiliary predictions in DNNs to generate anytime predictions, and optimize these predictions simultaneously by minimizing a carefully constructed weighted sum of losses, where the weights also oscillate during training.
The proposed anytime neural networks (ANNs) produce reasonable anytime predictions without sacriﬁcing the ﬁnal performance or incurring noticeable extra computation.
This enables us to assemble a sequence of exponentially deepening ANNs, and it achieves, both theoretically and practically, near-optimal anytime predictions at every budget after spending a constant fraction of extra cost.
The proposed methods are shown to produce anytime predictions at the state-of-the-art level on visual recognition data-sets, including ILSVRC2012.
I NTRODUCTION
An increasing number of machine learning applications now require fast and accurate responses.
For examples, autonomous vehicles require real-time object detectors, and spam ﬁlters on email servers desire low latency.
Furthermore, the computational budget limits can vary at test-time, e.g., autonomous vehicles need faster object detection at high speed, and servers have less time per sample during peak trafﬁc.
Hence, it is often difﬁcult to balance the trade-off between computation and accuracy.
One approach to such problems is through anytime predictors (Grass & Zilberstein, 1996;
Yang et al., 2007), which output crude results ﬁrst quickly, and then continuously reﬁne them until the budgets are depleted.
Such versatile predictors not only automatically utilize all the available budgets, but also enable systems to exit early on easy samples in order to focus computation on difﬁcult ones.
Using anytime predictions, Bolukbasi et al.
(2017) speed up neural networks by three times, and (Mai et al., 2016) speed up a density-based clustering algorithm by orders of magnitude.
To produce competitive anytime predictions, an anytime predictor produces a sequence of predictions that are gradually more expensive and accurate, e.g., a wide array of works (Yang et al., 2007;
Grubb & Bagnell, 2012; Chen et al., 2012; Xu et al.
; Cai et al., 2015; Hu et al., 2016; Bolukbasi et al., 2017; Guan et al., 2017) have studied how to order the computation of features or predictions to form such a sequence.
These meta-algorithms, however, rely on assembling existing features and predictors, and have difﬁculty utilizing the modern computationally intensive deep neural networks (DNNs), because DNNs typically greatly improve prediction performance at the cost of long chains of computation without intermediate outputs.
Hence it would be ideal to have gradually improving predictions within a DNN so that we can trade off between computation and accuracy smoothly in order to adjust to, and utilize, any test-time computational budget.
To achieve competitive anytime predictions in DNNs, we propose to uniformly add auxiliary predictions and losses to intermediate layers of existing feed-forward networks to form Anytime Neural
Networks (ANNs), as shown in Fig.
1a.
Speciﬁcally, we desire the ﬁnal prediction’s performance to reach that of the original network.
Subject to this constraint, we desire the anytime prediction performance after any cost B to be near the optimal at B , where we deﬁne the optimal at B as the
[Figure]
performance of the network trained speciﬁcally to output at cost B .
We highlight the importance of the low ﬁnal performance gap, because previous works that utilize auxiliary predictions (Lee et al., 2015; Xie & Tu, 2015; Zamir et al., 2017) are using training strategies that may increase the ﬁnal error rates relatively by > 15% , which is signiﬁcant, as shown in Fig.
1b, because the gap can cost anytime networks three times more computation than a regular DNN to reach the same accuracy.
We make the following contributions to achieve competitive anytime predictions from the auxiliary predictors.
(1) We propose loss weighting schemes that utilize the high correlation among neighboring predictions, so that we maintain the accuracy of the ﬁnal prediction ( ∼ 2 .
5% relative gap) and have competitive auxiliary predictions in the deeper layers.
(2) We also propose to oscillate the loss weighting during training to further improve each individual anytime prediction by avoiding spurious solutions that are optimal for the sum of the losses but not individual ones.
Furthermore, thanks to these near-optimal late predictions, we assemble ANNs of exponentially increasing depths to perform near-optimally at every budget theoretically and practically.
R ELATED W ORKS
Prediction with test-time budget limits.
Various works have studied how to predict efﬁciently to meet test-time budget limits by organizing the test-time computation of predictors and features.
(Reyzin, 2011; Grubb & Bagnell, 2012;
Hu et al., 2016) apply boosting and greedy methods to order feature and predictor computation.
(Weinberger et al., 2009; Xu et al., 2012; 2013) sequentially generate features to empower the ﬁnal predictor.
(Karayev et al., 2012; Odena et al., 2017) form
Markov Decision Processes for computation of weak predictors and features, and learn policies to order them.
Cascade design (Viola & Jones, 2001; Lefakis & Fleuret, 2010; Chen et al., 2012; Xu et al.
; Cai et al., 2015; Nan & Saligrama, 2017) speed up test-time prediction by early exiting on easy samples and focusing computation on difﬁcult ones.
However, these meta-algorithms are not easily compatible with complex and accurate predictors like CNNs, because the anytime predictions before the CNN computation are inaccurate, and there are no intermediate results during the long computation of the CNN.
This work addresses this problem by learning competitive anytime predictions end-to-end within neural networks.
These anytime predictions can also utilize gating functions from adaptive models (Bolukbasi et al., 2017) to reduce average test-time computation.
Computationally efﬁcient neural networks.
Many works have studied how to compress neural networks without sacriﬁcing accuracy.
Reducing redundant information in neural networks, (Li et al., 2017;
Liu et al., 2017) prune network weights and connections; (Hubara et al., 2016; Rastegari et al., 2016; Iandola et al., 2016) quantize weights within networks to reduce computation and memory footprint.
(Ba & Caruana, 2014; Hinton et al., 2014) transfer knowledge of deep networks into shallow ones by changing the training target of shallow networks.
This work addresses the computational efﬁciency of deep networks from an orthogonal approach, where we, instead of making models small, train networks to be adaptive to varying budget limits.
Furthermore, this work is compatible with the above-mentioned methods to produce potentially more efﬁcient predictors.
Networks with auxiliary predictions.
Auxiliary predictions have been used in neural networks for various purposes.
(Lee et al., 2015; Szegedy et al., 2017; Zhao et al., 2017) regularize networks training using auxiliary predictions.
(Bengio et al., 2009; Zamir et al., 2017) train auxiliary predictions to target label sets instead one-hot labels to improve ﬁnal predictions.
An increasing number of works (Xie & Tu, 2015; Larsson et al., 2017; Bolukbasi et al., 2017; Huang et al., 2017) now also appreciate high-quality auxiliary predictions, i.e., anytime predictions, for analyzing images multiple scales and speeding up average test-time predictions.
We contribute to works on auxiliary predictions in two folds.
First, we propose better-performing and easy-to-implement strategies to train all the auxiliary predictions end-to-end.
In particular, the proposed methods leverage the correlation among neighboring auxiliary predictions to achieve near-optimal level of anytime predictions near the ﬁnal outputs, where the previous methods suffer.
Second, instead of competing against ensembles of networks as in the previous works, we show that assembling a sequence of exponentially deepening ANNs can greatly improve the cost-effectiveness of predictions.
T RAINING A NYTIME N EURAL N ETWORKS
As illustrated in Fig.
1a, given a sample ( x, y ) ∼ D , the initial feature map x 0 is set to x , and the subsequent feature transformations f 1 , f 2 , ..., f L generate a sequence of intermediate features x i =
f i ( x i − 1 ; θ i ) for i ≥ 1 using parameter θ i .
Each feature map x
i can then produce an auxiliary prediction ˆ y i using a prediction layer
g i : ˆ y i
=
g i ( x i ; w i ) with parameter w i .
Each auxiliary prediction ˆ y i then incurs an expected loss (cid:96) i :=
E ( x,y ) ∼ D [ (cid:96) ( y, ˆ y i )] .
We call such a feed-forward network with auxiliary predictions and losses an Anytime Neural Network (ANN) .
For instance, we base our ANNs on ResNets (He et al., 2016), so each f i contains s number of residual building blocks (whose implementation details are deferred to the appendix), where s is the prediction period.
Each g i outputs the predicted label distribution ˆ
y i using a global pooling layer followed by a linear layer and a softmax; the loss function (cid:96) is the cross-entropy loss.
During test-time, an ANN computes ˆ y i sequentially until interruption, and outputs the latest available prediction.
The interruption can be issued by users, or can be learned with additional computations as in (Bolukbasi et al., 2017).
3. J OINT O PTIMIZATION OF L OSSES VIA AN O SCILLATING W EIGHTED S UM
Let the parameters of the full ANN be θ =
( θ 1 , w 1 , ..., θ L , w L ) .
Let (cid:96) ∗ i :=
min θ (cid:96) i ( θ ) be the optimal loss at the i th prediction.
The goal of training an ANN is to ﬁnd a single θ ∗ such that (cid:96) i ( θ ∗ ) =
(cid:96) ∗ i for all i , i.e., θ ∗ ∈ ∩
Li
=1 { x : x = arg min θ (cid:96) i ( θ ) } .
Such an ideal θ ∗ does not exist in general as this is a multi-objective optimization, and it often does not exist in practice.
Hence, we instead minimize a weighted sum of the losses in hopes of approximating (cid:96) ∗ i with (cid:96) i ( θ ) , i.e., we solve min θ (cid:80) Li
=1 B i (cid:96) i ( θ ) , where B i is the weight of the loss (cid:96) i .
We call the various choices of the B i weight schemes .
The most intuitive scheme is the CONST scheme, which set B i = 1 for all i .
This is a part of the formulation of a number of previous works (Lee et al., 2015; Xie & Tu, 2015; Huang et al., 2017).
Other works have proposed LINEAR scheme, which set the weights to be linearly increasing (Zamir et al., 2017).
However, as we will see in Sec. 4.2, optimizing the weighted sum to convergence using these intuitive schemes lead to about 12% relative increase in test error, in comparison to optimizing only the ﬁnal loss.
This is signiﬁcant, as we explained in Fig.
1b.
Weight scheme.
We now explain how we form the proposed weight scheme.
We study on a ResNet of 15 residual building blocks for the CIFAR100 (Krizhevsky, 2009) data-set.
To know the best possible performance at each depth i , we train a network whose weights are only on (cid:96) i to convergence.
We call the resulting performance OPT, and compare the training loss from CONST against
OPT in Fig.
2a.
We observe that within an ANN, the CONST scheme is substantially worse than
OPT in deep layers.
To remedy this, we set B L to be half of the total weights, so that the ﬁnal loss gradient can overcome the sum of all other loss gradients when all gradients have equal 2-norms.
If we distribute the remaining half of the total weights evenly among the ﬁrst L − 1 layers, we get the “Half-End” scheme in Fig.
2a, which reduces the ﬁnal loss to be near the OPT, but increases the the early losses.
Luckily, we also found that increasing weightings at a depth will improve the performance of a neighborhood.
E.g., Half-End improves the penultimate layer, even its relative weighting is smaller in Half-End than in CONST.
For another example, if we triple the weights at
L/ 4 , 3 L/ 4 and quadruple the weights at L/ 2 in the Half-End, the depths around these milestone depths also have reduced loss even their weights are unaffected, as shown in the SIEVE scheme in
Fig.
2a.
We believe the more uneven weight scheme wins here because neighboring layers share most of their parameters, and the well optimized prediction at the milestone layers make the sur-
[Figure]
rounding layers easier to optimize.
To scale the SIEVE scheme to larger L , we form it iteratively: we ﬁrst add to B (cid:98) L 2 (cid:101) one unit of weight, where (cid:98)•(cid:101) means rounding.
We then add one unit to each
B (cid:98) kL 4 (cid:101) for k = 1 , 2 , 3 , and then to each B (cid:98) kL 8 (cid:101) for k = 1 , 2 , ..., 7 , and so on, until all predictors have non-zero weights.
We ﬁnally normalize B i so that (cid:80) L − 1 i =1 B i = 1 , and set B L = 1 .
The values of
B 1 to B L − 1 resemble the length of markings on a ruler: large weights are few and apart, and are interleaved by small weights, as shown on Fig. 2b.
Alternating weights.
We also propose to oscillate the training loss for each batch of samples, since minimizing (cid:80) Li =
1 B i (cid:96) i ( θ ) may lead to spurious solutions where the sum of the gradients is zero, (cid:80)
Li =1 B i ∇ (cid:96) i
( θ ) = 0 , but individual gradients ∇ (cid:96) i ( θ ) are not.
To do so, we choose, in each iteration, a loss (cid:96) i , and temporarily increase its weight, B i , for the current iteration, by a constant τ of the total weights, τ (cid:80) Li
=1 B i .
We choose layer i with the probability proportional to the weight
B i in the weight scheme.
We call this sampling strategy Alternating ANN (AANN), and will show experimentally in Sec. 4.1 that it improves anytime predictions at each budget limit.
We also learn the sampling strategy via no-regret online learning, but this requires more hyper-parameters and has almost no performance gains in our experiments, as shown in the appendix.
This ﬁnding echoes the recent results by Graves et al.
(2017) that sampling according to B i is a competitive baseline.
3. S EQUENCE OF E XPONENTIALLY D EEPENING A NYTIME N EURAL N ETWORKS
Since we have half of the total weights at the ﬁnal loss, we observe in Sec 4.2 that ANNs are much closer to the optimal in the second half of the network than in the ﬁrst half.
In order to improve the early predictions, we leverage the more accurate late predictions in ANNs, by forming a sequence of ANNs whose depths grow exponentially ( EANN ).
As shown in Fig 3, the EANN sequentially computes the independently-trained ANNs whose depths grow exponentially in base b = 2 .
Then at interruption, the EANN outputs the better prediction between the ﬁnal prediction of the latest ﬁnished ANN, and the latest anytime prediction of the currently-running ANN, where the comparison is determined on a validation set.
Intuitively, if for some b > 1 each ANN is competitive after 1 b of its layers are computed, then we can recursively delegate the predictions in the early 1 b fraction of each network to a smaller network that has 1 b fraction of the cost.
Then we only rely on the competitive predictions from the later 1 − 1 b fraction of each ANN.
Furthermore, because the
ANN depths grow exponentially, we only pay a constant C − 1 fraction of extra costs to evaluate all the shallower ANNs ﬁrst for some C > 1 .
Formally, the following proposition proves that EANN is competitive at any budget with a constant fraction of additional computational complexity.
Proposition 3.1.
Let b > 1 .
Assume for any L , any ANN of depth L has competitive anytime prediction at depth i >
Lb against the optimal of depth i .
Then after B layers of computation, EANN produces anytime predictions that are competitive against the optimal of depth BC for some C > 1 , such that sup B C
= 2 + 1 b − 1 , and C has expectation E B ∼ uniform (1 ,L ) [ C ] ≤ 1 − 12 b + 1+ln( b ) b − 1 .
This proposition implies that if we multiply the allowed budget by C , we can build an anytime predictor that is competitive everywhere.
Furthermore, the stronger each anytime model is, the larger b becomes, so that the worst case inﬂation rate, sup B C , shrinks to 2, and the average rate, E
[ C ] , shrinks to 1.
Moreover, if we have M number of parallel workers instead of one, we can speed up EANNs by computing ANNs in parallel in a ﬁrst-in-ﬁrst-out schedule, so that we effectively increase the constant b to b M for computing C .
It is also worth noting that if we form the sequence using regular networks instead of ANNs, then we will lose the ability to output frequently, as we only produce Θ(log( B )) intermediate predictions instead of the Θ( B ) predictions in an EANN.
We will further have a larger cost inﬂation, C , such that sup B C ≥ 4 and E [ C ] ≥ 1 .
5 + √ 2 ≈ 2 .
91 , so that the average cost inﬂation is at least about 2 .
We defer the proofs to the appendix.
Weight scheme inspired by EANN.
Inspired by the EANN, which achieves near-optimal guarantee by having near-optimal performance at exponentially spaced locations, we also consider the following weight scheme.
The EXP-LIN weight scheme sets B (cid:98)
L 2 k (cid:101) =
2 − k for k = 1 , ..., log 2 ( L ) , and then adds 1 /L to each B i .
The idea of this scheme is to repeatedly apply the following idea: if a location is important, then its gradient should have enough weight to overcome the sum of the less important gradients when all the gradients have equal norms.
This idea suggest we exponentially decrease the weight in the order of priority.
Besides the exponentially spaced locations suggested by EANN, we also want to prioritize the layer at 3 L 4 , because it is natural to prioritize the milestones at 1/4, 1/2, and 3/4 of the total costs, and the relative loss increase from the OPT at 3 L 4 is high for EXP-LIN as in Fig.
2a. We set 3 L 4 to have priority after L 2 and L 4 .
We again normalize the weights so that (cid:80) L − 1 i =1 B i = 1 , and set B L = 1 , as shown in Fig. 2b.
E XPERIMENTS
We experiment with ANNs on CIFAR10, CIFAR100 (Krizhevsky, 2009), and SVHN (Netzer et al., 2011) 1 .
We evaluate our ANN augmentations on 13 different ResNets (He et al., 2016).
We refer to these models as ( n, c ) , where n is the number of basic residual building blocks (He et al., 2016), each of which contains two BN-ReLU-3x3conv and a BN-ReLU-addition with the input, and c is the number of channels of the initial convolution.
Each network contains 3 n blocks or 6 n + 2 conv layers.
The 13 models are in { 7 , 9 , 13 , 17 , 25 } × { 16 , 32 } and { (9 , 64) , (9 , 128) , (13 , 64) } .
We set the prediction period s = 1 , i.e., an anytime prediction is expected at every building block, unless speciﬁed otherwise.
We defer a detailed study of the prediction period to the appendix.
We optimize all networks with stochastic gradient descent with a momentum of 0.9.
The learning rate starts at
[Table]
0.1, and is divided by 10 after 12 and 34 of all epochs.
We train for 300 epochs on CIFAR andepochs on SVHN.
We evaluate using single-crop.
To quantitatively evaluate each training technique, we ﬁrst train the optimal ( OPT ) predictions at 1/4, 1/2, 3/4 and 1 of the total depth for each network, where the optimal at depth i is from training an ANN speciﬁcally to predict ˆ y i .
Then we can compute for each ANN the relative increase in toperror rate at these four milestone depths.
The ANN error rate is bi-linearly interpolated if the ANN doesn’t predict at these milestones.
We compare the proposed weight schemes against the popular
CONST scheme ( B i = 1 for all i ), which is used in (Lee et al., 2015; Xie & Tu, 2015; Huang et al., 2017) for its simplicity, and LINEAR scheme, which is used by Zamir et al.
and increases the weight linearly from 0.25 at B 1 to 1 at B L .
4. AANN: A G ENERAL T ECHNIQUE TO I MPROVE A LL P REDICTIONS
We showcase AANN as a useful technique independent of the weight schemes.
As explained in Sec. 3, AANN samples a layer i with the probability proportional to B i in each iteration, and increases the weight B i by τ (cid:80) Li
=1 B i for the iteration.
We choose τ = 0 .
5 from the set { 0 . 25 , 0 . 5 , 1 . 0 , 2 . 0 } by experiments on the validation network.
We experiment on all 39 settings to compare AANNs against ANNs.
Table 1a presents the percentage of the 39 settings such that
AANNs outperform ANNs.
Treating the experimental settings as 39 independent coin ﬂips, we apply hypothesis testing to understand these percentages.
For each entry in Table 1a, let X be the probability listed in the entry and p ∈ [0 , 1] be the probability that AANN outperforms ANN using the associated weight scheme at the associated milestone cost.
Let the null hypothesis H 0 be “ p ≤ 0 . 5 ” and the alternative hypothesis H 1 be “ p > 0 . 5 ”.
Using Hoeffding’s inequality 2 , we have
Pr ( X | H 0 ) ≤ 0 .
if X ≥ 0 .
598 , and Pr ( X | H 0 ) ≤ 0 .
001 if X ≥ 0 .
Then since almost all
X > 0 .
6 in Table 1a for the 3/4 and 1 milestones, we conclude that sampling according to B i to form AANNs allows us to improve predictions in the second half of ANNs independent of weight schemes.
This also suggests that optimizing a static weighted sum of anytime losses hinders the optimization of the late predictions more than that of the early predictions.
From now on we assume all ANNs are trained with AANN.
4. W EIGHTING S CHEME C OMPARISONS
Table 1b presents the average relative performance gap from the OPT by various weight schemes.
We observe that the CONST scheme makes 15 ∼ 18% more errors than the OPT, and the relative gap widens at deeper depths.
The LINEAR scheme also has a 13% relative gap in the ﬁnal prediction.
In CIFAR100, such a relative ﬁnal error translates to 3 ∼ 4% absolute error, which means the ﬁnal performances of ANNs with CONST scheme are often equivalent to those of ResNets of < 13 of the cost, e.g., on CIFAR100, (9 , 128) achieves 22.01% ﬁnal error using CONST and 21.25% using LINEAR, but the four-times-cheaper (9 , 64) achieves 21.14% and 20.85% using EXP-LIN and SIEVE.
Sec 4.3 will further show that these ﬁnal performance gaps are highly detrimental to the cost-efﬁciency of EANNs.
These two intuitive weight schemes fail at the ﬁnal prediction, possibly because the ﬁnal prediction is more difﬁcult to train than the early ones, but it only was assigned with a weighting on the order of 1 L .
On the other hand, the proposed SIEVE and EXP-LIN schemes both enjoy very small relative performance gap from the OPT at the ﬁnal prediction ( ∼ 2 . 5% ).
Furthermore, we also found ANNs with the proposed schemes can outperform the OPT at the ﬁnal predictions of expensive models, such as (9 , 64) , (9 , 128) , where OPT has trouble converging without dropouts(Zagoruyko & Komodakis, 2016).
This is possibly because the regularization effect of the anytime predictions, used by Lee et al.
; Szegedy et al.
; Zhao et al.
The
[Figure]
optimal-level ﬁnal performances of the proposed schemes, however, cost higher error rates in the ﬁrst half of layers.
Since EXP-LIN is designed to have higher weights at depth 2 k for each k , it outperforms SIEVE in the early layers.
4. EANN: C LOSING E ARLY P ERFORMANCE G APS
In this section, we leverage the near-optimal ﬁnal predictions ANNs with SIEVE and EXP-LIN schemes to close their early performance gap from the OPT using EANN as described in Sec. 3.2.
We train individually ANN models that have c = 32 and n = 7 , 13 , 25 in order to form an EANN with exponential base b ≈ 2 .
By proposition 3.1, E [ C ] ≈ 2 . 44 for b = 2 , and the EANN should compete against the optimal with n = (7 + 13 + 25) / 2 .
44 ≈ 20 .
We instead compare this EANN against a more challenging OPT+ , where we ﬁrst collect the four OPT for each of model that also has c = 32 , and then for each depth i , we set the error rate of OPT+ at i to be the minimum among the collection that also has depth no greater than i .
Hence, OPT+ equates to running all the OPT in parallel and outputting the best at each cost, whereas the EANN computes its AANNs sequentially.
As we noted in Sec. 3.2, parallelism in EANN effectively exponentiate the base b with the number of parallel workers, and we will discuss it in Sec. 4.4.
Fig.
4 compares EANNs with various weight schemes against the OPT+.
Without any parallelism,
EANNs with SIEVE and EXP-LIN schemes are shown in Fig.
4a to be near OPT+ at every budget, closing the early-mid performance gaps of AANNs.
AANNs are worse than EANNs in the early cost (Fig. 4b) because the early predictions of a large AANN (n=25) is less accurate than the ﬁnal predictions of a small AANN (n=7).
Then since the early AANNs are exponentially cheaper,
EANNs only pay a constant fraction of extra FLOPS to compute them ﬁrst.
Fig.
4a also shows that when combined with EANNs, the popular CONST and LINEAR schemes have clearly higher error rate for most budgets, because these schemes fail to reach near-optimal ﬁnal predictions in small AANNs.
For instance, to achieve the error rate 25%, CONST and LINEAR require two to six times more computation than SIEVE and EXP-LIN do.
Fig.
4b further shows the better earlymid AANN predictions of CONST and LINEAR than SIEVE and EXP-LIN to be irrelevant, since the EANN+SIEVE is much more cost-efﬁcient than the AANNs.
These comparisons suggest that it is imperative to have near-optimal ﬁnal predictions in ANNs, because large ANNs with ﬁnal performance gaps are outperformed by small ANNs without gaps, like the early AANNs in the
EANN+SIEVE.
The latter are also much more suitable for the cost-efﬁcient EANNs.
4. E VALUATION ON ILSVRC
To showcase the practical use of ANNs, we train ResNet+AANNs with SIEVE scheme on
ILSVRC2012 (Russakovsky et al., 2015), which is a visual recognition data-set that contains around 1.2 million natural images and 50000 validation images for 1000 classes.
We report the top-1 error rates on the validation set using a single-crop of size 224x224.
ResNet50 and ResNet101 are designed by He et al.
and they have 16 and 33 residual “bottleneck” building blocks, each of which contains a BN-ReLU-interleaved sequence of 1x1conv-3x3conv-1x1conv and a summation with the input.
We set the prediction period to s = 2 and s = 3 blocks for these two networks.
To
[Figure]
form an EANN we also design ResNet14, which contains 4 residual bottleneck blocks (one block for each scale).
We set the period to s = 1 .
AANNs based on these ResNet models are trained using SGD with momentum 0.9 for 120 epochs with batch size 256.
The learning rate starts at 0.05, and is divided by 10 at epoch 60, 90 and 105.
Since each anytime prediction only costs a global pooling followed by a linear product, anytime predictions in total cost < 0 .
1% of the total FLOPS.
In Fig.
5a, we observe that the AANN with the SIEVE scheme on each of ResNet14,50,101 can achieve the optimal level of performance at the ﬁnal layer, and the error rates are near-optimal in the later half of the depths.
AANN50 also outperforms signiﬁcantly the underlying anytime ResNet used by Adaptive Networks (Bolukbasi et al., 2017).
Hence, combining their early stopping policies with the proposed AANNs could drastically improve the cost-efﬁciency.
We also note that ANNs with deeper ﬁnal depths have better ﬁnal performances but worse early ones, so there is a trade-off between the ﬁnal and early accuracy.
In Fig.
5b, the EANN using these three AANNs achieves the optimal performance at a constant fraction of additional cost, as suggested by Proposition 3.1.
In addition, by running AANNs in parallel in a FIFO manner on two worker GPUs, we can speed up the EANN signiﬁcantly as the exponential base b effectively becomes b 2 .
We also compare against an ensemble of ResNets whose depths grow linearly from 10 to 50.
Linear ensembles are used by
Huang et al.
(2017) and Zamir et al.
(2017) as baselines for anytime predictors.
We observe in
Fig.
5b that both AANN101 and the EANN outperform this baseline.
Moreover, thanks to the ANN outputs, EANN predicts more frequently and more accurately than the linear ensemble, even though the EANN evaluate large networks exponentially earlier.
Finally, we also show the performance of the recent MSDNet (Huang et al., 2017), which is slightly faster, but produces 0.8% more errors than AANN50.
We also note that the proposed alternating weights, and the exponential ensembles are not restricted to ResNet based ANNs, so it will be interesting future work to see whether the proposed techniques are beneﬁcial to ANNs based on other networks.
C ONCLUSION AND D ISCUSSION
In this work, we propose weighting techniques to achieve anytime predictions in deep neural networks without degrading the ﬁnal performance.
Such anytime predictors can be combined using an ensemble technique to achieve near-optimal level of performance at all budgets in theory and in practice.
These versatile and near-optimal anytime predictions allow applications to automatically adjust to varying test-time budget limits, and we can use cascade early-exit policies with the proposed methods to reduce average prediction time.
For future work, ANNs can possibly beneﬁt from more complex anytime predictors than the simple linear predictors on the globally pooled features.
EANNs can also be formed based on both the computational cost and accuracy of ANNs.
This can potentially be addressed by adaptive models (Bolukbasi et al., 2017), predictor cascades (Viola &
Jones, 2001; Chen et al., 2012; Cai et al., 2015; Guan et al., 2017), and anytime model selection (Grubb & Bagnell, 2012).
A S KETCH OF P ROOF OF P ROPOSITION 3.
Proof.
For each budget consumed x , we compute the cost x (cid:48) of the optimal that EANN is competitive against.
The goal is then to analyze the ratio C = xx (cid:48) .
The ﬁrst ANN in EANN has depth 1.
The optimal and the result of EANN are the same.
Now assume EANN is on depth z of ANN number n + 1 for n ≥ 0 , which has depth b n .
(Case 1) For z ≤ b n − 1 ,
EANN reuse the result from the end of ANN number n .
The cost spent is
[Equation]
=
[Equation]
+ (cid:80) n
[Equation]
i =
[Equation]
i =
z +
[Equation]
n − 1 b − 1 .
The optimal we compete has cost of the last ANN, which is b n −
The ratio satisﬁes:
[Equation]
Furthermore, since C increases with z ,
[Equation]
(Case 2) For b n − 1 < z ≤ b n , EANN outputs anytime results from ANN number n + 1 at depth z .
The cost is still x
= z + b n − 1 b − 1 .
The optimal competitor has cost x (cid:48)
=
z .
Hence the ratio is
[Equation]
Furthermore, since C decreases with z ,
[Equation]
Finally, since case 1 and case 2 happen with probability 1 b and (1 − 1 b ) , we have sup B
[Equation]
and E B ∼ Uniform
[Equation] [Equation]
We also note that with large b , sup B C → 2 and E
[ C ] → 1 from above.
If we form a sequence of regular networks that grow exponentially in depth instead of ANN, then the worst case happen right before a new prediction is produced.
Hence the ratio between the consumed budget and the cost of the optimal that the current anytime prediction can compete, C , right before the number n + 1 network is completed, is
[Equation]
Note that ( b − 1) + 1 b − 1 ≥
and the inequality is tight at b = 2 .
Hence we know sup B C is at least 4.
Furthermore, the expected value of C , assume B is uniformly sampled such that the interruption happens on the ( n + 1) th network, is:
[Equation]
The inequality is tight at b = 1+ √ 2 .
With large n , since almost all budgets are consumed by the last few networks, we know the overall expectation E B ∼ Uniform (0 ,L ) [ C ] approaches 1 .
5 + b − 12 + 1 b − 1 , which is at least 1 . 5 + √ 2 .
[Figure]
B W ALL -C LOCK S PEED OF ANN S
For reference, on a single GTX 1080 Ti, our implementation of ResNet+AANNs using Tensorﬂow (Abadi & et al., 2015) has the following FLOPS cost and test-time wall-clock speed on 224x images: 3.1e9 FLOPS and 690 FPS for ResNet14, 8.4e9 FLOPS and 310 FPS for ResNet50, and 15.9e9 FLOPS and 190 FPS for ResNet101.
C R ES N ET R ESIDUAL B LOCKS D ETAILS
For CIFAR10, CIFAR100, and SVHN, we ﬁrst apply a 3x3 convolution of c output channels with no stride on the input image, where c is speciﬁed by the model name ( n, c ) .
The initial conv layer is followed by three groups of residual blocks and each group has n basic residual building blocks.
A residual building block applies a transformation φ i ( x i ) on the input feature map x i , and then add the transformation and the input together to form the next feature map,
i.e., x i +
= x
i + φ i ( x i ) .
In a basic residual block with pre-activation, φ i is BN-ReLU-3x3conv-BN-ReLU-3x3conv-BN.
The two conv have the same channel size as the input within a group, and both have a stride of 1.
In between two groups, the second conv of the previous residual block produces twice as many channels but use a stride of 2.
The input of this block, i.e., x n − 1 or x 2 n − 1 , are also ﬁrst average pooled and padded with zero channels before added to the second convolution result.
For ILSVRC, we ﬁrst, apply a 7x7conv with 64 output channels and a stride of 2 on the input 224x224 image.
Then we apply BN-ReLU and max-pooling with kernel size 3.
In a bottleneck residual block (for ILSVRC), φ i is BN-ReLU-1x1conv-BN-ReLU-3x3conv-BN-ReLU-1x1convBN.
The ﬁrst bottleneck residual block have its two conv to have 64 output channels, and the third conv has 64x4=256 channels.
For other bottleneck block not at scale transition, the ﬁrst 1x1 conv outputs has 1/4 of the channel size as its input; the 3x3conv maintains the same channel size as its input; the second 1x1 conv outputs four times the channel size as its input.
At the scale transition, the ﬁrst 1x1 conv outputs 1/2 of the channel size as its input; the other two conv have the same output channel size multiplier relative to their inputs as before; the 3x3 conv, however, uses a stride of 2.
The input of the transition residual blocks are again ﬁrst pooled and padded with zero channels before adding to the result of φ i .
D F REQUENCY OF A NYTIME P REDICTION
We note that not every residual bottleneck unit is required to compute an anytime output, and it is possible that allowing more layers to be computed without interruption can lead to better predictions.
This section studies the relationship between frequency and performance of anytime predictions.
We expect that a model with more frequent anytime predictions to perform worse, because optimizing the extra auxiliary losses limits the freedom of the network.
Additionally, with more anytime outputs, the average weight of anytime losses decreases, assuming the total loss weight is a constant.
Then the less relative weight a layer has, the less optimized the layer will be in general.
Following (Zamir et al., 2017), we call an ANN that makes an anytime prediction every i units of feature transformations (residual bottlenecks) as stack- i (i.e., the prediction period is i ).
The layers that predict bottlenecks whose index can be written as L − ci , where c is a non-negative integer, so that the last layer always predict.
Fig.
6 and Table 2 demonstrate the performances of ANNs on CIFAR with different stack-i on networks with n = 7 , 9 , 13 .
In all cases all anytime predictions are able to achieve near-optimal ﬁnal predictions due to the ﬁnal large weights from the sieve scheme.
From
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
the slightly larger network n = 9 and n =
, we observe the general trend of improvement of anytime performances in all metrics as the period s increases.
However, such increase is not without a limit.
For instance, when n = 7 , if have a relatively large period, s = 5 , then it is possible that a milestone cost ( 12 in this case) is right before a new prediction, so that the anytime performance at such a milestone suffers.
In applications, we should choose large period s as long as it does not cause much waste of budgets right before milestones.
E F ULL T ABLE FOR AANN WITH VARIOUS WEIGHT SCHEMES
Table 3 displays the performance of AANNs with each weight schemes on the 13 models anddata-sets listed in Sec. 4.
All models have s = 1 , except for (25 , 16) and (25 , 32) , which have s = 3 .
F N O - REGRET O NLINE LEARNERS FOR C HOOSING L AYERS TO A MPLIFY F. L EARN TO C HOOSE L AYERS VIA M IN - MAX O PTIMIZATION
This section consider to replace the heuristic and static strategy in AANN with strategies that are learned during training.
Inspired by Shalev-Shwartz & Wexler (2016), we choose B i to amplify in each iteration by applying no-regret online learner on the maximization of the following min-max optimization:
[Equation]
where ∆ L is the L -dimensional probability simplex, and L i ( θ ) is a heuristic loss objective of choosing layer i at parameter θ .
Note that L may not be the same as the loss of layer i , (cid:96) i .
Intuitively, this min-max optimization can be considered as a two player zero-sum game, where the max player chooses layer i using v to generate the maximal loss, and the min player updates θ to reduce the chosen loss.
As suggested by Shalev-Shwartz & Wexler (2016), applying no-regret online learner on the maximization w.r.t.
v leads to a no-regret strategy of choosing which layers against any static strategies.
There are multiple options for no-regret online learners.
For instance, EXP3(Auer et al., 2002) and Random Weighted Majority (RWM)(Littlestone & Warmuth, 1994) both exponential gradient ascend on variable v to ﬁnd layers that consistently results in high losses.
Ideally, these no-regret online learners will discover high loss layer for optimization to focus on, so that eventually all loss
L i will be equal.
We also consider another heuristic sampling strategy called AVGL : we sample layer i with probability proportional to the exponential average of L i .
F. M AXIMIZATION O BJECTIVE
Now we explain our choice of heuristic loss objective L i so that minimization of Eq.
1 leads to low loss (cid:96) i .
Ideally, we would like to set L i = max( (cid:96) i − (cid:96) ∗ i (cid:96) ∗ i , 0) to measure the relative difference
[Table] [Table]
LIN
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
LIN 22.
[Table]
SIEVE 24.
[Table]
OPT 9.
[Table]
32 CONST 11.
[Table] [Table]
LIN 14.
1.88 17-32 SIEVE - - - - 51.
[Table]
16 OPT 12.
1.68 25-16 CONST 15.
[Table]
16 LINEAR 17.
[Table]
[Table] [Table]
[Table]
of (cid:96) i from its optimal (cid:96) ∗ i .
In fact Lee et al. (2015) apply intermediate loss of this very form, and set the (cid:96) ∗ i to be a hyper parameter to tune.
However, in general tuning this parameter is difﬁcult, and we cannot compute (cid:96) ∗ i efﬁciently without training a model speciﬁcally for each i .
To avoid the dependency on (cid:96) ∗ i , we consider the relative loss difference between neighboring layers, and use the following min-max optimization to select losses (cid:96) i to optimize,
[Equation]
where η is a constant chosen by cross-validation.
Intuitively, for i <
L , L i ( θ ) =
[Equation]
i
[Equation]
θ ) represents the relative reduction in loss from layer i to i + 1 , and if this value is high, layer i is performing much worse than its successor while their structures are relative the same.
This suggests layer i could be improved signiﬁcantly with more optimization focus.
The ﬁnal loss objective is set to be L L = η max i =1 ,...,L − 1 (cid:96)
i ( θ ) − (cid:96) i +1 ( θ ) (cid:96)
i ( θ ) , a constant fraction of the maximal L i of the previous L − 1 layers, because the ﬁnal layer does not have a successor and we desire a relative high probability of choosing the ﬁnal layer.
F. N O - REGRET O NLINE L EARNING A LGORITHMS
As suggested by Shalev-Shwartz & Wexler (2016), we let the max player use EXP3, a no-regret online learning algorithm, to update v in the min-max optimization in Eq.
1 as follows.
In each iteration, we ﬁrst sample a loss (cid:96) i using EXP3 according to the probability distribution p i =
(1 − γ ) v i + γL , where γ is a hyper parameter of EXP3 that represents the probability that the algorithm should explore a loss at random.
Then the chosen (cid:96) i is added to a ﬁxed sum objective (cid:80)
Lj =1 B j (cid:96)
j to form the total loss (cid:96) total =
(cid:80)
Lj =1 B j (cid:96)
j
+ τ (cid:80) Lj =1 B j (cid:96)
i , where τ is a hyper parameter.
The extra weight of (cid:96) i is τ (cid:80)
Lj =1 B j , a fraction of the total weight of the sum objective, so that the chosen (cid:96) i has signiﬁcant inﬂuence on the total loss.
After computing (cid:96) total , we next update network parameter θ gradient of (cid:96) total .
Finally, we apply EXP3 update rules to update v i to be v i exp ( γ L i ( θ )
Lp i ) and normalize vector v onto the probability simplex ∆ L .
If we use Random Weighted Majority (RWM) (Littlestone & Warmuth, 1994) instead of EXP3, the procedure is almost identical: the only difference is that in each iteration we update all v j , j = 1 , ..., L , by exponential gradient descent using the signal L j .
F. H YPER P ARAMETER S ET - UP
There are three hyper parameters for applying EXP3 or RWM to alternating optimization: a probability γ for the online learner to explore uniformly at random, a ratio η between the reward for the ﬁnal layer and the maximum reward among previous layers, and the additional weight of the chosen layer as τ (cid:80) Li =1 B i .
We run grid search on CIFAR10 and CIFAR100 validation network with γ from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } , η from { 0 . 7 , 0 . 8 , 0 . 9 , 1 . 0 } , and τ from { 2 , 1 , 0 . 5 , 0 . 25 } , and then sort the settings by average error and ﬁnal error.
We choose the setting that ﬁrst appears on both ranking: γ = 0 .
3 , η = 0 . 8 , and τ = 0 .
A true optimal parameter setting may be hard to determine due to the random nature of SGD and the randomized strategy itself, but luckily we found on validation set that the performance is not overly sensitive to parameters.
We set the momentum constant in the exponential gradient average of the AVGL strategy to be 0.
F. E XPERIMENTAL R ESULTS OF V ARIOUS S AMPLING S TRATEGIES
In Fig.
7, we plot the probability that each bottleneck unit i is chosen by various strategies on model with n = 7 , and c = 32 .
The overall behavior of the probabilities are similar across models, even
[Figure]
the performance ranking are not the same.
We plot the stack graph versus the training epochs, so that we can view how the strategy evolve over time.
We found that both no-regret algorithms, EXP and RWM, learned to pick the ﬁnal layer with more than half of the probability.
The remaining probability are roughly split evenly among layers.
AVGL learns to put more weight on the ﬁnal layer than the other layers as well, but since AVGL update its probabilities with gradient descent instead of exponentiated gradient descent in EXP3 and RWM, AVGL does not select the ﬁnal layer with too high a probability.
Moreover, AVGL learns to focus layers around layer 7 and 14, which are around the transition layers that subsample the feature maps and double the channel sizes via 1x1 convolutions.
AANN based on sieve weight scheme naturally focus half of the weights on the ﬁnal layer, and it does not spread the remaining weights evenly.
Instead, we see intermittent large weights and small weights, so that every layer is somewhat focused on due to its correlation with its neighbors, and no neighborhood other than the ﬁnal layer is overly focused on.
Table 4 lists for each sampling strategy and each anytime evaluation metric the percentage of the 39 experiments in Table 5 such that the corresponding strategy is better than the vanilla ANN in the corresponding metric.
Missing experiments are ignored.
We see that RWM degrades performance in every metric.
EXP3 is only able to improve at the 3/4 milestone, and degrades performance at all other milestones.
AVGL is able to somewhat improve performance at 1/2 and 3/4 milestones.
However, this is at the cost of the ﬁnal prediction performance, even though we note the difference is usually small.
In contrast, AANN is able to improve performances in every metric, except in the ﬁnal prediction, which cannot be improved because ANN is already at the optimal level in this metric.
Overall, all of dynamic strategies, EXP3, RWM and AVGL are not as effective as the proposed simple AANN strategy that samples layers according to the normalized static weights, B i .
Since each of these dynamic strategies has extra parameters to tune, and the chosen heuristic loss objective L is probably sub-optimal, our experiments cannot fully prove that they are not helpful for improving performance of ANN.
However, we can conclude that even if they can be helpful, they require extensive parameter tuning.
G F ULL T ABLE FOR E VALUATING S AMPLING S TRATEGIES
Table 5 lists the 13 different models that we experiment with sampling strategies on.
The table also contains the performance of the sampling strategy milestone , which sample evenly at 1/4, 1/2, 3/4, and 1 of the total depths.
[Table]