M ULTI -S CALE D ENSE N ETWORKS FOR R ESOURCE E FFICIENT I MAGE C LASSIFICATION
A BSTRACT
In this paper we investigate image classiﬁcation with computational resource limits at test time.
Two such settings are: 1.
anytime classiﬁcation , where the network’s prediction for a test example is progressively updated, facilitating the output of a prediction at any time; and 2.
budgeted batch classiﬁcation , where a ﬁxed amount of computation is available to classify a set of examples that can be spent unevenly across “easier” and “harder” inputs.
In contrast to most prior work, such as the popular Viola and Jones algorithm our approach is based on convolutional neural networks.
We train multiple classiﬁers with varying resource demands which we adaptively apply during test time.
To maximally re-use computation between the classiﬁers , we incorporate them as early-exits into a single deep con volutional neural network and inter-connect them with dense connectivity.
To fa cilitate high quality classiﬁcation early on , we use a two-dimensional multi- scale network architecture that maintains coarse and ﬁne level features all-throughout the network
Experiments on three image -classiﬁcation tasks demonstrate that our framework substantially improves the existing state-of-the-art in both settings
I NTRODUCTION
Recent years have witnessed a surge in demand for applications of visual object recognition, for instance, in self-driving cars (Bojarski et al., 2016) and content-based image search (Wan et al., 2014).
This demand has in part been fueled through the promise generated by the astonishing progress of convolutional networks (CNNs) on visual object recognition benchmark competition datasets, such as ILSVRC (Deng et al., 2009) and COCO (Lin et al., 2014), where state-of-the-art models may have even surpassed human-level performance (He et al., 2015; 2016).
However, the requirements of such competitions differ from realworld applications, which tend to incentivize resource-hungry models with high computational demands at inference time.
For example, the COCO 2016 competition was won by a large ensemble of computationally intensive CNNs 1 — a model likely far too computationally expensive for any resource-aware application.
Although much smaller models would also obtain decent error, very large, computationally intensive models seem necessary to correctly classify the hard examples that make up the bulk of the remaining misclassiﬁcations of modern algorithms.
To illustrate this point, Figure 1 shows two images of horses.
The left image depicts a horse in canonical pose and is easy to classify, whereas the right image is taken from a rare viewpoint and is likely in the tail of the data distribution.
Computationally intensive models are needed to classify such tail examples correctly, but are wasteful when applied to canonical images such as the left one.
In real-world applications, computation directly translates into power consumption, which should be minimized for environmental and economical reasons, and is a scarce commodity on mobile devices.
This begs the question: why do we choose between either wasting computational resources by applying an unnecessarily computationally expensive model to easy images, or making mistakes by using an efﬁcient model that fails to recognize difﬁcult images?
Ideally, our systems should automatically use small networks when test images are easy or computational resources limited, and use big networks when test images are hard or computation is abundant.
Such systems would be beneﬁcial in at least two settings with computational constraints at testtime: anytime prediction , where the network can be forced to output a prediction at any given point in time; and budgeted batch classiﬁcation , where a ﬁxed computational budget is shared across a large set of examples which can be spent unevenly across “easy” and “hard” examples.
A practical use-case of anytime prediction is in mobile apps on Android devices: in 2015, there existed 24 ,distinct Android devices 2 , each with its own distinct computational limitations.
It is infeasible to train a different network that processes video frame-by-frame at a ﬁxed framerate for each of these devices.
Instead, you would like to train a single network that maximizes accuracy on all these devices, within the computational constraints of that device.
The budget batch classiﬁcation setting is ubiquitous in large-scale machine learning applications.
Search engines, social media companies, on-line advertising agencies, all must process large volumes of data on limited hardware resources.
For example, as of 2010, Google Image Search had over 10 Billion images indexed 3 , which has likely grown to over 1 Trillion since.
Even if a new model to process these images is only 1/10s slower per image, this additional cost would add 3170 years of CPU time.
In the budget batch classiﬁcation setting, companies can improve the average accuracy by reducing the amount of computation spent on “easy” cases to save up computation for “hard” cases.
Motivated by prior work in computer vision on resource-efﬁcient recognition (Viola & Jones, 2001), we aim to develop CNNs that “slice” the computation and process these slices one-by-one, stopping the evaluation once the CPU time is depleted or the classiﬁcation sufﬁciently certain (through “early exits”).
Unfortunately, the architecture of CNNs is inherently at odds with the introduction of early exits.
CNNs learn the data representation and the classiﬁer jointly, which leads to two problems with early exits: 1.
The features in the last layer are extracted directly to be used by the classiﬁer, whereas earlier features are not.
The inherent dilemma is that different kinds of features need to be extracted depending on how many layers are left until the classiﬁcation.
The features in different layers of the network may have different scale.
Typically, the ﬁrst layers of a deep nets operate on a ﬁne scale (to extract low-level features), whereas later layers transition (through pooling or strided convolution) to coarse scales that allow global context to enter the classiﬁer.
Both scales are needed but happen at different places in the network.
We propose a novel network architecture that addresses both of these problems through careful design changes, allowing for resource-efﬁcient image classiﬁcation.
Our network uses a cascade of intermediate classiﬁers throughout the network.
The ﬁrst problem, of classiﬁers altering the internal representation, is addressed through the introduction of dense connectivity (Huang et al., 2017).
By connecting all layers to all classiﬁers, features are no longer dominated by the most imminent earlyexit and the trade-off between early or later classiﬁcation can be performed elegantly as part of the loss function.
The second problem, the lack of coarse-scale features in early layers, is addressed by adopting a multi-scale network structure.
At each layer we produce features of all scales (ﬁne-tocoarse), which facilitates good classiﬁcation early on but also extracts low-level features that only become useful after several more layers of processing.
Our network architecture is illustrated in
Figure 2, and we refer to it as Multi-Scale DenseNet
(MSDNet) .
We evaluate MSDNets on three image-classiﬁcation datasets.
In the anytime classiﬁcation setting, we show that it is possible to provide the ability to output a prediction at any time while maintain high accuracies throughout.
In the budget batch classiﬁcation setting we show that MSDNets can be effectively used to adapt the amount of computation to the difﬁculty of the example to be classiﬁed, which allows us to reduce the computational requirements of our models drastically whilst performing on par with state-of-the-art CNNs in terms of overall classiﬁcation accuracy.
To our knowledge this is the ﬁrst deep learning architecture of its kind that allows dynamic resource adaptation with a single model and obtains competitive results throughout.
concatenation regular conv strided conv . .
classifier onelayer depth h ( · ) ` =` =˜
h ( · ) features s ca l e
x s` c identity c
c c c c c c c c c c c c c ` =` =c f ( · ) or or
R ELATED W ORK
We brieﬂy review related prior work on computation-efﬁcient networks, memory-efﬁcient networks, and resource-sensitive machine learning, from which our network architecture draws inspiration.
Computation-efﬁcient networks.
Most prior work on (convolutional) networks that are computationally efﬁcient at test time focuses on reducing model size after training.
In particular, many studies propose to prune weights (LeCun et al., 1989;
Hassibi et al., 1993; Li et al., 2017) or quantize weights (Hubara et al., 2016; Rastegari et al., 2016) during or after training.
These approaches are generally effective because deep networks often have a substantial number of redundant weights that can be pruned or quantized without sacriﬁcing (and sometimes even improving) performance.
Prior work also studies approaches that directly learn compact models with less parameter redundancy.
For example, the knowledge-distillation method (Bucilua et al., 2006; Hinton et al., 2014) trains small student networks to reproduce the output of a much larger teacher network or ensemble.
Our work differs from those approaches in that we train a single model that trades off computation for accuracy at test time without any re-training or ﬁnetuning.
Indeed, weight pruning and knowledge distillation can be used in combination with our approach, and may lead to further improvements.
Resource-efﬁcient machine learning.
Various prior studies explore computationally efﬁcient variants of traditional machine-learning models (Viola & Jones, 2001; Grubb & Bagnell, 2012; Karayev et al., 2014; Trapeznikov & Saligrama, 2013; Xu et al., 2012; 2013; Nan et al., 2015; Wang et al., 2015).
Most of these studies focus on how to incorporate the computational requirements of computing particular features in the training of machine-learning models such as (gradient-boosted) decision trees.
Whilst our study is certainly inspired by these results, the architecture we explore differs substantially: most prior work exploits characteristics of machine-learning models (such as decision trees) that do not apply to deep networks.
Our work is possibly most closely related to recent work on FractalNets (Larsson et al., 2017), which can perform anytime prediction by progressively evaluating subnetworks of the full network.
FractalNets differ from our work in that they are not explicitly optimized for computation efﬁciency and consequently our experiments show that
MSDNets substantially outperform FractalNets.
Our dynamic evaluation strategy for reducing batch computational cost is closely related to the the adaptive computation time approach (Graves, 2016;
Figurnov et al., 2016), and the recently proposed method of adaptively evaluating neural networks (Bolukbasi et al., 2017).
Different from these works, our method adopts a specially designed network with multiple classiﬁers, which are jointly optimized during training and can directly output conﬁdence scores to control the evaluation process for each test example.
The adaptive computation time method (Graves, 2016) and its extension (Figurnov et al., 2016) also perform adaptive evaluation on test examples to save batch computational cost, but focus on skipping units rather than layers.
In (Odena et al., 2017), a “composer”model is trained to construct the evaluation network from a set of sub-modules for each test example.
By contrast, our work uses a single CNN with multiple intermediate classiﬁers that is trained end-to-end.
The Feedback Networks (Zamir et al., 2016) enable early predictions by making predictions in a recurrent fashion, which heavily shares parameters among classiﬁers, but is less efﬁcient in sharing computation.
Related network architectures.
Our network architecture borrows elements from neural fabrics (Saxena & Verbeek, 2016) and others (Zhou et al., 2015; Jacobsen et al., 2017; Ke et al., 2016)
[Figure]
r e l a t i v e a cc u r a c
[Figure]
Relative accuracy of the intermediate classiﬁer
[Figure]
to rapidly construct a low-resolution feature map that is amenable to classiﬁcation, whilst also maintaining feature maps of higher resolution that are essential for obtaining high classiﬁcation accuracy. Our design differs from the neural fabrics (Saxena & Verbeek, 2016) substantially in that MSDNets have a reduced number of scales and no sparse channel connectivity or up-sampling paths.
MSDNets are at least one order of magnitude more efﬁcient and typically more accurate — for example, an MSDNet with less than 1 million parameters obtains a test error below 7 . 0% on CIFAR-10 (Krizhevsky & Hinton, 2009), whereas Saxena & Verbeek (2016) report 7 .
43% with over 20 million parameters.
We use the same feature-concatenation approach as DenseNets (Huang et al., 2017), which allows us to bypass features optimized for early classiﬁers in later layers of the network.
Our architecture is related to deeply supervised networks (Lee et al., 2015) in that it incorporates classiﬁers at multiple layers throughout the network.
In contrast to all these prior architectures, our network is speciﬁcally designed to operate in resource-aware settings.
P ROBLEM S ETUP
We consider two settings that impose computational constraints at prediction time.
Anytime prediction.
In the anytime prediction setting (Grubb & Bagnell, 2012), there is a ﬁnite computational budget B > 0 available for each test example x .
The computational budget is nondeterministic, and varies per test instance.
It is determined by the occurrence of an event that requires the model to output a prediction immediately.
We assume that the budget is drawn from some joint distribution P ( x , B ) .
In some applications P ( B ) may be independent of P ( x ) and can be estimated.
For example, if the event is governed by a Poisson process, P ( B ) is an exponential distribution.
We denote the loss of a model f ( x ) that has to produce a prediction for instance x within budget B by
L ( f ( x ) , B ) .
The goal of an anytime learner is to minimize the expected loss under the budget distribution:
L ( f ) =
E [ L ( f ( x ) , B )] P ( x ,B ) .
Here, L ( · ) denotes a suitable loss function.
As is common in the empirical risk minimization framework, the expectation under P ( x , B ) may be estimated by an average over samples from P ( x , B ) .
Budgeted batch classiﬁcation.
In the budgeted batch classiﬁcation setting, the model needs to classify a set of examples D test = { x 1 , . . .
, x M } within a ﬁnite computational budget B > 0 that is known in advance.
The learner aims to minimize the loss across all examples in
D test within a cumulative cost bounded by B , which we denote by L ( f ( D test ) , B ) for some suitable loss function
L ( · ) .
It can potentially do so by spending less than BM computation on classifying an “easy” example whilst using more than BM computation on classifying a “difﬁcult” example.
Therefore, the budget B considered here is a soft constraint when we have a large batch of testing samples.
M ULTI - SCALE D ENSE C ONVOLUTIONAL N ETWORKS
A straightforward solution to the two problems introduced in Section 3 is to train multiple networks of increasing capacity, and sequentially evaluate them at test time (as in Bolukbasi et al.
In the anytime setting the evaluation can be stopped at any point and the most recent prediction is returned.
In the batch setting, the evaluation is stopped prematurely the moment a network classiﬁes the test sample with sufﬁcient conﬁdence.
When the resources are so limited that the execution is terminated after the ﬁrst network, this approach is optimal because the ﬁrst network is trained for exactly this computational budget without compromises.
However, in both settings, this scenario is rare.
In the more common scenario where some test samples can require more processing time than others the approach is far from optimal because previously learned features are never re-used across the different networks.
An alternative solution is to build a deep network with a cascade of classiﬁers operating on the features of internal layers: in such a network features computed for an earlier classiﬁer can be re-used by later classiﬁers.
However, na¨ıvely attaching intermediate early-exit classiﬁers to a stateof-the-art deep network leads to poor performance.
There are two reasons why intermediate early-exit classiﬁers hurt the performance of deep neural networks: early classiﬁers lack coarse-level features and classiﬁers throughout interfere with the feature generation process.
In this section we investigate these effects empirically (see Figure 3) and, in response to our ﬁndings, propose the MSDNet architecture illustrated in Figure 2.
Problem: The lack of coarse-level features.
Traditional neural networks learn features of ﬁne scale in early layers and coarse scale in later layers (through repeated convolution, pooling, and strided convolution).
Coarse scale features in the ﬁnal layers are important to classify the content of the whole image into a single class.
Early layers lack coarse-level features and early-exit classiﬁers attached to these layers will likely yield unsatisfactory high error rates.
To illustrate this point, we attached 4 intermediate classiﬁers to varying layers of a ResNet (He et al., 2016) and a
DenseNet (Huang et al., 2017) on the CIFAR-100 dataset (Krizhevsky & Hinton, 2009).
The blue and red dashed lines in the left plot of Figure 3 show the relative accuracies of these classiﬁers.
All three plots gives rise to a clear trend: the accuracy of a classiﬁer is highly correlated with its position within the network.
Particularly , one can observe a visible “staircase”
Solution: Multi-scale feature maps.
To address this issue, MSDNets maintain a feature representation at multiple scales throughout the network, and all the classiﬁers only use the coarse-level features .
The feature maps at a particular layer 5 and scale are computed by concatenating the results of one or two convolutions: 1.
the result of a regular convolution applied on the same-scale features from the previous layer (horizontal connections) and, if possible, 2.
the result of a strided convolution applied on the ﬁner-scale feature map from the previous layer (diagonal connections).
The horizontal connections preserve and progress high-resolution information, which facilitates the construction of high-quality coarse features in later layers.
The vertical connections produce coarse features throughout that are amenable to classiﬁcation.
The dashed black line in Figure 3 shows that
MSDNets substantially increase the accuracy of early classiﬁers.
Problem: Early classiﬁers interfere with later classiﬁers.
The right plot of Figure 3 shows the accuracies of the ﬁnal classiﬁer as a function of the location of a single intermediate classiﬁer, relative to the accuracy of a network without intermediate classiﬁers.
The results show that the introduction of an intermediate classiﬁer harms the ﬁnal ResNet classiﬁer (blue line), reducing its accuracy by up to 7% .
We postulate that this accuracy degradation in the ResNet may be caused by the intermediate classiﬁer inﬂuencing the early features to be optimized for the short-term and not for the ﬁnal layers.
This improves the accuracy of the immediate classiﬁer but collapses information required to generate high quality features in later layers.
This effect becomes more pronounced when the ﬁrst classiﬁer is attached to an earlier layer.
Solution: Dense connectivity.
By contrast, the DenseNet (red line) suffers much less from this effect.
Dense connectivity (Huang et al., 2017) connects each layer with all subsequent layers and allows later layers to bypass features optimized for the short-term, to maintain the high accuracy of the ﬁnal classiﬁer.
If an earlier layer collapses information to generate short-term features, the lost information can be recovered through the direct connection to its preceding layer.
The ﬁnal classiﬁer’s performance becomes (more or less) independent of the location of the intermediate
[Table]
x s` ` =` =` =` =
s =h 11 ( x 10 ) h x
h⇥ x
11 , x⇤ h⇥ x 11 , x 12 , x⇤ s =
˜
h 21 ( x 11 )  ˜ h x ,
h x   ˜ h⇥ x 11 , x⇤ , h⇥ x 21 , x⇤   ˜ h⇥ x 11 , x 12 , x⇤ , h⇥ x 21 , x 22 , x⇤  s =
˜
h 31 ( x 21 )  ˜ h x ,
h x   ˜ h⇥ x 21 , x⇤ , h⇥ x 31 , x⇤   ˜ h⇥ x 21 , x 22 , x
⇤ , h⇥ x 31 , x 32 , x⇤  . . .
[Table]
` =` =` =` =h 11 ( x 10 ) h x h x 11 , x h x 11 , x 12 , x s =
s =s =˜
h x  ˜ h⇥ x
⇤ h⇥ x
⇤  ˜ h⇥ x
11 , x⇤ h⇥ x 21 , x⇤  ˜ h⇥ x 11 , x 12 , x⇤ h⇥ x 21 , x 22 , x⇤  ˜
h x  ˜ h⇥ x
⇤ h⇥ x
⇤  ˜ h⇥ x 21 , x⇤ h⇥ x 31 , x⇤
x sl  ˜ h⇥ x 21 , x 22 , x
⇤ h⇥ x 31 , x 32 , x⇤  classiﬁer.
As far as we know, this is the ﬁrst paper that discovers that dense connectivity is an important element to early-exit classiﬁers in deep networks, and we make it an integral design choice in MSDNets.
4. THE MSDNET ARCHITECTURE
The MSDNet architecture is illustrated in Figure 2.
We present its main components below.
Additional details on the architecture are presented in Appendix A.
First layer.
The ﬁrst layer ( (cid:96) =1 ) is unique as it includes vertical connections in Figure 2.
Its main purpose is to “seed” representations on all S scales.
One could view its vertical layout as a miniature “S-layers” convolutional network (S=3 in Figure 2).
Let us denote the output feature maps at layer (cid:96) and scale s as x s(cid:96) and the original input image as x 10 .
Feature maps at coarser scales are obtained via down-sampling.
The output x s 1 of the ﬁrst layer is formally given in the top row of Figure 4.
Subsequent layers.
Following Huang et al.
(2017), the output feature maps x s(cid:96) produced at subsequent layers, (cid:96)> 1 , and scales, s , are a concatenation of transformed feature maps from all previous feature maps of scale s and s − 1 (if s > 1 ).
Formally, the (cid:96) -th layer of our network outputs a set of features at S scales (cid:8) x 1 (cid:96) , . . .
, x S(cid:96) (cid:9) , given in the last row of Figure 4.
Classiﬁers.
The classiﬁers in MSDNets also follow the dense connectivity pattern within the coarsest scale, S , i.e., the classiﬁer at layer (cid:96) uses all the features (cid:2)
x S 1 , . . .
, x S(cid:96) (cid:3) .
Each classiﬁer consists of two convolutional layers, followed by one average pooling layer and one linear layer.
In practice, we only attach classiﬁers to some of the intermediate layers, and we let f k ( · ) denote the k th classiﬁer.
During testing in the anytime setting we propagate the input through the network until the budget is exhausted and output the most recent prediction.
In the batch budget setting at test time, an example traverses the network and exits after classiﬁer f k if its prediction conﬁdence (we use the maximum value of the softmax probability as a conﬁdence measure) exceeds a pre-determined threshold θ k .
Before training, we compute the computational cost, C k , required to process the network up to the k th classiﬁer.
We denote by 0 < q ≤ 1 a ﬁxed exit probability that a sample that reaches a classiﬁer will obtain a classiﬁcation with sufﬁcient conﬁdence to exit.
We assume that q is constant across all layers, which allows us to compute the probability that a sample exits at classiﬁer k as: q k = z (1 − q ) k − 1 q , where z is a normalizing constant that ensures that (cid:80) k p ( q k )
=
At test time, we need to ensure that the overall cost of classifying all samples in D test does not exceed our budget B (in expectation).
This gives rise to the constraint |D test | (cid:80) k q k C k ≤ B .
We can solve this constraint for q and determine the thresholds θ k on a validation set in such a way that approximately |D test | q k validation samples exit at the k th classiﬁer.
Loss functions.
During training we use cross entropy loss functions L ( f k ) for all classiﬁers and minimize a weighted cumulative loss: 1 |D| (cid:80) ( x ,y ) ∈D (cid:80) k w k L ( f k ) .
Herein, D denotes the training set and w k ≥
the weight of the k -th classiﬁer.
If the budget distribution P ( B ) is known, we can use the weights w k to incorporate our prior knowledge about the budget B in the learning.
Empirically, we ﬁnd that using the same weight for all loss functions ( i.e. , setting ∀ k : w k = 1 ) works well in practice.
Network reduction and lazy evaluation.
There are two straightforward ways to further reduce the computational requirements of MSDNets.
First, it is inefﬁcient to maintain all the ﬁner scales until the last layer of the network.
One simple strategy to reduce the size of the network is by splitting it into S blocks along the depth dimension, and only keeping the coarsest ( S − i + 1) scales in the i th block (a schematic layout of this structure is shown in Figure 9).
This reduces computational cost for both training and testing.
Every time a scale is removed from the network, we add a transition layer between the two blocks that merges the concatenated features using a 1 × 1 convolution and cuts the number of channels in half before feeding the ﬁne-scale features into the coarser scale via a strided convolution (this is similar to the DenseNet-BC architecture of Huang et al.
Second, since a classiﬁer at layer (cid:96) only uses features from the coarsest scale, the ﬁner feature maps in layer (cid:96) (and some of the ﬁner feature maps in the previous S − 2 layers) do not inﬂuence the prediction of that classiﬁer.
Therefore, we group the computation in “diagonal blocks” such that we only propagate the example along paths that are required for the evaluation of the next classiﬁer.
This minimizes unnecessary computations when we need to stop because the computational budget is exhausted.
We call this strategy lazy evaluation .
E XPERIMENTS
We evaluate the effectiveness of our approach on three image classiﬁcation datasets, i.e., the CIFAR10, CIFAR-100 (Krizhevsky & Hinton, 2009) and ILSVRC 2012 (ImageNet; Deng et al.
datasets.
Code to reproduce all results is available at https://anonymous-url .
Details on architectural conﬁgurations of MSDNets are described in Appendix A.
Datasets.
The two CIFAR datasets contain 50 , 000 training and 10 , 000 test images of 32 × 32 pixels; we hold out 5 , 000 training images as a validation set.
The datasets comprise 10 and 100 classes, respectively.
We follow He et al.
and apply standard data-augmentation techniques to the training images: images are zero-padded with 4 pixels on each side, and then randomly cropped to produce 32 × 32 images.
Images are ﬂipped horizontally with probability 0 . 5 , and normalized by subtracting channel means and dividing by channel standard deviations.
The ImageNet dataset comprises 1 , 000 classes, with a total of 1.2 million training images and 50,000 validation images.
We hold out 50,000 images from the training set to estimate the conﬁdence threshold for classiﬁers in MSDNet.
We adopt the data augmentation scheme of He et al.
at training time; at test time, we classify a 224 × 224 center crop of images that were resized to 256 × 256 pixels.
Training Details.
We train all models using the framework of Gross & Wilber (2016).
On the two
CIFAR datasets, all models (including all baselines) are trained using stochastic gradient descent (SGD) with mini-batch size 64.
We use Nesterov momentum with a momentum weight of 0.
without dampening, and a weight decay of 10 − 4 .
All models are trained for 300 epochs, with an initial learning rate of 0.1, which is divided by a factor 10 after 150 and 225 epochs.
We apply the same optimization scheme to the ImageNet dataset, except that we increase the mini-batch size to 256, and all the models are trained for 90 epochs with learning rate drops after 30 and 60 epochs.
5. A NYTIME P REDICTION
In the anytime prediction setting, the model maintains a progressively updated distribution over classes, and it can be forced to output its most up-to-date prediction at an arbitrary time.
Baselines.
There exist several baseline approaches for anytime prediction: FractalNets (Larsson et al., 2017), deeply supervised networks (Lee et al., 2015), and ensembles of deep networks of varying or identical sizes.
FractalNets allow for multiple evaluation paths during inference time, which vary in computation time.
In the anytime setting, paths are evaluated in order of increasing computation.
In our result ﬁgures, we replicate the FractalNet results reported in the original paper (Larsson et al., 2017) for reference.
Deeply supervised networks introduce multiple early-exit classiﬁers throughout a network, which are applied on the features of the particular layer they are attached to.
Instead of using the original model proposed in Lee et al.
(2015), we use the more competitive
ResNet and DenseNet architectures (referred to as DenseNet-BC in Huang et al.
as the base networks in our experiments with deeply supervised networks.
We refer to these as ResNet MC and
DenseNet MC , where MC stands for multiple classiﬁers .
Both networks require about 1 . 3 ×
FLOPs when fully evaluated; the detailed network conﬁgurations are presented in the supplementary material.
In addition, we include ensembles of ResNets and DenseNets of varying or identical sizes.
At test time, the networks are evaluated sequentially (in ascending order of network size) to obtain predictions for the test data.
All predictions are averaged over the evaluated classiﬁers.
On
[Figure] [Figure] ImageNet, we compare MSDNet against a highly competitive ensemble of ResNets and DenseNets, with depth varying from 10 layers to 50 layers, and 36 layers to 121 layers, respectively.
Anytime prediction results are presented in Figure 5.
The left plot shows the top-1 classiﬁcation accuracy on the ImageNet validation set.
Here, for all budgets in our evaluation, the accuracy of
MSDNet substantially outperforms the ResNets and DenseNets ensemble.
In particular, when the budget ranges from 0 .
1 × 10 10 to 0 .
3 × 10 10 FLOPs, MSDNet achieves ∼ 4% − 8% higher accuracy.
We evaluate more baselines on CIFAR-100 (and CIFAR-10; see supplementary materials).
We observe that MSDNet substantially outperforms ResNets MC and DenseNets MC at any computational budget within our range.
This is due to the fact that after just a few layers, MSDNets have produced low-resolution feature maps that are much more suitable for classiﬁcation than the high-resolution feature maps in the early layers of ResNets or DenseNets.
MSDNet also outperforms the other baselines for nearly all computational budgets, although it performs on par with ensembles when the budget is very small.
In the extremely low-budget regime, ensembles have an advantage because their predictions are performed by the ﬁrst (small) network, which is optimized exclusively for the low budget.
However, the accuracy of ensembles does not increase nearly as fast when the budget is increased.
The MSDNet outperforms the ensemble as soon as the latter needs to evaluate a second model: unlike MSDNets, this forces the ensemble to repeat the computation of similar low-level features repeatedly.
Ensemble accuracies saturate rapidly when all networks are shallow.
5. B UDGETED BATCH CLASSIFICATION
In budgeted batch classiﬁcation setting, the predictive model receives a batch of M instances and a computational budget B for classifying all M instances.
In this setting, we use dynamic evaluation : we perform early-exiting of “easy” examples at early classiﬁers whilst propagating “hard” examples through the entire network, using the procedure described in Section 4.
Baselines.
On ImageNet, we compare the dynamically evaluated MSDNet with ﬁve ResNets (He et al., 2016) and ﬁve DenseNets (Huang et al., 2017), AlexNet (Krizhevsky et al., 2012), and GoogleLeNet (Szegedy et al., 2015); see the supplementary material for details.
We also evaluate an ensemble of the ﬁve ResNets that uses exactly the same dynamic-evaluation procedure as MSDNets at test time: “easy” images are only propagated through the smallest ResNet-10, whereas “hard” images are classiﬁed by all ﬁve ResNet models (predictions are averaged across all evaluated networks in the ensemble).
We classify batches of M =128 images.
On CIFAR-100, we compare MSDNet with several highly competitive baselines, including ResNets (He et al., 2016), DenseNets (Huang et al., 2017) of varying sizes, Stochastic Depth Networks (Huang et al., 2016), Wide ResNets (Zagoruyko & Komodakis, 2016) and FractalNets (Larsson et al., 2017).
We also compare MSDNet to the ResNet MC and DenseNet MC models that were used in Section 5.1, using dynamic evaluation at test time.
We denote these baselines as ResNet MC /
DenseNet MC with early-exits .
To prevent the result plots from becoming too cluttered, we present
CIFAR-100 results with dynamically evaluated ensembles in the supplementary material.
We classify batches of M =256 images at test time.
Budgeted batch classiﬁcation results on ImageNet are shown in the left panel of Figure 7.
We trained three MSDNets with different depths, each of which covers a different range of compu-
[Figure] batch classiﬁcation
[Figure]
ImageNet
[Figure]
tational budgets.
We plot the performance of each MSDNet as a gray curve; we select the best model for each budget based on its accuracy on the validation set, and plot the corresponding accuracy as a black curve.
The plot shows that the predictions of MSDNets with dynamic evaluation are substantially more accurate than those of ResNets and DenseNets that use the same amount of computation.
For instance, with an average budget of 1 . 7 × 10 9 FLOPs, MSDNet achieves a topaccuracy of ∼ 75%, which is ∼ 6% higher than that achieved by a ResNet with the same number of
FLOPs.
Compared to the computationally efﬁcient DenseNets, MSDNet uses ∼ 2 − 3 × times fewer FLOPs to achieve the same classiﬁcation accuracy.
Moreover, MSDNet with dynamic evaluation allows for very precise tuning of the computational budget that is consumed, which is not possible with individual ResNet or DenseNet models.
The ensemble of ResNets or DenseNets with dynamic evaluation performs on par with or worse than their individual counterparts (but they do allow for setting the computational budget very precisely).
The right panel of Figure 7 shows our results on CIFAR-100.
The results show that MSDNets consistently outperform all baselines across all budgets.
Notably, MSDNet performs on par with a 110layer ResNet using only 1/10th of the computational budget and it is up to ∼ 5 times more efﬁcient than DenseNets, Stochastic Depth Networks, Wide ResNets, and FractalNets.
Similar to results in the anytime-prediction setting, MSDNet substantially outperform ResNets MC and DenseNets MC with multiple intermediate classiﬁers, which provides further evidence that the coarse features in the
MSDNet are important for high performance in earlier layers.
[Table]
Visualization.
To illustrate the ability of our approach to reduce the computational requirements for classifying “easy” examples, we show twelve randomly sampled test images from two ImageNet classes in Figure 6.
The top row shows “easy” examples that were correctly classiﬁed and exited by the ﬁrst classiﬁer.
The bottom row shows “hard” examples that would have been incorrectly classiﬁed by the ﬁrst classiﬁer but were passed on because its uncertainty was too high.
The ﬁgure suggests that early classiﬁers recognize prototypical class examples, whereas the last classiﬁer recognizes non-typical images.
5. M ORE C OMPUTATIONALLY E FFICIENT D ENSE N ETS
Here, we discuss an interesting ﬁnding during our exploration of the MSDNet architecture.
We found that following the DenseNet structure to design our network, i.e. , by keeping the number of output channels (or growth rate ) the same at all scales, did not lead to optimal results in terms of the accuracy-speed trade-off.
The main reason for this is that compared to network architectures like
ResNets, the DenseNet structure tends to apply more ﬁlters on the high-resolution feature maps in the network.
This helps to reduce the number of parameters in the model, but at the same time, it greatly increases the computational cost.
We tried to modify DenseNets by doubling the growth rate [Figure] after each transition layer, so that more ﬁlters are applied to low-resolution feature maps.
It turns out that the resulting network, which we denote as DenseNet*, signiﬁcantly outperform the original
DenseNet in terms of computational efﬁciency.
We experimented with DenseNet* in our two settings with test time budget constraints.
The left panel of Figure 8 shows the anytime prediction performance of an ensemble of DenseNets* of varying depths.
It outperforms the ensemble of original DenseNets of varying depth by a large margin, but is still slightly worse than MSDNets.
In the budgeted batch budget setting, DenseNet* also leads to signiﬁcantly higher accuracy over its counterpart under all budgets, but is still substantially outperformed by MSDNets.
C ONCLUSION
We presented the MSDNet, a novel convolutional network architecture, optimized to incorporate
CPU budgets at test-time.
Our design is based on two high-level design principles, to generate and maintain coarse level features throughout the network and to inter-connect the layers with dense connectivity.
The former allows us to introduce intermediate classiﬁers even at early layers and the latter ensures that these classiﬁers do not interfere with each other.
The ﬁnal design is a two dimensional array of horizontal and vertical layers, which decouples depth and feature coarseness.
Whereas in traditional convolutional networks features only become coarser with increasing depth, the MSDNet generates features of all resolutions from the ﬁrst layer on and maintains them throughout.
The result is an architecture with an unprecedented range of efﬁciency.
A single network can outperform all competitive baselines on an impressive range of computational budgets ranging from highly limited CPU constraints to almost unconstrained settings.
As future work we plan to investigate the use of resource-aware deep architectures beyond object classiﬁcation, e.g. image segmentation (Long et al., 2015).
Further, we intend to explore approaches that combine MSDNets with model compression (Chen et al., 2015; Han et al., 2015), spatially adaptive computation (Figurnov et al., 2016) and more efﬁcient convolution operations (Chollet, 2016;
Howard et al., 2017) to further improve computational efﬁciency.
A CKNOWLEDGMENTS
The authors are supported in part by grants from the National Science Foundation ( III-1525919,
IIS-1550179, IIS-1618134, S&AS 1724282, and CCF-1740822), the Ofﬁce of Naval Research DOD (N00014-17-1-2175), and the Bill and Melinda Gates Foundation.
A D ETAILS OF MSDN ET A RCHITECTURE AND B ASELINE N ETWORKS
We use MSDNet with three scales on the CIFAR datasets, and the network reduction method introduced in 4.1 is applied.
Figure 9 gives an illustration of the reduced network.
The convolutional layer functions in the ﬁrst layer, h s 1 , denote a sequence of 3 × 3 convolutions (Conv), batch normalization (BN; Ioffe & Szegedy (2015)), and rectiﬁed linear unit (ReLU) activation.
In the computation of ˜ h s 1 , down-sampling is performed by applying convolutions using strides that are powers of two.
For subsequent feature layers, the transformations h s(cid:96) and ˜ h s(cid:96) are deﬁned following the design in
DenseNets (Huang et al., 2017): Conv( 1 × 1 )-BN-ReLU-Conv( 3 × 3 )-BN-ReLU.
We set the number of output channels of the three scales to 6, 12, and 24, respectively.
Each classiﬁer has two down-sampling convolutional layers with 128 dimensional 3 × 3 ﬁlters, followed by a 2 × 2 average pooling layer and a linear layer.
The MSDNet used for ImageNet has four scales, respectively producing 16, 32, 64, and 64 feature maps at each layer.
The network reduction is also applied to reduce computational cost.
The original images are ﬁrst transformed by a 7 × 7 convolution and a 3 × 3 max pooling (both with stride 2), before entering the ﬁrst layer of MSDNets.
The classiﬁers have the same structure as those used for the CIFAR datasets, except that the number of output channels of each convolutional layer is set to be equal to the number of its input channels.
c
c c c c c c c c . . .
c c . . .
c ` =c
c c c c . . .
[Figure]
Network architecture for anytime prediction.
The MSDNet used in our anytime-prediction experiments has 24 layers (each layer corresponds to a column in Fig.
1 of the main paper), using the reduced network with transition layers as described in Section 4.
The classiﬁers operate on the output of the 2 × ( i +1)
th layers, with i =1 , . .
On ImageNet, we use MSDNets with four scales, and the i th classiﬁer operates on the ( k × i +3)
th layer (with i =1 , . . .
where k =4 , 6 and 7 .
For simplicity, the losses of all the classiﬁers are weighted equally during training.
Network architecture for budgeted batch setting.
The MSDNets used here for the two CIFAR datasets have depths ranging from 10 to 36 layers, using the reduced network with transition layers as described in Section 4.
The k th classiﬁer is attached to the ( (cid:80) ki
=1 i ) th layer.
The MSDNets used for ImageNet are the same as those described for the anytime learning setting.
ResNet MC and DenseNet MC .
The ResNet MC has 62 layers, with 10 residual blocks at each spatial resolution (for three resolutions): we train early-exit classiﬁers on the output of the 4 th and 8 th residual blocks at each resolution, producing a total of 6 intermediate classiﬁers (plus the ﬁnal classiﬁcation layer).
The DenseNet MC consists of 52 layers with three dense blocks and each of them has 16 layers.
The six intermediate classiﬁers are attached to the 6 th and 12 th layer in each block, also with dense connections to all previous layers in that block.
B A DDITIONAL R ESULTS B. A BLATION STUDY
We perform additional experiments to shed light on the contributions of the three main components of MSDNet, viz.
, multi-scale feature maps, dense connectivity, and intermediate classiﬁers.
[Figure]
We start from an MSDNet with six intermediate classiﬁers and remove the three main components one at a time.
To make our comparisons fair, we keep the computational costs of the full networks similar, at around 3 .
× 10 8 FLOPs, by adapting the network width, i.e., number of output channels at each layer.
After removing all the three components in an MSDNet, we obtain a regular
VGG-like convolutional network.
We show the classiﬁcation accuracy of all classiﬁers in a model in the left panel of Figure 10.
Several observations can be made: 1.
the dense connectivity is crucial for the performance of MSDNet and removing it hurts the overall accuracy drastically (orange vs. black curve); 2. removing multi-scale convolution hurts the accuracy only in the lower budget regions, which is consistent with our motivation that the multi-scale design introduces discriminative features early on; 3.
the ﬁnal canonical
CNN (star) performs similarly as MSDNet under the speciﬁc budget that matches its evaluation cost exactly, but it is unsuited for varying budget constraints.
The ﬁnal CNN performs substantially better at its particular budget region than the model without dense connectivity (orange curve).
This suggests that dense connectivity is particularly important in combination with multiple classiﬁers.
B. R ESULTS ON CIFAR-
For the CIFAR-10 dataset
[Figure]
we use the same MSDNets and baseline models as we used for CIFAR100, except that the networks used here have a 10-way fully connected layer at the end.
The results under the anytime learning setting and the batch computational budget setting are shown in the left and right panel of Figure 11, respectively.
Similar to what we have observed from the results on
CIFAR-100 and ImageNet, MSDNets outperform all the baselines by a signiﬁcant margin in both settings.
As in the experiments presented in the main paper, ResNet and DenseNet models with multiple intermediate classiﬁers perform relatively poorly
[Figure]