L
EARN TO P AY A TTENTION
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose an end-to-end-trainable attention module for convolutional neural network (CNN) architectures built for image classiﬁcation.
The module takes as input the 2D feature vector maps which form the intermediate representations of the input image at different stages in the CNN pipeline, and outputs a 2D matrix of scores for each map.
Standard CNN architectures are modiﬁed through the incorporation of this module, and trained under the constraint that a convex combination of the intermediate 2D feature vectors, as parametrised by the score matrices, must alone be used for classiﬁcation.
Incentivised to amplify the relevant and suppress the irrelevant or misleading, the scores thus assume the role of attention values.
Our experimental observations provide clear evidence to this effect: the learned attention maps neatly highlight the regions of interest while suppressing background clutter.
Consequently, the proposed function is able to bootstrap standard CNN architectures for the task of image classiﬁcation, demonstrating superior generalisation over 6 unseen benchmark datasets.
When binarised, our attention maps outperform other CNN-based attention maps, traditional saliency maps, and top object proposals for weakly supervised segmentation as demonstrated on the Object Discovery dataset.
We also demonstrate improved robustness against the fast gradient sign method of adversarial attack.
I NTRODUCTION
Feed-forward convolutional neural networks (CNNs) have demonstrated impressive results on a wide variety of visual tasks, such as image classiﬁcation, captioning, segmentation, and object detection.
However, the visual reasoning which they implement in solving these problems remains largely inscrutable, impeding understanding of their successes and failures alike.
One approach to visualising and interpreting the inner workings of CNNs is the attention map: a scalar matrix representing the relative importance of layer activations at different 2D spatial locations with respect to the target task (Simonyan et al., 2013).
This notion of a nonuniform spatial distribution of relevant features being used to form a task-speciﬁc representation, and the explicit scalar representation of their relative relevance, is what we term ‘attention’.
Previous works have shown that for a classiﬁcation CNN trained using image-level annotations alone, extracting the attention map provides a straightforward way of determining the location of the object of interest (Cao et al., 2015; Zhou et al., 2016) and/or its segmentation mask (Simonyan et al., 2013), as well as helping to identify discriminative visual properties across classes (Zhou et al., 2016).
More recently, it has also been shown that training smaller networks to mimic the attention maps of larger and higher-performing network architectures can lead to gains in classiﬁcation accuracy by those smaller networks (Zagoruyko & Komodakis, 2016).
The works of Simonyan et al.
; Cao et al.
; Zhou et al.
(2016) form a progressive series of increasingly sophisticated techniques for estimating attention maps in CNNs.
However, these approaches share a crucial limitation: all are implemented as post-hoc additions to fully trained networks.
Further, using these attention maps to crop regions for reclassiﬁcation in the manner of Cao et al.
(2015) splits the pipeline into two or more independently tuned steps, meaning that there is no joint optimisation with respect to the ﬁnal classiﬁcation loss.
Thus, for the ﬁrst time in the context of classiﬁcation CNNs, we propose a probabilistic attention estimator, embedded within the network architecture, whose parameters are learned over the course of end-to-end training of the entire network.
Our efforts are motivated in part by recent improvements
[Figure]
in the tasks of machine translation (Bahdanau et al., 2014) and image caption generation (Xu et al., 2015) via the use of attention in the context of RNNs.
Note that while Mnih et al.
(2014) and Jaderberg et al.
(2015) integrate the learning of their respective attention representations into their training pipelines, they both model attention non-probabilistically.
Our attention matrix is probabilistic in its representation of attention.
This difference becomes crucial in cases where higher-order probabilistic reasoning over the attention maps is sought, e.g. when applying conditional random ﬁelds to reﬁne attention maps for weakly supervised segmentation.
The matrix is itself a deterministic function of the input image.
Fig.
1 provides an overview of the proposed method.
Henceforth, we will use the terms ‘local features’ and ‘global features’ to refer to features extracted by some layer of the CNN whose effective receptive ﬁelds are, respectively, contiguous proper subsets of the image (‘local’) and the entire image (‘global’).
By deﬁning a compatibility measure between local and global features, we redesign standard architectures such that they must classify the input image using only a weighted combination of local features, with the weights represented here by the attention map.
The network is thus forced to learn a pattern of attention relevant to solving the task at hand.
We experiment with applying the proposed attention mechanism to the popular CNN architectures of
VGGNet (Simonyan & Zisserman, 2014) and ResNet (He et al., 2015), and capturing coarse-to-ﬁne attention maps at multiple levels.
We observe that the proposed mechanism can bootstrap baseline
CNN architectures for the task of image classiﬁcation: for example, adding attention to the VGG model offers an accuracy gain of 7% on CIFAR-100.
Our use of attention-weighted representations leads to improved ﬁne-grained recognition and superior generalisation on 6 benchmark datasets for domain-shifted classiﬁcation.
As observed on models trained for ﬁne-grained bird recognition, attention aware models offer limited resistance to adversarial fooling at low and moderate L ∞ -noise norms.
The trained attention maps outperform other CNN-derived attention maps (Zhou et al., 2016), traditional saliency maps ( Jiang et al.
; Zhang & Sclaroff (2013)), and top object proposals on the task of weakly supervised segmentation of the Object Discovery dataset ( Rubinstein et al.
In § 5, we present sample results which suggest that these improvements may owe to the method’s tendency to highlight the object of interest while suppressing background clutter.
R ELATED W ORK
Attention in deep classiﬁcation networks : Early approaches to understanding the performance of
CNNs as stacked feature hierarchies focus on capturing the image semantics that activate speciﬁc features, particularly at higher-up network layers (Girshick et al., 2014;
Simonyan et al., 2013;
Donahue et al., 2014).
For example, Girshick et al.
(2014) cluster the image patches that activate features in a given CNN layer: this enables visualisation of the patterns learned by the network, but does not identify the visual support that the network relies upon in making a classiﬁcation decision.
Simonyan et al.
(2013) approximate CNNs as linear functions, interpreting the gradient of a class output score with respect to the input image as that class’s spatial support in the image domain, i.e. its attention map.
Importantly, they are one of the ﬁrst to successfully demonstrate the use of attention for localising objects of interest using image-level category labels alone.
Zhou et al.
(2016) apply the classiﬁer weights learned for image-level descriptors to patch descriptors, and the resulting class scores are used as a proxy for attention.
Their improved localisation performance comes at the cost of classiﬁcation accuracy.
Cao et al.
(2015) introduce attention in the form of binary nodes between the network layers of Simonyan et al.
While the attention maps are adapted to the end task, these nodes require additional parameter tuning at test time.
Notably, all of the above methods extract attention from fully trained CNN classiﬁcation models, i.e. in a posthoc manner.
Again, we demonstrate the classiﬁcation and generalisation advantages of training an attention mechanism that is built into a CNN pipeline.
Robustness to adversarial attacks : The work by Goodfellow et al.
(2014) explores the ease of fooling the current generation of deep networks by adding an imperceptible perturbation to the input image in the approximate direction of the negative class score gradient.
This suggests that the sample images, at least in some dimensions, are situated dangerously close to the decision hyperplanes.
Wang et al.
(2016) argue that this weakness emerges from the reliance on spurious non-causal image features for classiﬁcation which makes the network vulnerable along these extra dimensions.
Our experiments conﬁrm this to the extent that when parts of the input image, and hence the corresponding features, may be selectively ignored we can achieve some increased robustness to adversarial attacks.
Tackling domain shift : CNNs are successful at the task of object recognition on experimental datasets such as PascalVOC (Everingham et al., 2015), where the train and test data distributions are extremely similar to one another.
However, real-world images are challenging as they include wide variations in object pose, background content, resolution, occlusion, and illumination conditions.
As these variations are not necessarily fully represented by the experimental datasets used to train networks, successfully applying those networks in the real world involves addressing the domain shift problem.
A traditional approach to handling domain shift in CNNs involves ﬁne-tuning on the new dataset, which may require thousands of images from each target category for successful adaptation.
Past studies have also explored domain adaptation by feature transformation, a process that requires sampling the target distribution to approximate the data transformation from the source to the target.
Here, we show that attention-aware models can offer signiﬁcant beneﬁts in handling domain changes, particularly those involving background content, occlusion, and object pose variation, by selectively attending to the object of interest.
In a manner similar to Girshick et al.
(2014) and Donahue et al.
(2014), we compare how well out-of-the-box CNN features from attention-aware models and their standard counterparts are able to classify previously unseen datasets.
Weakly supervised semantic segmentation :
Some of the most effective approaches to semantic segmentation require expensive annotation of thousands of images per category (Everingham et al., 2015).
To counter this, weakly supervised methods leverage scribbles, bounding boxes and imagelevel category labels for image segmentation.
We demonstrate the use of an attention mechanism built into standard CNN architectures to obtain segmentations as a by-product of training for the task of image classiﬁcation.
We compare with other works in this area including those that use traditional deﬁnitions of saliency (Jiang et al., 2013; Zhang & Sclaroff, 2013), object proposal techniques (Arbel´aez et al., 2014) and joint-inference for co-segmentation (Joulin et al., 2010; 2012; Chen et al., 2014; Dutt Jain & Grauman, 2016) for the task of weakly supervised semantic segmentation.
A PPROACH
The core goal of this work is to use attention maps to identify and exploit the effective spatial support of the information used by CNNs in making their classiﬁcation decisions.
This approach is premised on the hypothesis that there is beneﬁt to identifying salient regions and amplifying their inﬂuence, while likewise suppressing the irrelevant and potentially confusing information in other regions.
In particular, we expect that enforcing a more focused and parsimonious use of information over image space should aid in generalisation over changes in the data distribution, as occurs for instance when training on one set and testing on another.
Thus, we propose a trainable attention estimator and illustrate how to integrate it into standard CNN pipelines so as to inﬂuence their output as intended.
The method is based on enforcing a notion of compatibility between local feature vectors
3 x 3 C on v ,3 x 3 C on v ,3 x 3 C on v ,3 x 3 C on v ,3 x 3 C on v ,3 x 3 C on v ,3 x 3 C on v ,/3 x 3 C on v ,3 x 3 C on v ,3 x 3 C on v ,/3 x 3 C on v ,3 x 3 C on v ,3 x 3 C on v ,/
F C - 1 ,
F C - 2 , 3 x 3 C on v ,/3 x 3 C on v ,/
Attention EstimatorAtt.
[Figure]
Attention EstimatorAtt.
[Figure] [Figure]
Weighted
[Figure]
L
L
Lg ag ag
a
F C - 2 , extracted at intermediate stages in the CNN pipeline and
the global feature vector normally fed to the linear classiﬁcation layers at the end of the pipeline.
We implement attention-aware classiﬁcation by restricting the classiﬁer to use only a collection of local feature vectors, as chosen and weighted by the compatibility scores, in classifying examples.
We will ﬁrst discuss the modiﬁcation to the network architecture and the method of training it, given a choice of compatibility score.
We will then conclude the method description by presenting alternate choices of the score.
3. D ESIGN AND TRAINING OF ATTENTION SUBMODULE
The proposed approach is illustrated in Fig.
Denote by L s = { (cid:96) s 1 , (cid:96) s 2 , · · · , (cid:96) sn } the set of feature vectors extracted at a given convolutional layer denoted by s ∈ { 1 , · · · , S } , where each (cid:96) si is the vector of output activations at the spatial location i of n total spatial locations in the layer.
g is the global feature vector which has the entire input image as support and is output by the network’s series of convolutional and nonlinear layers, having only to pass through the ﬁnal fully connected layers to produce the original architecture’s class score for that input.
Assume for now the existence of a compatibility function C which takes two vectors of equal dimension as arguments and outputs a scalar compatibility score: this will be speciﬁed in 3.2.
The method proceeds by computing, for each of one or more layers s , the set of compatibility scores
C ( ˆ L s , g ) =
{ c s 1 , c s 2 , ...
c sn } , where ˆ L s is the image of L s under a linear mapping of the (cid:96) si to the dimensionality of g .
The compatibility scores are then normalised by a softmax operation:
[Equation]
The normalised compatibility scores A s = { a s 1 , a s 2 , · · · a sn } are then used to produce a single vector g sa =
(cid:80) ni
=1 a si · (cid:96) si for each layer s , by simple element-wise weighted averaging.
Crucially, the g sa now replace g as the global descriptor(s) for the image.
For a network trained under the restriction that the g sa alone are used to classify the input image, A corresponds to ‘attention’ as we have deﬁned it earlier.
In the case of a single layer ( S = 1 ), the attention-incorporating global vector g a is computed as described above, then mapped onto a T -dimensional vector which is passed through a softmax layer to obtain class prediction probabilities { ˆ p 1 , ˆ p 2 , · · · ˆ p T } , where T is the number of target classes.
In the case of multiple layers ( S > 1 ), we compare two options: concatenating the global vectors into a single vector g a =
[ g 1 a , g 2 a , · · · g Sa ] and using this as the input to the linear classiﬁcation step as above, or, using S different linear classiﬁers and averaging the output class probabilities.
All free network parameters are learned in end-to-end training under a cross-entropy loss function.
3. C HOICE OF COMPATIBILITY FUNCTION C
The compatibility score function C can be deﬁned in various ways.
The alignment model from Bahdanau et al.
Xu et al.
(2015) can be re-purposed as a compatibility function as follows:
[Equation]
Given the free parameters between the local and the global image descriptors in a CNN pipeline, we simplify the concatenation of the two descriptors to an addition operation and learn a single fully connected mapping to the compatibility scores.
Here, the weight vector u can be interpreted as learning the universal set of features relevant to the object categories in the dataset.
In that sense, the weights may be understood as learning the general concept of objectness.
Alternatively, we can use the dot product between g and (cid:96) i as a measure of their compatibility:
[Equation]
In this case, the relative magnitude of the scores would depend on the alignment between g and (cid:96) i in the high dimensional feature space and the strength of activation of (cid:96) i .
3. I NTUITION
In a standard CNN architecture, a global image descriptor g is derived from the input image and passed through a fully connected layer to obtain class prediction probabilities.
The network must express g via mapping of the input into a high-dimensional space in which salient higher-order visual concepts are represented by different dimensions, so as to render the classes linearly separable from one another.
Our method encourages the ﬁlters earlier in the CNN pipeline to learn mappings compatible with the mapping that produces g in the original architecture, by allowing a local descriptor (cid:96) i of an image patch to contribute to the ﬁnal classiﬁcation step only in proportion to its compatibility with g as detailed above.
That is, C (ˆ (cid:96) i , g ) should be high if and only if the corresponding patch contains parts of the dominant image category.
Note that this implies that the effective ﬁlters operating over image patches in the layers s must represent relatively ‘mature’ features with respect to the classiﬁcation goal.
We thus expect to see the greatest beneﬁt in deploying attention relatively late in the pipeline.
Note also that the use of the softmax function in normalising the compatibility scores enforces 0 ≤ a i ≤ 1 ∀ i ∈ { 1 · · · n } and (cid:80) i a i = 1 , i.e. that the combination of feature vectors is convex.
This ensures that features at different spatial locations must effectively compete against one another for their share of the attention map.
The compatibility scores thus serve as a robust proxy for attention in the classiﬁcation pipeline.
E XPERIMENTAL S ETUP
Baselines and proposed architectures: For our attention model, we remove the ﬁrst 2 max-pooling layers of the VGG network and place them after two additional convolutional layers at the end of the pipeline.
This ensures that the local layers used for estimating attention are reasonably sized.
Thus our modiﬁed model has 17 layers - 15 convolutional and 2 fully-connected layers.
The output activations of layer- 16 (FC layer) deﬁne our global feature vector g .
We use g to experiment with the proposed attention framework at layers 7 , 10 and 13 (Conv layers).
We contrast our approach with the modiﬁcation proposed by Zhou et al.
(2016), namely VGG-GAP.
For ResNets (He et al., 2015), we use a 164 -layered network.
We replace the spatial average-pooling step after the computational block 4 by convolutional and max-pooling steps which gives us our global feature vector g .
We use the representations at the output of blocks 2 , 3 and 4 as the local descriptors for estimating attention.
For more details refer to the appendix.
We refer to the network Net with attention at the last level as Net-att , attention at the last two levels as Net-att2 and attention at three levels as Net-att3 .
We denote by dp the use of the dot product for matching the global and local descriptors and by pc the use of parametrised compatibility.
We denote by concat the concatenation of descriptors from different levels for the ﬁnal linear classiﬁcation step.
Alternatively, we use indep to denote an independent prediction of probabilities at different levels using separate linear classiﬁers.
The probabilities are later averaged to obtain a single score per class.
Task-speciﬁc processing: We generate the adversarial images using the fast gradient sign method of Goodfellow et al.
and observe the network fooling behaviour at increasing L ∞ norms of the perturbations.
For cross-domain classiﬁcation, we extract features at the input of the ﬁnal fully connected layer in each model and use these to train a linear SVM with C = 1 similar to the setup used by Zhou et al.
We report the results of a 5-fold cross validation.
At no point do we ﬁne-tune the networks on the target dataset.
For weakly supervised segmentation, we combine the attention maps from the last 2 levels using element-wise multiplication, take the square root of the result to re-interpret it as a probability distribution (in the limit of the two attention distributions approaching each other), and binarise the result using the Otsu binarization threshold.
Details of the dataset pre-processing, architecture modiﬁcations for ResNet and network training can be found in § A.
R ESULTS AND DISCUSSION 5. I MAGE CLASSIFICATION AND FINE - GRAINED RECOGNITION
[Table]
[Table]
[Figure] [Figure]
Incorporating attention in a standard VGG architecture provides noticeable performance improvement over baseline and other counterparts e.g. , VGG-GAP, ResNet(RN) as seen for the classiﬁcation task on CIFAR datasets in Table 1.
Speciﬁcally, VGG-att2-concat-pc model achieves a 2 .
5% and 7 . 4% improvement over baseline for CIFAR-10 and CIFAR-100.
For the task of ﬁne-grained recognition of CUB-200-2011 and SVHN categories the same model gives 7 . 8% and 0 .
5% improvement w.r.t.
vanilla VGG, see Table 2.
Thus, for the rest of the experiments concat-pc is the implicit design of attention unless speciﬁed otherwise.
We visualise the attention maps learned by this network over the earlier layers in Fig.
Observably, the attention mechanism enables the network to focus on the object of interest which suppressing the background regions.
For the task of ﬁne-grained recognition, the same layers learn specialized focus on different object parts as seen in Fig.
As per our intuition, adding attention at more than two levels starts to degrade performance.
Note that due to the inclusion of additional layers, the number of parameters in the attention model is more than in the baseline.
However, training under the constraints of the attention mechanism allows the networks to generalise well on the test set.
When the same attention mechanism is introduced in ResNets (RN) we observe a marginal drop in performance, 0 .
9% on CIFAR-10 and 1 . 5% on CIFAR-100.
We believe that the skip-connections in these models that combine the features from shallow network layers with those of deeper layers could be working in a manner similar to the proposed attention mechanism i.e. allowing the network a systematic focus on some local regions while downplaying others.
However, unlike the skipconnection architecture, our method is able to learn explicit maps of network attention and exploit these for other tasks such as weak segmentation.
5. R OBUSTNESS TO ADVERSARIAL ATTACK
[Figure]
From Fig.
6, the fooling rate of attention aware VGG is 5% less than the baseline VGG at an L ∞ noise norm of 1 .
As the norm increases, the fooling rate saturates for the two networks and the gap gradually decreases.
Interestingly, when the noise begins to be perceptible (see col. 5 , Fig. 5), the fooling rate of VGG-att2 is around a percentage higher than that of VGG.
[Figure]
[Figure]
[Table]
CIFAR images cover a wider variety of natural object categories in comparison to SVHN and CUB, hence we use these to train different network architectures and use the networks as off-the-shelf feature extractors to evaluate their generalizability on new unseen datasets.
As seen in Table 3, attention aware models improve over the baseline models by an average margin of 6% across all unseen datasets.
We make two additional observations.
Firstly, low resolution CIFAR images contain useful visual properties that are transferrable and useful in classifying high resolution images such as the 600 × 600 images of Event-8 dataset (Li & Fei-Fei, 2007).
Secondly, training for diversity is better.
CIFAR-10 and CIFAR-100 datasets contain same corpus of images organized differently into 10 and 100 categories respectively.
From the accuracy pairs in Table 3, and the attention maps of Fig.
7, it appears that while learning to distinguish a larger set of categories the network is able to capture more nuanced features that transfer well to classifying new datasets.
(Note that STL dataset shares the same category set as CIFAR-10.
Hence, attention-mounted and baseline models are directly evaluated on STL without the use of an SVM.)
[Figure]
[Figure] [Figure]
5. W EAKLY - SUPERVISED SEMANTIC SEGMENTATION
[Figure]
[Figure]
(Zhang & Sclaroff, 2013) 51.
39.52 — Top object proposal-based —
MCG (Arbel´aez et al., 2014) 32.
37.85 — Joint segmentation-based —
Joulin et al. (Joulin et al., 2010) 15.
30.16 Object-discovery (Rubinstein et al., 2013) 55.
51.65 Chen et al. (Chen et al., 2014) 54.
44.46 Jain et al. (Dutt Jain & Grauman, 2016) 58.
From Table 9, the proposed attention maps perform signiﬁcantly better at weak segmentation than those obtained using the GAP approach of Zhou et al.
(2016); the minimum improvement in terms of Jaccard index being 30% .
We compare favourably to the top object-proposal based approach; outperforming for all three categories of airplane, car and horse by a minimum margin of 3%.
Note that we do not compare with the state-of-the-art CNN-based object proposal approaches given that they are trained using additional bounding box annotations.
We surpass the saliency-based approaches for segmentation of car category, but perform less well for the other two categories of airplane and horse.
This could be due to the detailed structure and small size of instances of the latter two categories, see Fig. 8.
We are not far behind the top performing methods that use joint inference on a group of test images for segmenting out the common object category.
C ONCLUSION
We propose a trainable attention module for generating probabilistic landscapes that highlight where and in what proportion a network attends to different regions of the input image for the task of classiﬁcation.
We demonstrate that the method, when deployed at multiple levels within a network, affords signiﬁcant performance gains in classiﬁcation of seen and unseen categories by focusing on the object of interest
We also show that the attention landscapes can facilitate weakly supervised segmentation of the predominant object.
Further, the proposed attention scheme is amenable to popular post-processing techniques such as conditional random ﬁelds for reﬁning the segmentation masks and has shown promise in learning robustness to certain kinds of adversarial attacks
A A PPENDICES A. D ATASETS
We evaluate the proposed attention models on CIFAR-10 (Krizhevsky & Hinton, 2009), CIFAR100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011) and CUB-200-2011 (Wah et al., 2011) for the task of classiﬁcation.
We use the attention incorporating VGG model trained on
CUBS-200-2011 for investigating robustness to adversarial attacks.
For cross-domain classiﬁcation, we test on 6 standard benchmarks including STL (Coates et al., 2010), Caltech-256 (Grifﬁn et al., 2007) and Action-40 (Yao et al., 2011).
We use the Object Discovery dataset (Rubinstein et al., 2013) for evaluating weakly supervised segmentation performance.
A detailed summary of the these datasets can be found in 4.
For all datasets except SVHN and CUB, we perform mean and standard-deviation normalization as well as color normalization.
For CUB-200-2011, the images are cropped using ground truth bounding box annotations and resized.
For cross-domain image classiﬁcation we downsample the input images to avoid memory overhead.
[Figure]
[Table]
A. N ETWORK ARCHITECTURES
ResNet: Our baseline ResNet implementation contains 4 distinct levels that map the RGB input into a 256 -dimensional space through 16 , 64 and 128 dimensional embedding spaces respectively.
Each level excepting the ﬁrst, which contains 2 convolutional layers separated by a non-linearity, contains n -residual blocks and each residual block contains a maximum of 3 convolutional layers interleaved by non-linearities.
This yields a network deﬁnition of 9 n + 2 parameterized layers (He et al., 2015).
We work with an n of 18 for a 164-layered network.
Attention in ResNet: We replace the spatial average pooling layer after the ﬁnal level 4 by convolutional and max-pooling steps which gives us our global feature vector g .
We refer to the network with attention on the output of the last level as RN-att and attention at the output of last two levels as RN-att2.
Following the results of VGG network, we train these attention units in the concat-pc framework.
A. T RAINING ROUTINES
VGG networks for CIFAR-10, CIFAR-100 and SVHN are trained from scratch.
We use a stochastic gradient descent (SGD) optimizer with a batch size of 128 , initial learning rate of 1 , learning rate decay of 10 − 7 , weight decay of 5 × 10 − 4 , and momentum of 0 . 9 .
The learning rate is scaled by 0 .every 25 epochs
and we train over 300 epochs for convergence.
For CUB, since the training data is limited, we initialise the model with the weights learned for CIFAR-100.
We use the transferlearning training schedule inspired by Redmon et al.
Thus the training starts at a learning rate of 0 . 1 for ﬁrst 30 epochs, is multiplied by 2 twice over the next 60 epochs, and then scaled by 0 .
5 every 30 epochs for the next 200 epochs.
For ResNet, the networks are trained using an SGD optimizer with a batch size of 64 , initial learning rate of 0 . 1 , weight decay of 5 × 10 − 4 , and a momentum of 0 . 9 .
The learning rate is multiplied by 0 .
2 after 60 , 120 and 160 epochs.
The network is trained for 200 epochs until convergence.
We train the models for CIFAR-10 and CIFAR-100 from scratch.
All models are implemented in Torch and trained with an NVIDIA Titan-X GPU.
Training takes around one to two days depending on the model and datasets.