A P AINLESS A TTENTION M ECHANISM FOR C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel attention mechanism to enhance Convolutional Neural Networks for ﬁne-grained recognition.
The proposed mechanism reuses CNN feature activations to ﬁnd the most informative parts of the image at different depths with the help of gating mechanisms and without part annotations.
Thus, it can be used to augment any layer of a CNN to extract low- and high-level local information to be more discriminative.
Differently, from other approaches, the mechanism we propose just needs a single pass through the input and it can be trained end-to-end through SGD.
As a consequence, the proposed mechanism is modular, architecture-independent, easy to implement, and faster than iterative approaches.
Experiments show that, when augmented with our approach, Wide Residual Networks systematically achieve superior performance on each of ﬁve different ﬁnegrained recognition datasets: the Adience age and gender recognition benchmark,
Caltech-UCSD Birds-200-2011, Stanford Dogs, Stanford Cars, and UEC Food100, obtaining competitive and state-of-the-art scores.
I NTRODUCTION
Humans and animals process vasts amounts of information with limited computational resources thanks to attention mechanisms which allow them to focus resources on the most informative chunks of information.
These biological mechanisms have been extensively studied (see Anderson (1985);
Desimone & Duncan (1995)), concretely those mechanisms concerning visual attention, e.g. the work done by Ungerleider & G (2000).
In this work, we incorporate the advantages of visual and biological attention mechanisms for for ﬁne-grained visual recognition with Convolutional Neural Networks (CNN) (see LeCun et al.
This is a particularly difﬁcult task since it involves looking for details in large amounts of data (images) while remaining robust to deformation and clutter.
In this sense, different attention mechanisms for ﬁne-grained recognition exist in the literature: (i) iterative methods that process images using ”glimpses” with recurrent neural networks (RNN) or long short-term memory (LSTM)
(e.g. the work done by Sermanet et al. (2015); Zhao et al.
(2017b))
, (ii) feed-forward attention mechanisms that augment vanilla CNNs, such as the Spatial Transformer Networks (STN) by Jaderberg et al.
(2015), or a top-down feed-forward attention mechanism (FAM)
(Rodr´ıguez et al.
Although it is not applied to ﬁne-grained recognition, the Residual Attention introduced by Wang et al.
(2017) is another example of feed-forward attention mechanism that takes advantage of residual connections (He et al.
(2016)) to enhance or dampen certain regions of the feature maps in an incremental manner.
Inspired by all the previous research about attention mechanisms in computer vision, we propose a novel feed-forward attention architecture (see Figure 1) that accumulates and enhances most of the desirable properties from previous approaches:
Detect and process in detail the most informative parts of an image: more robust to deformation and clutter.
Feed-forward trainable with SGD: faster inference than iterative models, faster convergence rate than Reinforcement Learning-based (RL) methods like the ones presented by Sermanet et al.
; Liu et al.
P r ed
C l a ss
C on v B l
o ck
C on v B l
o ck
C on v B l
o ck Σ
Input conv1x1 Softmax
Attention P r ed c c
Input conv1x1 Softmax
Attention P r ed c
Input conv1x1 Softmax
Attention P r ed c
Preserve low-level detail:
unlike Residual Attention (Wang et al.
(2017)), where low-level features are subject to noise after traversing multiple residual connections, our architecture directly uses them to make predictions.
This is important for ﬁne-grained recognition where low-level patterns such as textures can help to distinguish two similar classes.
Moreover, the proposed mechanism possesses other interesting properties such as:
Modular and incremental : the attention mechanism can be replicated at each layer on any convolutional architecture, and it is easy to adapt to the task at hand.
Architecture independent : the mechanism can accept any pre-trained architecture such as VGG (Simonyan & Zisserman (2014)) or ResNet.
Low computational impact : While STNs use a small convnet to predict afﬁne-transform parameters and Residual Attention uses the hourglass architecture, our attention mechanism consists of a single 1 × 1 convolution and a small fully-connected layer.
Simple : the proposed mechanism can be implemented in few lines of code, making it appealing to be used in future work.
The proposed attention mechanism has been included in a strong baseline like Wide Residual Networks (WRN) (Zagoruyko & Komodakis (2016)), and applied on ﬁve ﬁne-grained recognition datasets.
The resulting network, called Wide Residual Network with Attention (WRNA) systematically enhances the performance of WRNs, obtains competitive results using low resolution training images, and surpasses the state of the art in the Adience gender recognition task, Stanford dogs, and
UEC Food-100.
Table 1 shows the gain in performance of WRNA w.r.t.
WRN for all the datasets considered in this paper.
In the next section, we review the most relevant work concerning attention mechanisms for visual ﬁne-grained recognition.
R ELATED W ORK
As reviewed by Zhao et al. (2017a), there are different approaches to ﬁne-grained recognition: (i) vanilla deep CNNs, (ii) CNNs as feature extractors for localizing parts and do alignment, (iii) ensembles, (iv) attention mechanisms.
In this paper we focus on (iv), the attention mechanisms, which aim to discover the most discriminative parts of an image to be processed in greater detail, thus ignoring clutter and focusing on the most distinctive traits.
These parts are central for ﬁne-grained recognition, where the inter-class variance is small and the intra-class variance is high.
[Table]
Different ﬁne-grained attention mechanisms can be found in the literature.
Xiao et al.
(2015) proposed a two-level attention mechanism for ﬁne-grained classiﬁcation on different subsets of the
ICLR2012 (Russakovsky et al.
(2012)) dataset, and the CUB200 2011.
In this model, images are ﬁrst processed by a bottom-up object proposal network based on R-CNN (Zhang et al.
and selective search (Uijlings et al.
Then, the softmax scores of another ILSVRC2012 pretrained CNN, which they call FilterNet , are thresholded to prune the patches with the lowest parent class score.
These patches are then classiﬁed to ﬁne-grained categories with a DomainNet .
Spectral clustering is also used on the DomainNet ﬁlters in order to extract parts (head, neck, body, etc.), which are classiﬁed with an SVM.
Finally, the part- and object-based classiﬁer scores are merged to get the ﬁnal prediction.
The two-level attention obtained state of the art results on CUB200with only class-level supervision.
However, the pipeline must be carefully ﬁne-tuned since many stages are involved with many hyper-parameters.
Differently from two-level attention , which consists of independent processing and it is not endto-end, Sermanet et al.
proposed to use a deep CNN and a Recurrent Neural Network (RNN) to accumulate high multi-resolution ”glimpses” of an image to make a ﬁnal prediction (Sermanet et al.
, however, reinforcement learning slows down convergence and the RNN adds extra computation steps and parameters.
A more efﬁcient approach was presented by Liu et al.
in (Liu et al.
(2016)), where a fullyconvolutional network is trained with reinforcement learning to generate conﬁdence maps on the image and use them to extract the parts for the ﬁnal classiﬁers whose scores are averaged.
Compared to previous approaches, in the work done by Liu et al. (2016), multiple image regions are proposed in a single timestep thus, speeding up the computation.
A greedy reward strategy is also proposed in order to increase the training speed.
The recent approach presented by Fu et al.
(2017) uses a classiﬁcation network and a recurrent attention proposal network that iteratively reﬁnes the center and scale of the input (RA-CNN).
A ranking loss is used to enforce incremental performance at each iteration.
Zhao et al.
proposed Diversiﬁed Visual Attention Network (DVAN), i.e. enforcing multiple nonoverlapped attention regions (Zhao et al.
(2017b)).
The overall architecture consists of an attention canvas generator, which extracts patches of different regions and scales from the original image; a
VGG-16 (Simonyan & Zisserman (2014))
CNN is then used to extract features from the patches, which are aggregated with a DVAN long short-term memory (Hochreiter & Schmidhuber (1997)) that attends to non-overlapping regions of the patches.
Classiﬁcation is performed with the average prediction of the DVAN at each region.
All the previously described methods involve multi-stage pipelines and most of them are trained using reinforcement learning (which requires sampling and makes them slow to train).
In contrast,
STNs, FAM, and our approach jointly propose the attention regions and classify them in a single pass.
Moreover, they possess interesting properties compared to previous approaches such as (i) simplicity (just a single model is needed), (ii) deterministic training (no RL), and (iii) feed-forward training (only one timestep is needed).
In addition, since our approach only uses one CNN stream, it is far more computationally efﬁcient than STNs and FAM, as described next.
[Figure]
O UR APPROACH
Our approach consists of a universal attention module that can be added after each convolutional layer without altering pre-deﬁned information pathways of any architecture.
This is helpful since it allows to seamlessly augment any architecture such as VGG and ResNet with no extra supervision, i.e. no part labels are necessary.
The attention module consists of three main submodules: (i) the attention heads H , which deﬁne the most relevant regions of a feature map, (ii) the output heads
O , generate an hypothesis given the attended information, and (iii) the conﬁdence gates G , which output a conﬁdence score for each attention head.
Each of these modules is explained in detail in the following subsections.
3. A TTENTION HEAD
Inspired by DVAN (Zhao et al. (2017b)) and the transformer architecture presented by Vaswani et al.
(2017), and following the notation established by Zagoruyko & Komodakis (2016), we have identiﬁed two main dimensions to deﬁne attentional mechanisms: (i) the number of layers using the attention mechanism, which we call attention depth (AD), and (ii) the number of attention heads in each attention module, which we call attention width (AW).
Thus, a desirable property for any universal attention mechanism is to be able to be deployed at any arbitrary depth and width .
This property is fulﬁlled by including K attention heads H k (width), depicted in Figure 2a, into each attention module (depth) 1 .
Then, the attention heads at layer l , receive the feature activations
Z l of that layer as input, and output K weighted feature maps, see Equations 1 and 2:
[Equation]
where H l is the output matrix of the l th attention module, W H is a 1 × 1 convolution kernel with output dimensionality K used to compute the attention masks corresponding to the attention heads
H k , ∗ denotes the convolution operator, and (cid:12) is the element-wise product.
Please note that M lk is a 2d ﬂat mask and the product with each of the N input channels of Z is done by broadcasting.
Likewise, the dimensionality of H l k is the same as Z l .
Since the different attention heads in an attention module are sometimes focusing on the same exact part of the image, similarly to DVAN, we have introduced a regularization loss L R that forces the multiple masks to be different.
In order to simplify the notation, we set m lk to be k th ﬂattened version of the attention mask M in Equation 1.
Then, the regularization loss is expressed as:
[Equation]
i.e. , it minimizes the Frobenius norm of the cross-correlation between each pair of attention masks.
This loss is added to the network loss L net weighted by a constant factor γ = 0 .
1 which was found to work best across all tasks:
[Equation]
3. O UTPUT HEAD
The output of each attention module consists of a spatial dimensionality reduction layer:
[Equation]
followed by a fully-connected layer that produces an hypothesis on the output space, see Figure 2b.
[Equation]
We consider two different dimensionality reductions: (i) a channel-wise inner product by W 1 × n F , where W F is a dimensionality reduction projection matrix with n the number of input channels; and (ii) an average pooling layer.
We empirically found (i) to work slightly better than (ii) but at a higher computational cost.
W F is shared across all attention heads in an attention module.
3. A TTENTION GATES
Each attention module makes a class hypothesis given its local information.
However, in some cases, the local features are not good enough to output a good hypothesis.
In order to alleviate this problem, we make each attention module, as well as the network output, to predict a conﬁdence score c by means of an inner product by the gate weight matrix W G :
[Equation]
The gate weights g are then obtained by normalizing the set of scores by means of a softmax function:
[Equation]
where |G| is the total number of gates, and c i is the i th conﬁdence score from the set of all conﬁdence scores.
The ﬁnal output of the network is the weighted sum of the output heads:
[Equation]
Please note that setting the output of G to 1 |G| , corresponds to averaging all the outputs.
Likewise, setting {G \ G output } = 0 , G output = 1 , i.e. the set of attention gates is set to zero and the output gate to one, corresponds to the original pre-trained model without attention.
In Table 2 we show the importance of each submodule of our proposal on WRN.
Attention modules are placed after each pooling layer, starting from the fourth and going backward when AD increases.
[Table]
As it can be seen, just adding a single attention module with a single attention head is enough to increase the mean accuracy by 1 . 2% .
Adding extra heads and gates increase an extra 0 .
1% each.
The fact that the attention mask is generated by just one 1 × 1 convolution and the direct connection to the output makes the module fast to learn, thus being able to generate foreground masks from the beginning of the training and reﬁning them during the following epochs. A sample of these attention masks for each dataset is shown on Figure 3. [Figure]
In the next section, we test the performance of the previously described mechanisms on different datasets.
E XPERIMENTS
In this section, we demonstrate the effectiveness of the proposed mechanisms for ﬁne-grained recognition on ﬁve different datasets: and age and gender (Adience dataset by Eidinger et al.
, birds (CUB200-2011 by Wah et al.
Stanford cars Krause et al.
Stanford dogs (Khosla et al.
(2011)), and food (UECFOOD-100 by Matsuda et al.
In order to demonstrate that the proposed generalized attention can easily augment any recent architecture, we have trained a strong baseline, namely a Wide Residual Network (WRN) (Zagoruyko &
Komodakis (2016)) pre-trained on the ImageNet.
The modules described in the previous sections have been implemented on pytorch, and trained in a single workstation with two NVIDIA 1080Ti.
[Figure]
All the experiments are trained for 100 epochs, with a batch size of 64.
The learning rate is ﬁrst set to 10 − 3 to all layers except the attention modules and the classiﬁer, for which it ten times higher.
The learning rate is reduced by a factor of 0 .
5 every 30 iterations and the experiment is automatically stopped if a plateau is reached.
The network is trained with standard data augmentation, i.e. random 224 × 224 patches are extracted from 256 × 256 images with random horizontal ﬂips 2 .
For the sake of clarity and since the aim of this work is to demonstrate that the proposed mechanism universally improves CNNs for ﬁne-grained recognition, we follow the same training procedure in all datasets.
Thus, we do not use 512 × 512 images which are central in STNs, RA-CNNs, or
B-CNNs to reach state of the art performances.
Accordingly, we do not perform color jitter and other advanced augmentation techniques such as the ones used by Hassannejad et al.
(2016) for food recognition.
The proposed method is able to obtain state of the art results in Adience Gender,
Stanford dogs and UEC Food-100 even when trained with lower resolution.
In the following subsections the proposed approach is evaluated on the ﬁve datasets.
4. A DIENCE DATASET
The adience dataset consists of 26.5 K images distributed in eight age categories (02, 46, 813, 1520, 2532, 3843, 4853, 60+), and gender labels.
A sample is shown in Figure 4a.
The performance on this dataset is measured by both the accuracy in gender and age recognition tasks using 5-fold cross-validation in which the provided folds are subject-exclusive.
The ﬁnal score is given by the mean of the accuracies of the ﬁve folds.
This dataset is particularly challenging due to the high level of deformation of face pictures taken in the wild, occlusions and clutter such as sunglasses, hats, and even multiple people in the same image.
As it can be seen in Table 3, the Wide ResNet augmented with generalized attention surpasses the baseline performance, etc.
4. C ALTECH -UCSD B IRDS
The birds dataset (see Figure 4b) consists of 6K train and 5.8K test bird images distributed incategories.
The dataset is especially challenging since birds are in different poses and orientations, and correct classiﬁcation often depends on texture and shape details.
Although bounding box, bough
[Table]
segmentation, and attributes are provided, we perform raw classiﬁcation as done by Jaderberg et al.
[Table]
In Table
[Table]
the performance of our approach is shown in context with the state of the art.
Please note that even our approach is trained in lower resolution crops, i.e. 224 × 224 instead of 448 × 448 , we reach the same accuracy as the recent fully convolutional attention by Liu et al.
4. S TANFORD C ARS
The Cars dataset contains 16K images of 196 classes of cars, see Figure 4c.
The data is split into 8K training images and 8K testing images.
The difﬁculty of this dataset resides in the identiﬁcation of the subtle differences that distinguish between two car models.
[Table]
In Table 5 the performance of our approach with respect to the baseline and other state of the art is shown.
The augmented WRN shows better performance than the baseline, and even surpasses recent approaches such as FCAN.
4. S TANFORD D OGS
The Stanford Dogs dataset consists of 20.5K images of 120 breeds of dogs, see Figure 4d.
The dataset splits are ﬁxed and they consist of 12k training images and 8.5K validation images.
Pictures are taken in the wild and thus dogs are not always a centered, unique, pose-normalized object in the image but a small, cluttered region.
[Table]
Table 6 shows the results on Stanford dogs.
As it can be seen, performances are low in general and nonetheless, our model was able to increase the accuracy by a 0 .
1% w/o gates), being the highest score obtained on this dataset to the best of our knowledge.
This performance has been achieved thanks to the gates, which act as a detection mechanism, giving more importance to those attention masks that correctly guessed the position of the dog.
4. UEC F OOD
UEC Food100 is a Japanese food dataset with 14K images of 100 different dishes, see Figure 4e.
Pictures present a high level of variation in the form of deformation, rotation, clutter
[Table]
and noise.
In order to follow the standard procedure in the literature ( e
[Table]
g. Chen & Ngo (2016); Hassannejad et al.
(2016)), we use the provided bounding boxes to crop the images before training
[Table]
Table 7 shows the performance of our model compared to the state of the art.
As it can be seen, our model is able to improve the baseline by a relative 7% with a 85 .
5% of accuracy, the best-reported result compared to previous publications.
C ONCLUSION
We have presented a novel attention mechanism to improve CNNs for ﬁne-grained recognition.
The proposed mechanism ﬁnds the most informative parts of the CNN feature maps at different depth levels and combines them with a gating mechanism to update the output distribution.
Unlike previous work, the proposed mechanism is modular, architecture independent, fast, and simple and yet WRN augmented with it show higher accuracy in each of the following tasks:
Age and Gender Recognition (Adience dataset), CUB200-2011 birds, Stanford Dogs, Stanford Cars, and
UEC Food-100.
Moreover, state of the art performance is obtained on gender, dogs, and cars.