U NSUPERVISED R EPRESENTATION L EARNING BY P REDICTING I MAGE R OTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Over the last years, deep convolutional neural networks (ConvNets) have transformed the ﬁeld of computer vision thanks to their unparalleled capacity to learn high level semantic image features.
However, in order to successfully learn those features, they usually require massive amounts of manually labeled data, which is both expensive and impractical to scale.
Therefore, unsupervised semantic feature learning, i.e., learning without requiring manual annotation effort, is of crucial importance in order to successfully harvest the vast amount of visual data that are available today.
In our work we propose to learn image features by training
ConvNets to recognize the 2d rotation that is applied to the image that it gets as input.
We demonstrate both qualitative and quantitative that this apparently simple task actually provides a very powerful supervisory signal for semantic feature learning.
We exhaustively evaluate our method in various unsupervised feature learning benchmarks and we exhibit in all of them remarkably good performance.
Even more, our results on those benchmarks demonstrate dramatic improvements w.r.t.
prior state-of-the-art, thus they signiﬁcantly close the gap between unsupervised and supervised feature learning.
For instance, in PASCAL VOCdetection task our unsupervised pre-trained AlexNet model achieves the state-ofthe-art (among unsupervised methods) mAP of 54 .
4% that is only 2.4 points lower from the supervised case.
Similar striking results we get when transfer our unsupervised learned features on various other tasks, such as ImageNet classiﬁcation,
PASCAL classiﬁcation, and CIFAR-10 classiﬁcation.
I NTRODUCTION
In recent years, there is a tremendous progress on the ﬁeld of computer vision that is thanks to the use of deep convolutional neural networks (LeCun et al., 1998) (ConvNets) for learning powerful image representations for semantic image understanding.
Typically, those ConvNets learn high level image features by being trained on the object recognition (Russakovsky et al., 2015) or the scene classiﬁcation (Zhou et al., 2014) tasks using a massive amount of manually labeled data.
The image features learned by the ConvNets in this supervised manner have been proven extremely helpful when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).
However, supervised feature learning has the main limitation of requiring intensive manual labeling effort, which is both expensive and infeasible to scale on the vast amount of visual data that are available today.
Due to that, there is lately an increased interest to learn high level ConvNet based representations in an unsupervised manner that avoids manual annotation of visual data.
Among the unsupervised feature learning approaches, a prominent paradigm is the so-called self-supervised learning that deﬁnes an annotation free pretext task, using only the visual information present on the images or videos, in order to provide a surrogate supervision signal for feature learning.
For example, in order to learn features, Zhang et al.
(2016a)
and Larsson et al.
(2016) train ConvNets to colorize gray scale images, Doersch et al.
and Noroozi & Favaro (2016) predict the relative position of image patches, and Agrawal et al.
(2015) predict the egomotion (i.e., self-motion) of a moving vehicle between two consecutive frames.
The rationale behind such self-supervised tasks is that solving them will force the ConvNet to learn semantic image features that can be useful for other vision task.
In fact, image representations learned with the above self-supervised tasks, although they have not managed to match the performance of supervised-learned representation, they have proved to be good alternatives for transferring on other vision tasks, such as object recognition, object detection, and semantic segmentation (Zhang et al., 2016a; Larsson et al., 2016;
Zhang et al., 2016b; Larsson et al., 2017; Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2017).
Our work follows this self-supervised paradigm and proposes to learn image representations by training ConvNets to recognize the geometric transformation that is applied to the image that it gets as input.
More speciﬁcally, we ﬁrst deﬁne a small set of discrete geometric transformations, then each of those geometric transformations are applied to each image on the dataset and the produced transformed images are fed to the ConvNet model that is trained to recognize the transformation of each image.
In this formulation, it is the set of geometric transformations that actually deﬁnes the classiﬁcation pretext task that the ConvNet model has to learn.
Therefore, in order to achieve unsupervised semantic feature learning, it is of crucial importance to properly choose those geometric transformations (we further discuss this aspect of our methodology in section 2.2).
What we propose is to deﬁne the geometric transformations as the image rotations by 0, 90, 180, anddegrees.
Thus, the ConvNet model is trained on the 4-way image classiﬁcation task of recognizing one of the four image rotations (see Figure 2).
We argue that in order a ConvNet model to be able recognize the rotation transformation that was applied to an image it will require to understand the concept of the objects depicted in the image (see Figure 1), such as their location in the image, their type, and their pose.
Throughout the paper we support that argument both qualitatively and quantitatively.
Furthermore we demonstrate on the experimental section of the paper that despite the simplicity of our self-supervised approach, the task of predicting rotation transformations provides a powerful surrogate supervision signal for feature learning and leads to dramatic improvements on the relevant benchmarks.
Note also that our self-supervised task is different from the work of Dosovitskiy et al.
(2014) that also employs geometric (as well as chromatic) transformations.
Basically, they train a ConvNet model to yield representations that are discriminative between images and at the same time invariant on geometric and chromatic transformations.
In contrast, we train a ConvNet model to recognize the geometric transformation applied to an image.
Our contributions are:
• We propose a new self-supervised task that is very simple and at the same time, as we demonstrate throughout the paper, offers a powerful supervisory signal for semantic feature learning.
• We exhaustively evaluate our self-supervised method under various settings (e.g. semisupervised or transfer learning settings) and in various vision tasks (i
e., CIFAR-10, ImageNet, and PASCAL classiﬁcation or detection).
• In all of them, our novel self-supervised formulation demonstrates extremely good performance and achieves state-of-the-art results with dramatic improvements w.r.t.
prior unsupervised approaches.
• As a consequence we show that for several important vision tasks, our self-supervised learning approach signiﬁcantly narrows the gap between unsupervised and supervised feature learning.
In the following sections, we describe our self-supervised methodology in § 2, we provide experimental results in § 3, and ﬁnally we conclude in § 4.
M ETHODOLOGY 2. O VERVIEW
The goal of our work is to learn ConvNet based semantic features in an unsupervised manner.
To achieve that goal we propose to train a ConvNet model F ( . ) to estimate the geometric transformation applied to an image that is given to it as input.
Speciﬁcally, we deﬁne a set of K discrete geometric transformations G =
{ g ( .
| y ) }
Ky =1 , where g ( .
| y ) is the operator that applies to image X the geometric transformation with label y that yields the transformed image
X y =
g ( X | y ) .
The ConvNet model
F ( . ) gets as input an image X y ∗ (where the label y ∗ is unknown to model F ( . ) ) and yields as output
[Figure]
a probability distribution over all possible geometric transformations:
[Equation]
where F y ( X y ∗ | θ ) is the predicted probability for the geometric transformation with label y and θ are the learnable parameters of model F ( . ) .
Therefore, given a set of N training images D =
{ X i } Ni =0 , the self-supervised training objective that the ConvNet model must learn to solve is:
[Equation]
where the loss function loss ( . ) is deﬁned as:
[Equation]
In the following subsection we describe the type of geometric transformations that we propose in our work.
2. C HOOSING GEOMETRIC TRANSFORMATIONS : IMAGE ROTATIONS
The type of geometric transformation G in the above formulation is of crucial importance since they deﬁne a classiﬁcation task that solving it must require learning semantic features useful for vision perception tasks (e.g., object detection or image classiﬁcation).
In our work we propose to deﬁne the set of geometric transformations G as all the image rotations by multiples of 90 degrees, i.e., 2d image rotations by 0, 90, 180, and 270 degrees (see Figure 2).
More formally, if Rot ( X, φ ) is an operator that rotates image X by φ degrees, then our set of geometric transformations consists of the K = 4 image rotations G =
{ g ( X | y ) } 4 y =1 , where g ( X | y ) =
Rot ( X, ( y − 1)90) .
Forcing the learning of semantic features: The core intuition behind using these image rotations as the set of geometric transformations relates to the simple fact that it is essentially impossible for a
ConvNet model to effectively perform the above rotation recognition task unless it has ﬁrst learnt to recognize and detect classes of objects as well as their semantic parts in images.
More speciﬁcally, to successfully predict the rotation of an image the ConvNet model must necessarily learn to localize salient objects in the image, recognize their orientation and object type, and then relate the object orientation with the dominant orientation that each type of object tends to be depicted within the available images.
In Figure 3 we visualize some attention maps generated by a model trained on the rotation recognition task.
These attention maps are computed based on the magnitude of activations at each spatial cell of a convolutional layer and essentially reﬂect where the network puts most of its focus in order to classify an input image.
We observe, indeed, that in order for the model to accomplish the rotation prediction task it learns to focus on high level object parts in the image, such as eyes, nose, tails, and heads.
Furthermore, in Figure 4 we visualize the ﬁrst layer ﬁlters that were learnt by an AlexNet model trained on the proposed rotation recognition task.
As can be seen, they appear to have a big variety of edge ﬁlters on multiple orientations and multiple frequencies.
Rotated image: X Rotated image: X
Rotated image: X Rotated image: X
ConvNet model F(.) ConvNet model F(.)
ConvNet model F(.) ConvNet model F(.)
Image X
Predict 270 degrees rotation (y=3)
Rotate 270 degrees g
( X , y = 3 )
Rotate 180 degrees g
( X , y = 2 )
Rotate 90 degrees g
( X , y = 1 )
Rotate 0 degrees g ( X , y = 0 )
Maximize prob.
F 3 ( X 3 )
Predict 0 degrees rotation (y=0)
Maximize prob.
F 2 ( X 2 )
Maximize prob.
F 1 ( X 1 )
Maximize prob.
F
( X 0 )
Predict 180 degrees rotation (y=2)
Predict 90 degrees rotation (y=1)
Objectives:
Given four possible geometric transformations, the 0, 90, 180, and 270 degrees rotations, we train a ConvNet model F ( . ) to recognize the rotation that is applied to the image that it gets as input.
F y ( X y ∗ ) is the probability of rotation transformation y predicted by model F ( . )
when it gets as input an image that has been transformed by the rotation transformation y ∗ .
Remarkably, these ﬁlters seem to have a greater amount of variety even than the ﬁlters learnt by the supervised object recognition task.
Absence of low-level visual artifacts: An additional important advantage of using image rotations by multiples of 90 degrees over other geometric transformations, is that they can be implemented by ﬂip and transpose operations (as we will see below) that do not leave any easily detectable low-level visual artifacts that will lead the ConvNet to learn trivial features with no practical value for the vision perception tasks.
In contrast, had we decided to use as geometric transformations, e.g., scale and aspect ratio image transformations, in order to implement them we would need to use image resizing routines that leave easily detectable image artifacts.
Well-posedness: Furthermore, human captured images tend to depict objects in an “up-standing” position, thus making the rotation recognition task well deﬁned, i.e., given an image rotated by 0, 90, 180, or 270 degrees, there is usually no ambiguity of what is the rotation transformation (with the exception of images that only depict round objects).
In contrast, that is not the case for the object scale that varies signiﬁcantly on human captured images.
Implementing image rotations:
In order to implement the image rotations by 90, 180, anddegrees (the 0 degrees case is the image itself), we use ﬂip and transpose operations.
Speciﬁcally, for 90 degrees rotation we ﬁrst transpose the image and then ﬂip it vertically (upside-down ﬂip), for 180 degrees rotation we ﬂip the image ﬁrst vertically and then horizontally (left-right ﬂip), and ﬁnally for 270 degrees rotation we ﬁrst ﬂip vertically the image
and then we transpose it.
2. D ISCUSSION
The simple formulation of our self-supervised task has several advantages.
It has the same computational cost as supervised learning, similar training convergence speed (that is signiﬁcantly faster than image reconstruction based approaches; our AlexNet model trains in around 2 days using a single
Titan X GPU), and can trivially adopt the efﬁcient parallelization schemes devised for supervised learning (Goyal et al., 2017), making it an ideal candidate for unsupervised learning on internetscale data (i.e., billions of images).
Furthermore, our approach does not require any special image
[Figure] [Figure]
pre-processing routine in order to avoid learning trivial features, as many other unsupervised or self-supervised approaches do.
Despite the simplicity of our self-supervised formulation, as we will see in the experimental section of the paper, the features learned by our approach achieve dramatic improvements on the unsupervised feature learning benchmarks.
[Table]
[Table]
E XPERIMENTAL R ESULTS
In this section we conduct an extensive evaluation of our approach, on the most commonly used image datasets, such as ImageNet, PASCAL, and CIFAR-10, and on various vision tasks, such as object detection and image classiﬁcation.
We also consider several learning scenarios, including transfer learning and semi-supervised learning.
In all cases, we compare our approach with corresponding state-of-the-art methods.
3. CIFAR EXPERIMENTS
We start by evaluating the performance of our self-supervised ConvNet based learned features (using the rotation prediction task) when transferred on the object recognition task of CIFAR10 (Krizhevsky & Hinton, 2009).
We will here after call a ConvNet model that is trained on the self-supervised task of rotation prediction RotNet model.
Implementation details:
In our CIFAR-10 experiments we implement the RotNet models with
Network-In-Network (NIN) architectures (Lin et al., 2013).
In order to train them on the rotation prediction task, we use SGD with batch size 128 , momentum 0 .
9 , weight decay 5 e − 4 and lr of 0.1.
We drop the learning rates by a factor of 5 after epochs 30, 60, and 80 epochs.
We train in total for 100 epochs.
In our preliminary experiments we found that we get signiﬁcant improvement when during training we train the network by feeding it all the four rotated copies of an image simultaneously instead of each time randomly sampling a single rotation transformation.
Therefore, at each training batch the network sees 4 times more images than the batch size.
Evaluation of the learned feature hierarchies: First, we explore how the quality of the learned features depends from their depth (i.e., the depth of the layer that they come from) as well as from the total depth of the RotNet model.
For that purpose, we ﬁrst train 3 RotNet models which have 3, 4,
[Figure]
along trained and 5 convolutional blocks respectively on the rotation prediction task using the CIFAR-10 training images (note that each conv.
block in the NIN architectures that implement the RotNet models have 3 conv.
layers; therefore, the total number of conv.
layers of the examined RotNet models is 9, 12, and 15 for 3, 4, and 5 conv.
blocks respectively).
Afterwards, we learn classiﬁers on top of the feature maps generated by each conv.
block of each
RotNet model.
Those classiﬁers are trained in a supervised way on the object recognition task of
CIFAR-10.
They consist of 3 fully connected layers (plus batch-norm and relu after the ﬁrst two layers) and the 2 hidden layers have 200 feature channels each.
We report the accuracy results of CIFAR-10 test set in Table 1.
We observe that in all cases the best performance comes from the feature maps generated by the 2nd conv.
block (that actually has depth 6 in terms of the total number of conv.
layer till that point)
and they rich very high accuracy, i.e., between 88.26% and 89.06%).
After the 2nd conv.
block the self-supervised learned features start degrade the object recognition accuracy, which we assume is because they start becoming more and more speciﬁc on the self-supervised task of rotation prediction.
Also, we observe that increasing the total depth of the RotNet models leads to increased object recognition performance by the feature maps generated by earlier layers (and after the 1st conv. block).
We assume that this is because increasing the depth of the model and thus the complexity of its head (i.e., top ConvNet layers) allows the features of earlier layers to be less speciﬁc to the rotation prediction task.
Comparison against supervised and other unsupervised methods: In Table 2 we compare our unsupervised learned features against other unsupervised (or hand-crafted) features on CIFAR-10.
For our entry we use the feature maps generated by the 2nd conv.
block of a RotNet model withconv.
blocks in total.
On top of those RotNet features we train 2 different classiﬁers: (a) a non-linear classiﬁer with 3 fully connected layers as before (entry (Ours) RotNet + non-linear ), (b) three conv.
layers plus a linear prediction layer (entry (Ours) RotNet +conv. ; note that this entry is basically ablocks NIN model with the ﬁrst 2 blocks coming from a RotNet model and the 3rd being randomly initialized and trained on the recognition task).
We observe that we signiﬁcantly improve over the prior unsupervised approaches and we achieve state-of-the-art results in CIFAR-10 .
More notably, the accuracy gap between the RotNet based model and the fully supervised NIN model is very small , only 1.64 percentage points (92.80% vs 91.16%).
[Table] [Table]
Correlation between object classiﬁcation task and rotation prediction task:
In Figure 5a
[Table]
with the yellow curve we plot the rotation prediction accuracy and the object classiﬁcation accuracy (of a classiﬁer that is applied on self-supervised features) as a function of the training epochs used for solving the self-supervised task of recognizing rotations.
We observe that the rotation prediction accuracy is correlated with the object recognition accuracy and also that the object recognition accuracy converges fast w.r.t.
the number of training epochs used for solving the pretext task of rotation prediction.
Semi-supervised setting: Motivated by the very high performance of our unsupervised feature learning method, we also evaluate it on a semi-supervised setting.
More speciﬁcally, we ﬁrst train a 4 block RotNet model on the rotation prediction task using the entire image dataset of CIFAR-10 and then we train on top of its feature maps object classiﬁers using only a subset of the available images and their corresponding labels.
As feature maps we use those generated by the 2nd conv.
block of the
RotNet model.
As a classiﬁer we use a set convolutional layers that actually has the same architecture as the 3rd conv.
block of a NIN model plus a linear classiﬁer, all randomly initialized.
For training the object classiﬁer we use for each category 20, 100, 400, 1000, or 5000 image examples.
Note that 5000 image examples is the extreme case of using the entire CIFAR-10 training dataset.
Also, we compare our method with a supervised model that is trained only on the available examples each time.
In Figure 5b we plot the accuracy of the examined models as a function of the available training examples.
We observe that our unsupervised trained model exceeds in this semi-supervised setting the supervised model when the number of examples per category drops below 1000.
Furthermore, as the number of examples decrease the performance gap in favor of our method becomes more striking.
This empirical evidence demonstrates the usefulness of our method on semi-supervised settings.
3. I MAGE N ET AND PASCAL EXPERIMENTS
Here we evaluate the performance of our self-supervised ConvNet models on the ImageNet and
PASCAL VOC datasets.
Speciﬁcally, we ﬁrst train a rotation prediction model RotNet on the training images of the ImageNet dataset
and then we evaluate the performance of the self-supervised features on the image classiﬁcation tasks of ImageNet and PASCAL datasets and on the object detection task of PASCAL VOC 2007.
Implementation details: For the ImageNet and PASCAL VOC experiments we implement our
RotNet model with an AlexNet architecture.
Our implementation of the AlexNet model does not have local response normalization units or groups and it includes batch normalization units after
To obtain those results we use the publicly available testing frameworks of Kr¨ahenb¨uhl et al. (2015) and Girshick (2015).
For classiﬁcation, we either ﬁx the features before conv5 (column fc6-8 ) or we ﬁne-tune the whole model (column all ).
For detection we use multi-scale training and single scale testing.
After training the RotNet model on ImageNet, we absorb the batch normalization units on the linear layers and we use the weight rescaling technique proposed by Kr¨ahenb¨uhl et al.
(2015) (which is common among the unsupervised methods).
We report mean average prevision as customary on P ASCAL VOC.
[Table]
each linear layer (either convolutional or fully connected).
In order to train the AlexNet based
RotNet model, we use SGD with batch size 192 , momentum 0 .
9 , weight decay 5 e − 4 and lr of 0.01.
We drop the learning rates by a factor of 10 after epochs 10, and 20 epochs.
We train in total forepochs.
As in the CIFAR experiments, during training we feed the RotNet model all the four rotated copies of an image simultanouesly (in the same mini-batch).
ImageNet classiﬁcation task: We evaluate the task generalization of our self-supervised learned features by training on top of them object classiﬁers for the ImageNet classiﬁcation task (following the evaluation scheme of (Noroozi & Favaro, 2016)).
In Table 3 we report the classiﬁcation performance of our self-supervised features and we compare it with the other unsupervised approaches.
We observe that our approaches surpass all the other methods by a signiﬁcant margin .
For the feature maps generated by the Conv4 layer, our improvement is more than 4 percentage points and for the feature maps generated by the Conv5 layer, our improvement is even bigger, around 8 percentage points.
Furthermore, our approach narrows signiﬁcantly the gap between unsupervised features and supervised features.
Transfer learning evaluation on PASCAL VOC: We also explore the ability of our self-supervised features to transfer on the PASCAL VOC classiﬁcation and detection datasets.
The performance of our models on those tasks is reported in Table 4 along with the performance of other unsupervised methods.
As with the ImageNet classiﬁcation task, we outperform by signiﬁcant margin all the competing methods both in the classiﬁcation and the detection tasks thus signiﬁcantly narrowing the gap with the supervised features.
Notably, the PASCAL VOC 2007 object detection performance that our self-supervised features achieve is 54 .
4% mAP, which is only 2.4 points lower than the supervised case.
C ONCLUSIONS
In our work we propose a novel formulation for self-supervised feature learning that trains a ConvNet model to be able to recognize the image rotation that is applied to the images that it gets as input.
Despite the simplicity of our self-supervised task, we demonstrate that it successfully forces the ConvNet model that tries to solve it, to learn semantic features useful for perception vision tasks, such as object recognition and object detection.
We exhaustively evaluate our method in var- ious unsupervised and semi-supervised benchmarks and we achieve in all of them extremely good performance.
Even more, our self-supervised approach achieves state-of-the-art results on unsupervised feature learning for ImageNet classiﬁcation, PASCAL classiﬁcation, PASCAL detection, and
CIFAR-10 classiﬁcation, surpassing the prior approaches by signiﬁcant margin and thus drastically reducing the gap between unsupervised and supervised feature learning.