S ELF - LABELLING VIA SIMULTANEOUS CLUSTERING AND REPRESENTATION LEARNING
A BSTRACT
Combining clustering and representation learning is one of the most promising approaches for unsupervised learning of deep neural networks.
However, doing so naively leads to ill posed learning problems with degenerate solutions.
In this paper, we propose a novel and principled learning formulation that addresses these issues.
The method is obtained by maximizing the information between labels and input data indices.
We show that this criterion extends standard crossentropy minimization to an optimal transport problem, which we solve efﬁciently for millions of input images and thousands of labels using a fast variant of the
Sinkhorn-Knopp algorithm.
The resulting method is able to self-label visual data so as to train highly competitive image representations without manual labels.
Our method achieves state of the art representation learning performance for AlexNet and ResNet-50 on SVHN, CIFAR-10, CIFAR-100 and ImageNet and yields the ﬁrst self-supervised AlexNet that outperforms the supervised Pascal VOC detection baseline.
Code and models are available 1 .
I NTRODUCTION
Learning from unlabelled data can dramatically reduce the cost of deploying machine learning algorithms to new applications, thus amplifying their impact in the real world.
Self-supervision is an increasingly popular framework for learning without labels.
The idea is to deﬁne pretext learning tasks that can be constructed from raw data alone but that still result in neural networks that transfer well to useful applications.
Much of the research in self-supervision has focused on designing new pretext tasks.
However, given supervised data such as ImageNet (Deng et al., 2009), the standard classiﬁcation objective of minimizing the cross-entropy loss still results in better or at least as good pre-training than any of such methods (for a given amount of data and for a given model complexity).
This suggests that the task of classiﬁcation is sufﬁcient for pre-training networks, provided that suitable data labels are available.
In this paper, we thus focus on the problem of obtaining the labels automatically by designing a self-labelling algorithm.
Learning a deep neural network together while discovering the data labels can be viewed as simultaneous clustering and representation learning .
The latter can be approached by combining cross-entropy minimization with an off-the-shelf clustering algorithm such as K -means.
This is precisely the approach adopted by the recent DeepCluster method (Caron et al., 2018), which achieves excellent results in unsupervised representation learning.
However, combining representation learning, which is a discriminative task, with clustering is not at all trivial.
In particular, we show that the combination of cross-entropy minimization and K -means as adopted by DeepCluster cannot be described as the optimization of an overall learning objective; instead, there exist degenerate solutions that the algorithm avoids via particular implementation choices.
In order to address this technical shortcoming, in this paper, we contribute a new principled formulation for simultaneous clustering and representation learning.
The starting point is to minimize a single loss, the cross-entropy loss, for learning the deep network and for estimating the data labels.
This is often done in semi-supervised learning and multiple instance learning.
However, when applied naively to the unsupervised case, it immediately leads to a degenerate solution where all data points are mapped to the same cluster.
We solve this issue by adding the constraint that the labels must induce an equipartition of the data, which we show maximizes the information between data indices and labels.
We also show that the resulting label assignment problem is the same as optimal transport, and can therefore be solved in polynomial time by linear programming.
However, since we want to scale the algorithm to millions of data points and thousands of labels, standard transport solvers are inadequate.
Thus, we also propose to use a fast version of the Sinkhorn-Knopp algorithm for ﬁnding an approximate solution to the transport problem efﬁciently at scale, using fast matrix-vector algebra.
Compared to methods such as DeepCluster, the new formulation is more principled and allows to more easily demonstrate properties of the method such as convergence.
Most importantly, via extensive experimentation, we show that our new approach leads to signiﬁcantly superior results than DeepCluster, achieving the new state of the art for representation learning approaches.
In fact, the method’s performance surpasses others that use a single type of supervisory signal for self-supervision, and is on par or better than very recent contributions as well (Tian et al., 2019;
He et al., 2019; Misra & van der Maaten, 2019; Oord et al., 2018).
R ELATED W ORK
Our paper relates to two broad areas of research: (a) self-supervised representation learning, and (b) more speciﬁcally, training a deep neural network using pseudo-labels, i.e. the assignment of a label to each image.
We discuss closely related works for each.
Self-supervised learning: A wide variety of methods that do not require manual annotations have been proposed for the self-training of deep convolutional neural networks.
These methods use various cues and proxy tasks namely, in-painting (Pathak et al., 2016), patch context and jigsaw puzzles (Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2018; Mundhenk et al., 2017), clustering (Caron et al., 2018; Huang et al., 2019; Zhuang et al., 2019; Bautista et al., 2016), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et al., 2017; Donahue & Simonyan, 2019), geometry (Dosovitskiy et al., 2016), predicting transformations (Gidaris et al., 2018; Zhang et al., 2019) and counting (Noroozi et al., 2017).
Most recently, contrastive methods have shown great performance gains, (Oord et al., 2018; Hénaff et al., 2019; Tian et al., 2019;
He et al., 2019) by leveraging augmentation and adequate losses.
In (Feng et al., 2019), predicting rotation (Gidaris et al., 2018) is combined with instance retrieval (Wu et al., 2018) and multiple tasks are combined in (Doersch & Zisserman, 2017).
Pseudo-labels for images: In the self-supervised domain, we ﬁnd a spectrum of methods that either give each data point a unique label (Wu et al., 2018; Dosovitskiy et al., 2016) or train on a ﬂexible number of labels with K -means (Caron et al., 2018), with mutual information (Ji et al., 2018) or with noise (Bojanowski & Joulin, 2017).
In (Noroozi et al., 2018) a large network is trained with a pretext task and a smaller network is trained via knowledge transfer of the clustered data.
Finally, (Bach &
Harchaoui, 2008; Vo et al., 2019) use convex relaxations to regularized afﬁne-transformation invariant linear clustering, but can not scale to larger datasets.
Our contribution is a simple method that combines a novel pseudo-label extraction procedure from raw data alone and the training of a deep neural network using a standard cross-entropy loss.
M ETHOD
We will ﬁrst derive our self-labelling method, then interpret the method as optimizing labels and targets of a cross-entropy loss and ﬁnally analyze similarities and differences with other clusteringbased methods.
3. S ELF - LABELLING
Neural network pre-training is often achieved via a supervised data classiﬁcation task.
Formally, consider a deep neural network x = Φ( I ) mapping data
I (e.g. images) to feature vectors x ∈ R D .
The model is trained using a dataset (e.g. ImageNet) of N data points I 1 , . .
, I N with corresponding labels y 1 , . . .
, y N ∈ { 1 , . . .
, K } , drawn from a space of K possible labels.
The representation is followed by a classiﬁcation head h : R D
→ R K , usually consisting of a single linear layer, converting the feature vector into a vector of class scores.
The class scores are mapped to class probabilities via the softmax operator:
[Equation]
The model and head parameters are learned by minimizing the average cross-entropy loss
[Equation]
Training with objective (1) requires a labelled dataset.
When labels are unavailable, we require a self-labelling mechanism to assign the labels automatically.
In semi-supervised learning, self-labelling is often achieved by jointly optimizing (1) with respect to the model h ◦ Φ and the labels y 1 , . . .
, y N .
This can work if at least part of the labels are known, thus constraining the optimization.
However, in the fully unsupervised case, it leads to a degenerate solution: eq.
(1) is trivially minimized by assigning all data points to a single (arbitrary) label.
To address this issue, we ﬁrst rewrite eq.
(1) by encoding the labels as posterior distributions q ( y | x i ) :
[Equation]
If we set the posterior distributions q ( y | x i ) =
δ ( y − y i ) to be deterministic, the formulations in eqs.
(1) and (2) are equivalent, in the sense that E ( p, q ) =
E ( p | y 1 , . . .
, y N ) .
In this case, optimizing q is the same as reassigning the labels, which leads to the degeneracy.
To avoid this, we add the constraint that the label assignments must partition the data in equally-sized subsets.
Formally, the learning objective objective 2 is thus:
[Equation]
The constraints mean that each data point x i is assigned to exactly one label and that, overall, the N data points are split uniformly among the K classes.
The objective in eq.
(3) is combinatorial in q and thus may appear very difﬁcult to optimize.
However, this is an instance of the optimal transport problem , which can be solved relatively efﬁciently.
In order to see this more clearly, let P yi =
p
( y | x i )
N be the K × N matrix of joint probabilities estimated by the model.
Likewise, let Q yi =
q ( y | x i )
N be K × N matrix of assigned joint probabilities.
Using the notation of (Cuturi, 2013), we relax matrix Q to be an element of the transportation polytope
[Equation]
Here 1 are vectors of all ones of the appropriate dimensions, so that r and c are the marginal projections of matrix Q onto its rows and columns, respectively.
In our case, we require Q to be a matrix of conditional probability distributions that split the data uniformly, which is captured by:
[Equation]
With this notation, we can rewrite the objective function in eq.
(3), up to a constant shift, as
[Equation]
where (cid:104)·(cid:105) is the Frobenius dot-product between two matrices and log is applied element-wise.
Hence optimizing eq.
(3) with respect to the assignments Q is equivalent to solving the problem:
[Equation]
This is a linear program , and can thus be solved in polynomial time.
Furthermore, solving this problem always leads to an integral solution despite having relaxed Q to the continuous polytope
U ( r, c ) , guaranteeing the exact equivalence to the original problem.
In practice, however, the resulting linear program is large, involving millions of data points and thousands of classes.
Traditional algorithms to solve the transport problem scale badly to instances of this size.
We address this issue by adopting a fast version (Cuturi, 2013) of the Sinkhorn-Knopp algorithm .
This amounts to introducing a regularization term
[Equation]
where KL is the Kullback-Leibler divergence and rc (cid:62) can be interpreted as a K × N probability matrix.
The advantage of this regularization term is that the minimizer of eq.
(7) can be written as:
[Equation]
where exponentiation is meant element-wise and α and β are two vectors of scaling coefﬁcients chosen so that the resulting matrix Q is also a probability matrix (see (Cuturi, 2013) for a derivation).
The vectors α and β can be obtained, as shown below, via a simple matrix scaling iteration.
For very large λ , optimizing eq.
(7) is of course equivalent to optimizing eq.
(6), but even for moderate values of λ the two objectives tend to have approximately the same optimizer (Cuturi, 2013).
Choosing λ trades off convergence speed with closeness to the original transport problem.
In our case, using a ﬁxed λ is appropriate as we are ultimately interested in the ﬁnal clustering and representation learning results, rather than in solving the transport problem exactly.
Our ﬁnal algorithm’s core can be described as follows.
We learn a model h ◦ Φ and a label assignment matrix Q by solving the optimization problem eq.
(6) with respect to both Q , which is a probability matrix, and the model h ◦ Φ , which determines the predictions P yi = softmax y ( h ◦ Φ( x i )) .
We do so by alternating the following two steps:
Step 1: representation learning.
Given the current label assignments Q , the model is updated by minimizing eq.
(6) with respect to (the parameters of) h ◦ Φ .
This is the same as training the model using the common cross-entropy loss for classiﬁcation.
Step 2: self-labelling.
Given the current model h ◦
Φ , we compute the log probabilities P .
Then, we ﬁnd Q using eq.
(8) by iterating the updates (Cuturi, 2013)
[Equation]
Each update involves a single matrix-vector multiplication with complexity O ( NK ) , so it is relatively quick even for millions of data points and thousands of labels and so the cost of this method scales linearly with the number of images N .
In practice, convergence is reached within 2 minutes on
ImageNet when computed on a GPU.
Also, note that the parameters α and β can be retained between steps, thus allowing a warm start of Step 2.
3. I NTERPRETATION
As shown above, the formulation in eq.
(2) uses scaled versions of the probabilities.
We can interpret these by treating the data index i as a random variable with uniform distribution p ( i ) = 1 /N and by rewriting the posteriors p ( y | x i ) =
p ( y | i ) and q ( y | x i ) =
q ( y | i ) as conditional distributions with respect to the data index i instead of the feature vector x i .
With these changes, we can rewrite eq.
(5) as
[Equation]
which is the cross-entropy between the joint label-index distributions q ( y, i ) and p ( y, i ) .
The minimum of this quantity w.r.t.
q is obtained when p = q , in which case E ( q, q ) + log N reduces to the entropy H q ( y, i ) of the random variables y
and i .
Additionally, since we assumed that q ( i ) = 1 /N , the marginal entropy H q ( i ) = log N is constant and, due to the equipartition condition q ( y ) = 1 /K ,
H q ( y ) =
log K is also constant.
Subtracting these two constants from the entropy yields: min p E ( p, q )+log N = E ( q, q )+log N =
H q ( y, i ) =
H q ( y )
+ H q ( i ) −
I q ( y, i ) = const.
− I q ( y, i ) .
Thus we see that minimizing E ( p, q ) is the same as maximizing the mutual information between the label y and the data index i .
In our formulation, the maximization above is carried out under the equipartition constraint.
We can instead relax this constraint and directly maximize the information I ( y, i ) .
However, by rewriting information as the difference I
( y, i )
=
H ( y ) − H
( y | i ) , we see that the optimal solution is given by
H ( y | i ) = 0 , which states each data point i is associated to only one label deterministically, and by
H ( y ) =
ln K , which is another way of stating the equipartition condition.
In other words, our learning formulation can be interpreted as maximizing the information between data indices and labels while explicitly enforcing the equipartition condition, which is implied by maximizing the information in any case.
Compared to minimizing the entropy alone, maximizing information avoids degenerate solutions as the latter carry no mutual information between labels y and indices i .
Similar considerations can be found in (Ji et al., 2018).
3. R ELATION TO SIMULTANEOUS REPRESENTATION LEARNING AND CLUSTERING
In the discussion above, self-labelling amounts to assigning discrete labels to data and can thus be interpreted as clustering.
Most of the traditional clustering approaches are generative .
For example,
K -means takes a dataset x 1 , . . .
, x N of vectors and partitions it into K classes in order to minimize the reconstruction error
[Equation]
where y i ∈ { 1 , . .
, K } are the data-to-cluster assignments and µ y are means approximating the vectors in the corresponding clusters.
The K -means energy can thus be interpreted as the average data reconstruction error.
It is natural to ask whether a clustering method such as K -means, which is based on approximating the input data, could be combined with representation learning, which uses a discriminative objective.
In this setting, the feature vectors x = Φ( I ) are extracted by the neural network Φ from the input data
I .
Unfortunately, optimizing a loss such as eq.
(10) with respect to the clustering and representation parameters is meaningless: in fact, the obvious solution is to let the representation send all the data points to the same constant feature vector and setting all the means to coincide with it, in which case the K -means reconstruction error is zero (and thus minimal).
Nevertheless, DeepCluster (Caron et al., 2018) does successfully combine K -means with representation learning.
DeepCluster can be related to our approach as follows.
Step 1 of the algorithm, namely representation learning via cross-entropy minimization, is exactly the same.
Step 2, namely self-labelling, differs: where we solve an optimal transport problem to obtain the pseudo-labels, they do so by running K -means on the feature vectors extracted by the neural network.
DeepCluster does have an obvious degenerate solution: we can assign all data points to the same label and learn a constant representation, achieving simultaneously a minimum of the cross-entropy loss in Step 1 and of the K -means loss in Step 2.
The reason why DeepCluster avoids this pitfall is due to the particular interaction between the two steps.
First, during Step 2, the features x i are ﬁxed so K -means cannot pull them together.
Instead, the means spread to cover the features as they are, resulting in a balanced partitioning.
Second, during the classiﬁcation step, the cluster assignments
y i are ﬁxed, and optimizing the features x i with respect to the cross-entropy loss tends to separate them.
Lastly, the method in (Caron et al., 2018) also uses other heuristics such as sampling the training data inversely to their associated clusters’ size, leading to further regularization.
However, a downside of DeepCluster is that it does not have a single, well-deﬁned objective to optimize, which means that it is difﬁcult to characterize its convergence properties.
By contrast, in our formulation, both Step 1 and Step 2 optimize the same objective , with the advantage that convergence to a (local) optimum is guaranteed.
3. A UGMENTING SELF - LABELLING VIA DATA TRANSFORMATIONS
Methods such as DeepCluster extend the training data via augmentations.
In vision problems, this amounts to (heavily) distorting and cropping the input images at random.
Augmentations are applied so that the neural network is encouraged to learn a labelling function which is transformation invariant .
In practice, this is crucial to learn good clusters and representations, so we adopt it here.
This is achieved by setting P yi =
E t [log softmax y h ◦ Φ( t x i )] where the transformations t are sampled at random.
In practice, in Step 1 (representation learning), this is implemented via the application of the random transformations to data batches during optimization via SGD, which is corresponds to the usual data augmentation scheme for deep neural networks.
As noted in (YM.
et al., 2020), and as can be noted by an analysis of recent publications (Hénaff et al., 2019; Tian et al., 2019; Misra & van der
Maaten, 2019), augmentation is critical for good performance.
3. M ULTIPLE SIMULTANEOUS SELF - LABELINGS
Intuitively, the same data can often be clustered in many equally good ways.
For example, visual objects can be clustered by color, size, typology, viewpoint, and many other attributes.
Since our main objective is to use clustering to learn a good data representation Φ , we consider a multi-task setting in which the same representation is shared among several different clustering tasks, which can potentially capture different and complementary clustering axis.
In our formulation, this is easily achieved by considering multiple heads (Ji et al., 2018)
h 1 , . .
. , h T , one for each of T clustering tasks (which may also have a different number of labels).
Then, we optimize a sum of objective functions of the type eq.
(6), one for each task, while sharing the parameters of the feature extractor Φ among them.
E XPERIMENTS
In this section, we evaluate the quality of the representations learned by our Self Labelling (SeLa) technique.
We ﬁrst test variants of our method, including ablating its components, in order to ﬁnd an optimal conﬁguration.
Then, we compare our results to the state of the art in self-supervised representation learning, where we ﬁnd that our method is the best among clustering-based techniques and overall state-of-the-art or at least highly competitive in many benchmarks.
In the appendix, we also show qualitatively that the labels identiﬁed by our algorithm are usually meaningful and group visually similar concepts in the same clusters, often even capturing whole ImageNet classes.
4. S ETUP
Linear probes.
In order to quantify if a neural network has learned useful feature representations, we follow the standard approach of using linear probes (Zhang et al., 2017).
This amounts to solving a difﬁcult task, such as ImageNet classiﬁcation, by training a linear classiﬁer on top of a pre-trained feature representation, which is kept ﬁxed.
Linear classiﬁers heavily rely on the quality of the representation since their discriminative power is low.
We apply linear probes to all intermediate convolutional blocks of representative networks.
While linear probes are conceptually straightforward, there are several technical details that can affect the ﬁnal accuracy, so we follow the standard protocol further outlined in the Appendix.
Data.
For training data we consider ImageNet LSVRC-
(Deng et al., 2009) and other smaller scale datasets.
We also test our features by transferring them to MIT Places (Zhou et al., 2014).
All of these are standard benchmarks for evaluation in self-supervised learning.
Architectures.
Our base encoder architecture is AlexNet (Krizhevsky et al., 2012), since this is the most frequently used in other self-supervised learning works for the purpose of benchmarking.
We inject the probes right after the ReLU layer in each of the ﬁve blocks, and denote these entry points conv1 to conv5 .
Furthermore, since the conv1 and conv2 can be learned effectively from data augmentations alone (YM.
et al., 2020), we focus the analysis on the deeper layers conv2 to conv which are more sensitive to the quality of the learning algorithm.
In addition to AlexNet, we also test
ResNet-50 (He et al., 2016) models.
Further experimental details are given in the Appendix.
4. O PTIMAL CONFIGURATION AND ABLATIONS
In tables 1 and 5, we ﬁrst validate various modelling and conﬁguration choices.
Two key hyperparameters are the number of clusters K and the number of clustering heads T , which we denote in the experiments below with the shorthand “SeLa [ K × T ] ”.
We run SeLa by alternating steps 1 andas described in section 3.1.
Step 1 amounts to standard CE training, which we run for a ﬁxed number of epochs.
Step 2 can be interleaved at any point in the optimization; to amortize its cost, we run it
[Table]
[Table] [Table]
[Table] [Table]
at most once per epoch, and usually less, with a schedule described and validated below.
For these experiments, we train the representation and the linear probes on ImageNet.
Number of clusters K .
Table 2, compares different values for K : moving from 1k to 3k improves the results, but larger numbers decrease the quality slightly.
Ablation: number of heads T .
Table 3 shows that increasing the number of heads from T = 1 to
T = 10 yields a large performance gain: +2% for AlexNet and +10% for ResNet.
The latter more expressive model appears to beneﬁt more from a more diverse training signal.
Ablation: number of self-labelling iterations.
First, in table 1, we show that self-labelling (step 2) is essential for good performance, as opposed to only relying on the initial random label assignments and the data augmentations.
For this, we vary the number of times the self-labelling algorithm (step 2) is run during training (#opts), from zero to once per step 1 epoch.
We see that self-labelling is essential, with the best value around 80 (for 160 step 1 epochs in total).
Architectures.
Table 4 compares a smaller variant of AlexNet which uses (64, 192) ﬁlters in its ﬁrst two convolutional layers (Krizhevsky, 2014), to the standard variant with (96, 256) (Krizhevsky et al., 2012), all the way to a ResNet-50.
SeLa works well in all cases, for large models such as
ResNet but also smaller ones such as AlexNet, for which methods such as BigBiGAN (Donahue &
Simonyan, 2019) or CPC (Hénaff et al., 2019) are unsuitable.
4. L ABEL TRANSFER
An appealing property of SeLa is that the label it assigns to the images can be used to train another model from scratch, using standard supervised training.
For instance, table 5 shows that, given the labels assigned by applying SeLa to AlexNet, we can re-train AlexNet from scratch using a shorter 90-epochs schedule with achieving the same ﬁnal accuracy.
This shows that the quality of the learned representation depends only the ﬁnal label assignment, not on the fact that the representation is learned jointly with the labels.
More interestingly, we can transfer labels between different architectures.
For example, the labels obtained by applying SeLa [ 3 k × 1 ] and SeLa [ 3 k × 10 ] to ResNet-50 can be used to train a better AlexNet model than applying SeLa to the latter directly.
For this reason, we publish on our website the self-labels for the ImageNet dataset in addition to the code and trained models.
4. S MALL - SCALE D ATASETS
Here, we evaluate our method on relatively simple and small datasets, namely
CIFAR-10/100 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011).
For this, we follow the experimental and evaluation protocol from the current state of the art in self-supervised learning in these datasets, AND (Huang et al., 2019).
In table 6, we compare our method with the settings [ 128 × 10 ] for CIFAR-10, [ 512 × 10 ] for CIFAR-100 and [ 128 × 1 ] for SVHN to other published methods; details on the evaluation method are provided in the appendix.
We observe that our proposed method outperforms the best previous method by 5 . 8% for CIFAR-10, by 9 . 5% for
CIFAR-100 and by 0 .
8% for SVHN when training a linear classiﬁer on top of the frozen network.
The relatively minor gains on SVHN can be explained by the fact that the gap between the supervised
[Table] [Table]
[Table] [Table]
baseline and the self-supervised results is already very small ( < 3% ).
We also evaluate our method using weighted kNN using an embedding of size 128 .
We ﬁnd that the proposed method consistently outperforms the previous state of the art by around 2% across these datasets, even though AND is based on explicitly learning local neighbourhoods.
4. I MBALANCED D ATA E XPERIMENTS
In order to understand if our equipartition regularization is affected by the underlying class distribution of a dataset, we perform multiple ablation experiments on artiﬁcially imbalanced datasets in table 8.
We consider three training datasets based on CIFAR-10.
The ﬁrst is the original dataset withimages for each class ( full in table 8).
Second, we remove 50% of the images of one class (truck) while the rest remains untouched ( light imbalance ) and ﬁnally we remove 10% of one class, 20% of the second class and so on ( heavy imbalance ).
On each of the three datasets we compare the performance of our method “ours (SK)” with a baseline that replaces our Sinkhorn-Knopp optimization with
K -means clustering “ours ( K -means)”.
We also compare the performance to training a network under full supervision.
The evaluation follows Huang et al. and is based on linear probing and kNN classiﬁcation — both on CIFAR-10 and, to understand feature generalization, on CIFAR-100.
[Table]
To our surprise, we ﬁnd that our method generalizes better to CIFAR-100 than the supervised baseline during kNN evaluation, potentially due to overﬁtting when training with labels.
We also ﬁnd that using SK optmization for obtaining pseudo-labels is always better than K -means on all metrics and datasets.
When comparing the imbalance settings, we ﬁnd that under the light imbalance scenario, the methods’ performances are ranked the same and no method is strongly affected by the imbalance.
Under the heavy imbalance scenario, all methods drop in performance.
However, compared to full data and light imbalance, the gap between supervised and self-supervised even decreases slightly for both K -means and our method, indicating stronger robustness of self-supervised methods compared to a supervised one.
In conclusion, our proposed method does not rely on the data to contain the same number of images for every class and outperforms a K -means baseline even in very strong imbalance settings.
This conﬁrms the intuition that the equipartioning constraint acts as a regularizer and does not exploit the class distribution of the dataset.
4. L ARGE S CALE B ENCHMARKS
To compare to the state of the art and concurrent work, we evaluate several architectures using linear probes on public benchmark datasets.
AlexNet.
The main benchmark for feature learning methods is linear probing of an AlexNet trained on ImageNet.
In table 9 we compare the performance across layers also on the Places dataset.
We ﬁnd that across both datasets our method outperforms DeepCluster and local Aggregation at every layer.
From our ablation studies in tables 1-5 we also note that even our single head variant [3k × 1] outperforms both methods.
Given that our method provides labels for a dataset that can be used for retraining a network quickly, we ﬁnd that we can improve upon this initial performance.
And by adopting a hybrid approach, similar to (Feng et al., 2019), of training an AlexNet with 10 heads and one additional head for computing the RotNet loss, we ﬁnd further improvement.
This result (SeLa
[Table]
[ 3 k × 10 ] − + Rot) achieves state of the art in unsupervised representation learning for AlexNet, with a gap of 1 . 3% to the previous best performance on ImageNet and surpasses the ImageNet supervised baseline transferred to Places by 1 . 7% .
ResNet.
Training better models than AlexNets is not yet standardized in the feature learning community.
In Table 10 we compare a ResNet-50 trained with our method to other works.
With topaccuracy of 61 . 5 , we outperform than all other methods including Local Aggregation, CPCv1 and
MoCo that use the same level of data augmentation.
We even outperform larger architectures such as
BigBiGAN’s RevNet-50x4 and reach close to the performance of models using AutoAugment-style transformations.
4. F INE - TUNING : C LASSIFICATION , OBJECT DETECTION AND SEMANTIC SEGMENTATION
Finally, since pre-training is usually aimed at improving down-stream tasks, we evaluate the quality of the learned features by ﬁne-tuning the model for three distinct tasks on the PASCAL VOC benchmark.
In Table 7 we compare results with regard to multi-label classiﬁcation, object detection and semantic segmentation on PASCAL VOC (Everingham et al., 2010).
As in the linear probe experiments, we ﬁnd our method better than the current state of the art in detection and classiﬁcation with both ﬁne-tuning only the last fully connected layers and when ﬁne-tuning the whole network (“ all ”.
Notably, our ﬁne-tuned AlexNet outperforms its supervised
ImageNet baseline on the VOC detection task.
Also for segmentation the method is very close ( 0 . 2% ) to the best performing method.
This shows that our trained network does not only learn useful feature representations but is also able to perform well when ﬁne-tuned on actual down-stream tasks.
C ONCLUSION
We present a self-supervised feature learning method that is based on clustering.
In contrast to other methods, ours optimizes the same objective during feature learning and during clustering.
This becomes possible through a weak assumption that the number of samples should be equal across clusters.
This constraint is explicitly encoded in the label assignment step and can be solved for efﬁciently using a modiﬁed Sinkhorn-Knopp algorithm.
Our method outperforms all other feature learning approaches and achieves SOTA on SVHN, CIFAR-10/100 and ImageNet for AlexNet and
ResNet-50.
By virtue of the method, the resulting self-labels can be used to quickly learn features for new architectures using simple cross-entropy training.
A CKNOWLEDGMENTS
Yuki Asano gratefully acknowledges support from the EPSRC Centre for Doctoral Training in
Autonomous Intelligent Machines & Systems (EP/L015897/1).
We are also grateful to ERC IDIU638009, AWS Machine Learning Research Awards (MLRA) and the use of the University of Oxford
Advanced Research Computing (ARC)
A A PPENDIX A. I MPLEMENTATION D ETAILS
Learning Details Unless otherwise noted, we train all our self-supervised models with SGD and intial learning rate 0.05 for 400 epochs with two learning rate drops where we divide the rate by ten at 150 and 300 and 350 epochs.
We spread our pseudo-label optimizations throughout the whole training process in a logarithmic distribution.
We optimize the label assignment at t i = (cid:16) iM − 1 (cid:17) 2 , i ∈ { 1 , . . .
, M } , where M is the user-deﬁned number of optimizations and t i is expressed as a fraction of total training epochs.
For the Sinkhorn-Knopp optimization we set λ
=as in (Cuturi, 2013).
We use standard data augmentations during training that consist of randomly resized crops, horizontal ﬂipping and adding noise, as in (Wu et al., 2018).
Quantitative Evaluation – Technical Details.
Unfortunately, prior work has used several slightly different setups, so that comparing results between different publications must be done with caution.
In our ImageNet implementation, we follow the original proposal (Zhang et al., 2017) in pooling each representation to a vector with 9600 , 9216 , 9600 , 9600 , 9216 dimensions for conv1-5 using adaptive max-pooling, and absorb the batch normalization weights into the preceding convolutions.
For evaluation on ImageNet we follow RotNet to train linear probes: images are resized such that the shorter edge has a length of 256 pixels, random crops of 224 × 224 are computed and ﬂipped horizontally with 50% probability.
Learning lasts for 36 epochs and the learning rate schedule starts from 0 .
and is divided by ﬁve at epochs 5 , 15 and 25 .
The top-1 accuracy of the linear classiﬁer is then measured on the ImageNet validation subset by optionally extracting 10 crops for each validation image (four at the corners and one at the center along with their horizontal ﬂips) and averaging the prediction scores before the accuracy is computed or just taking the a centred crop.
For CIFAR-10/100 and SVHN we train AlexNet architectures on the resized images with batchsize 128 , learning rate 0 .
03 and also the same image augmentations (random resized crops, color jitter and random grayscale) as is used in prior work (Huang et al., 2019).
We use the same linear probing protocol as for our ImageNet experiments but without using 10 crops.
For the weighted kNN experiments we use k = 50 , σ = 0 .
and we use an embedding of size 128 as done in previous works.
In Table 9, when retraining an AlexNet using ResNet generated labels, we can apply heavier augmentation strategies as the labels are kept constant.
Hence for the experiments denoted by "+ more aug."
, in addition to the usual augmentations, we further randomly apply one of equalize, autoconstrast and sharpening.
We ﬁnd that this raises the performance for ImageNet but lowers the performance on Places by a small amount, hence illuminating the need to always also report performance on both datasets.
A. F URTHER DETAILS
NMI over time In ﬁg. A.
we ﬁnd that most learning takes place in the early epochs, and we reach a ﬁnal NMI value of around 66%.
Similarly, we ﬁnd that due to the updating of the pseudo-labels at regular intervals and our data augmentation, the pseudo-label accuracies keep continuously rising without overﬁtting to these labels.
Clustering metrics In table A.1, we report standard clustering metrics (see (Vinh et al., 2010) for detailed deﬁnitions) of our trained models with regards to the ImageNet validation set groundtruth labels.
These metrics include chance-corrected metrics which are the adjusted normalized mutual information (NMI) and the adjusted Rand-Index, as well as the default NMI, also reported in
DeepCluster (Caron et al., 2018).
Conv1 ﬁlters
In ﬁg. A.3 we show the ﬁrst convolutional ﬁlters of two of our trained models.
We can ﬁnd the typical Gabor-like edge detectors as well as color blops and dot-detectors.
Entropy over time In ﬁg.
A.4, we show how the distribution of entropy with regards to the true
ImageNet labels changes with training time.
We ﬁnd that while at ﬁrst, all 3000 pseudo-labels contain random real ImageNet labels, yielding high entropy of around 6 ≈ ln(400) =
ln(1 .
Towards the end of training we arrive at a broad spectrum of entropies with some as low as
[Figure]
[Table]
≈ ln(1 .
07) (see Fig.
A.5 and A.6 for low entropy label visualizations) and the mean around 4 .
2 ≈ ln(66)
(see Fig.
A.7 and A.8 for randomly chosen labels’ visualizations).
[Figure]
during
[Figure]
A. C OMPLETE TABLES
In the following, we report the unabridged tables with all related work.
[Table] [Table]
[Table] [Table]
[Figure]
[Figure] [Figure]