E XPLORING N ON -C ONTRASTIVE R EPRESENTATION L EARNING FOR D EEP C LUSTERING
Anonymous authors
Paper under double-blind review
A BSTRACT
Existing deep clustering methods rely on contrastive learning for representation learning, which requires negative examples to form an embedding space where all instances are well-separated.
However, the negative examples inevitably give rise to the class collision issue, compromising the representation learning for clustering.
In this paper, we explore non-contrastive representation learning for deep clustering, termed NCC, which is based on BYOL, a representative method without negative examples.
First, we propose to align one augmented view of instance with the neighbors of another view in the embedding space, called positive sampling strategy, which avoids the class collision issue caused by the negative examples and hence improves the within-cluster compactness.
Second, we propose to encourage alignment between two augmented views of one prototype and uniformity among all prototypes, named prototypical contrastive loss or ProtoCL, which can maximize the inter-cluster distance.
Moreover, we formulate NCC in an Expectation-Maximization (EM) framework, in which E-step utilizes spherical k-means to estimate the pseudo-labels of instances and distribution of prototypes from a target network and M-step leverages the proposed losses to optimize an online network.
As a result, NCC forms an embedding space where all clusters are well-separated and within-cluster examples are compact.
Experimental results on several clustering benchmark datasets including ImageNet-1K demonstrate that
NCC outperforms the state-of-the-art methods by a signiﬁcant margin.
I NTRODUCTION
Deep clustering is gaining considerable attention as it can learn representation of images and perform clustering in an end-to-end fashion.
Remarkably, contrastive learning-based methods (Wang et al., 2021; Van Gansbeke et al., 2020; Li et al., 2021a;b; Tao et al., 2021;
Tsai et al., 2021; Niu et al., 2021) have become the main thrust to advance the representation of images on several complex benchmark datasets, signiﬁcantly contributing to the clustering performance.
In addition, some contrastive learning methods such as MoCo (He et al., 2020) and SimCLR (Chen et al., 2020) usually require specially designed losses (Wang et al., 2021;
Li et al., 2021a;b; Tao et al., 2021;
Tsai et al., 2021) or an extra pre-training stage for more discriminative representations (Van Gansbeke et al., 2020; Niu et al., 2021).
Although achieving promising clustering results, contrastive learning requires a large number of negative examples to achieve the instance-wise discrimination in an embedding space where all instances are well-separated.
The constructed negative pairs usually require a large batch size (Chen et al., 2020), memory queue (He et al., 2020), or memory bank (Wu et al., 2018), which not only bring extra computational cost but also give rise to class collision issue (Saunshi et al., 2019).
Here, the class collision issue refers to that different instances from the same semantic class are regarded as negative pairs, hurting the representation learning for clustering.
A question naturally arises: are negative examples necessary for deep clustering?
Another kind of self-supervised learning is the non-contrastive methods such as BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), which use the representations of one augmented view to predict another view.
Their success demonstrates that negative examples are not the key to avoiding representation collapse.
However, to the best of our knowledge, almost all recent successful literature of deep clustering is built upon contrastive learning-based methods such as MoCo (He et al.,
2020) and SimCLR (Chen et al., 2020).
There is a general consensus that the negative examples are helpful to stabilize the training of representation learning for deep clustering.
As discussed in (Wang & Isola, 2020), the typical contrastive loss can be identiﬁed into two properties: 1) alignment term to improve the closeness of positive pairs; and 2) uniformity term to encourage instances to be uniformly distributed on a unit hypersphere.
In contrast, non-contrastive methods such as BYOL only optimize the alignment term, leading to unstable training and suffering from the representation collapse—which may be worsen when adding extra losses.
To tackle the class collision issue, we explore the non-contrastive representation learning for deep clustering, termed non-contrastive clustering or NCC, which is based on BYOL, a representative method without negative examples.
First, instead of negative sampling that is a double-edged sword, i .
e ., causing class collision issue but improving the training stability, we propose to align one augmented view of the instance with the neighbors of another view in the embedding space, called positive sampling strategy, which can avoid the class collision issue and hence improve the withincluster compactness.
Second, as for the clustering task, the different clusters are truly negative pairs for contrastive loss.
To this end, we propose to encourage the alignment between two augmented views of prototypes and the uniformity among all prototypes, named prototypical contrastive loss or ProtoCL, which can maximize the inter-cluster distance.
Moreover, we formulate our method into an EM framework, in which we iteratively perform E-step as estimating the pseudo-labels of instances and distribution of prototypes via spherical k-means based on the target network and Mstep as optimizing the online network via the proposed losses.
As a result, NCC is able to form an embedding space where all clusters are well-separated and within-cluster examples are compact.
The contributions of this paper are summarized as follows:
• We explore the non-contrastive representation learning for deep clustering by proposing noncontrastive clustering or NCC, which is based on the Bootstrap Your Own Latent (BYOL), a representative method without negative examples.
• We propose a positive sampling strategy to augment instance alignment by taking into account neighboring positive examples in the embedding space, which can avoid the class collision issue and hence improve the within-cluster compactness.
• We propose a novel prototypical contrastive loss or ProtoCL, which can align one augmented view of prototypes with another view and encourage the uniformity among all prototypes on a unit hypersphere, hence maximizing the inter-cluster distance.
• We formulate our method into an EM framework, in which we can iteratively estimate the pseudolabels and distribution of prototypes via spherical k-means based on the target network and optimize the online network via the proposed losses.
• Extensively experimental results on several benchmark datasets as well as ImageNet-1K demonstrate that NCC outperforms the existing state-of-the-art methods by a signiﬁcant margin.
R ELATED W ORK
Deep clustering can be signiﬁcantly advanced by discriminative representations.
Examples of traditional deep clustering methods include: Xie et al.
; Yang et al.
(2017) use autoencoders to simultaneously perform representation learning and clustering; Chang et al.
; Haeusser et al.
; Wu et al.
; Ji et al.
(2019) learn pair-wise relationships between original and augmented instances.
However, they often suffer from inferior performance on some complex datasets such as CIFAR-20.
Inspired by the success of contrastive learning, recent studies turn to exploit the discriminative representations learned from contrastive learning to assist the downstream clustering tasks (Van Gansbeke et al., 2020; Niu et al., 2021) or simultaneously optimize representation learning and clustering (Tao et al., 2021;
Tsai et al., 2021; Li et al., 2021a;
Shen et al., 2021).
SCAN (Van Gansbeke et al., 2020) uses the model pre-trained by SimCLR to yield the conﬁdent pseudo-labels.
IDFD
(Tao et al., 2021) proposes to perform both instance discrimination and feature decorrelation.
GCC (Zhong et al., 2021) and WCL (Zheng et al., 2021) build a graph to label the neighbor samples as pseudo-positive examples, however, they still suffer from the class collision issue due to the contrastive loss involved and these pseudo-positive examples that may not be truly positive.
All of them are built upon the contrastive learning framework, which means that they require a large number of negative examples for training stability, inevitably giving rise to class collision issue.
Different from prior work, this paper explores the non-contrastive self-supervised methods, i .
e ., BYOL, to achieve both representation learning and clustering.
We note that Regatti et al.
; Lee et al.
(2020) have tried to build the clustering framework based on BYOL, however, their methods do not consider improving within-cluster compactness and maximizing inter-cluster distance like ours.
Therefore, to the best of our knowledge, this is the ﬁrst successful attempt that introduces the non-contrastive representation learning into deep clustering that yields a substantial performance improvement over previous state-of-the-art methods.
In Appendix A, we present related work on self-supervised learning and difference from existing methods including CC (Li et al., 2021b), GCC (Zhong et al., 2021), WCL (Zheng et al., 2021), and PCL (Li et al., 2021a).
P RELIMINARY
The most successful self-supervised learning methods in recent years can be roughly divided into contrastive (Chen et al., 2020;
He et al., 2020) and non-contrastive (Grill et al., 2020; Chen & He, 2021).
Here, we brieﬂy summarize their formulas and discuss their difference.
Contrastive learning.
Contrastive learning methods perform instance-wise discrimination (Wu et al., 2018) using the InfoNCE loss (Oord et al., 2018).
Formally, assume that we have one instance x , its augmented version x + by using random data augmentation, and a set of M negative examples drawn from the dataset , { x − 1 , x − 2 , . . .
, x − M } .
The contrastive learning aims to learn an embedding function f that maps x onto a unit hypersphere, in which the InfoNCE loss can be deﬁned as:
[Equation]
where the ﬁrst and second terms in Eq.
(2) refer to as instance alignment and instance uniformity, respectively.
Here, we assume that the output of f ( · ) is (cid:96) 2 normalized.
That is, the representation is on a unit hypersphere.
The temperature τ controls the concentration level of representations; please refer to (Wang & Liu, 2021) for detailed behaviors of τ in the contrastive loss.
Intuitively, the InfoNCE loss aims to pull together the positive pair ( x , x + ) from two different data augmentations of the same instance, and push x away from M negative examples of other instances.
As discussed in (Wang & Isola, 2020), when M → ∞ , the InfoNCE loss in Eq.
(1) can be approximately decoupled into two terms: alignment and uniformity , as shown in Eq.
Despite the alignment term closes the positive pair, the key to avoiding representation collapse is the uniformity term, which makes the negative examples uniformly distributed on the hypersphere.
Although beneﬁcial, the negative examples inevitably lead to the class collision issue, hurting the representation learning for clustering.
Non-contrastive learning.
Non-contrastive learning-based methods have shown more promising results than contrastive learning for representation learning and downstream tasks (Ericsson et al., 2021).
Non-contrastive methods only optimize the alignment term in Eq.
(2) to match the representations between two augmented views.
Without negative examples, they leverage an online and a target network for two views, and use a predictor network to bridge the gap between these two views.
They also stop the gradient of the target network to avoid the representation collapse.
In particular, if τ = 0 . 5 , the loss used in (Grill et al., 2020; Chen & He, 2021) can be written as:
[Equation]
where g the predictor; f and f (cid:48) are the online and target networks, respectively; the outputs of g ( f ( · )) and f (cid:48) ( · ) are (cid:96) 2 -normalized.
However, as mentioned in (Fetterman & Albrecht, 2020), the non-contrastive learning methods often suffer from unstable training and highly rely on the batch-statistics and hyper-parameter tuning to avoid representation collapse.
Even though Grill et al.
Richemond et al.
(2020) have proposed to use some tricks such as SyncBN (Ioffe & Szegedy, 2015) and weight normalization (Qiao et al., 2019) to alleviate this issue, the additional computation cost is signiﬁcant.
Without negative examples, the collapse issue could be worsen when adding additional clustering losses for clustering task; see Fig.
A1 for the analysis of applying
PCL (Li et al., 2021a) to the BYOL.
In a nutshell, most of existing successful deep clustering methods are based on contrastive learning for representation learning—giving rise to class collision issue—while the non-contrastive learning, due to unstable training with additional losses, is not yet ready for deep clustering.
To that end, we explore the non-contrastive learning, i .
e .
BYOL, for deep clustering with positive sampling strategy and prototypical contrastive loss to avoid the class collision issue, improve the within-cluster compactness, and maximize the inter-class distance.
N ON -C ONTRASTIVE C LUSTERING
[Figure]
Fig.
1 presents the overall framework of the proposed NCC.
Based on BYOL, NCC is comprised of three networks: an online, a target, and a predictor.
In Sec. 4.1, we propose a positive sampling strategy to augment instance alignment to improve the within-cluster compactness.
In Sec. 4.2, a prototypical contrastive loss is introduced to maximize the inter-cluster distance using the pseudolabels from k-means clustering, which can encourage uniform representations.
Finally, we formulate
NCC into an EM framework to facilitate the understanding of training procedure in Sec. 4.3.
[Figure]
The negative examples are essential for contrastive learning-based deep clustering to stabilize the training of representation learning, at the cost of inevitable class collision issue (Saunshi et al., 2019), as shown in Fig.
2(a).
This issue can hurt the representation learning for clustering as the instances from the same class/cluster—should be close to each other—could be treated as negative pairs and are pushed away during training, discouraging the within-cluster compactness.
To address the class collision issue, we resort to non-contrastive learning-based methods for representation learning, which no longer need negative examples.
Although we cannot optimize the uniformity term like contrastive loss, our idea is to optimize the opposite of the uniformity instead.
That is, we aim to encourage the neighboring examples around one augmented view, sampled in the embedding space, to be aligned with another view, as shown in Fig. 2(b).
Our motivation is that although we cannot guarantee the negative examples in contrastive loss are from different classes, we can certainly assume that the neighboring samples around one view in the embedding space are positive with respect to another view and belong to the same class.
Therefore, we propose a positive sampling strategy to augment the instance alignment in Eq.
(3) by taking into account the neighboring samples towards within-cluster compactness.
Speciﬁcally, we model the representation of one augmented view of an instance as a Gaussian distribution in the embedding space, which can be formulated as follows:
[Equation]
where I represents the identity matrix and σ is a positive hyperparameter controlling how many samples around one view can be treated as positive pairs with another view.
However, the sampled examples from Eq.
(4) cannot allow the error to be backpropagated through the network to update the network parameters.
We employ the reparametrization trick (Kingma & Welling, 2014) to achieve the backpropagation.
As a result, the positive sampling strategy can be implemented as follows:
[Equation]
Therefore, we can augment the instance alignment in Eq.
(3) by taking into account the neighboring samples to encourage the within-cluster compactness.
With only sampling one example from the
Gaussian distribution, the augmented instance alignment term can be formally deﬁned as:
[Equation]
The beneﬁts of the proposed positive sampling are summarized as follows.
Improved within-cluster compactness.
The conventional instance alignment in Eq.
(3) only encourages the representation of one augmented view to be close to another view.
In the context of clustering, such compactness is instance-wise and neutral for the clustering.
Put differently, all instances are treated as cluster centers and the semantic class information cannot be captured at only instance level.
In contrast, our augmented instance alignment in Eq.
(6) encourages neighboring examples around one augmented view—either different augmented examples of the same instance or same/different augmented examples of different instances within the same cluster—to be positive pairs with another view.
This is helpful to improve within-cluster compactness.
Avoidable class collision issue.
As we mentioned before, class collision issue induced by the negative examples indicates that we cannot guarantee that the negative examples are from different clusters.
However, our positive sampling strategy can guarantee that the positive examples around one instance, sampled in the embedding space, are from the same cluster as the instance, getting rid of class collision issue.
The difference from recent work (Zhong et al., 2021; Zheng et al., 2021) is discussed in Appendix A.
We note that our positive sampling strategy does not consider uniformity, which is solved by the proposed prototypical contrastive loss in Sec. 4.2.
4. P ROTOTYPICAL C ONTRASTIVE L OSS
A good clustering is supposed to have distinct semantic prototypes/clusters.
Assume that the dataset has K clusters, where K is a predeﬁned hyperparameter, it naturally constructs a contrastive loss for these K prototypes as for one prototype, the remaining K − 1 prototypes are deﬁnitely negative examples.
Therefore, we propose a prototypical contrastive loss or ProtoCL, which encourages the prototypical alignment between two augmented views and the prototypical uniformity, hence maximizing the inter-cluster distance.
Speciﬁcally, assume we have K prototypes from the online network, { µ 1 , µ 2 , . .
. , µ K } , and another K prototypes from the target network, { µ (cid:48)
[Equation]
, µ
[Equation]
. , µ (cid:48) K } , our proposed ProtoCL, illustrated in Fig. 2(c), is given as follows:
[Equation]
where the ﬁrst and second terms in Eq.
(8) refer to as prototypical alignment and prototypical uniformity, respectively.
Here, the cluster centers µ k and µ (cid:48)
k are computed within a mini-batch B as follows: µ k =
[Equation]
and
[Equation]
x
[Equation]
x
[Equation]
2 , and p ( k | x ) is the cluster assignment posterior probability.
When K > |B| , it is obvious that the mini-batch cannot cover all clusters.
To this end, we zero out the losses and logits of empty clusters for each iteration; see the pseudocode in Appendix D for more details.
Clearly, our ProtoCL is quite similar to conventional contrastive loss in Eq.
(1) but for prototypes with non-contrastive representation learning framework.
The prototypical alignment is to align the prototypes derived from the online network with the ones from the target network, which can stabilize the update of the prototypes.
The prototypical uniformity is to encourage the prototypes to be uniformly distributed on a unit hypersphere, which can maximize the inter-cluster distance.
The difference from ProtoNCE in (Li et al., 2021a) is discussed in Appendix A.
4. EM F RAMEWORK
We formulate our NCC into an EM framework to facilitate the understanding of the training procedure, detailed in Fig.
1 and derived in Appendix B.
E-step.
This step aims to estimate p ( k | x ) .
We perform spherical k-means algorithm on the features extracted from the target network since the target network performs more stable and yields more consistent clusters, similar to BYOL and MoCo.
Although we need an additional k-means clustering to obtain the cluster pseudo-labels p ( k | x ) for every r epochs, we found that even with a larger r , rather than every epoch r = 1 , our method can still produce consistent performance improvement over the baseline methods.
Therefore, our method will not introduce much computation cost and is robust to the cluster pseudo-labels; see detailed results in Fig.
A2.
The analysis of computational cost is discussed in Appendix C. Finally, with p ( k | x ) , we build the cluster centers within a mini-batch without additional memory like queue (He et al., 2020) or bank (Wu et al., 2018).
M-step.
Combining the augmented instance alignment loss in Eq.
(6) and the proposed ProtoCL in Eq.
(7) yields our objective function for M-step as follows:
[Equation]
where λ pcl controls the balance between two loss components.
Therefore, there are only two additional hyper-parameters compared to original BYOL, including: σ in L aug − ins and the loss weight λ pcl ; see detailed results of these two hyper-parameters in Figs.
A3 and A4.
E XPERIMENTS
[Table]
We conducted experiments on six benchmark datasets, including CIFAR10 (Krizhevsky et al., 2009), CIFAR20 (Krizhevsky et al., 2009),
STL10 (Coates et al., 2011), ImageNet10 (Chang et al., 2017), ImageNetDogs
(Chang et al., 2017), and ImageNet1K (Deng et al., 2009), which are summarized in Table 1.
We note that CIFAR-20 containssuperclasses of CIFAR-100.
This paper follows the experimental settings widely used in deep clustering work (Chang et al., 2017; Wu et al., 2019; Ji et al., 2019;
Tsai et al., 2021; Tao et al., 2021), including the image size, backbone and train-test split.
We employ three common metrics to evaluate the clustering performance, including Normalized Mutual Information (NMI), Cluster
Accuracy (ACC), and Adjusted Rand Index (ARI) for the ﬁrst ﬁve datasets.
Following (Li et al., 2021a), we report Adjusted Mutual Information (AMI) to evaluate the clustering performance for
ImageNet-1K. The results are presented in percentage (%) and the higher the better clustering performance.
For fair comparisons, we use ResNet-34 (He et al., 2016) as the backbone to report the results in Table 2.
Unless noted otherwise, we use ResNet-18 for the rest of experiments.
We run each experiment three times and report the mean and standard deviation as the ﬁnal results.
We provided detailed training settings in Appendix C. We also provide the pseudocode of NCC for better understanding in Appendix D.
The source code will be publicly available upon acceptance.
5. M AIN R ESULTS
[Table]
Quantitative results We compared NCC with previous state-of-the-art clustering methods in Table 2.
NCC achieves signiﬁcant performance improvement on all ﬁve benchmark datasets, demonstrating the superiority of NCC for deep clustering to capture the semantic class information.
Interestingly, directly using the representations learned by BYOL for k-means clustering outperforms previous work including the contrastive-based ones (Li et al., 2021b;
Tsai et al., 2021; Tao et al., 2021), which suggests a great potential for non-contrastive representation learning for deep clustering without suffering class collision issue.
On the ImageNet-10, our NCC achieves competitive performance as compared to IDFD
(Tao et al., 2021) since this dataset is relatively small with only 13K images, which cannot arise discriminative differences for current state-of-the-art methods.
On the ImageNet-Dogs, a ﬁne-grained dataset containing different species of dogs from the ImageNet dataset, there are almost 20% improvements over previous SOTA work.
The contrastive-based methods cannot handle this kind of dataset due to severe class collision issue that pushes away the instances from the same class.
Meanwhile, IDFD can deal with this problem to some degree thanks to the feature decorrelation along with the instance discrimination.
Without the need of negative examples, BYOL can achieve signiﬁcant improvement, although its performance is unstable.
Our NCC, built upon BYOL with a positive sampling strategy and prototypical contrastive loss, has shown its signiﬁcant and stable performance against vanilla
BYOL and contrastive-based methods.
[Table] [Table]
Table 3 further presents the results between our NCC and baseline methods including DeepCluster (Caron et al., 2018), MoCo (He et al., 2020), and PCL (Li et al., 2021a) on ImageNet-1K dataset, showing that NCC achieves signiﬁcantly higher AMI score.
Although we employed the fair conditions, some work has trained the network with different split (Van Gansbeke et al., 2020) for
CIFAR-10 and CIFAR-20, or large image size (Li et al., 2021b) for
ImageNet-10 and ImageNet-Dogs.
For the sake of fair comparisons with different image sizes and splits, we report these comparison results in Table A1.
In addition, we also reported the clustering results on ImageNet subsets like SCAN (Van Gansbeke et al., 2020) in Table A2 and Tiny-ImageNet (Le & Yang, 2015) in Table A3.
We also conducted additional experiments in Table A4 to demonstrate the ability of NCC handling the long-tailed datasets.
Qualitative results.
Fig.
3 visualizes the learned representations by t-SNE (Van der Maaten &
Hinton, 2008) for four different training epochs throughout the training process.
At the beginning, the random-initialized model cannot distinguish the instances from different semantic classes, where all instances are mixed together.
As the training process goes, NCC gradually attracts the instances from the same cluster while pushing the clusters away from each other.
Obviously, at the end of the training, NCC produces clear boundary between clusters and within-cluster compactness.
Visualization for the outlier points produced by the model at 1000-th epoch is shown in Fig.
A5.
[Figure]
5. A BLATION S TUDY
Here, we perform detailed ablation studies with both quantitative and qualitative comparisons to provide more insights into why NCC performs so well for deep clustering.
[Table]
Quantitative ablation study.
We report the quantitative results of ablation studies in Table 4.
BYOL outperforms MoCo v2 and SimSiam by a large margin on both two datasets.
The difference between BYOL and MoCo v2 is that MoCo v2 uses a memory queue to store the consistent negative examples while BYOL directly aligns two augmented views with a predictor network.
Different from the BYOL that employs a momentum-updated network as the target network to yield the positive representations, SimSiam shares the weights of the target and online networks.
Therefore, BYOL outperforms MoCo v2 by dealing with the class collision issue and SimSiam by the momentum-updated target network.
Compared to vanilla BYOL, simply using the positive sampling strategy can stable and further improve the performance, especially when the number of semantic classes increases for CIFAR-20.
Although ProtoCL improves the baseline results by a large margin, positive sampling can further boost the clustering performance.
This is because ProtoCL only considers inter-cluster distance, and cannot beneﬁt within-cluster compactness.
Therefore, the combination of the positive sampling and ProtoCL achieves the best clustering results, where positive sampling strategy can improve the within-cluster compactness and ProtoCL encourages the prototypical alignment between two augmented views and maximizes the inter-cluster distance.
To further explore the effect of the proposed ProtoCL, we split this loss function into prototypical alignment and uniformity as shown in Eq.
It is clear that the performance gain from the alignment term is marginal while the gain from the uniformity term is signiﬁcant.
Note that for only alignment term, we compute the loss after predictor network instead of feature extractor, otherwise representation collapse will turn out.
This indicates that prototypical uniformity is more important than prototypical alignment since BYOL has already performed instance alignment at an augmented instance level.
However, we note that the prototypical alignment term is essential to stabilize the training process, as demonstrated in the results for CIFAR-20 with more clusters.
To demonstrate that BYOL is not robust to additional losses for deep clustering tasks, we integrate
CC (Li et al., 2021b) and PCL (Li et al., 2021a) into BYOL; the results in Table 4 show that both of them compromise clustering performance and become unstable.
This is because CC contrasts the cluster probability not helpful for representation learning, and the representations of PCL would collapse without negative examples; see Fig.
A1 for detailed analysis.
For fair comparisons of the self-supervised learning framework, we integrate our ProtoCL into CC by replacing the cluster head with our ProtoCL on the representations while keeping other ofﬁcial hyper-parameters unchanged.
Although class collision issue remains, the signiﬁcant improvements over CC on both datasets suggest that class-level contrastive loss over representations of cluster centers is better than the one over cluster probabilities and ProtoCL can be generalized to other self-supervised learning frameworks.
Qualitative ablation study.
Fig.
4 visualizes the distribution of representations learned from
MoCo v2, BYOL, NCC w/o ProtoCL, and NCC.
The representations from MoCo v2 are mixed up at the center due to the class collision issue.
The rest can reduce this phenomenon where NCC w/o ProtoCL produces more compact clusters than BYOL, and NCC further maintains distinct borders between different clusters.
In addition, we also visualize the training and clustering results of
BYOL and NCC in Fig.
A7, demonstrating that NCC can produce more uniform representations, more balanced clusters, and better clustering performance than BYOL.
[Figure]
Additional ablation studies.
To explore the inﬂuences of different hyper-parameters in NCC, we perform the following ablation studies: 1) performing k-means clustering for every r epochs in Fig.
A2; 2) σ in positive sampling in Fig.
A3; 3) λ pcl for ProtoCL in Fig.
A4; 4) predeﬁned number of clusters K in Fig.
A6; 5) projection dimension for self-supervised learning in Fig.
A8; 6) data augmentation in Fig.
A9 for self-supervised learning; and 7) different ResNet architectures in
Table A5.
All ablation study results verify that the performance gain of NCC does not come from backbone, projection dimension, or any other hyper-parameters.
The results also suggest that NCC is robust to the choice of hyper-parameters.
C ONCLUSION
We have explored the non-contrastive representation learning for deep clustering.
The proposed positive sampling strategy and prototypical contrastive loss can lead to within-cluster compactness and well-separated clusters towards the goodness of clustering.
The results suggest that the proposed
NCC outperforms the state-of-the-art methods by a signiﬁcant margin.
We hope our study will attract the community’s attention to the non-contrastive representation learning methods for deep clustering, which do not suffer from class collision issue.
A PPENDICES A A DDITIONAL R ELATED W ORK
Self-supervised learning.
Previous self-supervised learning (SSL) methods for representation learning attempt to capture the data distribution using generative models (Donahue et al., 2017;
Donahue & Simonyan, 2019) or learn the representations through some special designed pretext tasks (Doersch et al., 2015; Noroozi & Favaro, 2016; Zhang et al., 2016; Caron et al., 2018).
In recent years, contrastive learning methods (Wu et al., 2018;
He et al., 2020; Chen et al., 2020) have shown promising results for both representation learning and downstream tasks.
For example,
MoCo (He et al., 2020) uses a memory queue to store the consistent representations output by a moving-averaged encoder.
However, the class collision issue remains unavoidable; i .
e ., the semantic similar instances are pushed away since they could be regarded as negative pairs (Saunshi et al., 2019).
Some attempts have been made to address this issue (Khosla et al., 2020;
Hu et al., 2021;
Chuang et al., 2020).
On the contrary, the recent studies of SSL demonstrate that the negative examples are not necessary, termed non-contrastive methods (Caron et al., 2020;
Grill et al., 2020; Chen et al., 2020).
In summary, SSL methods mainly focus on inducing transferable representations for the downstream tasks instead of grouping the data into different semantic classes in deep clustering.
Relation to CC.
Although both L pcl and CC (Li et al., 2021b) are class-level contrastive loss, which perform contrastive learning at the cluster level, they have the following difference.
• The class-level contrastive loss in CC implements the contrastive loss on the cluster probabilities while ours on the representation of cluster centers. Implementing contrastive loss on the cluster probability in (Li et
al. , 2021b) would lose the semantic information of the learned representations, which is not helpful for representation learning. Speciﬁcally
given
the x ∈ B , CC obtains the cluster assignments P k
=
[ p ( k | x (1) ) , . . .
, p ( k | x ( N ) )] from one view and P k (cid:48) from another view, and then contrasts P k and P k (cid:48) at the cluster level using the InfoNCE loss.
In contrast, L pcl implements the contrastive loss on the representation of the cluster centers within a mini-batch using the pseudo-labels from k-means clustering.
As a result, L pcl is able to sense the semantic information of the latent space and make the representations of clusters more discriminative and suitable for the clustering task.
• The class-level contrastive loss in CC does not encourage cluster uniformity while our ProtoCL does.
CC still needs the instance-wise contrastive loss to encourage the instance uniformity, which inevitably introduces the class collision issue.
We have also conducted experiments that integrate CC into BYOL and reported the results in Table 4.
The performance of BYOL drops and the training becomes unstable.
Under the same conditions, NCC achieves signiﬁcant improvements over CC.
Relation to GCC and WCL.
GCC (Zhong et al., 2021) and WCL (Zheng et al., 2021) built a graph to label the neighbor samples as pseudo-positive examples.
Then, they enforce the two data augmentations of one example to be close to its multiple pseudo-positive examples using a supervised contrastive loss.
GCC adopted a moving-averaged memory bank for the graph-based pseudo-labeling while WCL built the graph within a mini-batch.
GCC and WCL mainly focus on how to effectively select positive examples from mini-batch/memory bank to alleviate the class collision issue.
Here, we divide the class collision issue into the following two cases:
• Negative class collision issue: negative examples may not be truly negative, which is the case we discussed in the paper.
• Positive class collision issue: positive examples may not be truly positive, which is a new case raised in GCC and WCL.
Consequently, they still suffer from the positive class collision issue as the selected pseudo-positive examples may not be truly positive.
In addition to this, they also suffer from the negative class collision issue since they still need negative examples for instance-wise contrastive learning.
We summarize the difference between our positive sampling strategy and theirs in the following four aspects.
• GCC and WCL select the examples that exist in the dataset (mini-batch/memory bank) while ours samples augmented examples from the latent space that may not exist in the dataset.
• GCC and WCL select neighbor examples in a graph as pseudo-positive examples that may not be truly positive while ours samples examples around the instance that can be assumed to be positive in the semantic space.
• GCC and WCL still rely on instance-wise contrastive loss that could lead to class collision issue while ours can avoid class collision issue by using BYOL.
• GCC and WCL require additional computational cost for graph construction while ours is rather cheap in sampling one augmented example.
Relation to PCL.
Here, we summarize the difference between our NCC and PCL (Li et al., 2021a) in terms of the losses and EM frameworks.
First, we summarize the difference of our ProtoCL loss and ProtoNCE loss in PCL as follow.
• Our NCC can avoid class collision issue while PCL cannot.
NCC is based on BYOL that does not require negative examples for representation learning while PCL is based on instance-wise contrastive loss that requires a number of negative examples for representation learning, inevitably leading to class collision issue.
• The proposed ProtoCL in NCC is conceptually different from the ProtoNCE in PCL.
ProtoCL is to maximize the inter-cluster distance to form a uniformly distributed space while ProtoNCE is to minimize the instance-to-cluster distance to improve the within-cluster compactness.
The within-cluster compactness of NCC is improved by the proposed positive sampling strategy.
• Pure ProtoCL can work well for deep clustering while ProtoNCE requires another InfoNCE to form uniformly distributed space.
This is a direct result of the different designs of the losses.
ProtoCL can maximize the inter-cluster distance to form a uniformly distributed space while ProtoNCE suffers from collapse without the help of another InfoNCE to form such a space.
Second, we summarize difference between our NCC and PCL in the EM framework.
Formulating NCC into an EM
framework can offer more
insights about NCC and make it easy to understand
Although both in an EM framework, the M-step in PCL is signiﬁcantly different from the one in our NCC.
More speciﬁcally, the M-step in PCL is to optimize the ProtoNCE, which is an instanceto-prototypes contrastive loss to improve the within-cluster compactness while the M-step in our
NCC is to optimize the proposed ProtoCL, which is a prototypes-to-prototypes contrastive loss to maximize the inter-cluster distance for better clustering performance.
In addition, NCC also proposes a positive sampling strategy by sampling positive examples around each sample to improve within-cluster compactness.
Finally, Table 3 demonstrates that NCC outperforms PCL by almost 10% AMI on the ImageNet-1K dataset.
B EM F RAMEWORK
In this section, we ﬁrst describe the von Mises-Fisher (vMF) distribution on the hypersphere, and then derive the ELBO for our EM framework, followed by detailed E-step and M-step.
Finally, we describe our proposed prototypical contrastive loss and provide proof for convergence analysis.
Von Mises-Fisher distribution.
Since the features in current SSL methods are usually (cid:96) 2normalized, it is more proper to employ the spherical distribution to describe the features.
The von Mises-Fisher (vMF) distribution, often seen as the Gaussian distribution on a hypersphere, is parameterized by µ ∈ R d the mean direction and κ ∈ R + the concentration around µ .
For the special case of κ = 0 , the vMF distribution represents a uniform distribution on the hypersphere.
The PDF of vMF distribution for the random unit vector v ∈ R d is deﬁned as:
[Equation]
where d is the feature dimension, (cid:107) µ (cid:107) 2 = 1 , C d ( κ ) is the normalizing constant, and I v denotes the modiﬁed Bessel function of the ﬁrst kind at order v .
The standard Gaussian distribution z ∼ N (0 , I ) can be approximately seen as the uniform vMF distribution if the z is (cid:96) 2 -normalized and d is large for the high dimension data.
Derivation of ELBO.
Given the dataset D = { x ( n ) } Nn =1 with N observed data points that are related to a set of K cluster latent variables, k ∈ K = { 1 , 2 , . . .
, K } , the marginal likelihood can be written as:
[Equation]
where θ denotes the model parameters.
Eq.
(A2) is usually maximized to train the neural network.
However, it is hard to directly optimize the log-likelihood function.
Using an inference model q ( k ) like VAE (Kingma & Welling, 2014) to approximate the distribution of K , especially (cid:80) k ∈K q ( k ) = 1 , we can re-write the log-likelihood function for one example as: [Equation] A
[Equation]
where p ( x , k ; θ ) =
p ( k | x ; θ ) p ( x ; θ )
so we have log p ( x ; θ )
= log p ( x , k ; θ ) − log p ( k | x ; θ ) and the evidence lower bound (ELBO) is the lower bound of log-likelihood function since
KL( q ( k ) (cid:107) p ( k | x ; θ )) ≥ 0 .
When KL( q ( k ) (cid:107) p ( k | x ; θ ))
= 0 , the ELBO reaches its maximum value log p ( x ; θ ) , making q ( k ) =
p ( k | x ; θ ) .
By replacing q ( k ) with p ( k | x ; θ ) and ignoring the constant value (cid:80) k ∈K − q ( k ) log q ( k ) , we are ready to maximize:
[Equation]
E-
step.
With the ﬁxed θ t at the iteration t , this step aims to estimate q t +1 ( k ) that makes q t +1 ( k )
= p ( k | x ; θ t ) so that ELBO( q t +1 , x ; θ t ) =
log p ( x ; θ t ) .
Here, we perform the spherical k-means algorithm to estimate p ( k | x ; θ t ) .
We extract features from the target network since the target network performs more stable and yields more consistent clusters, similar to BYOL and
MoCo.
M-step.
With the ﬁxed suboptimal q t +1 ( k ) =
p ( k | x ; θ t ) after E-step, we turn to optimize the θ to maximize the ELBO:
[Equation]
Using a uniform prior for k as p ( k ) = 1 /K , we can obtain p ( x , k ; θ ) =
p ( k ) p ( x | k ; θ ) = p ( x | k ; θ ) /K .
By replacing p ( x , k ; θ ) in Eq.
(A7)
and ignoring constant value, in this step, we should maximize:
[Equation]
where p ( k | x ; θ ) = 1 ( x ∈ k ) .
1 ( · ) is an indicator function using the hard labels estimated from Estep
so that 1 ( x ∈ k ) =
if x belongs to k -th cluster; otherwise 1 ( x ∈ k )
= 0 .
Following (Li et al., 2021a), if we assume that the distribution for each cluster is the vMF distribution with a constant κ as the temperature of softmax function, we can further obtain the follow:
[Equation]
where v = f ( x ; θ ) , τ = 1 /κ , and µ k is the cluster center of k -th cluster.
Combining Eqs.
(A9) and (A10), we can achieve the maximum log-likelihood estimation to ﬁnd the optimal θ ∗ by minimizing the following negative log-likelihood:
[Equation]
where y ( n ) is the pseudo-label for x ( n ) estimated by the k-means algorithm in E-step.
Directly optimizing Eq.
(A11) usually leads to improve the cluster compactness, which, however, will compromise the stability of BYOL since it does not consider the uniformity term.
To this end, we propose a prototypical contrastive loss (ProtoCL) to maximize the log-likelihood at the cluster level by employing the clusters centers as the special instances, or prototypes from a set of instances.
The ProtoCL is deﬁned as:
[Equation]
where { µ 1 , µ 2 , . .
. , µ K } and { µ (cid:48) 1 , µ (cid:48) 2 , . .
. , µ (cid:48) K } are K prototypes from target and online networks, respectively.
Here, instead of using the centroids computed from k-means, our cluster center µ k and µ (cid:48) k is empirically estimated within mini-batch B as follows:
[Equation]
where p ( k | x ) is estimated from E-step.
When K > |B| , it is obvious that the mini-batch cannot cover all clusters.
To this end, we zero out the losses and logits of empty clusters for each iteration; see the pseudocode in Appendix D for more details.
Intuitively, ProtoCL can encourage the prototypical alignment between two augmented views and the prototypical uniformity, hence maximizing the inter-cluster distance.
Convergence analysis.
At E-step of the iteration t , we estimate q t +1 ( k ) to make
ELBO( q t +1 , x ; θ t ) = log p ( x ; θ t ) .
At M-step after E-step, we obtain the optimized θ t +1 with the ﬁxed q t +1 ( k ) so that ELBO( q t +1 , x ; θ t +1 ) ≥ ELBO( q t +1 , x ; θ t ) .
Consequently, we obtain the following sequence:
[Equation]
Given log p ( x ; θ t +1 ) ≥ log p ( x ; θ t ) , we can guarantee the convergence of our NCC.
C E XPERIMENTAL S ETUP
Datasets.
We conducted experiments on six benchmark datasets, including CIFAR10 (Krizhevsky et al., 2009), CIFAR-20 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), ImageNet-10 (Chang et al., 2017), ImageNet-Dogs (Chang et al., 2017), ImageNet1K (Deng et al., 2009), which are summarized in Table 1.
We note that CIFAR-20 contains 20 superclasses of CIFAR-100.
STL-10 includes extra unlabeled images.
ImageNet-10 and
ImageNet-Dogs are the subset of ImageNet-1K, containing 10 and 15 classes, respectively.
This paper follows the experimental settings widely used in deep clustering work (Chang et al., 2017; Wu et al., 2019; Ji et al., 2019;
Tsai et al., 2021; Tao et al., 2021), including the image size, backbone and train-test split.
For image size, we have used 32 × 32 for CIFAR-10 and CIFAR-20, 96 × 96 for
STL-10, ImageNet-10 and ImageNet-Dogs, and 224 × 224 for ImageNet-1K. For train-test split, we use the whole datasets including training and testing set for CIFAR-10 and CIFAR-20.
Backbone.
We use ResNet-34 (He et al., 2016) as the backbone for fair comparisons to report the results in Table 2.
Unless noted otherwise, we use ResNet-18 for the rest of the experiments.
Since the image sizes of CIFAR-10 and CIFAR-100 are relatively small, following (Chen et al., 2020), we replace the ﬁrst convolution layer of kernel size 7 × 7 and stride 2 with a 3 × 3 Conv of stride 1 and remove the ﬁrst max-pooling layer for all experiments on CIFAR-10 and CIFAR-100.
Metrics.
We employ three common metrics to evaluate the clustering performance for the former ﬁve datasets, including Normalized Mutual Information (NMI), Cluster Accuracy (ACC), and
Adjusted Rand Index (ARI).
Following (Li et al., 2021a), we report Adjusted Mutual Information (AMI) to evaluate the clustering performance for ImageNet-1K. All the metrics are presented in percentage (%), and the higher the better clustering performance.
We run each experiment three times and report the mean and standard deviation as the ﬁnal results.
Optimization.
We train all methods with 1000 epochs, strictly following the literature (Tsai et al., 2021; Tao et al., 2021), and adopt the SGD optimizer and the cosine decay learning rate schedule with 50 epochs for learning rate warmup.
The base learning rate for MoCo v2, BYOL, and NCC were 0.05, scaled linearly with the batch size (LearningRate = 0.05 × BatchSize/256).
Note that the learning rates for predictor networks of BYOL and NCC are 10 × as the learning rate of feature extractor.
It is relatively important to achieve satisfactory performance, as discussed in (Grill et al., 2020; Chen & He, 2021).
For other hyperparameters of NCC, the temperature τ , λ pcl for prototypical contrastive loss, and σ for positive sampling were set as 0 .
5 , 0 . 1 , and 0 .
001 , respectively.
The mini-batch size was 512 for MoCo and 256 for the rest methods, trained on 4 NVIDIA V100 GPUs.
In terms of the results of CC (Li et al., 2021b) and PCL (Li et al., 2021a) in Table 4, we tried our best to reproduce their results for fair comparisons.
For CC, we used their ofﬁcial code.
For PCL, under the fair conditions of MoCo, we set the loss weight of ProtoNCE to 0 .
01 and the number of clusters to { 250 , 500 , 1000 } following the suggestions of authors, which we found can achieve the best results.
We integrated CC and PCL into BYOL by adding their losses without changing other settings.
Conﬁgurations of self-supervised learning frameworks.
We adopt the same data augmentations as SimCLR (Chen et al., 2020), including ResizedCrop, ColorJitter, Grayscale, and HorizontalFlip.
We have removed GaussianBlur since we only used a small image size for all datasets.
We also strictly follow the settings of BYOL (Grill et al., 2020).
Speciﬁcally, despite the standard ResNet backbones, the projection and predictor networks have the architectures of FC-BN-ReLU-FC, where the projection dimension and hidden size were 256 and 4096 for both two networks, respectively.
In the context of the whole paper, we have included the backbone and projection network as the encoder network, or feature extractor, since the architectures except predictor are symmetric across views.
For fair comparisons, we have also set the projection dimension of MoCo v2 as 256.
We have used symmetric loss for all methods, i .
e ., swapping two data augmentations to compute twice loss.
Given the target network f (cid:48) θ (cid:48) and online network f θ , the momentum rule describes that the θ (cid:48) is updated as θ (cid:48)
= m θ (cid:48)
+ (1 − m ) θ with a momentum hyperparameter m ∈ [0 , 1) .
We set the m to 0.996 for both BYOL and NCC same as (Grill et al., 2020) and 0.99 for MoCo v2.
For MoCo v2, the queue size, temperature for InfoNCE loss, weight decay were 4096, 1.0, 1 .
× 10 − 4 , respectively.
We have not employed SyncBN in NCC like BYOL.
We note that SyncBN will introduce 1 / 3 additional computation costs.
Instead, we adopt the shufﬂingBN in MoCo to avoid the trivial solution of noncontrastive learning.
Conﬁgurations of ImageNet-1K. For ImageNet-1K, the weight decay and base learning rate were 10 − 4 and 0.4, respectively.
We employ ResNet-50 as the backbone, train our method for 200 epochs without symetric loss to reduce training time, under the same conditions of PCL (Li et al., 2021a).
We include the GaussianBlur to the data augmentations since we use the original image size for training.
During evaluation, images are resized to 256 × 256 and center cropped.
Due to the large computation cost, we run NCC on ImageNet-1K only once.
Computational cost.
Both BYOL and MoCo use a momentum-updated network f (cid:48) θ (cid:48) .
The computation cost between them lies on the fact that MoCo needs a large number of negative examples while BYOL uses a large hidden size for the predictor network.
Our NCC is built upon BYOL, the main additional computational cost is the k-means clustering procedure.
We have implemented the k-means algorithm with k-means++ (Arthur & Vassilvitskii, 2006) initialization using PyTorch to utilize the GPU and accelerate the clustering process.
Taking CIFAR-10 and CIFAR-20 datasets for example, it takes only 5 seconds for k-means clustering, leading to only 5 hours of training time.
Besides, as suggested in the results of Fig.
A2, NCC is robust to the different r , so that there is no need to perform k-means for every epoch.
The computational cost of the ProtoCL is also small since we build the cluster centers within mini-batch, saying that NCC does not need additional memory to store the cluster centers.
Consequently, considering the promising performance improvements, the additional computational cost is relatively affordable.
D P SEUDOCODE OF NCC
→ backward()
[Table]
( θ , ϕ ) # network; θ
-m) θ )
E A DDITIONAL E XPERIMENTAL R ESULTS
[Figure]
[Figure]
for every r epoch.
Here we study how different r inﬂuences the clustering performance.
The results demonstrate NCC is robust to large r and the cluster pseudo-labels, which means it is not necessary to perform clustering for every epoch so that the computation cost can be signiﬁcantly reduced.
In summary, we suggest that r can be set to [1 , 8] by considering the datasets and computation resources.
[Figure] [Figure]
possible reason is that CIFAR-20 is more diverse and has more semantic classes than CIFAR-10 (100 versus 10).
Anyway, we suggest that the loss weight for ProtoCL loss can be set to [0 . 01 , 0 . 1] for different situations, which has demonstrated superior performance on both two datasets.
[Figure] [Figure]
ba l an
[Figure]
a t i
[Figure]
decreasing STDs.
Moreover, we visualize the fourth cluster statistic in (d), with the sorted number of samples.
Unsurprisingly, the samples for NCC are approximately and equally assigned to different clusters, compared to almost long-tailed assignments of BYOL.
[Table]
[Figure] [Figure]
[Table] [Table]
with ResNet-50 for 300 epochs.
We note that SCAN has used the pre-trained model of MoCo trained on the full ImageNet forepochs.
The results are directly referred from their published paper including k-means with pretrained MoCo 1 , SCAN after the clustering step 2 , and SCAN after the self-labeling step 3 .
With much fewer training epochs and training data, NCC still produces better performance with a clear margin, demonstrating the superiority of NCC.
[Table] [Table]
[Table] [Table]