C LUSTERING - FRIENDLY R EPRESENTATION L EARNING VIA I NSTANCE D ISCRIMINATION AND F EATURE D ECORRELATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Clustering is one of the most fundamental tasks in machine learning
Recently , deep clustering has become a major trend in clustering techniques
Representa tion learning often plays an important role in the effectiveness of deep clustering , and thus can be a principal cause of performance degradation.
In this paper, we propose a clustering-friendly representation learning method using instance dis - crimination and feature decorrelation.
Our deep-learning-based representation learning method is motivated by the properties of classical spectral clustering.
Instance discrimination learns similarities among data and feature decorrelation re- moves redundant correlation among features
We utilize an instance discrimination method in which learning individual instance classes leads to learning similarity among instances
Through detailed experiments and examination, we show that the approach can be adapted to learning a latent space for clustering.
We design novel softmax-formulated decorrelation constraints for learning.
In evaluations of image clustering using CIFAR-10 and ImageNet-10, our method achieves accuracy of 81 .
5% and 95 .
4% , respectively.
We also show that the softmax-formulated constraints are compatible with various neural networks.
I NTRODUCTION
Clustering is one of the most fundamental tasks in machine learning.
Recently, deep clustering has become a major trend in clustering techniques.
In a fundamental form, autoencoders are used for feature extraction, and classical clustering techniques such as k -means are serially applied to the features.
Recent deep clustering techniques integrate learning processes of feature extraction and clustering, yielding high performance for large-scale datasets such as handwritten digits Hu et al.
(2017); Shaham et al.
; Xie et al. (2016); Tao et al.
However, those methods have fallen short when targets become more complex, as in the case of real-world photograph dataset
CIFAR-10 Krizhevsky et al.
Several works report powerful representation learning leads to improvement of clustering performance on complex datasets Chang et al.
; Wu et al.
Learning representation is a key challenge to unsupervised clustering.
In order to learn representations for clustering, recent works utilize metric learning which automatically learns similarity functions from data Chang et al.
; Wu et al.
They assign pseudo-labels or pseudo-graph to unlabeled data by similarity measures in latent space, and learn discriminative representations to cluster data.
These works improve clustering performance on real world images such as CIFAR-10 and ImageNet-10, and indicate the impact of representation learning on clustering.
Although features from learned similarity function and pseudo-labels work well for clustering, algorithms still seem to be heuristic; we design a novel algorithm which is based on knowledge from established clustering techniques.
In this work, we exploit a core idea of spectral clustering which uses eigenvectors derived from similarities.
Spectral clustering has been theoretically and experimentally investigated, and known to outperform other traditional clustering methods Von Luxburg (2007).
The algorithm involves similarity matrix construction, transformation from similarity matrix to Laplacian, and eigendecomposition.
Based on eigenvectors, data points are mapped into a lower dimensional representation which carries informa- tion of similarities and is preferable for clustering.
We bring this idea of eigenvector representation into deep representation learning.
We design the representation learning with two aims: 1) learning similarities among instances; and 2) reducing correlations within features.
The ﬁrst corresponds to Laplacian, and the second corresponds to feature orthogonality constrains in the spectral clustering algorithm.
Learning process integrating both is relevant to eigendecomposition of Laplacian matrix in the spectral clustering.
For the ﬁrst aim, we adopt the instance discrimination method presented in Wu et al.
(2018), where each unlabeled instance is treated as its own distinct class, and discriminative representations are learned to distinguish between individual instance classes.
This numerous-class discriminative learning enables learning partial but important features, such as small foreground objects in natural images.
Wu et al.
(2018) showed that the representation features retain apparent similarity among images and improve the performance of image classiﬁcation by the nearest neighbor method.
We extend their work to the clustering tasks.
We clarify their softmax formulation works like similarity matrix in spectral clustering under the condition that temperature parameter τ , which was underexplored in Wu et al.
(2018), is set to be a larger value .
For the second aim, we introduce constraints which have the effect of making latent features orthogonal.
Orthogonality is often an essential idea in dimension reduction methods such as principal components analysis, and it is preferable for latent features to be independent to ensure that redundant information is reduced.
Orthogonality is also essential to a connection between proposed method and spectral clustering, as stated in Section 3.4.
In addition to a simple soft orthogonal constraint, we design a novel softmax-formulated decorrelation constraint.
Our softmax constraint is "softer" than the soft orthogonal constraint for learning independent feature spaces, but realizes stable improvement of clustering performance.
Finally, we combine instance discrimination and feature decorrelation into learning representation to improve the performance of complex image clustering.
For the CIFAR-10 and ImageNet-10 datasets, our method achieves accuracy of 81 .
5% and 95 .
4% , respectively.
Our main contributions are as follows:
• We propose a clustering-friendly representation learning method combining instance discrimination and feature decorrelation based on spectral clustering properties.
• We adapt deep representation learning by instance discrimination to clustering and clarify the essential properties of the temperature parameter.
• We design a softmax-formulated orthogonal constraint for learning latent features and realize stable improvement of clustering performance.
• Our representation learning method achieves performance comparable to state-of-the-art levels for image clustering tasks with simple k -means.
R ELATED W ORK
Deep clustering methods offer state-of-the-art performance in various ﬁelds.
Most early deep clustering methods, such as Vincent et al. (2010); Tian et al.
(2014), are two-stage methods that apply clustering after learning low-dimensional representations of data in a nonlinear latent space.
The autoencoder method proposed in Hinton & Salakhutdinov (2006) is one of the most effective methods for learning representations.
Recent works have simultaneously performed representation learning and clustering Song et al.
; Xie et al. (2016); Yang et al. (2017); Guo et al.
; Tao et al.
Several methods based on generative models have also been proposed Jiang et al.
Dilokthanakul et al.
These methods outperform conventional methods, and sometimes offer performance comparable to that of supervised learning for simple datasets.
Deep-learning-based unsupervised image clustering is also being developed Chang et al.
; Wu et al.
; Ji et al.
; Gupta et al.
Van Gansbeke et al.
Several approaches focus on learning discriminative representations via deep learning.
Bojanowski &
Joulin (2017) found a mapping between images on a uniformly discretized target space, and enforced their representations to resemble a distribution of pairwise relationships.
Caron et al.
(2018) applied pseudo-labels to output as supervision by k -means and then trained a deep neural network.
Donahue et al.
(2016) proposed bidirectional generative adversarial networks for learning generative models that map simple latent distributions to complex real distributions, in order for generators to capture semantic representations.
Hjelm et al.
(2018) proposed deep infomax to maximize mutual information between the input and output of an encoder.
Wu et al.
(2018) was motivated by observations in supervised learning that the probabilities of similar image classes become simultaneously high.
They showed that discriminating individual instance classes leads to learning representations that retain similarities among data.
IIC Ji et al. and SCAN Van Gansbeke et al. (2020) are two recent works focusing on image clustering and obtained high performance. IIC Ji et al.
(2019) directly learns semantic labels without learning representations based on mutual information between image pairs.
SCAN
Van Gansbeke et al.
(2020) focuses on the clustering phase and largely improved performance based on a given pre-designed representation learning.
By contrast, we focus on learning a clusteringfriendly representation space where objects can be simply clustered.
Our method exploits the idea of spectral clustering Shi & Malik (2000); Meila & Shi (2001);
Von Luxburg (2007); Ng et al. (2002).
From one perspective, spectral clustering ﬁnds a low dimensional embedding of data in the eigenspace of the Laplacian matrix, which is derived from pairwise similarities between data.
By using the embedded representations, we can proceed to cluster the data by the k -means algorithm in the low-dimensional space.
Spectral clustering often outperforms earlier algorithms such as k -means once pair similarities are properly calculated.
Shaham et al.
(2018) incorporated the concept of spectral clustering into deep a neural network structure.
Similarities were calculated by learning a Siamese net Shaham & Lederman (2018) where the input positive and negative pairs were constructed according to the Euclidean distance.
P ROPOSED M ETHOD
Given an unlabeled dataset X = { x i } ni =1 and a predeﬁned number of clusters k , where x i denotes the i th sample, we perform the clustering task in two phases, namely, representation learning and clustering.
This work focuses on the ﬁrst phase, which aims to learn an embedding function v = f θ ( x ) mapping data x to representation v so that v is preferable for clustering.
f θ is modeled as a deep neural network with parameter θ .
We use V = { v i } ni =1 to denote the whole representation set.
3. I NSTANCE D ISCRIMINATION
We apply the instance discrimination method proposed by Wu et al.
(2018) to learn clustering-friendly representations that capture similarity between instances.
The objective function is formulated based on the softmax criterion.
Each instance is assumed to represent a distinct class.
For given data x 1 , . . .
, x n , the corresponding representations are v 1 , . .
. , v n , and data x i is classiﬁed into the i th class.
Accordingly, the weight vector for the i th class can be approximated by a vector v i .
The probability of representation v being assigned into the i th class is
[Equation]
where v Tj v measures how well v matches the j th class, τ is a temperature parameter that controls the concentration of the distribution Hinton et al.
and v is normalized to ||
v || = 1 .
The objective maximizes the joint probability (cid:81) ni
=1 P θ
( i | f θ ( x i )) as
[Equation]
Wu et al.
(2018) shows that features obtained by minimizing the objective retain similarity between image instances and improve the performance of nearest neighbor classiﬁcation.
For clustering, we note that the parameter τ , which is underexplored in Wu et al.
(2018), has a large impact on clustering performance.
The effect of τ is discussed later and experimental results are shown in 4.2.1.
3. F EATURE D ECORRELATION
We deﬁne a set of latent feature vectors f and use f l to denote the l th feature vector.
Transposition of latent vectors V coincides with { f l } dl =1 , where d is the dimensionality of representations.
The simple constraint for orthogonal features is,
[Equation]
Our novel constraint is based on a softmax formulation of
[Equation]
Q ( l | f ) is analogous to P ( i | v ) .
Q ( l | f ) measures how correlated a feature vector is to itself and how dissimilar it is to others.
τ 2 is the temperature parameter.
We formulate the feature decorrelation constraint as
[Equation]
Both constrains in Eq.
(3) and Eq.
(5) aim to construct independent features.
Conventionally, it is preferable for features to be independent to ensure that redundant information is reduced, and orthogonality is a common technique.
Compare Eq.
(3) and Eq.
(5), we can see that minimizing
L F and L FO can result in a similar effect, f Tl f l → 1 and f Tj f l → − 1 or 0( l (cid:54) =
j ) , and both try to decorrelate latent features.
Our softmax constraint in Eq.
(5) shows practical advantages in ﬂexibility and stability.
Eq.
(3) is called a soft orthogonal constraint, but is still strict enough to force the features to be orthogonal.
If d is larger than underlying structures that are hidden and unknown, all features are forcibly orthogonalized and the resultant features may not be appropriate.
Softmax formulation allows off-diagonal elements to be non-zero and alleviates the problem of strict orthogonality.
Partial derivatives of L F and L FO with respect to z jl =
f
Tj f l are calculated as ∂L F ∂z jl = − 1 τ 2 δ jl + 1 τ 2 exp
( z jl /τ 2 ) (cid:80) dj exp( z jl /τ 2 ) and ∂L FO ∂z jl = − 2 δ jl + 2 z jl , where δ jl is an indicator function.
Since the derivatives nearly equal zero due to z jl = 1 in the case of j = l , we focus on the case of j (cid:54)
=
l .
When j (cid:54) =
l , the ranges of partial derivatives are 0 ≤ ∂L F ∂z jl ≤ 1 τ 2 and − 2 ≤ ∂L FO ∂z jl ≤ 2 .
The monotonicity of L F can lead to more stable convergence.
The advantages of L F are conﬁrmed by experiments in section 4.
O BJECTIVE F UNCTION AND L EARNING M ODEL
Combining instance discrimination and feature decorrelation learning, we formulate our objective function L IDFD as follows:
[Equation]
Where α is a weight that balances the contributions of two terms L I and L F .
Figure 1 shows the learning process for the motif of image clustering.
Input images X are converted into feature representations V in a lower d -dimensional latent space, via nonlinear mapping with deep neural networks such as ResNet He et al.
The d -dimensional vectors are simultaneously learned through instance discrimination and feature decorrelation.
A clustering method, such as classical k -means clustering, is then used on the learned representations to obtain the clustering results.
Optimization can be performed by mini-batch training.
To compute the probability P ( i | v ) in Eq.
(1), { v j } is needed for all images.
Like Wu et al.
Xiao et al.
, we maintain a feature memory bank for storing them.
For Q ( l | f ) in Eq.
(4), all { f m } of d dimensions in the current mini-batch can be obtained, we simply calculate the Q ( l | f ) within the mini-batches.
We combine L I and L FO to formulate an alternative loss L IDFO in E.q.
[Equation]
I [Equation] We refer to representation learning using L IDFD , L IDFO , and L I loss as instance discrimination and feature decorrelation (IDFD), instance discrimination and feature orthogonalization (IDFO), and instance discrimination (ID), respectively.
3. C ONNECTION WITH S PECTRAL C LUSTERING
We explain the connection between IDFD and spectral clustering.
We consider a fully connected graph consisting of all representation points, and the similarity matrix W and degree matrix D can be written as W ij =
exp( v Ti v j /τ ) and D ii = (cid:80) nm exp( v Ti v m /τ ) .
The loss function of spectral clustering Shaham et al.
(2018) can be reformulated as
[Equation]
where L is Laplacian matrix, f are feature vectors.
Spectral clustering is performed by minimizing
L SP subject to orthogonal condition of f , and when L SP takes minimum value f become eigenvectors of Laplacian L .
According to Section 3.2, minimizing L F can approximate the orthogonal condition.
Under this condition, minimizing L I can approximate the minimizing L SP , which is explained as follows.
According to Eq.(2), minimizing loss L I means maximizing v Ti v i and minimizing v Ti v j .
When i =
j , we have ||
v
i − v j || 2 = 0 , L SP becomes zero.
We need consider only the inﬂuence on L SP from minimizing v Ti v j .
As v are normalized, L SP can be rewritten using cosine metric as
[Equation]
then ∂L SP ∂θ can be calculated as
[Equation]
(cid:19) . According to Eq.(10), we get ∂L SP ∂θ ≥ 0 when τ ≥ 2 .
This means L SP monotonically decreases when we minimize v Ti v j .
Therefore, the impact from minimizing v Ti v j is good for minimizing L SP .
Even if τ is a little smaller than 2 , because τ controls the scale of derivatives and the range of θ where the derivative is negative, large τ decreases the scale and narrows the range, resulting in a small inﬂuence on the total loss.
From this viewpoint, the effectiveness of minimizing L I using large τ is approximately the same as that of L SP .
By adding feature decorrelation constraints, IDFD becomes analogous to spectral clustering.
E XPERIMENTS
We conducted experiments using ﬁve datasets: CIFAR-10 Krizhevsky et al.
(2009), CIFARKrizhevsky et al.
(2009), STL-10 Coates et al.
(2011), ImageNet-10 Deng et al.
(2009), and
ImageNet-Dog Deng et al.
We adopted ResNet
He et al.
as the neural network architecture in our main experiments.
The same architecture is used for all datasets.
Our experimental settings are in accordance with that of Wu et al.
Data augmentation strategies often used on images are also adopted in experiments.
Details about datasets and experimental setup are given in Appendix A.
For IDFD, the weight α is simply ﬁxed at 1 .
Orthogonality constraint weights for IDFO were α =on CIFAR-10 and CIFAR-100, and α = 0 .
5 on STL-10 and ImageNet subsets.
The weight α was set according to the orders of magnitudes of losses.
In the main experiments, we set temperature parameter τ = 1 for IDFO and IDFD, and τ 2 = 2 for IDFD.
In order to fully investigate our work, we also constructed two versions of instance discrimination (ID) that uses only L I loss, ID(original) with small τ = 0 .
07 and ID(tuned)
with large τ = 1 .
We compared ID(tuned), IDFO, and IDFD with ID(original) and six other competitive methods, clustering with an autoencoder (AE) Hinton & Salakhutdinov (2006), deep embedded clustering (DEC)
Xie et al.
(2016), deep adaptive image clustering (DAC)
Chang et al.
(2017), deep comprehensive correlation mining (DCCM)
Wu et al.
(2019), invariant information clustering (IIC)
Ji et al.
(2019), and semantic clustering by adopting nearest neighbors (SCAN)
Van Gansbeke et al.
(2020) .We use three metrics to measure clustering performance: standard clustering accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI).
These metrics give values in [0 , 1] , with higher scores indicating more accurate clustering assignments.
4. M AIN R ESULTS
Table 1 lists the best performances for each method.
The results for the four methods AE, DEC, DAC, and DCCM are cited from Wu et al. (2019), and results for two methods IIC and SCAN are cited from Van Gansbeke et al.
Comparing these results, we conclude that ID(tuned), IDFO, and IDFD, clearly outperform these methods excluding SCAN for all datasets, according to the metrics ACC, NMI, and ARI.
For dataset CIFAR-10, ID(tuned), IDFO, and IDFD yielded ACC values of 77 .
8% , and 81 .
5% , respectively.
For dataset ImageNet-10, ID(tuned), IDFO, and IDFD achieved ACC values of 93 .
7% , 94 . 2% , and 95 .
The high performance is comparable with that of supervised and semi-supervised methods.
Gaps between the results of ID(tuned) and those of IDFO and IDFD reﬂect the effect of the feature constraint term.
The performance is improved for all datasets by introducing feature orthogonalization and decorrelation.
Impressively, ID(tuned) signiﬁcantly outperformed ID(original) on all datasets, showing strong impact of temperature parameter.
This will be discussed separately in section 4.2.1.
In addition, we note that IDFD differs from SCAN in that IDFD focuses on the representation leaning while SCAN focuses on clustering by given a representation learning.
Both SCAN and IDFD demonstrate signiﬁcant improvement on performance compared with other methods.
Results of IDFD and SCAN showed effectiveness of efforts on both representation learning and clustering phases of deep clustering.
We also examine the learning stability of ID(tuned), IDFO, and IDFD.
Figure 2 illustrates the accuracy on CIFAR-10 running each of ID(tuned), IDFO, and IDFD.
We can see that both IDFO and IDFD obtained higher peak ACC values than ID(tuned).
In particular, IDFD yielded higher performance than ID over the entire learning process.
IDFO performed better than the other two methods and obtained the highest ACC value in earlier epochs.
However, the ACC widely ﬂuctuated [Table] [Table] over the learning process and dropped in later epochs.
As analyzed in 3.2, our proposed IDFD makes performance higher than ID and more stable than IDFO.
4. D ISCUSSION 4.2. A NALYSIS ON T EMPERATURE P ARAMETER
Gaps between results of ID(original) and ID(tuned) in Table 1 show strong impact of temperature parameter.
We theoretically and intuitively analyze the essential change caused by the temperature parameter in this subsection.
First, we consider why instance-level discrimination works and under what conditions.
Difference in the performance of ID(original) and ID(tuned) suggests optimal distribution in latent space changes with the magnitude of τ .
According to empirical investigation and theoretical analysis, we ﬁnd that a large τ in L I encourages data points to follow a compact distribution when minimizing the loss, while a small τ drives them to follow a uniform distribution.
This means minimizing L I with a large τ can reach a good clustering-friendly solution.
This property was explained by demonstrating examples and calculation, details are given in Appendix B.
In the deﬁnition of P ( i | v ) in Eq.
(1), when τ is small, we compute softmax on larger logits, resulting in higher prediction, and obtain a more conﬁdent model.
From this viewpoint, we can leverage a small τ to decrease class entanglement if we can learn an accurate class-weight vector.
In the general classiﬁcation problem, since the weight of each class can be learned according to the real labels, it is preferable for models to be more conﬁdent.
Most works therefore recommend setting a small value, such as τ = 0 .
07 Wu et al.
In clustering, however, instance-level discrimination is used to learn similarity among samples, with only one sample in each class.
Because the model is highly conﬁdent, each sample tends to be completely independent from each other.
Similarity among samples is seemingly encouraged to approach close to zero, even for samples from the same class.
This clearly deviates from the original intent of adopting instance-level discrimination to learn sample entanglements under the condition that each sample can be discriminative.
A larger τ than that used for classiﬁcation is thus needed.
More experiments over different temperature settings on ID and IDFD were conducted on CIFAR-10.
Figure 3 shows the accuracy of ID for τ = { 0 . 07 , 0 . 2 , 0 . 5 , 0 . 8 , 1 , 2 , 5 , 10 } .
We calculated the mean and standard deviation of ACC values over the last 500 epochs for each experiment.
From the results, we can see that ID can suffer signiﬁcant performance degradation when τ is too small or too large.
This agrees with our analysis above.
We also investigate the impact of τ 2 by ﬁxing τ = 1 .
Figureshows the accuracy of the IDFD for τ 2 = { 0 . 1 , 0 .
Experimental results show that
IDFD is relatively robust to the parameter τ 2 and enables stable representation learning.
4.2. R EPRESENTATION D ISTRIBUTION AND F EATURE B EHAVIOR
Figure 5 visualizes the results of representations learned in four experiments: (a) ID(original), (b)
ID(tuned), (c) IDFO with τ = 1 and α = 10 , and (d) IDFD with τ = 1 , τ 2 = 2 , and α = 1 on CIFAR10.
128-dimension representations were embedded into two dimensions by t-SNE (t-distributed stochastic neighbor embedding) Maaten & Hinton (2008).
Colors indicate ground truth classes.
The distributions for the ID(original) and ID(tuned) again show the signiﬁcant difference between them.
Data distribution when τ = 1 is apparently more clustering-friendly than when τ = 0 .
Furthermore, compared with ID(tuned), IDFO and IDFD can separate samples from different classes with certain margins.
IDFO tended to construct a patch-like distribution within one class.
In contrast,
IDFD maintained a tighter connection among samples of the same class and more distinct borders between different classes.
Figure 6 shows distribution of feature representations on ImageNet-10 learned by IDFD.
We can see that representations of ImageNet-10 are clustering-friendly and even better than that of CIFAR-10.
This is consistent with the results in Table 1 evaluated by metrics ACC, NMI and ARI.
In addition to that, we also plot sample images corresponding to points lying near the border between clusters.
We can see that these samples are certainly similar in appearance.
We investigate the effects of orthogonal and decorrelation constraints L FO and L F .
Figure 7 illustrates the feature correlations of ID(tuned), IDFO and IDFD on dataset CIFAR-10.
We see that IDFO clearly decorrelates features and IDFD retains a moderate level of feature correlation between ID and IDFD.
Taken together with Figure 2, these results suggest that the softmax formulation of IDFD alleviates the problem of strict orthogonality and enables stable representation learning.
4.2. I NVESTIGATION FOR P RACTICAL U SE
We investigate the dependencies of our method on networks through experiments on other networks:
ConvNet Wu et al.
VGG16 Simonyan & Zisserman (2014), and ResNet34 He et al. (2016).
Performance was evaluated using the CIFAR-10 dataset.
Results listed in Table 2 show that IDFD can work on various networks.
IDFD outperforms ID(tuned), and FD term shows more obvious effect on these networks.
We also conﬁrm the effect of cooperation between L I and L F from the viewpoint of spectral clustering, combinations of AE and L F were evaluated in terms of clustering performance.
We found that AE cannot beneﬁt from L F as L I did.
This result veriﬁed that L F has a deep relation with L I , and IDFD is not a simple combination.
We also investigate the importance of data augmentation in performance through experiments.
Due to the page limit, our extended experiments are given in Appendix C
[Table]
[Table]
C ONCLUSION
We present a clustering-friendly representation learning method combining instance discrimination and feature decorrelation based on spectral clustering properties.
Instance discrimination learns similarities among data and feature decorrelation removes redundant correlation among features.
We analyzed why instance discrimination works for clustering and clariﬁed the conditions.
We designed a softmax-formulated feature decorrelation constraint for learning the latent space to realize stable improvement of clustering performance.
We also explained the connection between our method and spectral clustering.
The proposed representation learning method achieves accuracies comparable to state-of-the-art values on the CIFAR-10 and ImageNet-10 datasets with simple k -means.
We also veriﬁed IDFD loss works on multiple neural network structures, and our method is expected to be effective for various kinds of problems
A PPENDICES A D ATASETS AND E XPERIMENTAL S ETUP
Five datasets were used to conduct experiments: CIFAR-10 Krizhevsky et al.
(2009), CIFAR100 Krizhevsky et al.
(2009), STL-10 Coates et al.
(2011), ImageNet-10 Deng et al.
(2009), and
ImageNet-Dog Deng et al.
Table 3 lists the numbers of images, number of clusters, and image sizes of these datasets.
Speciﬁcally, the training and testing sets of dataset STL-10 were jointly used in our experiments.
Images from the three ImageNet subsets were resized to 96 × 96 × 3 .
[Table]
We adopted ResNet He et al.
as the neural network architecture in our main experiments.
For simplicity, we used ResNet18, which according to our preliminary experiments yields sufﬁciently high performance.
The same architecture was used for all datasets except the input layer.
In accordance with the experimental settings of Wu et al.
(2018), the dimension of latent feature vectors was set to d = 128 , and a stochastic gradient descent optimizer with momentum β = 0 .
9 was used.
The learning rate lr was initialized to 0 .
03 , then gradually scaled down after the ﬁrst 600 epochs using a coefﬁcient of 0 .
1 every 350 epochs.
The total number of epochs was set to 2000 , and the batch size was set to B = 128 .
Orthogonality constraint weights for IDFO were α = 10 for CIFARand CIFAR-100 and α = 0 .
5 for the STL-10 and ImageNet subsets.
The weight for IDFO α was set according to the orders of magnitudes of the two losses L I and L FO .
For IDFD, the weight α was simply ﬁxed at 1 .
In the main experiments, we set the default temperature parameter value τ = 1 for
ID(tuned), IDFO, and IDFD, and τ 2 = 2 for IDFD.
B O PTIMAL S OLUTIONS OF C LUSTERING AND I NSTANCE D ISCRIMINATION
In Section 4.2.1, we concluded that minimizing L I under the condition that τ is large can reach a clustering-friendly solution.
Details about the analysis and calculation was demonstrated by a two-dimensional toy model as follows.
Empirically, we observe that visually similar images tend to get similar assignment probabilities.
Similar images can thus be projected to close locations in the latent space.
This also motivated ID
Wu et al.
In the case of ID, similar images x i and x j yield respective highest probabilities p ii and p jj , and also receive relatively high p ij and p ji values.
This property can retain over the process of approximation to the optimal solution.
Because instance-level discrimination tries to maximally scatter embedded features of instances over the unit sphere Wu et al.
(2018), all representations are thus uniformly spread over the latent space with each representation relatively similar to its surroundings, we call this uniform case.
We also consider another case that yields an optimal clustering solution where all samples from the same class are compacted to one point and k clusters are uniformly spread over the space.
We call this compact case.
Figure 8 shows the representation distributions in the two cases.
Because we normalize v , two-dimensional representations form a circle.
In the uniform case, n representations are uniformly located on a circle with an angular interval of θ = 2 π/n , and the inner product between two neighboring representations is cos θ .
Without loss of generality, we can start with an arbitrary point v i and orderly mark all samples as v i + j .
The cosine similarity between v i and v i + j can then be calculated by v T i + j v i = cos jθ .
Accordingly, the loss contributed by sample i in the uniform case can be calculated as
[Equation]
Similarly, in the compact case, n/k data from the same class are exactly compacted to a point and k corresponding points located on a circle at an angular interval of θ (cid:48)
=
2 π/k .
The inner product between an arbitrary start sample v i and the j -th sample can be calculated as v Ti v i + j = cos lθ (cid:48) , where l = j mod n/k .
The probability of assigning i to the cluster with j becomes p ij = exp
[Equation]
θ
[Equation] [Equation]
/τ ) .
Accordingly, the loss contributed by sample i in the compact case can be calculated as
[Equation]
Comparing Eq.
(11) and (12)
[Equation]
we see that the difference between L iuniform and L icompact comes only from the denominator part of the logarithm.
These are two discrete forms of the same integral (cid:82) exp(cos θ/τ ) dθ .
Clearly, L iuniform equals L icompact when k, n → + ∞ .
We therefore need to consider only the general case where n is sufﬁciently large and k (cid:28) n .
Figure 9 shows a plot of function values exp( cos θ τ ) with different τ settings over the domain θ ∈ [0 , 2 π ] .
We can see that the curve becomes ﬂatter as τ increases.
A ﬂat function f means that for an arbitrary ( θ, θ (cid:48) ) pair in its domain of deﬁnition, we have f ( θ )
≈ f
( θ (cid:48) ) .
In this situation even k (cid:28) n , the difference between the summations of these two discrete functions is not large.
Accordingly, we can say L i compact is approximate to L i uniform for a large τ .
In other words, minimizing L I can approach the compact situation where same-class samples assemble and differing samples separate.
Learning instance-level discrimination for clustering is therefore reasonable.
C E XTENDED E XPERIMENTS
In Section 4.2.3, we have reported some investigations of our method for practical use.
Details about several important experiments are supplemented as follows.
C. I MPACT OF N ETWORK A RCHITECTURE
As Table 2 shows, IDFD can be applied to various networks, and the performance gaps between
IDFD and ID(turned) on networks like ConvNet Wu et al.
and
VGG16 Simonyan & Zisserman (2014) are more signiﬁcant than on ResNet He et al.
We added the feature correlation matrix of VGG16 in Figure 10.
IDFD on VGG16 obtained sparse correlations similar to the case of
ResNet18 in Figure 7, while ID on VGG16 obtained denser and stronger correlations than ResNet18, presumably constructing redundant features that degraded clustering.
In the case of VGG16, the feature decorrelation term L F exhibits a larger effect on clustering performance than that of ResNet.
Our proposed losses work on all network architectures, and we expect to introduce the losses to various networks that are suitable for individual problems.
C. C OMBINATION OF A UTOENCODER AND F EATURE D ECORRELATION
In order to further conﬁrm the cooperation effect of instance discrimination and feature decorrelation from the viewpoint of spectral clustering, a combination of autoencoder and feature decorrelation was evaluated in terms of clustering performance.
Autoencoder has been veriﬁed by datasets such as handwritten digits to be an effective method for deep clustering.
In this experiment, we used ConvNet
Wu et al.
for the autoencoder architecture and trained it on the CIFAR-10 dataset.
We applied k -means to representations learned from autoencoder only and autoencoder combined with feature decorrelation, which are called AE and AEFD, respectively.
According to our experiments, the ACC value of AE was 26 . 0% , and the ACC value of AEFD was 22 .
Compared to the improvement from ID to IDFD (from 26 .
8% to 42 .
0% as shown in Table 2), we see that AE cannot beneﬁt from
FD as ID.
This result again indicates that FD has a deep relation with ID as we analyzed in Section 3.
C. I MPACT OF D ATA A UGMENTATION
For reproduction of our results and practical use, we note that data augmentation (DA) has strong impact on the performance.
DA is known to have impact on image classiﬁcation and representation learning.
Like in Wu et al.
(2018), several generic and accepted techniques, such as cropping and grayscale, were used for data augmenting in this work.
The details of the augmentation in the original code can be linked to Wu et al.
In order to investigate the impact of DA, we conducted experiments on ﬁve datasets with and without DA and compared their clustering results.
Tableshows the results.
We can see that methods without DA suffered signiﬁcant performance degradations for clustering, as well as for classiﬁcation Chen et al.
This reminds us not to ignore the effects of DA in practical use.
[Table]
To further ﬁnd out main factors affecting the performance
[Table]
we also executed experiments by removing each technique used for DA.
Take the example of CIFAR-10, techniques used for data augmentation include: ColorJitter , RandomResizedCrop , RandomGrayscale
[Table]
and RandomHorizontalFlip .
All these techniques are generic and easy to be implemented.
They have been integrated into general deep learning frameworks such as PyTorch.
According to our experimental results as shown in Figure 11, we ﬁnd that RandomResizedCrop , RandomGrayscale [Table] and ColorJitter have strong effect on image clustering.
For practice, we also applied IDFD to our private images produced by manufacturing process.
Generic
DA like above were used to these images.
IDFD showed good performance on these images according to our experiments.
This indicates that our method can be simply applied to practical images.
For other types of data such as text and time series, corresponding data augmentation techniques are needed to cooperate with our method.