D EEP C LUSTERING AND R EPRESENTATION L EARNING THAT P RESERVES G EOMETRIC S TRUCTURES
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose a novel framework for Deep Clustering and Representation Learning (DCRL) that preserves the geometric structure of data.
In the proposed DCRL framework, manifold clustering is done in the latent space guided by a clustering loss.
To overcome the problem that clustering-oriented losses may deteriorate the geometric structure of embeddings in the latent space, an isometric loss is proposed for preserving intra-manifold structure locally and a ranking loss for inter-manifold structure globally.
Experimental results on various datasets show that the DCRL framework leads to performances comparable to current state-of-the-art deep clustering algorithms, yet exhibits superior performance for downstream tasks.
Our results also demonstrate the importance and effectiveness of the proposed losses in preserving geometric structure in terms of visualization and performance metrics. Source code to reproduce all results will be released.
I NTRODUCTION
Clustering, a fundamental tool for data analysis and visualization, has been an essential research topic in data science and machine learning.
Conventional clustering algorithms such as K -Means (MacQueen, 1965), Gaussian Mixture Models (GMM) (Bishop, 2006), and spectral clustering (Shi & Malik, 2000) perform clustering based on distance or similarity.
However, handcrafted distance or similarity measures are rarely reliable for large-scale high-dimensional data, making it increasingly challenging to achieve effective clustering.
An intuitive solution is to transform the data from the high-dimensional input space to the low-dimensional latent space and then to cluster the data in the latent space.
This can be achieved by applying dimensionality reduction techniques such as PCA (Wold et al., 1987), t-SNE (Maaten & Hinton, 2008), and UMAP (McInnes et al., 2018).
However, since these methods are not speciﬁcally designed for clustering tasks, some of their properties may be contrary to our expectations, e.g., two data points from different manifolds that are close in the input space will be closer in the latent space derived by UMAP.
Therefore, the ﬁrst question here is how to learn the representation that favors clustering?
The two main points of the multi-manifold representation learning are (1) preserving the local geometric structure within each manifold and (2) ensuring the discriminability between different manifolds.
However, it is challenging to decouple complex cross-over relations and ensure the discriminability between different manifolds, especially in unsupervised situations.
One natural strategy is to perform clustering in the input space to get pseudo-labels and then perform representation learning for each manifold.
However, in that case, representation learning’s performance depends heavily on the clustering effect, but commonly used clustering algorithms such as K -Means do not work well on high-dimensional data.
Thus, the second question here is how to cluster data that favors representation learning?
To answer these two questions, some pioneering work has proposed integrating deep clustering and representation learning into a uniﬁed framework by deﬁning a clustering-oriented loss.
Though promising performance has been demonstrated on various datasets, we observe that a vital factor has been ignored by these work that the deﬁned clustering-oriented loss may deteriorate the geometric structure of the latent space, which in turn hurts the performance of visualization, clustering generalization, and downstream tasks.
In this paper, we propose to jointly perform deep clustering and representation learning with geometric structure preservation.
Inspired by Xie et al.
(2016), the clustering centers are deﬁned as a set of learnable parameters, and we use a clustering loss to simultaneously guide the separation of data points from different manifolds and the learning of the clustering centers.
To prevent clustering loss from deteriorating the latent space, an isometric loss and a ranking loss are proposed to preserve the intra-manifold structure locally and inter-manifold structure globally.
Our experimental results show that our method exhibits far superior performance to counterparts in terms of clustering and representation learining, which demonstrates the importance and effectiveness of preserving geometric structure.
The contributions of this work are summarized as below:
• Proposing to integrate deep clustering and representation learning into a uniﬁed framework with local and global structure preservation.
• Unlike conventional multi-manifold learning algorithms that deal with all point pair relationships between different manifolds simultaneously, we set the clustering centers as a set of learnable parameters and achieve global structure preservation in a faster, more efﬁcient, and easier to optimize manner by applying ranking loss to the clustering centers.
• Analyzing the contradiction between two optimization goals of clustering and local structure preservation, and proposing an elegant training strategy to alleviate it.
• The proposed DCRL algorithm outperforms competing algorithms in terms of clustering effect, generalizability to out-of-sample, and performance in downstream tasks.
R ELATED W ORK
Clustering analysis .
As a fundamental tool in machine learning, it has been widely applied in various domains.
One branch of classical clustering is K -Means (MacQueen, 1965) and Gaussian
Mixture Models (GMM) (Bishop, 2006), which are fast, easy to understand, and can be applied to a large number of problems.
However, limited by Euclidean measure, their performance on high-dimensional data is often unsatisfactory.
Spectral clustering and its variants (such as SC-Ncut (Bishop, 2006)) extend clustering to high-dimensional data by allowing more ﬂexible distance measures.
However, limited by computational efﬁciency of the full Laplace matrix, spectral clustering is challenging to extend to large-scale datasets.
Deep clustering .
The success of deep learning has contributed to the growth of deep clustering .
One branch of deep clustering performs clustering after learning a representation through existing unsupervised techniques.
For example, Tian et al.
(2014) use autoencoder to learn low dimensional features and then run K -Means to get clustering results (AE+ K -Means).
Considering the geometric structure of the data, N2D applies UMAP to ﬁnd the best clusterable manifold of the obtained embedding, and then run K -Means to discover higher-quality clusters (McConville et al., 2019).
The other category of algorithms tries to optimize clustering and representation learning jointly.
The closest work to us is Deep Embedding for Clustering (DEC)
(Xie et al., 2016), which learns a mapping from the input space to a lower-dimensional latent space through iteratively optimizing a clustering objective.
As a modiﬁed version of DEC, while IDEC claims to preserve the local structure of the data (Guo et al., 2017), in reality, their contribution is nothing more than adding a reconstruction loss.
JULE proposes a recurrent framework for integrating clustering and representation learning into a single model with a weighted triplet loss and optimizing it end-to-end (Yang et al., 2016b).
DSC devises a dual autoencoder to embed data into latent space, and then deep spectral clustering (Shaham et al., 2018) is applied to obtain label assignments (Yang et al., 2019).
Manifold Representation Learning .
Isomap, as a representative algorithm of single-manifold learning, aims to capture global nonlinear features and seek an optimal subspace that best preserves the geodesic distance between data points (Tenenbaum et al., 2000).
In contrast, some algorithms, such as the LLE (Roweis & Saul, 2000), are more concerned with the preservation of local neighborhood information.
Combining DNN with manifold learning, the recently proposed
MLDL algorithm achieves the preservation of local and global geometries by imposing LIS prior constraints (Li et al., 2020).
Furthermore, multi-manifold learning is proposed to obtain intrinsic properties of different manifolds.
Yang et al.
(2016a) proposed a supervised MMD-Isomap where data points are partitioned into different manifolds according to label information.
Similarly, Zhang et al.
(2018) proposed a semi-supervised local multi-manifold learning framework, termed SSMMIsomap, that applies the labeled and unlabeled training samples to perform the joint learning of local neighborhood-preserving features.
In most previous work on multi-manifold learning, the problem is considered from the perspective that the label is known or partially known, which signiﬁcantly simpliﬁes the problem.
For unsupervised multi-manifold learning, it is still very challenging to decouple multiple overlapping manifolds, and that is exactly what this paper aims to explore.
P ROPOSED M ETHOD
Consider a dataset X with N samples, and each sample x i ∈ R
d is sampled from C different manifolds { M c } Cc =1 .
Assume that each category in the data set lies in a compact low-dimensional manifold, and the number of manifolds C is prior knowledge.
Deﬁne two nonlinear mapping z
i
= f
( x i , θ f ) and y i =
g ( z i , θ g ) , where z i ∈ R m is the embedding of x i in the latent space, y i is the reconstruction of x i .
The j -th cluster center is denoted as µ j ∈ R m , where { µ j } Cj =1 is deﬁned as a set of learnable parameters.
We aim to ﬁnd optimal parameters θ f
so that the embedding features { z i } Ni =1 can achieve clustering with local and global structure preservation.
To this end, a denoising autoencoder (Vincent et al., 2010) shown in Fig 1 is ﬁrst pre-trained in an unsupervised manner to learn an initial latent space.
Denoising autoencoder aims to optimize the self-reconstruction loss L AE =
MSE (ˆ x, y ) , where the ˆ x is a copy of x with Gaussian noise added, that is, ˆ x = x + N (0 , σ 2 ) .
Then the autoencoder is ﬁnetuned by optimizing the following clustering-oriented loss { L cluster ( z, µ ) } and structure-oriented losses { L rank ( x, µ ) , L LIS ( x, z ) , L align ( z, µ ) } .
Since the clustering should be performed on features of clean data, instead of noised data ˆ x that is used in denoising autoencoder, the clean data x is used for ﬁne-tuning.
[Figure]
3. C LUSTERING - ORIENTED L OSS
First, the cluster centers { µ j } Cj =1 in the latent space Z are initialized (the initialization method will be introduced in Sec 4.1).
Then the similarity between the embedded point z i and cluster centers { µ j } Cj =1 is measured by Student’s t -distribution:
[Equation]
The auxiliary target distribution is designed to help manipulate the latent space, deﬁned as:
[Equation]
where f j = (cid:88) i q ij (2) where f j is the normalized cluster frequency, used to balance the size of different clusters.
Then the encoder is optimized by the following objective:
[Equation]
The gradient of L cluster with respect to each learnable cluster center µ j can be computed as:
[Equation]
L cluster facilitates the aggregation of data points within the same manifold, while data points from different manifolds are kept away from each other.
However, we ﬁnd that the clustering-oriented loss may deteriorate the geometric structure of the latent space, which hurts the clustering accuracy and leads to meaningless representation.
To prevent the deterioration of clustering loss, we introduce isometry loss L LIS and ranking loss L rank to preserve the local and global structure, respectively.
3. S TRUCTURE - ORIENTED L OSS
Intra-manifold Isometry Loss.
The intra-manifold local structure is preserved by optimizing the following objective:
[Equation]
where N Z i represents the neighborhood of data point z i in the feature space Z , and the k NN is applied to determine the neighborhood.
π ( · ) ∈ { 0 , 1 } is an indicator function, and l ( x i ) is a manifold determination function that returns the manifold s i where sample x i is located, that is s i
= l ( x i ) =
arg max j p ij .
Then we can derive C manifolds { M c } Cc =1 : M c = { x i ; s i
=
c, i = 1 , 2 , ..., N } .
In a nutshell, the loss L LIS constrains the isometry within each manifold.
Inter-manifold Ranking Loss.
The inter-manifold global structure is preserved by optimizing the following objective:
[Equation]
where { v Xj } Cj =1 is deﬁned as the centers of different manifolds in the original input space X with v Xj = 1 | M j | (cid:80) i ∈
M j x i ( j = 1 , 2 , ..., C ).
The parameter scale determines the extent to which different manifolds move away from each other.
The larger scale is, the further away the different manifolds are from each other.
The derivation for the gradient of L rank with respect to each learnable cluster center µ j is placed in Appendix A.1 .
Additionally, contrary to us, the conventional methods for dealing with inter-manifold separation typically impose push-away constraints on all data points from different manifolds (Zhang et al., 2018;
Yang et al., 2016a), deﬁned as:
[Equation]
The main differences between L rank and L sep are as follows: (1) L sep imposes constraints on embedding points { z i } Ni =1 , which in turn indirectly affects the network parameters θ f .
In contrast,
L rank imposes rank-preservation constrains directly on learnable parameters { µ j } Cj =1 in the form of regularization item to control the separation of the clustering centers.
(2) L sep involves N × N pointto-point relationships, while L rank involves only C × C cluster-to-cluster relationships, so L rank is easier to optimize, faster to process, and more accurate.
The parameter scale introduced in L rank allows us to control the extent of separation between manifolds for speciﬁc downstream tasks.
Alignment Loss.
Note that the global ranking loss L rank is imposed directly on the learnable parameter { µ j } Cj =1 , so optimizing L rank will only update { µ j } Cj =1 rather the encoder’s parameter θ f .
Thus here we need to introduce an auxiliary item L align to align learnable cluster centers { µ j } C j =1 with real cluster centers { v Z j } C j =1 :
[Equation]
where { v Zj } Cj =1 are deﬁned as v Zj =
[Equation]
j |
[Equation]
i ∈ M j z i ( j = 1 , 2 , ..., C ).
We place the derivation for the gradient of L align with respect to each learnable cluster center µ j in Appendix A.1 .
3. T RAINING S TRATEGY 3.3. C ONTRADICTION
The contradiction between clustering and local structure preservation is analyzed from the forces analysis perspective.
As shown in Fig 2, we assume that there exists a data point (red point) and its three nearest neighbors (blue points) around a cluster center (gray point).
When clustering and local structure preserving are optimized simultaneously, it is very easy to fall into a local optimum, where the data point is in steady-state, and the resultant force from its three nearest neighbors is equal in magnitude and opposite to the gravitational forces of the cluster.
Therefore, the following training strategy is applied to prevent such local optimal solutions.
3.3. A LTERNATING T RAINING AND W EIGHT G RADUALITY
Alternating Training.
To solve the above problem and integrate the goals of clustering and structure preservation into a uniﬁed framework, we take an alternating training strategy.
Within each epoch, we ﬁrst jointly optimize L cluster and L rank in a mini-batch , with joint loss deﬁned as
[Equation]
where α is the weighting factor that balances the effects of clustering and global rank-preservation.
Then at each epoch, we optimize isometry loss L LIS and L align on the whole dataset , deﬁned as
[Equation]
Weight Graduality.
At different stages of training, we have different expectations for the clustering and structure-preserving.
At the beginning of training, to successfully decouple the overlapping manifolds, we hope that the L cluster will dominate and L LIS will be auxiliary.
When the margin between different manifolds is sufﬁciently pronounced, the weight α for L cluster can be gradually reduced, while the weight β for L LIS can be gradually increased, focusing on the preservation of the local isometry.
The whole algorithm is summarized in Algorithm 1 in Appendix A.2 .
Three-stage explanation.
The entire training process can be roughly divided into three stages, as shown in Fig 3, to explain the training strategy more vividly.
At ﬁrst, four different manifolds overlap each other.
At Stage 1, L cluster dominates, thus data points within each manifold are converging towards the clustering center to form a sphere, and the local structure of manifolds is destroyed.
At
Stage 2, L rank dominates, thus different manifolds in the latent space move away from each other to increase the manifold margin and enhance the discriminability.
At stage 3, the manifolds gradually recover their original local structure from the spherical shape with L LIS dominating.
It is worth noting that all of the above losses coexist rather than independently at different stages , but that the role played by different losses varies due to the alternating training and weight graduality.
E XPERIMENTS 4. E XPERIMENTAL SETUPS
In this section, the effectiveness of the proposed framework is evaluated in 5 benchmark datasets:
USPS 1 , MNIST-full, MNIST-test (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017) and
REUTERS-10K (Lewis et al., 2004), on which our method is compared with 8 other methods mentioned in Sec 2 in 8 evaluation metrics including metrics designed speciﬁcally for clustering and representation learning.
The brief descriptions of the datasets are given in Appendix A.3 .
Parameters settings.
The encoder is a multilayer perceptron (MLP) with dimensions d -500-5002000-10 where d is the dimension of the input data, and the decoder is its mirror.
After pretraining, in order to initialize the learnable clustering centers, the t-SNE is applied to transform the latent space
Z to 2 dimensions further, and then the K -Means algorithm is run to obtain the label assignments for each data point.
The centers of each category in the latent space Z are set as initial clustering centers { µ j } Cj =1 .
The batch size is set to 256, the epoch is set to 300, the parameter k for nearest neighbor is set to 3, and the parameter scale is set to 3 for all datasets.
Besides, Adam optimizer (Kingma & Ba, 2014) with learning rate λ =0.001 is used.
As described in Sec 3.3.2, the weight graduality is applied to train the model.
The weight parameter α for L cluster decreases linearly from 0.1 to 0 within epoch 0-150.
In contrast, the weight parameter β for loss L LIS increases linearly fromto 1.0 within epoch 0-150.
The implementation is based on the PyTorch running on NVIDIA v
GPU.
Evaluation Metrics.
Two standard evaluation metrics: Accuracy (ACC) and Normalized Mutual
Information (NMI) (Xu et al., 2003) are used to evaluate clustering performance.
Besides, six evaluation metrics are adopted in this paper to evaluate the performance of representation learning, including Relative Rank Error (RRE), Trustworthiness (Trust), Continuity (Cont), Root Mean Reconstruction Error (RMRE), Locally Geometric Distortion (LGD) and Cluster Rank Accuracy (CRA).
Limited by space, their precise deﬁnitions are available in Appendix A.4 .
4. E VALUATION OF C LUSTERING 4.2. Q UANTITATIVE C OMPARISON
The metrics ACC/NMI of different methods on various datasets are reported in Tab 1.
For those comparison methods whose results are not reported on some datasets, we run the released code using the hyperparameters provided in their paper and label them with (*).
We ﬁnd that our method outperforms K -Means and SC-Ncut with a signiﬁcant margin and surpasses the other six competing
DNN-based algorithms on all datasets except MNIST-test.
With even the MNIST-test dataset, we still rank second, outperforming the third by 1.1%.
In particular, we obtained the best performance on the Fashion-MNIST dataset and, more notably, our clustering accuracy exceeds the current best method (N2D) by 3.8%.
Despite L cluster is inspired by and highly consistent with the design of
DEC, our method achieves much better clustering results than them.
With MNIST-full, for example, our clustering accuracy is 11.7% and 9.9% higher than DEC and IDEC, respectively.
[Table]
4.2. G ENERALIZABILITY EVALUATION
[Table]
Tab 2 demonstrates that a learned DCRL can generalize well to unseen data with high clustering accuracy.
Taking
MNIST-full as an example, DCRL was trained using 50,000 training samples and then tested on the remaining 20, testing samples using the learned model.
In terms of the metrics ACC and MNI, our method is optimal for both training and test samples.
More importantly, there is hardly any degradation in the performance of our method on the test samples compared to the training samples, while all other methods showed a signiﬁcant drop in performance, e.g., DEC from 84.1% to 74.8%.
This demonstrates the importance of geometric structure maintenance for good generalizability.
The testing visualization available in Appendix A.5 shows that DCRL still maintains clear inter-cluster boundaries even on the test samples, which demonstrates the great generalizability of our method.
4.2. C LUSTERING V ISUALIZATION
The visualization of DCRL with several comparison methods is shown in Fig 4 (visualized using
UMAP).
From the perspective of clustering, our method is much better than the other methods.
Among all methods, only DEC, IDEC and DCRL can hold clear boundaries between different clusters, while the cluster boundaries of the other methods are indistinguishable.
Although DEC and
IDEC can successfully separate different clusters, they group many data points from different classes into the same cluster.
Most importantly, due to the use of the clustering-oriented loss, the embedding learned by algorithms such as DEC, IDEC, JULE, and DSC (especially DSC) tend to form spheres and disrupt the original topological structure.
Instead, our method overcomes these problems and achieves almost perfect separation between different clusters while preserving the local and global structure.
Additionally, the embedding of latent space during training process is visualized in Appendix A.6 , which is highly consistent with the three-stage explanation mentioned in Sec 3.3.2.
[Figure]
4. E VALUATION OF R EPRESENTATION L EARNING 4.3. Q UANTITATIVE C OMPARISON
Although numerous previous work has claimed that they brought clustering and representation learning into a uniﬁed framework, they all, unfortunately, lack an analysis of the effectiveness of the learned representations.
In this paper, we compare DCRL with the other ﬁve methods in six evaluation metrics on ﬁve datasets.
(Limited by space, only MNIST-full results are provided in the Taband the complete results are in Appendix A.7 ).
The results show that DCRL outperforms all other methods, especially in the CRA metric, which is not only the best on all datasets but also reaches 1.0.
This means that the “rank” between different manifolds in the latent space is completely preserved and undamaged, which proves the effectiveness of our global ranking loss L rank .
Moreover, statistical analysis is performed in this paper to show the extent to which local and global structure is preserved in the latent space for each algorithm.
Limited by space, they are placed in Appendix A.8 .
[Table] [Table]
4.3. D OWNSTREAM TASKS
Recently
[Table]
numerous deep clustering algorithms have claimed to obtain meaningful representations, however, they do not analyze and experiment with the so-called “meaningful”.
Therefore, we are interested to see whether these proposed methods can indeed learn representations that are useful for downstream tasks.
Tab 4 compares DCRL with the other six methods on ﬁve datasets (Limited by space, only MNIST-full results are provided in the paper, and the complete results are in Appendix
A.9 ).
Four different classiﬁers, including a linear classiﬁer (Logistic Regression; LR), two nonlinear classiﬁers (MLP, SVM), and a tree-based classiﬁer (Random Forest Classiﬁer; RFC) are used as downstream tasks, all of which use default parameters and default implementations in sklearn (Pedregosa et al., 2011) for a fair comparison.
The learned representations are frozen and used as input for training.
The classiﬁcation accuracy evaluated on the test set serves as a metric to evaluate the effectiveness of learned representations.
On the MNIST-full dataset, our method outperforms all the other methods.
Moreover, we surprisingly ﬁnd that with MLP and RFC as downstream tasks, all methods except DCRL could not even match the accuracy of AE.
Signiﬁcantly, the performance of
DEC on downstream tasks deteriorates sharply and even shows a large gap with the simplest AEs, which once again shows that the clustering-oriented loss may damage the data geometric structure.
4. A BLATION S TUDY
This evaluates the effects of the loss terms and training strategies in the DCRL with ﬁve sets of experiments: the model without (A) Structure-oriented Loss (SL); (B) Clustering-oriented Loss (CL); (C) Weight Graduality (WG); (D)
Alternating Training (AT), and (E) the full model.
Limited by space, only MNIST-full results are provided in the paper, and results for the other four datasets are in Appendix A.10 .
After analyzing the results, we can conclude: (1) CL is the most important factor for obtaining good clustering, the lack of which leads to unsuccessful clustering, hence the numbers in the table are not very meaningful and are shown in gray color.
(2) SL not only brings subtle improvements in clustering but also greatly improves the performance of representation learning.
(3) Our elegant training strategies (WG and AT) both improve the performance of clustering and representation learning to some extent, especially on metrics such as RRE, Trust, Cont, and CRA.
[Table]
C ONCLUSION
The proposed DCRL framework imposes clustering-oriented and structure-oriented constraints to optimize the latent space for simultaneously performing clustering and representation learning with local and global structure preservation.
Extensive experiments on image and text datasets demonstrate that DCRL is not only comparable to the state-of-the-art deep clustering algorithms but also able to learn effective and robust representation, which is beyond the capability of those clustering methods that only care about clustering accuracy .
Future work will focus on the adaptive determination of manifolds (clusters) number and extend our work to datasets with larger scale
[Table]
A PPENDIX A.1 G RADIENT D ERIVATION
In the paper, we have emphasized time and again that { µ j } Cj =1 is a set of learnalbe parameters, which means that we can optimize it while optimizing the network parameter θ f .
In Eq.
(4) of the paper, we have presented the gradient of L cluster with respect to µ j .
In addition to L cluster , both
L rank and L align are involving µ j .
Hence, the detailed derivations for the gradient of L rank and
L align with respect to µ j are also provided.
The gradient of L rank with respect to each learnalbe cluster center µ j can be computed as:
[Equation]
The Euclidean metric is used for both the input space and the hidden layer space, i.e., d Z ( µ i (cid:48) , µ j (cid:48) )
= (cid:107) µ
i (cid:48) − µ j (cid:48)
(cid:107) .
In addition, the symbols are somewhat abused for clear derivation, representing scale ∗ d X (cid:0) v Xi (cid:48) , v Xj (cid:48) (cid:1) with K .
Accordingly, Eq.
(11) can be further derived as follows:
[Equation]
µ j − µ j (cid:48) (cid:107) − K (cid:12)(cid:12) (cid:107) µ j − µ j (cid:48) (cid:107) − K (cid:12)(cid:12) =
C (cid:88) i (cid:48)
=
∂ (cid:107) µ i (cid:48) − µ j (cid:107) ∂µ j · (cid:107) µ i (cid:48)
− µ j (cid:107) − K
(cid:12)(cid:12) (cid:107) µ i (cid:48)
− µ j (cid:107) − K
(cid:12)(cid:12)
+
C (cid:88) j (cid:48)
=
∂ (cid:107) µ j − µ j (cid:48)
(cid:107) ∂µ j · (cid:107) µ j − µ j (cid:48) (cid:107) − K
(cid:12)(cid:12) (cid:107) µ j − µ j (cid:48)
(cid:107) − K
cid:12)(cid:12) =
C (cid:88) i (cid:48)
=
µ j − µ i (cid:48) (cid:107) µ j − µ
i (cid:48) (cid:107) · (cid:107) µ j − µ i (cid:48) (cid:107) − K (cid:12)(cid:12) (cid:107) µ j − µ i (cid:48) (cid:107) − K (cid:12)(cid:12) +
C (cid:88) j (cid:48)
=
µ j − µ j (cid:48) (cid:107) µ j − µ j (cid:48) (cid:107) · (cid:107) µ j − µ j (cid:48) (cid:107) − K (cid:12)(cid:12) (cid:107) µ j − µ j (cid:48) (cid:107) − K (cid:12)(cid:12) =
C (cid:88) i (cid:48)
=
µ j − µ i (cid:48) (cid:107) µ j − µ
i (cid:48) (cid:107) · (cid:107) µ j − µ i (cid:48) (cid:107) − K (cid:12)(cid:12) (cid:107) µ j − µ i (cid:48) (cid:107) − K (cid:12)(cid:12) =
C (cid:88) i (cid:48)
=
µ j − µ i (cid:48) (cid:107) µ j − µ
i (cid:48) (cid:107) · (cid:107) µ j − µ i (cid:48) (cid:107) −
[Table]
v Xi (cid:48) , v Xj (cid:1) (cid:12)(cid:12) (cid:107) µ j − µ i (cid:48) (cid:107) −
[Table]
d X (cid:0) v Xi (cid:48) , v Xj (cid:1) (cid:12)(cid:12) =
C (cid:88) i (cid:48)
=
µ j − µ i (cid:48)
d Z ( µ j , µ i (cid:48) ) ·
d Z ( µ j , µ i (cid:48) ) − scale ∗ d X (cid:0) v Xi (cid:48) , v Xj (cid:1) (cid:12)(cid:12)
d Z ( µ j , µ i (cid:48) )
− scale ∗ d X (cid:0) v X i (cid:48) , v X j (cid:1)(cid:12)(cid:12)
The gradient of L align with respect to each learnalbe cluster center µ j can be computed as: ∂L align
[Equation]
A.2 A LGORITHM
A.3 D ATASETS
To show that our method works well with various kinds of datasets, we choose the following ﬁve image and text datasets.
Some example images are shown in Fig A1, and the brief descriptions of the datasets are given in Tab A1.
• MNIST-full (LeCun et al., 1998):
The MNIST-full dataset consists of 70,000 handwritten digits of 28 × 28 pixels.
Each gray image is reshaped to a 784-dimensional vector.
• MNIST-test (LeCun et al., 1998):
The MNIST-test is the testing part of the MNIST dataset, which contains a total of 10000 samples.
• USPS 2 : The USPS dataset is composed of 9298 gray-scale handwritten digit images with a size of 16x16 pixels.
[Table]
• Fashion-MNIST
(Xiao et al., 2017):
This Fashion-MNIST dataset has the same number of images and the same image size as MNIST-full, but it is fairly more complicated.
Instead of digits, it consists of various types of fashion products.
REUTERS-10K: REUTERS (Lewis et al., 2004) is composed of around 810000 English news stories labeled with a category tree.
Four root categories (corporate/industrial, government/social, markets, and economics) are used as labels and excluded all documents with multiple labels.
Following DEC (Xie et al., 2016), a subset of 10000 examples are randomly sampled, and the tf-idf features on the 2000 most frequent words are computed.
The sampled dataset is denoted REUTERS-10K.
[Figure]
A.4 D EFINITIONS OF PERFORMANCE METRICS
The following notations are used for the deﬁnitions:
d X ( i, j ) : the pairwise distance between x i and x j in input space X ; d Z ( i, j ) : the pairwise distance between z i and z j in latent space Z ; N k,Xi : the set of indices to the k -nearest neighbor ( k NN) of x i in input space X ; N k,Zi : the set of indices to the k -nearest neighbor ( k NN) of z i in latent space Z ; r X ( i, j ) : the rank of the closeness of x j to x i in input space X ; r Z ( i, j ) : the rank of the closeness of z j
to z i in latent space Z .
The eight evaluation metrics are deﬁned below:
(1) ACC (Accuracy) measures the accuracy of clustering:
[Equation]
N
where l i and s i are the true and predicted labels for data point x i , respectively, and m ( · ) is all possible one-to-one mappings between clusters and label categories.
(2) NMI (Normalized Mutual Information) NMI calculates the normalized measure of similarity between two labels of the same data
[Equation]
where I ( l, s ) is the mutual information between the real label l and predicted label s , and
H ( · ) represents their entropy.
(3) RRE (Relative Rank Change) measures the average of changes in neighbor ranking between two spaces X and Z :
[Equation]
where k 1 and k 2 are the lower and upper bounds of the k -NN.
[Equation]
where H k is the normalizing term, deﬁned as
[Equation]
(4) Trust (Trustworthiness) measures to what extent the k nearest neighbors of a point are preserved when going from the input space to the latent space:
[Equation]

[Equation]
where k 1 and k 2 are the bounds of the number of nearest neighbors.
(5) Cont (Continuity
) is deﬁned analogously to Trust , but checks to what extent neighbors are preserved when going from the latent space to the input space:
[Equation]
  where k 1 and k 2 are the bounds of the number of nearest neighbors.
(6) RMSE (Root Mean Square Error)
measures to
what extent the two
distributions of
dis
- tances coincide:
[Equation]
(7) LGD (Locally Geometric Distortion) measures how much corresponding distances between neighboring points differ in two metric spaces and is the primary metric for isometry, deﬁned as:
[Equation]
where k 1 and k 2 are the lower and upper bounds of the k -NN.
(8) CRA (Cluster Rank Accuracy) measures the changes in ranks of cluster centers from the input space X and to the latent space Z :
[Equation]
where C is the number of clusters, v X j is the cluster center of the j th cluster in the input space X ,
v Zj is the cluster center of the j th cluster in the latent space Z , r X ( v Xi , v Xj ) denotes the rank of the closeness of v Xi to v Xj in space X in the input space X , and r Z ( v Zi , v Zj ) denotes the rank of the closeness of v Zi to v Zj in space Z .
A.5 V ISUALIZATION IN GENERALIZABILITY
The visualization results on the testing samples are shown in Fig A2; even for testing samples, our method still shows distinguishable inter-cluster discriminability, while all the other methods without exception coupled different clusters together.
[Figure]
different algorithms on MNIST-full dateset.
A.6 V ISUALIZATION IN DIFFERENT STAGES
The embedding visualization of the latent space during the training process is visualized in Fig A for depicting how both clustering and structure-preserving is achieved.
We can see that the different clusters initialized by pretrained autoencoder are closely adjacent.
In the early stage of training, with clustering loss L cluster and global ranking loss L rank , different manifolds are separated from each other, each manifold loses its local structure, and all of them degenerate into spheres.
As the training progresses, the weight α for L cluster gradually decreases, while the weight β for L iso increases and the optimization is gradually focused from global to local , with each manifold gradually recovering its original geometric structure from the sphere.
Moreover, since our local isometry loss L iso is constrained within each manifold, the preservation of local structure will not disrupt the global ranking.
Finally, we obtain representations in which cluster boundaries are clearly distinguished, and local and global structures are perfectly preserved.
[Figure]
[Figure]
A.7 S TATISTICAL A NALYSIS
The statistical analysis is presented to show the extent to which local and global structure is preserved from the input space to the latent space.
Taking MNIST-full as an example, the statistical analysis of the global rank-preservation is shown in Fig A4 (a)-(f).
For the i -th cluster, if the rank between it and the j -th cluster is preserved from input space to latent space, then the grid in the i -th row and j -th column is set to blue, otherwise yellow.
As shown in the ﬁgure, only our method can fully preserve the global rank between different clusters, while all other methods fail.
Finally, we perform a statistical analysis for the local isometry property of each algorithm.
For each sample x i in the dataset, it forms a number of point pairs with its neighborhood samples { ( x i , x j ) | i = 1 , 2 , ..., N ; x j ∈ N Xi } .
We compute the difference in the distance of these point pairs from the input space to the latent space { d Z
( x i , x j ) − d X ( x i , x j ) | i = 1 , 2 , ..., N ; x j ∈ N i } , and plot it as a histogram.
As shown in Fig A4 (g), the curves of DCRL are distributed on both sides of the 0 value, with maximum peak height and minimum peak-bottom width, respectively, which indicates that DCRL achieves the best local isometry.
Although IDEC claims that they can preserve the local structure well, there is still a big gap between their results and ours.
A.8 Q UANTITATIVE E VALUATION OF R EPRESENTATION L EARINING
Our method is compared with the other ﬁve methods in six evaluation metrics on ﬁve datasets.
The complete results in Tab A2 demonstrate the superiority of our method, especially on metrics RRE,
Trust, Cont, and CRA.
[Table]
A.9 Q UANTITATIVE E VALUATION OF D OWNSTREAM T ASKS
Tab A3 compares DCRL with the other six methods on ﬁve datasets to see whether these methods can indeed learn representations that are useful for downstream tasks.
As shown in the table, our method outperforms the other methods on all ﬁve datasets with MLP, RFC, LR as downstream tasks.
[Table]
A.10 M ORE ABLATION EXPERIMENTS
The results of the ablation experiments on the MNIST-full dataset have been presented in Tabin Sec 4.3.
Here, we provide four more sets of ablation experiments on the other four datasets.
The conclusion is similar (note that for the clustering performance of the model without clusteringoriented losses is very poorly, so the “best” metric numbers are not meaningful and are shown in gray color): (1) CL is very important for obtaining good clustering.
(2) SL is beneﬁcial for both clustering and representation learning.
(3) Our training strategies (WG and AT) are very superior in improving metrics such as ACC, RRE, Trust, Cont
[Table]
and CRA.
[Table]