D EEP C ONTINUOUS C
LUSTERING
Anonymous authors
Paper under double-blind review
A BSTRACT
Clustering high-dimensional datasets is hard because interpoint distances become less informative in high-dimensional spaces.
We present a clustering algorithm that performs nonlinear dimensionality reduction and clustering jointly.
The data is embedded into a lower-dimensional space by a deep autoencoder.
The autoencoder is optimized as part of the clustering process.
The resulting network produces clustered data.
The presented approach does not rely on prior knowledge of the number of ground-truth clusters.
Joint nonlinear dimensionality reduction and clustering are formulated as optimization of a global continuous objective.
We thus avoid discrete reconﬁgurations of the objective that characterize prior clustering algorithms.
Experiments on datasets from multiple domains demonstrate that the presented algorithm outperforms state-of-the-art clustering schemes, including recent methods that use deep networks.
I NTRODUCTION
Clustering is a fundamental procedure in machine learning and data analysis.
Well-known approaches include center-based methods and their generalizations (Banerjee et al., 2005; Teboulle, 2007), and spectral methods (Ng et al., 2001; von Luxburg, 2007).
Despite decades of progress, reliable clustering of noisy high-dimensional datasets remains an open problem.
High dimensionality poses a particular challenge because assumptions made by many algorithms break down in high-dimensional spaces (Ball, 1997; Beyer et al., 1999; Steinbach et al., 2004).
There are techniques that reduce the dimensionality of data by embedding it in a lower-dimensional space (van der
Maaten et al., 2009).
Such general techniques, based on preserving variance or dissimilarity, may not be optimal when the goal is to discover cluster structure.
Dedicated algorithms have been developed that combine dimensionality reduction and clustering by ﬁtting low-dimensional subspaces (Kriegel et al., 2009; Vidal, 2011).
Such algorithms can achieve better results than pipelines that ﬁrst apply generic dimensionality reduction and then cluster in the reduced space.
However, frameworks such as subspace clustering and projected clustering operate on linear subspaces and are therefore limited in their ability to handle datasets that lie on nonlinear manifolds.
Recent approaches have sought to overcome this limitation by constructing a nonlinear embedding of the data into a low-dimensional space in which it is clustered (Dizaji et al., 2017;
Xie et al., 2016;
Yang et al., 2016; 2017).
Ultimately, the goal is to perform nonlinear embedding and clustering jointly, such that the embedding is optimized to bring out the latent cluster structure.
These works have achieved impressive results.
Nevertheless, they are based on classic center-based, divergencebased, or hierarchical clustering formulations and thus inherit some limitations from these classic methods.
In particular, these algorithms require setting the number of clusters a priori.
And the optimization procedures they employ involve discrete reconﬁgurations of the objective, such as discrete reassignments of datapoints to centroids or merging of putative clusters in an agglomerative procedure.
Thus it is challenging to integrate them with an optimization procedure that modiﬁes the embedding of the data itself.
We seek a procedure for joint nonlinear embedding and clustering that overcomes some of the limitations of prior formulations.
There are a number of characteristics we consider desirable.
First, we wish to express the joint problem as optimization of a single continuous objective.
Second, this optimization should be amenable to scalable gradient-based solvers such as modern variants of SGD.
Third, the formulation should not require setting the number of clusters a priori, since this number is often not known in advance.
While any one of these desiderata can be fulﬁlled by some existing approaches, the combination is challenging.
For example, it has long been known that the k -means objective can be optimized by SGD (Bottou & Bengio, 1994).
But this family of formulations requires positing the number of clusters k in advance.
Furthermore, the optimization is punctuated by discrete reassignments of datapoints to centroids, and is thus hard to integrate with continuous embedding of the data.
In this paper, we present a formulation for joint nonlinear embedding and clustering that possesses all of the aforementioned desirable characteristics.
Our approach is rooted in Robust Continuous
Clustering (RCC), a recent formulation of clustering as continuous optimization of a robust objective (Shah & Koltun, 2017).
The basic RCC formulation has the characteristics we seek, such as a clear continuous objective and no prior knowledge of the number of clusters.
However, integrating it with deep nonlinear embedding is still a challenge.
For example, Shah & Koltun (2017) presented a formulation for joint linear embedding and clustering (RCC-DR), but this formulation relies on a complex alternating optimization scheme with linear least-squares subproblems, and does not apply to nonlinear embeddings.
We present an integration of the RCC objective with dimensionality reduction that is simpler and more direct than RCC-DR, while naturally handling deep nonlinear embeddings.
Our formulation avoids alternating optimization and the introduction of auxiliary dual variables.
A deep nonlinear embedding of the data into a low-dimensional space is optimized while the data is clustered in the reduced space.
The optimization is expressed by a global continuous objective and conducted by standard gradient-based solvers.
The presented algorithm is evaluated on high-dimensional datasets of images and documents.
Experiments demonstrate that our formulation performs on par or better than state-of-the-art clustering algorithms across all datasets.
This includes recent approaches that utilize deep networks and rely on prior knowledge of the number of ground-truth clusters.
Controlled experiments conﬁrm that joint dimensionality reduction and clustering is more effective than a stagewise approach, and that the high accuracy achieved by the presented algorithm is stable across different dimensionalities of the latent space.
P RELIMINARIES
Let X =
[ x 1 , . . .
, x N ] be a set of points in R D that must be clustered.
Generic clustering algorithms that operate directly on X rely strongly on interpoint distances.
When D is high, these distances become less informative (Ball, 1997; Beyer et al., 1999).
Hence most clustering algorithms do not operate effectively in high-dimensional spaces.
To overcome this problem, we embed the data into a lower-dimensional space R d .
The embedding of the dataset into R d is denoted by Y =
[ y 1 , . . .
, y N ] .
The function that performs the embedding is denoted by f θ : R D
→ R d .
Thus y i
= f θ ( x i ) for all i .
Our goal is to cluster the embedded dataset Y and to optimize the parameters θ of the embedding as part of the clustering process.
This formulation presents an obvious difﬁculty: if the embedding f θ can be manipulated to assist the clustering of the embedded dataset Y , there is nothing that prevents f θ from distorting the dataset such that Y no longer respects the structure of the original data.
We must therefore introduce a regularizer on θ that constrains the low-dimensional image Y with respect to the original high-dimensional dataset X .
To this end, we also consider a reverse mapping g ω : R d → R D .
To constrain f θ to construct a faithful embedding of the original data, we require that the original data be reproducible from its low-dimensional image (Hinton & Salakhutdinov, 2006): minimize Ω
[Equation]
where Y = F θ ( X ) , Ω = { θ , ω } .
Here F θ ( X ) =
[ f θ ( x 1 ) , . .
, f θ ( x N )] , G ω ( Y ) =
[ g ω ( y 1 ) , . . .
, g ω ( y N )] , and (cid:107)·(cid:107) F denotes the
Frobenius norm.
Next, we must decide how the low-dimensional embedding Y will be clustered.
A natural solution is to choose a classic clustering framework: a center-based method such as k -means, a divergence-based formulation, or an agglomerative approach.
These are the paths taken in recent work on combining nonlinear dimensionality reduction and clustering (Dizaji et al., 2017; Xie et al., 2016; Yang et al., 2016; 2017).
However, the classic clustering algorithms have a discrete structure: associations between centroids and datapoints need to be recomputed or putative clusters need to be merged.
In either case, the optimization process is punctuated by discrete reconﬁgurations.
This makes it difﬁcult to coordinate the clustering of Y with the optimization of the embedding parameters Ω that modify the dataset Y itself.
Since we must conduct clustering in tandem with continuous optimization of the embedding, we seek a clustering algorithm that is inherently continuous and performs clustering by optimizing a continuous objective that does not need to be updated during the optimization.
The recent RCC formulation provides a suitable starting point (Shah & Koltun, 2017).
The key idea of RCC is to introduce a set of representatives
Z ∈ R
d × N and optimize the following nonconvex objective:
[Equation]
where ρ is a redescending M-estimator, E is a graph connecting the datapoints, { w i,j } are appropriately deﬁned weights, and λ is a coefﬁcient that balances the two objective terms.
The ﬁrst term in objective (2) constrains the representatives to remain near the corresponding datapoints.
The second term pulls the representatives to each other, encouraging them to merge.
This formulation has a number of advantages.
First, it reduces clustering to optimization of a ﬁxed continuous objective.
Second, each datapoint has its own representative in Z and no prior knowledge of the number of clusters is needed.
Third, the nonconvex robust estimator ρ limits the inﬂuence of outliers.
To perform nonlinear embedding and clustering jointly, we wish to integrate the reconstruction objective (1) and the RCC objective (2).
This idea is developed in the next section.
D EEP C ONTINUOUS C LUSTERING 3. O BJECTIVE
The Deep Continuous Clustering (DCC) algorithm optimizes the following objective:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where Y = F θ ( X ) .
This formulation bears some similarity to RCC-DR (Shah & Koltun, 2017), but differs in three major respects.
First, RCC-DR only operates on a linear embedding deﬁned by a sparse dictionary, while
DCC optimizes a more expressive nonlinear embedding parameterized by Ω .
Second, RCC-DR alternates between optimizing dictionary atoms, sparse codes, representatives Z , and dual line process variables; in contrast, DCC avoids duality altogether and optimizes the global objective directly.
Third, DCC does not rely on closed-form or linear least-squares solutions to subproblems; rather, the joint objective is optimized by modern gradient-based solvers, which are commonly used for deep representation learning and are highly scalable.
We now discuss objective (3) and its optimization in more detail.
The mappings F θ and G ω are performed by an autoencoder with fully-connected or convolutional layers and rectiﬁed linear units after each afﬁne projection (Hinton & Salakhutdinov, 2006; Nair & Hinton, 2010).
The graph E is constructed on X using the mutual kNN criterion (Brito et al., 1997), augmented by the minimum spanning tree of the kNN graph to ensure connectivity to all datapoints.
The role of M-estimators ρ 1 and ρ 2 is to pull the representatives of a true underlying cluster into a single point, while disregarding spurious connections across clusters.
For both estimators, we use scaled Geman-McClure functions (Geman & McClure, 1987): [Equation]
The parameters µ 1 and µ 2 control the radii of the convex basins of the estimators.
The weights w i,j are set to balance the contribution of each datapoint to the pairwise loss:
[Equation]
Here n i is the degree of z i in the graph E .
The numerator is simply the average degree.
The parameter λ balances the relative strength of the data loss and the pairwise loss.
To balance the different terms, we set λ
= (cid:107) Y (cid:107) 2 (cid:107) A (cid:107) 2 , where A = (cid:80) ( i,j ) ∈E w i,j ( e i − e j )
( e i − e j ) (cid:62) and (cid:107) · (cid:107) 2 denotes the spectral norm.
In contrast to RCC-DR, the parameter λ need not be updated during the optimization.
3. O PTIMIZATION
Objective (3) can be optimized using scalable modern forms of stochastic gradient descent (SGD).
Note that each z i is updated only via its corresponding loss and pairwise terms.
On the other hand, the autoencoder parameters Ω are updated via all data samples.
Thus in a single epoch, there is bound to be a difference between the update rates for Z and Ω .
To deal with this imbalance, an adaptive solver such as Adam should be used (Kingma & Ba, 2015).
Another difﬁculty is that the graph E connects all datapoints such that a randomly sampled minibatch is likely to be connected by pairwise terms to datapoints outside the minibatch.
In other words, the objective (3), and more speciﬁcally the pairwise loss, does not trivially decompose over datapoints.
This requires some care in the construction of minibatches.
Instead of sampling datapoints, we sample subsets of edges from E .
The corresponding minibatch B is deﬁned by all nodes incident to the sampled edges.
However, if we simply restrict the objective (3) to the minibatch and take a gradient step, the reconstruction and data terms will be given additional weight since the same datapoint can participate in different minibatches, once for each incident edge.
To maintain balance between the terms, we must weigh the contribution of each datapoint in the minibatch.
The rebalanced minibatch loss is given by
[Equation]
where y i =
f θ ( x i ) ∀ i ∈ B .
[Equation]
Here w i = n B i n i , where n B i is the number of edges connected to the i th node in the subgraph E B .
The gradients of L B with respect to the low-dimensional embedding Y and the representatives Z are given by
[Equation]
These gradients are propagated to the parameters Ω .
3. I NITIALIZATION , C ONTINUATION AND S TOPPING C RITERION
Initialization.
The embedding parameters Ω are initialized using the stacked denoising autoencoder (SDAE) framework (Vincent et al., 2010).
Each pair of corresponding encoding and decoding layers is pretrained in turn.
Noise is introduced during pretraining by adding dropout to the input of each afﬁne projection (Srivastava et al., 2014).
Encoder-decoder layer pairs are pretrained sequentially, from the outer to the inner.
After all layer pairs are pretrained, the entire SDAE is ﬁne-tuned end-to-end using the reconstruction loss.
This completes the initialization of the embedding parameters Ω .
These parameters are used to initialize the representatives Z , which are set to Z = Y =
F θ ( X ) .
Continuation.
The price of robustness is the nonconvexity of the estimators ρ 1 and ρ 2 .
One way to alleviate the dangers of nonconvexity is to use a continuation scheme that gradually sharpens the estimator (Blake & Zisserman, 1987; Mobahi & Fisher III, 2015).
Following Shah & Koltun (2017), we initially set µ i to a high value that makes the estimator
ρ
i effectively convex in the relevant range.
The value of µ i is decreased on a regular schedule until a threshold δ i 2 is reached.
We set δ 1 to the mean of the distance of each y i to the mean of Y , and δ 2 to the mean of the bottom 1% of the pairwise distances in E at initialization.
Stopping criterion.
Once the continuation scheme is completed, DCC monitors the computed clustering.
At the end of every epoch, a graph G =
( V , F ) is constructed such that f i,j = 1 if (cid:107)
z
i − z j (cid:107) <
δ 2 .
The cluster assignment is given by the connected components of G .
DCC compares this cluster assignment to the one produced at the end of the preceding epoch.
If less than 0 .
1% of the edges in E changed from intercluster to intracluster or vice versa, DCC outputs the computed clustering and terminates.
Complete algorithm.
The complete algorithm is summarized in Algorithm 1.
E XPERIMENTS 4. D ATASETS
We conduct experiments on six high-dimensional datasets, which cover domains such as handwritten digits, objects, faces, and text.
We used datasets from Shah & Koltun (2017) that had dimensionality above 100.
The datasets are further described in the appendix.
All features are normalized to the range [0 , 1] .
Note that DCC is an unsupervised learning algorithm.
Unlabelled data is embedded and clustered with no supervision.
There is thus no train/test split.
4. B ASELINES
The presented DCC algorithm is compared to 12 baselines, which include both classic and deep clustering algorithms.
The baselines include k -means++ (Arthur & Vassilvitskii, 2007), DBSCAN (Ester et al., 1996), two variants of agglomerative clustering: Ward (AC-W) and graph degree linkage (GDL) (Zhang et al., 2012), two variants of spectral clustering: spectral embedded clustering (SEC) (Nie et al., 2011) and local discriminant models and global integration (LDMGI)
(Yang et al., 2010), and two variant of robust continuous clustering: RCC and RCC-DR (Shah & Koltun, 2017).
The deep clustering baselines include four recent approaches that share our basic motivation and use deep networks for clustering: deep embedded clustering (DEC) (Xie et al., 2016), joint unsupervised learning (JULE)
(Yang et al., 2016), the deep clustering network (DCN) (Yang et al., 2017), and deep embedded regularized clustering (DEPICT) (Dizaji et al., 2017).
These are strong baselines that use deep autoencoders, the same network structure as our approach (DCC).
The key difference is in the loss function and the consequent optimization procedure.
The prior formulations are built on KLdivergence clustering, agglomerative clustering, and k -means, which involve discrete reconﬁguration of the objective during the optimization and rely on knowledge of the number of ground-truth clusters either in the design of network architecture, during the embedding optimization, or in post-processing.
In contrast, DCC optimizes a robust continuous loss and does not rely on prior knowledge of the number of clusters.
4. I MPLEMENTATION
We report experimental results for two different autoencoder architectures: one with only fullyconnected layers and one with convolutional layers.
This is motivated by prior deep clustering algorithms, some of which used fully-connected architectures and some convolutional.
For fully-connected autoencoders, we use the same autoencoder architecture as DEC (Xie et al., 2016).
Speciﬁcally, for all experiments on all datasets, we use an autoencoder with the following dimensions: D–500–500–2000–d–2000–500–500–D. This autoencoder architecture follows parametric t-SNE (van der Maaten, 2009).
For convolutional autoencoders, the network architecture is modeled on JULE (Yang et al., 2016).
The architecture is speciﬁed in the appendix.
As in Yang et al.
, the number of layers depends on image resolution in the dataset and it is set such that the output resolution of the encoder is about 4 × 4 .
In both architectures and for all datasets, the dimensionality of the reduced space is set to d = 10 .
(It is only varied for controlled experiments that analyze stability with respect to d .)
No dataset-speciﬁc hyperparameter tuning was done.
For autoencoder initialization, a minibatch size of 256 and dropout probability of 0.2 are used.
SDAE pretraining and ﬁnetuning start with a learning rate of 0.1, which is decreased by a factor of 10 every 80 epochs.
Each layer is pretrained for 200 epochs.
Finetuning of the whole SDAE is performed for 400 epochs.
For the fully-connected SDAE, the learning rates are scaled in accordance with the dimensionality of the dataset.
For m-kNN graph construction, the nearest-neighbor parameter k is set to 10 and the cosine distance metric is used.
The Adam solver is used with its default learning rate of 0.001 and momentum 0.99.
Minibatches are constructed by sampling 128 edges.
DCC was implemented using the PyTorch library.
For the baselines, we use publicly available implementations.
For k -means++, DBSCAN and ACW, we use the implementations in the SciPy library and report the best results across ten random restarts.
For a number of baselines, we performed hyperparameter search to maximize their reported performance.
For DBSCAN, we searched over values of Eps , for LDMGI we searched over values of the regularization constant λ , for SEC we searched over values of the parameter µ , and for GDL we tuned the graph construction parameter a .
The DCN approach uses a different network architecture for each dataset.
Wherever possible, we report results using their dataset-speciﬁc architecture.
For YTF, Coil100, and YaleB, we use their reference architecture for MNIST.
4. M EASURES
Common measures of clustering accuracy include normalized mutual information (NMI) (Strehl & Ghosh, 2002) and clustering accuracy (ACC).
However, NMI is known to be biased in favor of ﬁne-grained partitions and ACC is also biased on imbalanced datasets (Vinh et al., 2010).
To overcome these biases, we use adjusted mutual information (AMI)
(Vinh et al., 2010), deﬁned as
[Equation]
Here H ( · ) is the entropy, MI ( · , · ) is the mutual information, and c and ˆ c are the two partitions being compared.
AMI lies in a range [0 , 1] .
Higher is better.
For completeness, results according to ACC are reported in the appendix.
4. R ESULTS
The results are summarized in Table 1.
Among deep clustering methods that use fully-connected networks, DCN and DEC are not as accurate as fully-connected DCC and are also less consistent: the performance of DEC drops on the high-dimensional image datasets, while DCN is far behind on
MNIST and YaleB. Among deep clustering methods that use convolutional networks, the performance of DEPICT drops on COIL100 and YTF, while JULE is far behind on YTF.
The GDL algorithm failed to scale to the full MNIST dataset and the corresponding measurement is marked as ‘n/a’. [Table]
performs par better
A NALYSIS
Importance of joint optimization.
We now analyze the importance of performing dimensionality reduction and clustering jointly, versus performing dimensionality reduction and then clustering the embedded data.
To this end, we use the same SDAE architecture and training procedure as fully-connected DCC.
We optimize the autoencoder but do not optimize the full DCC objective.
This yields a standard nonlinear embedding, using the same autoencoder that is used by DCC, into a space with the same reduced dimensionality d .
In this space, we apply a number of clustering algorithms: k -means++, AC-W, DBSCAN, SEC, LDMGI, GDL, and RCC.
The results are shown in Table(top).
These results should be compared to results reported in Table 1.
The comparison shows that the accuracy of the baseline algorithms beneﬁts from dimensionality reduction.
However, in all cases their accuracy is still lower than that attained by DCC using joint optimization.
Furthermore, although
RCC and DCC share the same underlying nearest-neighbor graph construction and a similar clustering loss, the performance of DCC far surpasses that achieved by stagewise SDAE embedding followed by RCC.
Note also that the relative performance of most baselines drops on Coil100 and YaleB. We hypothesize that the fully-connected SDAE is limited in its ability to discover a good low-dimensional embedding for very high-dimensional image datasets (tens of thousands of dimensions for Coil and YaleB).
Next, we show the performance of the same clustering algorithms when they are applied in the reduced space produced by DCC.
These results are reported in Table 2 (bottom).
In comparison to
Table 2 (top), the performance of all algorithms improves signiﬁcantly and some results are now on par or better than the results of DCC as reported in Table 1.
The improvement for k -means++,
Ward, and DBSCAN is particularly striking.
This indicates that the performance of many clustering algorithms can be improved by ﬁrst optimizing a low-dimensional embedding using DCC and then clustering in the learned embedding space. [Table]
Visualization.
A visualization is provided in Figure 1.
Here we used Barnes-Hut t-SNE (van der
Maaten & Hinton, 2008; van der Maaten, 2014) to visualize a randomly sampled subset of 10K datapoints from the MNIST dataset.
We show the original dataset, the dataset embedded by the
SDAE into R d (optimized for dimensionality reduction), and the embedding into R d produced by
DCC.
As shown in the ﬁgure, the embedding produced by DCC is characterized by well-deﬁned, clearly separated clusters.
The clusters strongly correspond to the ground-truth classes (coded by color in the ﬁgure), but were discovered with no supervision.
[Figure]
Robustness to dimensionality of the latent space.
Next we study the robustness of DCC to the dimensionality d of the latent space.
For this experiment, we consider fully-connected DCC.
We vary d between 5 and 60 and measure AMI on the MNIST and Reuters datasets.
For comparison, we report the performance of DEC, which uses the same autoencoder architecture, as well as the accuracy attained by running k -means++ on the output of the SDAE, optimized for dimensionality reduction.
The results are shown in Figure 2.
[Figure]
The results yield two conclusions.
First, the accuracy of DCC, DEC, and SDAE+ k -means gradually decreases as the dimensionality d increases.
This supports the common view that clustering becomes progressively harder as the dimensionality of the data increases.
Second, the results demonstrate that
DCC is more robust to increased dimensionality than DEC and SDAE.
For example, on MNIST, as the dimensionality d changes from 5 to 60, the accuracy of DEC and SDAE drops by 28% and 35%, respectively, while the accuracy of DCC decreases by only 9%.
When d = 60 , the accuracy attained by DCC is higher than the accuracy attained by DEC and SDAE by 27% and 40%, respectively.
C ONCLUSION
We have presented a clustering algorithm that combines nonlinear dimensionality reduction and clustering.
Dimensionality reduction is performed by a deep network that embeds the data into a lower-dimensional space.
The embedding is optimized as part of the clustering process and the resulting network produces clustered data.
The presented algorithm does not rely on a priori knowledge of the number of ground-truth clusters.
Nonlinear dimensionality reduction and clustering are performed by optimizing a global continuous objective using scalable gradient-based solvers.
A PPENDIX A D ATASETS
MNIST (LeCun et al., 1998):
This is a popular dataset containing 70,000 images of handwritten digits.
Each image is of size 28 × 28 (784 dimensions).
The data is categorized into 10 classes.
Coil100 (Nene et al., 1996):
This dataset consists of 7,200 images of 100 object categories, each captured from 72 poses.
Each RGB image is of size 128 × 128 (49,152 dimensions).
YouTube Faces (Wolf et al., 2011): The YTF dataset contains videos of faces.
We use all the video frames of the ﬁrst 40 subjects sorted in chronological order.
Each frame is an RGB image of size 55 × 55 .
The number of datapoints is 10,056 and the dimensionality is 9,075.
YaleB (Georghiades et al., 2001):
This dataset contains 2,414 images of faces of 28 human subjects taken under different lightning condition.
Each image is of size 192 × 168 (32,256 dimensions).
Reuters :
This is a popular dataset comprising 21,578 Reuters news articles.
We consider the
Modiﬁed Apte split, which yields a total of 9,082 articles.
TF-IDF features on the 2,000 most frequently occurring word stems are computed and normalized.
The dimensionality of the data is thus 2,000.
RCV1 (Lewis et al., 2004):
This is a document dataset comprising 800,000 Reuters newswire articles.
Only the four root categories are considered and all articles labeled with more than one root category are pruned.
We report results on a randomly sampled subset of 10,000 articles.
2,000 TF-IDF features were extracted as in the case of the Reuters dataset.
B C ONVOLUTIONAL N ETWORK A RCHITECTURE
Table 3 summarizes the architecture of the convolutional encoder used for the convolutional conﬁguration of DCC.
Convolutional kernels are applied with a stride of two.
The encoder is followed by a fully-connected layer with output dimension d and a convolutional decoder with kernel size that matches the output dimension of conv5 .
The decoder architecture mirrors the encoder and the output from each layer is appropriately zero-padded to match the input size of the corresponding encoding layer.
All convolutional and transposed convolutional layers are followed by batch normalization and rectiﬁed linear units (Ioffe & Szegedy, 2015; Nair & Hinton, 2010).
[Table]
C H YPERPARAMETERS
DCC uses three hyperparameters: the nearest neighbor graph (mkNN) parameter k , the embedding dimensionality d , and the update period M for graduated nonconvexity.
For fair comparison to RCC and RCC-DR, we ﬁx k = 10 (the setting used in Shah & Koltun (2017)).
The other two hyperparameters were set to d = 10 and M = 20 based on grid search on MNIST.
The hyperparameters are ﬁxed at these values across all datasets.
No dataset-speciﬁc tuning is done.
However, note that the hyperparameter M is architecture-speciﬁc.
We set M = 10 for convolutional autoencoders and it is varied for varying dimensionality d during the controlled experiment reported in Figures 2 and 3.
The other hyperparameters such as λ, δ i , µ i are set automatically as described in Sections 3.2 and 3.
and in Shah & Koltun (2017).
D ACC M EASURE
For completeness, we report results according to the ACC measure.
Table 4 provides the ACC counterpart to Table 1.
Figure 3 provides the ACC counterpart to Figure 2. [Figure]
[Table]
We also report results according to the NMI measure. Table 5 provides the NMI counterpart to [Table]