Original: G ENERALIZED C LUSTERING BY L EARNING TO O PTIMIZE E XPECTED N ORMALIZED C UTS
Revised: G ENERALIZED C LUSTERING BY L EARNING TO O PTIMIZE E XPECTED N ORMALIZED C UTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce a novel end-to-end approach for learning to cluster in the absence of labeled examples.
Revised: We introduce a novel end-to-end approach for learning to cluster in the absence of labeled examples.

Original: Our clustering objective is based on optimizing normalized cuts, a criterion which measures both intra-cluster similarity as well as inter-cluster dissimilarity.
Revised: Our clustering objective is based on optimizing normalized cuts, a criterion which measures both intra-cluster similarity as well as inter-cluster dissimilarity.

Original: We deﬁne a differentiable loss function equivalent to the expected normalized cuts.
Revised: We deﬁne a differentiable loss function equivalent to the expected normalized cuts.

Original: Unlike much of the work in unsupervised deep learning, our trained model directly outputs ﬁnal cluster assignments, rather than embeddings that need further processing to be usable.
Revised: Unlike much of the work in unsupervised deep learning, our trained model directly outputs ﬁnal cluster assignments, rather than embeddings that need further processing to be usable.

Original: Our approach generalizes to unseen datasets across a wide variety of domains, including text, and image.
Revised: Our approach generalizes to unseen datasets across a wide variety of domains, including text, and image.

Original: Speciﬁcally, we achieve state-of-the-art results on popular unsupervised clustering benchmarks (e.g., MNIST, Reuters, CIFAR-10, and CIFAR-100), outperforming the strongest baselines by up to 10.9%.
Revised: Speciﬁcally, we achieve state-of-the-art results on popular unsupervised clustering benchmarks (e.g., MNIST, Reuters, CIFAR-10, and CIFAR-100), outperforming the strongest baselines by up to 10.9%.

Original: Our generalization results are superior (by up to 21.9%) to the recent top-performing clustering approach with the ability to generalize.
Revised: Our generalization results are superior (by up to 21.9%) to the recent top-performing clustering approach with the ability to generalize.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Clustering unlabeled data is an important problem from both a scientiﬁc and practical perspective.
Revised: Clustering unlabeled data is an important problem from both a scientiﬁc and practical perspective.

Original: As technology plays a larger role in daily life, the volume of available data has exploded.
Revised: As technology plays a larger role in daily life, the volume of available data has exploded.

Original: However, labeling this data remains very costly and often requires domain expertise.
Revised: However, labeling this data remains very costly and often requires domain expertise.

Original: Therefore, unsupervised clustering methods are one of the few viable approaches to gain insight into the structure of these massive unlabeled datasets.
Revised: Therefore, unsupervised clustering methods are one of the few viable approaches to gain insight into the structure of these massive unlabeled datasets.

Original: One of the most popular clustering methods is spectral clustering (Shi & Malik, 2000; Ng et al., 2002; Von Luxburg, 2007), which ﬁrst embeds the similarity of each pair of data points in the
Revised: One of the most popular clustering methods is spectral clustering (Shi & Malik, 2000; Ng et al., 2002; Von Luxburg, 2007), which ﬁrst embeds the similarity of each pair of data points in the

Original: Laplacian’s eigenspace and then uses k-means to generate clusters from it.
Revised: Laplacian’s eigenspace and then uses k-means to generate clusters from it.

Original: Spectral clustering not only outperforms commonly used clustering methods, such as k-means (Von Luxburg, 2007), but also allows us to directly minimize the pairwise distance between data points and solve for the optimal node embeddings analytically.
Revised: Spectral clustering not only outperforms commonly used clustering methods, such as k-means (Von Luxburg, 2007), but also allows us to directly minimize the pairwise distance between data points and solve for the optimal node embeddings analytically.

Original: Moreover, it is shown that the eigenvector of the normalized Laplacian matrix can be used to ﬁnd the approximate solution to the well known normalized cuts problem (Ng et al., 2002; Von Luxburg, 2007).
Revised: Moreover, it is shown that the eigenvector of the normalized Laplacian matrix can be used to ﬁnd the approximate solution to the well known normalized cuts problem (Ng et al., 2002; Von Luxburg, 2007).

Original: In this work, we introduce CNC , a framework for C lustering by learning to optimize expected
Revised: In this work, we introduce CNC , a framework for C lustering by learning to optimize expected

Original: N ormalized C uts .
Revised: N ormalized C uts .

Original: We show that by directly minimizing a continuous relaxation of the normalized cuts problem, CNC enables end-to-end learning approach that outperforms top-performing clustering approaches.
Revised: We show that by directly minimizing a continuous relaxation of the normalized cuts problem, CNC enables end-to-end learning approach that outperforms top-performing clustering approaches.

Original: We demonstrate that our approach indeed can produce lower normalized cut values than the baseline methods such as SpectralNet, which consequently results in better clustering accuracy.
Revised: We demonstrate that our approach indeed can produce lower normalized cut values than the baseline methods such as SpectralNet, which consequently results in better clustering accuracy.

Original: Let us motivate CNC through a simple example.
Revised: Let us motivate CNC through a simple example.

Original: In Figure 1, we want to cluster 6 images from
Revised: In Figure 1, we want to cluster 6 images from

Original: CIFAR-10 dataset into two clusters.
Revised: CIFAR-10 dataset into two clusters.

Original: The afﬁnity graph for these data points is shown in Figure 1(a)
Revised: The afﬁnity graph for these data points is shown in Figure 1(a)

Original: (details of constructing such graph is discussed in Section 4.2).
Revised: (details of constructing such graph is discussed in Section 4.2).

Original: In this example, it is obvious that the optimal clustering is the result of cutting the edge connecting the two triangles.
Revised: In this example, it is obvious that the optimal clustering is the result of cutting the edge connecting the two triangles.

Original: Cutting this edge will result in the optimal value for the normalized cuts objective.
Revised: Cutting this edge will result in the optimal value for the normalized cuts objective.

Original: In CNC , we deﬁne a new differentiable loss function equivalent to the expected normalized cuts objective.
Revised: In CNC , we deﬁne a new differentiable loss function equivalent to the expected normalized cuts objective.

Original: We train a deep learning model to minimize the proposed loss in an unsupervised manner without the need for any labeled datasets.
Revised: We train a deep learning model to minimize the proposed loss in an unsupervised manner without the need for any labeled datasets.

Original: Our trained model directly returns the probabilities of belonging to each cluster (Figure 1(b)).
Revised: Our trained model directly returns the probabilities of belonging to each cluster (Figure 1(b)).

Original: In this example, the optimal normalized cuts is 0.286 (Equation 1), and as we can see, the CNC loss also converges to this value (Figure 1(c)).
Revised: In this example, the optimal normalized cuts is 0.286 (Equation 1), and as we can see, the CNC loss also converges to this value (Figure 1(c)).

Original: [Figure]
Revised: [Figure]

Original: We compare the performance of CNC to several learning-based clustering approaches (SpectralNet (Shaham et al., 2018), DEC (Xie et al., 2016), DCN (Yang et al., 2017), VaDE (Jiang et al., 2017), DEPICT (Ghasedi Dizaji et al., 2017), IMSAT (Hu et al., 2017), and IIC (Ji et al., 2019)) on four datasets: MNIST, Reuters, CIFAR10, and CIFAR100.
Revised: We compare the performance of CNC to several learning-based clustering approaches (SpectralNet (Shaham et al., 2018), DEC (Xie et al., 2016), DCN (Yang et al., 2017), VaDE (Jiang et al., 2017), DEPICT (Ghasedi Dizaji et al., 2017), IMSAT (Hu et al., 2017), and IIC (Ji et al., 2019)) on four datasets: MNIST, Reuters, CIFAR10, and CIFAR100.

Original: Our results show up to 10.9% improvement over the baselines.
Revised: Our results show up to 10.9% improvement over the baselines.

Original: Moreover, generalizing spectral embeddings to unseen data points, a task commonly referred to as out-of-sample-extension (OOSE), is a non-trivial task (Bengio et al., 2003; Belkin et al., 2006; Mendoza Quispe et al., 2016).
Revised: Moreover, generalizing spectral embeddings to unseen data points, a task commonly referred to as out-of-sample-extension (OOSE), is a non-trivial task (Bengio et al., 2003; Belkin et al., 2006; Mendoza Quispe et al., 2016).

Original: Our results conﬁrm that CNC generalizes to unseen data.
Revised: Our results conﬁrm that CNC generalizes to unseen data.

Original: Our generalization results are superior (by up to 21.9%) to SpectralNet (Shaham et al., 2018), the recent top-performing clustering approach with the ability to generalize.
Revised: Our generalization results are superior (by up to 21.9%) to SpectralNet (Shaham et al., 2018), the recent top-performing clustering approach with the ability to generalize.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recent deep learning approaches to clustering attempt to embed the input data into a form that is amenable to clustering by k-means or Gaussian Mixture Models.
Revised: Recent deep learning approaches to clustering attempt to embed the input data into a form that is amenable to clustering by k-means or Gaussian Mixture Models.

Original: (Yang et al., 2017; Xie et al., 2016) focused on learning representations for clustering.
Revised: (Yang et al., 2017; Xie et al., 2016) focused on learning representations for clustering.

Original: To ﬁnd the clustering-friendly latent representations and to better cluster the data, DCN (Yang et al., 2017) proposed a joint dimensionality reduction (DR) and K-means clustering approach in which DR is accomplished via learning a deep neural network.
Revised: To ﬁnd the clustering-friendly latent representations and to better cluster the data, DCN (Yang et al., 2017) proposed a joint dimensionality reduction (DR) and K-means clustering approach in which DR is accomplished via learning a deep neural network.

Original: DEC (Xie et al., 2016) simultaneously learns cluster assignment and the underlying feature representation by iteratively updating a target distribution to sharpen cluster associations.
Revised: DEC (Xie et al., 2016) simultaneously learns cluster assignment and the underlying feature representation by iteratively updating a target distribution to sharpen cluster associations.

Original: Several other approaches rely on a variational autoencoder that utilizes a Gaussian mixture prior (Jiang et al., 2017;
Revised: Several other approaches rely on a variational autoencoder that utilizes a Gaussian mixture prior (Jiang et al., 2017;

Original: Dilokthanakul et al., 2016;
Revised: Dilokthanakul et al., 2016;

Original: Hu et al., 2017; Ji et al., 2019; Ben-Yosef & Weinshall, 2018).
Revised: Hu et al., 2017; Ji et al., 2019; Ben-Yosef & Weinshall, 2018).

Original: These approaches are mainly based on data augmentation, where the network is trained to maximize the mutual information between inputs and predicted clusters, while regularizing the network so that the cluster assignment of the data points is consistent with the assignment of the augmented points.
Revised: These approaches are mainly based on data augmentation, where the network is trained to maximize the mutual information between inputs and predicted clusters, while regularizing the network so that the cluster assignment of the data points is consistent with the assignment of the augmented points.

Original: Different clustering objectives, such as self-balanced k-means and balanced min-cut, have also been exhaustively studied (Liu et al., 2017;
Revised: Different clustering objectives, such as self-balanced k-means and balanced min-cut, have also been exhaustively studied (Liu et al., 2017;

Original: Chen et al., 2017; Chang et al., 2014).
Revised: Chen et al., 2017; Chang et al., 2014).

Original: One of the most effective techniques is spectral clustering, which ﬁrst generates node embeddings in the eigenspace of the graph Laplacian, and then applies k-means clustering to these vectors (Shi & Malik, 2000;
Revised: One of the most effective techniques is spectral clustering, which ﬁrst generates node embeddings in the eigenspace of the graph Laplacian, and then applies k-means clustering to these vectors (Shi & Malik, 2000;

Original: Ng et al., 2002;
Revised: Ng et al., 2002;

Original: Von Luxburg, 2007).
Revised: Von Luxburg, 2007).

Original: To address the fact that clusters with the lowest graph conductance tend to have few nodes (Leskovec, 2009; Zhang & Rohe, 2018), (Zhang & Rohe, 2018) proposed regularized spectral clustering to encourage more balanced clusters.
Revised: To address the fact that clusters with the lowest graph conductance tend to have few nodes (Leskovec, 2009; Zhang & Rohe, 2018), (Zhang & Rohe, 2018) proposed regularized spectral clustering to encourage more balanced clusters.

Original: Generalizing clustering to unseen nodes and graphs is nontrivial (Bengio et al., 2003; Belkin et al., 2006; Mendoza Quispe et al., 2016).
Revised: Generalizing clustering to unseen nodes and graphs is nontrivial (Bengio et al., 2003; Belkin et al., 2006; Mendoza Quispe et al., 2016).

Original: A recent work, SpectralNet (Shaham et al., 2018), takes a deep learning approach to spectral clustering that generalizes to unseen data points.
Revised: A recent work, SpectralNet (Shaham et al., 2018), takes a deep learning approach to spectral clustering that generalizes to unseen data points.

Original: This approach ﬁrst learns embeddings of the similarity of each pair of data points in Laplacian’s eigenspace and then applies k-means to those embeddings to generate clusters.
Revised: This approach ﬁrst learns embeddings of the similarity of each pair of data points in Laplacian’s eigenspace and then applies k-means to those embeddings to generate clusters.

Original: Unlike SpectralNet, we propose an end-to-end learning approach with a differentiable loss that directly minimizes the normalized cuts.
Revised: Unlike SpectralNet, we propose an end-to-end learning approach with a differentiable loss that directly minimizes the normalized cuts.

Original: We show that our approach indeed can produce lower normalized cut values than the baseline methods such as SpectralNet, which consequently results in better clustering accuracy.
Revised: We show that our approach indeed can produce lower normalized cut values than the baseline methods such as SpectralNet, which consequently results in better clustering accuracy.

Original: Our evaluation results show that CNC improves generalization accuracy on unseen data points by up to 21.9%.
Revised: Our evaluation results show that CNC improves generalization accuracy on unseen data points by up to 21.9%.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Since CNC objective is based on optimizing normalized cuts, in this section, we brieﬂy overview the formal deﬁnition of this metric.
Revised: Since CNC objective is based on optimizing normalized cuts, in this section, we brieﬂy overview the formal deﬁnition of this metric.

Original: 3. F ORMAL DEFINITION OF N ORMALIZED CUTS
Revised: 3. F ORMAL DEFINITION OF N ORMALIZED CUTS

Original: Let G =
Revised: Let G =

Original: ( V, E, W ) be a graph where V = { v i } and E = {
Revised: ( V, E, W ) be a graph where V = { v i } and E = {

Original: e ( v i , v j ) | v i ∈ V, v j ∈ V } are the set of nodes and edges in the graph and w ij ∈ W is the edge weight of the e ( v i , v j ) .
Revised: e ( v i , v j ) | v i ∈ V, v j ∈ V } are the set of nodes and edges in the graph and w ij ∈ W is the edge weight of the e ( v i , v j ) .

Original: Let n be the number of nodes.
Revised: Let n be the number of nodes.

Original: A graph G can be clustered into g disjoint sets S 1 , S 2 , . . .
Revised: A graph G can be clustered into g disjoint sets S 1 , S 2 , . . .

Original: S g , where the union of the nodes in those sets are V ( (cid:83) gk =1 S k = V ), and each node belongs to only one set ( (cid:84) gk =
Revised: S g , where the union of the nodes in those sets are V ( (cid:83) gk =1 S k = V ), and each node belongs to only one set ( (cid:84) gk =

Original: 1 S k = ∅ ), by simply removing edges connecting those sets.
Revised: 1 S k = ∅ ), by simply removing edges connecting those sets.

Original: For example, in Figure 1(a), by removing one edge two disjoint clusters are formed.
Revised: For example, in Figure 1(a), by removing one edge two disjoint clusters are formed.

Original: Normalized cuts ( Ncuts ) which is deﬁned based on the graph conductance, has been studied by (Shi & Malik, 2000; Zhang & Rohe, 2018), and the cost of a cut that forms disjoint sets S 1 , S 2 , . . .
Revised: Normalized cuts ( Ncuts ) which is deﬁned based on the graph conductance, has been studied by (Shi & Malik, 2000; Zhang & Rohe, 2018), and the cost of a cut that forms disjoint sets S 1 , S 2 , . . .

Original: S g is computed as:
Revised: S g is computed as:

Original: [Equation]
Revised: [Equation]

Original: Where ¯ S k represents the complement of S k , i.e., ¯ S k
Revised: Where ¯ S k represents the complement of S k , i.e., ¯ S k

Original: = (cid:83) i (cid:54) =
Revised: = (cid:83) i (cid:54) =

Original: k S i .
Revised: k S i .

Original: cut ( S k , ¯ S k ) is called cut and is the total weight of the edges that are removed from G in order to form disjoint sets S k and ¯ S k .
Revised: cut ( S k , ¯ S k ) is called cut and is the total weight of the edges that are removed from G in order to form disjoint sets S k and ¯ S k .

Original: vol ( S k , V ) is the total edge weights ( w ij ), whose end points ( v i , or v j ) belong to S k .
Revised: vol ( S k , V ) is the total edge weights ( w ij ), whose end points ( v i , or v j ) belong to S k .

Original: The cut and vol are: cut ( S k , ¯
Revised: The cut and vol are: cut ( S k , ¯

Original: [Equation]
Revised: [Equation]

Original: Note that in Equation 2, S k and ¯ S k are disjoint, i.e., S k ∩ ¯ S k = ∅ , while in vol , S k ⊂ V .
Revised: Note that in Equation 2, S k and ¯ S k are disjoint, i.e., S k ∩ ¯ S k = ∅ , while in vol , S k ⊂ V .

Original: In running example (Figure 1), since the edge weights are one, cut ( S 1 , ¯ S 1 )
Revised: In running example (Figure 1), since the edge weights are one, cut ( S 1 , ¯ S 1 )

Original: = cut ( S 2 , ¯ S 2 ) = 1 , and vol ( S 1 , V ) =
Revised: = cut ( S 2 , ¯ S 2 ) = 1 , and vol ( S 1 , V ) =

Original: vol ( S 2 , V )
Revised: vol ( S 2 , V )

Original: = 

Revised: =

Original: + 2 + 3 = 7 .
Revised: + 2 + 3 = 7 .

Original: Thus the Ncuts ( S 1 , S 2 )
Revised: Thus the Ncuts ( S 1 , S 2 )

Original: = 17 + 17 = 0 .
Revised: = 17 + 17 = 0 .

Original: In this example one can see that such clustering results in minimum value of the normalized cuts.
Revised: In this example one can see that such clustering results in minimum value of the normalized cuts.

Original: CNC aims to ﬁnd a cut that the normalized cuts (Equation 1) is minimized.
Revised: CNC aims to ﬁnd a cut that the normalized cuts (Equation 1) is minimized.

Original:  CNC F RAMEWORK
Revised:  CNC F RAMEWORK

Original: Finding the cluster assignments that minimizes the normalized cuts is NP-complete and an approximation to the this problem is based on the eigenvectors of the normalized graph Laplacian which has been studied in (Shi & Malik, 2000; Zhang & Rohe, 2018).
Revised: Finding the cluster assignments that minimizes the normalized cuts is NP-complete and an approximation to the this problem is based on the eigenvectors of the normalized graph Laplacian which has been studied in (Shi & Malik, 2000; Zhang & Rohe, 2018).

Original: CNC , on the other hand, is a neural network framework for learning to cluster in the absence of labeled examples by directly minimizing the continuous relaxation of the normalized cuts.
Revised: CNC , on the other hand, is a neural network framework for learning to cluster in the absence of labeled examples by directly minimizing the continuous relaxation of the normalized cuts.

Original: As shown in Algorithm 1, end-to-end training of the CNC contains two steps, i.e, (i) data points embedding (line 3), and (ii) clustering (lines 4-9).
Revised: As shown in Algorithm 1, end-to-end training of the CNC contains two steps, i.e, (i) data points embedding (line 3), and (ii) clustering (lines 4-9).

Original: In data points embedding, the goal is to learn embeddings that capture the afﬁnity of the data points, while the clustering step uses those embeddings to learn the CNC model and outputs the cluster assignments.
Revised: In data points embedding, the goal is to learn embeddings that capture the afﬁnity of the data points, while the clustering step uses those embeddings to learn the CNC model and outputs the cluster assignments.

Original: Next, we ﬁrst focus on the clustering step and we introduce our new diffrentiable loss function to train CNC model.
Revised: Next, we ﬁrst focus on the clustering step and we introduce our new differentiable loss function to train CNC model.

Original: Later in Section 4.2, we discuss the details of the embedding step.
Revised: Later in Section 4.2, we discuss the details of the embedding step.

Original: 4. C LUSTERING S TEP : L EARN CNC MODEL
Revised: 4. C LUSTERING S TEP : L EARN CNC MODEL

Original: In this section, we describe the clustering step in Algorithm 1 (lines 4-9).
Revised: In this section, we describe the clustering step in Algorithm 1 (lines 4-9).

Original: for each data point x i , the input to clustering step is embedding v i ∈ R d (detail in Section 4.2).
Revised: For each data point x i , the input to clustering step is embedding v i ∈ R d (detail in Section 4.2).

Original: The goal is to learn CNC model
Revised: The goal is to learn CNC model

Original: F θ : R d → R g that for a given embedding v i ∈
Revised: F θ : R d → R g that for a given embedding v i ∈

Original: R
Revised: R

Original: d it returns y i
Revised: d it returns y i

Original: = F θ ( v i )
Revised: = F θ ( v i )

Original: ∈ R g , which represents the assignment probabilities over g clusters.
Revised: ∈ R g , which represents the assignment probabilities over g clusters.

Original: Clearly for n data points, it returns Y ∈ R n × g where
Revised: Clearly for n data points, it returns Y ∈ R n × g where

Original: Y ik represents the probability that v i belongs to cluster S k .
Revised: Y ik represents the probability that v i belongs to cluster S k .

Original: The CNC model F θ is implemented using a neural network, where the parameter vector θ denotes the network weights.
Revised: The CNC model F θ is implemented using a neural network, where the parameter vector θ denotes the network weights.

Original: We propose a loss function based on output Y to calculate the expected normalized cuts.
Revised: We propose a loss function based on output Y to calculate the expected normalized cuts.

Original: Thus CNC learns the F θ by minimizing this loss.
Revised: Thus CNC learns the F θ by minimizing this loss (Equation 7).

Original: Recall that cut ( S k , ¯ S k ) is the total weight of the edges that are removed from G in order to form disjoint sets S k and ¯ S k .
Revised: Recall that cut ( S k , ¯ S k ) is the total weight of the edges that are removed from G in order to form disjoint sets S k and ¯ S k .

Original: In our setup, embeddings are the nodes in graph G , and neighbors of anInference, cluster assignments embedding v i are based on the k-nearest neighbors.
Revised: In our setup, embeddings are the nodes in graph G , and neighbors of an embedding v i are based on the k-nearest neighbors.

Original: Let Y ik be the probability that node v i belongs to cluster S k .
Revised: Let Y ik be the probability that node v i belongs to cluster S k .

Original: The probability that node v j does not belong to S k would be 1 − Y jk .
Revised: The probability that node v j does not belong to S k would be 1 − Y jk .

Original: Therefore, E [ cut ( S k , ¯ S k )] can be formulated by Equation 3, where N ( v i ) is the set of nodes adjacent to v i . [Equation] =
Revised: Therefore, E [ cut ( S k , ¯ S k )] can be formulated by Equation 3, where N ( v i ) is the set of nodes adjacent to v i . Inference, cluster assignments

Original: [Equation]
Revised: [Equation]

Original: Since the weight matrix W represents the edge weights adjacent nodes, we can rewrite Equation 3: 
Revised: Since the weight matrix W represents the edge weights adjacent nodes, we can rewrite Equation 3: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The element-wise product with the weight matrix ( (cid:12) W ) ensures that only the adjacent nodes are considered.
Revised: The element-wise product with the weight matrix ( (cid:12) W ) ensures that only the adjacent nodes are considered.

Original: Moreover, the result of Y : ,k (1 − Y : ,k ) (cid:124) (cid:12) W is an n × n matrix and reduce-sum is the sum over all of its elements.
Revised: Moreover, the result of Y : ,k (1 − Y : ,k ) (cid:124) (cid:12) W is an n × n matrix and reduce-sum is the sum over all of its elements.

Original: From Equation 2, vol ( S k , V ) is the total edge weights ( w ij ), whose end points ( v i , or v j ) belong to S k .
Revised: From Equation 2, vol ( S k , V ) is the total edge weights ( w ij ), whose end points ( v i , or v j ) belong to S k .

Original: Let D be a column vector of size n where D i is the total edge weights from node v i .
Revised: Let D be a column vector of size n where D i is the total edge weights from node v i .

Original: Given Y , we can calculate the E [ vol ( S k , V )] as follows:
Revised: We can update Equation 3 as follows to ﬁnd the expected normalized cuts.

Original: [Equation]
Revised: [Equation]

Revised: The matrix representation is given in Equation 6, where

Original: where Γ  is a vector in R g , and g is the number of sets/clusters.
Revised: Γ = Y (cid:124) D is a vector in R g , and g is the number of sets/clusters.

Original: With E [ cut ( S k , ¯ S k )] and E [ vol ( S k , V )] from Equations 4 and 5, we can calculate the expected normalized cuts as follows:
Revised: (cid:11) is element-wise division and the result of ( Y (cid:11) Γ)(1 − Y ) (cid:124) (cid:12) W is a n × n matrix where reduce-sum is the sum over all of its elements.

Original: [Equation]
Revised: [Equation]

Original: (cid:11) is element-wise division and the result of ( Y (cid:11) Γ)
Revised: CNC model F θ is implemented using a neural network, where the parameter θ denotes the network weights ( y i

Original: [Equation]
Revised: = F θ ( v i ) ).

Original: 1 − Y )
Revised: CNC is trained to optimize Equation 7 via backpropagation (Algorithm 1).

Original: [Equation]
Revised: [Equation]

Original: (cid:12) W is a n × n matrix where reduce-sum is the sum over all of its elements.

Original: As you can see the afﬁnity graph W is part of the CNC loss (Equation 6).
Revised: As you can see the afﬁnity graph W is part of the CNC loss (Equation 7).

Original: Clearly, when the number of data points ( n ) is large, such calculation can be expensive.
Revised: Clearly, when the number of data points ( n ) is large, such calculation can be expensive.

Original: However, in our experimental results, we show that for large dataset (e.g., Reuters contains 685,071 documents), it is possible to optimize the loss on randomly sampled minibatches of data.
Revised: However, in our experimental results, we show that for large dataset (e.g., Reuters contains 685,071 documents), it is possible to optimize the loss on randomly sampled minibatches of data.

Original: We also build the afﬁnity graph over a given minibach using the embeddings and based on their k nreast-neighbor (Algorithm 1 (lines 5-6)).
Revised: We also build the afﬁnity graph over a given minibach using the embeddings and based on their k nearest-neighbor (Algorithm 1 (lines 5-6)).

Original: Speciﬁcally, in our implementation, CNC model F θ is a fully connected layer followed by gumble softmax, trained on randomly sampled minibatches of data to minimize Equation 6. 
Revised: Speciﬁcally, in our implementation, CNC model F θ is a fully connected layer followed by gumble softmax, trained on randomly sampled minibatches of data to minimize Equation 6. In Section 5. through a sensitivity analysis we show that the minibatch size affects the accuracy of our model.

Original: When training is over, the ﬁnal assignment of a node v i to a cluster is the arg max of y i
Revised: When training is over, the ﬁnal assignment of a data point with embedding v i to a cluster is the arg max of y i

Original: = F θ ( v i ) (Algorithm 1 (line 9)).
Revised: = F θ ( v i ) (Algorithm 1 (line 9)).

Original: 4. E MBEDDING S TEP
Revised: 4. E MBEDDING S TEP

Original: In this section, we discuss the embedding step (line 3 in Algorithm 1).
Revised: In this section, we discuss the embedding step (line 3 in Algorithm 1).

Original: Different afﬁnity measures, such as simple euclidean distance or nearest neighbor pairs combined with a Gaussian kernel, have been used in spectral clustering.
Revised: Different afﬁnity measures, such as simple euclidean distance or nearest neighbor pairs combined with a Gaussian kernel, have been used in spectral clustering.

Original: Recently it is shown that unsupervised application of a Siamese network to determine the distances improves the quality of the clustering (Shaham et al., 2018).
Revised: Recently it is shown that unsupervised application of a Siamese network to determine the distances improves the quality of the clustering (Shaham et al., 2018).

Original: In this work, we also use Siamese networks to learn embeddings that capture the afﬁnities of the data points.
Revised: In this work, we also use Siamese networks to learn embeddings that capture the afﬁnities of the data points.

Original: Siamese network is trained to learn an adaptive nearest neighbor metric.
Revised: Siamese network is trained to learn an adaptive nearest neighbor metric.

Original: It learns the afﬁnities directly from euclidean proximity by ”labeling” points x i , x j positive if (cid:107) x
Revised: It learns the afﬁnities directly from euclidean proximity by ”labeling” points x i , x j positive if (cid:107) x

Original: i − x j (cid:107) is small and negative otherwise.
Revised: i − x j (cid:107) is small and negative otherwise.

Original: In other words, it generates embeddings such that adjacent nodes are closer in the embedding space and non-adjacent nodes are further.
Revised: In other words, it generates embeddings such that adjacent nodes are closer in the embedding space and non-adjacent nodes are further.

Original: Such network is typically trained to minimize contrastive loss:
Revised: Such network is typically trained to minimize contrastive loss:

Original: [Equation]
Revised: [Equation]

Original: where v i =
Revised: where v i =

Original: G θ siamese ( x i ) , and G θ siamese : R m → R d is a Siamese network that transforms representations in the input space x
Revised: G θ siamese ( x i ) , and G θ siamese : R m → R d is a Siamese network that transforms representations in the input space x

Original: i ∈
Revised: i ∈

Original: R m to embeddings v i ∈ R d .
Revised: R m to embeddings v i ∈ R d .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The main goals of our experiments are to evaluate: (a) The performance of CNC against the existing clustering approaches.
Revised: The main goals of our experiments are to evaluate: (a) The performance of CNC against the existing clustering approaches.

Original: (b) The ability of CNC to generalize to unseen data compared to the topperforming generalizable baseline.
Revised: (b) The ability of CNC to generalize to unseen data compared to the topperforming generalizable baseline.

Original: (c) The effectiveness of minimizing Normalized cuts on improving the clustering results.
Revised: (c) The effectiveness of minimizing Normalized cuts on improving the clustering results.

Original: (d) The generalization performance of CNC as we vary the number of data points in training dataset.
Revised: (d) The generalization performance of CNC as we vary the number of data points in training dataset.

Original: 5. D ATASETS AND B ASELINE M ETHODS
Revised: 5. D ATASETS AND B ASELINE M ETHODS

Original: We evaluate the performance of CNC in comparison to several deep learning-based clustering approaches on four real world datasets: MNIST, Reuters, CIFAR-10, and CIFAR-100.
Revised: We evaluate the performance of CNC in comparison to several deep learning-based clustering approaches on four real world datasets: MNIST, Reuters, CIFAR-10, and CIFAR-100.

Original: The details of the datasets are as follows:
Revised: The details of the datasets are as follows:

Original: • MNIST is a collection of 70,000 28×28 gray-scale images of handwritten digits, divided into 60,000 training images and 10,000 test images.
Revised: • MNIST is a collection of 70,000 28×28 gray-scale images of handwritten digits, divided into 60,000 training images and 10,000 test images.

Original: • The Reuters dataset is a collection of English news labeled by category.
Revised: • The Reuters dataset is a collection of English news labeled by category.

Original: Like SpectralNet, DEC, and VaDE, we used the following categories: corporate/industrial, government/social, markets, and economics as labels and discarded all documents with multiple labels.
Revised: Like SpectralNet, DEC, and VaDE, we used the following categories: corporate/industrial, government/social, markets, and economics as labels and discarded all documents with multiple labels.

Original: Each article is represented by a tﬁdf vector using the 2000 most frequent words.
Revised: Each article is represented by a tﬁdf vector using the 2000 most frequent words.

Original: The dataset contains 685,071 documents.
Revised: The dataset contains 685,071 documents.

Original: We divided the data randomly to a 90%-10% split to evaluate the generalization ability of CNC.
Revised: We divided the data randomly to a 90%-10% split to evaluate the generalization ability of CNC

Original: We also investigate the imapact of training data size on the generalization by consideringfollowing splits: 90%-10%, 70%-30%, 50%-50%, 20%-80%, and 10%-90%.
Revised: We also investigate the imapact of training data size on the generalization by considering

Original:  • CIFAR-10 consists of 60000 32x32 colour images in 10 classes, with 6000 images per class.
Revised: following splits: 90%-10%, 70%-30%, 50%-50%, 20%-80%, and 10%-90%. • CIFAR-10 consists of 60000 32x32 colour images in 10 classes, with 6000 images per class.

Original: There are 50000 training images and 10000 test images.
Revised: There are 50000 training images and 10000 test images.

Original: • CIFAR-100 has 100 classes containing 600 images each with a 500/100 train/test split per class.
Revised: • CIFAR-100 has 100 classes containing 600 images each with a 500/100 train/test split per class.

Original: In all runs we assume the number of clusters is given.
Revised: In all runs we assume the number of clusters is given.

Original: In MNIST and CIFAR-10 number of clusters (g) is 10, g = 4 in Reuters, g = 100 in CIFAR-100.
Revised: In MNIST and CIFAR-10 number of clusters (g) is 10, g = 4 in Reuters, g = 100 in CIFAR-100.

Original: We compare CNC to SpectralNet (Shaham et al., 2018),
Revised: We compare CNC to SpectralNet (Shaham et al., 2018),

Original: DEC (Xie et al., 2016), DCN (Yang et al., 2017), VaDE (Jiang et al., 2017), DEPICT (Ghasedi Dizaji et al., 2017), IMSAT (Hu et al., 2017), and IIC (Ji et al., 2019).
Revised: DEC (Xie et al., 2016), DCN (Yang et al., 2017), VaDE (Jiang et al., 2017), DEPICT (Ghasedi Dizaji et al., 2017), IMSAT (Hu et al., 2017), and IIC (Ji et al., 2019).

Original: While (Yang et al., 2017; Xie et al., 2016) focused on learning representations for clustering, other approaches (Jiang et al., 2017;
Revised: While (Yang et al., 2017; Xie et al., 2016) focused on learning representations for clustering, other approaches (Jiang et al., 2017;

Original: Dilokthanakul et al., 2016; Hu et al., 2017; Ji et al., 2019; Ben-Yosef & Weinshall, 2018) rely on a variational autoencoder that utilizes a Gaussian mixture prior.
Revised: Dilokthanakul et al., 2016; Hu et al., 2017; Ji et al., 2019; Ben-Yosef & Weinshall, 2018) rely on a variational autoencoder that utilizes a Gaussian mixture prior.

Original: SpectralNet (Shaham et al., 2018), takes a deep learning approach to spectral clustering that generalizes to unseen data points.
Revised: SpectralNet (Shaham et al., 2018), takes a deep learning approach to spectral clustering that generalizes to unseen data points.

Original: Tableshows the results reported for these six methods.
Revised: Tableshows the results reported for these six methods.

Original: [Table] Similar to (Shaham et al., 2018), for MNIST and Reuters we use publicly available and pre-trained autoencoders 1 .
Revised: Similar to (Shaham et al., 2018), for MNIST and Reuters we use publicly available and pre-trained autoencoders 1 .

Original: The autoencoder used to map the Reuters data to code space was trained based on a random subset of 10,000 samples from the full dataset.
Revised: The autoencoder used to map the Reuters data to code space was trained based on a random subset of 10,000 samples from the full dataset.

Original: Similar to (Hu et al., 2017), for CIFAR-10 and
Revised: Similar to (Hu et al., 2017), for CIFAR-10 and

Original: CIFAR-100 we applied 50-layer pre-trained deep residual networks trained on ImageNet to extract features and used them for clustering.
Revised: CIFAR-100 we applied 50-layer pre-trained deep residual networks trained on ImageNet to extract features and used them for clustering.https://github.com/slim1017/VaDE/tree/master/pretrain_weights [Table] The model is trained on all data. ( means values are not reported.

Revised: ( ∗ reported in DEC [(Xie et al., 2016)], ( ∗∗ ) reported in DCN [(Yang et al., 2017)], ( † ) reported in VaDE [( et al., 2017)], ( †† ) reported in DEPICT

Revised: [(Ghasedi Dizaji et al., 2017)], ( ‡‡ ) reported in IMSAT [(Hu et al., 2017)], ( ‡ ) reported in SpectralNet [(Shaham et al., 2018)], ( ††† ) reported in IIC [(Ji et al., 2019)].

Original: 5. P ERFORMANCE M EASURES
Revised: 5. P ERFORMANCE M EASURES

Original: We use two commonly used measures, the unsupervised clustering accuracy (ACC), and the normalized mutual information (NMI) in (Cai et al., 2011) to evaluate the accuracy of the clustering.
Revised: We use two commonly used measures, the unsupervised clustering accuracy (ACC), and the normalized mutual information (NMI) in (Cai et al., 2011) to evaluate the accuracy of the clustering.

Original: Both
Revised: Both

Original: ACC and NMI are in [0, 1], with higher values indicating better correspondence the clusters and the true labels.
Revised: ACC and NMI are in [0, 1], with higher values indicating better correspondence the clusters and the true labels.

Original: Note that true labels never used neither in training, nor in test.
Revised: Note that true labels never used neither in training, nor in test.

Original: Clustering Accuracy (ACC): For data points X = { x 1 , . . .
Revised: Clustering Accuracy (ACC): For data points X = { x 1 , . . .

Original: x n } , let l =
Revised: x n } , let l =

Original: ( l 1 , . . .
Revised: ( l 1 , . . .

Original: l n ) and c = ( c 1 , . . .
Revised: l n ) and c = ( c 1 , . . .

Original: c n ) be the true labels and predicted clusters respectively.
Revised: c n ) be the true labels and predicted clusters respectively.

Original: The ACC is deﬁned as:
Revised: The ACC is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:81) is the collection of all permutations of 1 , . . .
Revised: where (cid:81) is the collection of all permutations of 1 , . . .

Original: g .
Revised: g .

Original: The optimal permutation π can be computed using the Kuhn-Munkres algorithm (Munkres, 1957).
Revised: The optimal permutation π can be computed using the Kuhn-Munkres algorithm (Munkres, 1957).

Original: Normalized Mutual Information (NMI): Let I ( l ; c ) be the mutual information between l and c , and
Revised: Normalized Mutual Information (NMI): Let I ( l ; c ) be the mutual information between l and c , and

Original: H (.) be their entropy.
Revised: H (.) be their entropy.

Original: The NMI is:
Revised: The NMI is:

Original: [Equation]
Revised: [Equation]

Original: 5. E XPERIMENTAL R ESULTS
Revised: 5. E XPERIMENTAL R ESULTS

Original: For each dataset we trained a Siamese network (Hadsell et al., 2006; Shaham & Lederman, 2018) to learn embeddings which represents the afﬁnity of data points by only considering the k-nearest neighbors of each data.
Revised: For each dataset we trained a Siamese network (Hadsell et al., 2006; Shaham & Lederman, 2018) to learn embeddings which represents the afﬁnity of data points by only considering the k-nearest neighbors of each data.

Original: In Table 1, we compare clustering performance across four benchmark datasets.
Revised: In Table 1, we compare clustering performance across four benchmark datasets.

Original: Since most of the clustering approaches do not generalize to unseen data points, all data has been used for the training (Later in Section 5.4, to evaluate the generalizability we use 90%-10% split for training and testing).
Revised: Since most of the clustering approaches do not generalize to unseen data points, all data has been used for the training (Later in Section 5.4, to evaluate the generalizability we use 90%-10% split for training and testing).

Original: While the improvement of CNC is marginal over MNIST, it performs better across other three datasets.
Revised: While the improvement of CNC is marginal over MNIST, it performs better across other three datasets.

Original: Speciﬁcally, over CIFAR-10, CNC outperforms SpectralNet and IIC on ACC by 20.1% and 10.9%[Table] [Table] respectively.
Revised: Speciﬁcally, over CIFAR-10, CNC outperforms SpectralNet and IIC on ACC by 20.1% and 10.9% respectively.

Original: Moreover, the NMI is improved by 12.3%.
Revised: Moreover, the NMI is improved by 12.3%.

Original: The results over Reuters, and CIFAR-100, show 0.021% and 11% improvement on ACC.
Revised: The results over Reuters, and CIFAR-100, show 0.021% and 11% improvement on ACC.

Original: The NMI is also 27% better over CIFAR-100.
Revised: The NMI is also 27% better over CIFAR-100.

Original: The fact that our CNC outperforms existing approaches in most datasets suggests the effectiveness of using our deep learning approach to optimize normalized cuts for clustering. 
Revised: The fact that our CNC outperforms existing approaches in most datasets suggests the effectiveness of using our deep learning approach to optimize normalized cuts for clustering. We performed an ablation study to evaluate the impact of embeddings by omitting this step in

Original: [Figure]
Revised: Algorithm 1.

Original: min_start sec_start min_end sec_end
Revised: We ﬁnd that on both MNIST and Reuters datasets, adding the embedding step improves the performance, but CNC without embeddings still outperforms SpectralNet without embeddings. On MNIST, the ACC and NMI are 0.945 and 0.873, whereas with the embeddings, ACC and NMI increase to 0.972 and 0.924 (Table 1). Without embeddings, CNC outperforms SpectralNet (with ACC of 0.8 and NMI of 0.814).

Original: Total secend total min     13.08333333     7.8     5.516666667     3.
Revised: On Reuters, the ACC and NMI are 0.684 and 0.428, whereas with the embeddings, ACC and NMI increase to 0.824 and 0.583. Again, even without embeddings, CNC outperforms SpectralNet (with ACC of 0.605 and NMI of 0.401). [Table] [Table]

Original: 5. G ENERALIZATION
Revised: 5. G ENERALIZATION

Original: We further evaluate the generalization ability of CNC by dividing the data randomly to a 90%- 10% split and training on the training set and report the ACC and NMI on the test set (Table 2).
Revised: We further evaluate the generalization ability of CNC by dividing the data randomly to a 90%- 10% split and training on the training set and report the ACC and NMI on the test set (Table 2).

Original: Among seven methods in Table 1, only SpectralNet is able to generalize to unseen data points.
Revised: Among seven methods in Table [Table] only SpectralNet is able to generalize to unseen data points.

Original: CNC outperforms SpectralNet in most datasets by up to 21.9% on ACC and up to 10.7% on NMI.
Revised: CNC outperforms SpectralNet in most datasets by up to 21.9% on ACC and up to 10.7% on NMI.

Original: Note that simple arg max over the output of CNC retrieves the clustering assignments while SpectralNet relies on k-means to predict the ﬁnal clusters.
Revised: Note that simple arg max over the output of CNC retrieves the clustering assignments while SpectralNet relies on k-means to predict the ﬁnal clusters.

Original: 5. I MPACT OF N ORMALIZED CUTS IN CLUSTERING
Revised: 5. I MPACT OF N ORMALIZED CUTS IN CLUSTERING

Original: To evaluate the impact of normalized cuts for the clustering task, we calculate the numerical value of the Normalized cuts (Equation 1) over the clustering results of the CNC and SpectralNet.
Revised: To evaluate the impact of normalized cuts for the clustering task, we calculate the numerical value of the Normalized cuts (Equation 1) over the clustering results of the CNC and SpectralNet.

Original: Since such calculation over whole dataset is very expensive we only show this result over the test set.
Revised: Since such calculation over whole dataset is very expensive we only show this result over the test set.

Original: Table 3 shows the numerical value of the Normalized cuts over the clustering results of the CNC and SpectralNet.
Revised: Table 3 shows the numerical value of the Normalized cuts over the clustering results of the CNC and SpectralNet.

Original: As one can see CNC is able to ﬁnd better cuts than the SpectralNet.
Revised: As one can see CNC is able to ﬁnd better cuts than the SpectralNet.

Original: Moreover, we observe that for those datasets that the improvement of the CNC is marginal (MNIST and Reuters), the normalized cuts of CNC are also only slightly better than the SpectralNet, while for the CIFAR10 and CIFAR-100 that the accuracy improved signiﬁcantly the normalized cuts of CNC are also much smaller than SpectralNet.
Revised: Moreover, we observe that for those datasets that the improvement of the CNC is marginal (MNIST and Reuters), the normalized cuts of CNC are also only slightly better than the SpectralNet, while for the CIFAR10 and CIFAR-100 that the accuracy improved signiﬁcantly the normalized cuts of CNC are also much smaller than SpectralNet.

Original: The higher accuracy (ACC in Table 2) and smaller normalized cuts (Table 3), verify that indeed CNC loss function is a good notion for clustering task.
Revised: The higher accuracy (ACC in Table 2) and smaller normalized cuts (Table 3), verify that indeed CNC loss function is a good notion for clustering task.

Original: 5. I MAPACT OF TRAINING DATA SIZE ON THE GENERALIZATION
Revised: 5. I MAPACT OF TRAINING DATA SIZE ON THE GENERALIZATION

Original: As you may see in generalization result (Table 2), when we reduce the size of the training data to 90% the accuracy of CNC slightly changed in compare to training over the whole data (Table 1).
Revised: As you may see in generalization result (Table 2), when we reduce the size of the training data to 90% the accuracy of CNC slightly changed in compare to training over the whole data (Table 1).

Original: Based on this observation, we next investigate how varying the size of the training dataset affects the generalization.
Revised: Based on this observation, we next investigate how varying the size of the training dataset affects the generalization.

Original: In other words, how ACC and NMI of test data change when we vary the size of the training dataset.
Revised: In other words, how ACC and NMI of test data change when we vary the size of the training dataset.

Original: We ran experiment over Routers dataset by dividing the data randomly based on the following data splits: 90%-10%, 70%-30%, 50%-50%, 20%-80%, and 10%-90%.
Revised: We ran experiment over Routers dataset by dividing the data randomly based on the following data splits: 90%-10%, 70%-30%, 50%-50%, 20%-80%, and 10%-90%.

Original: For example, in 10%-90%, we train CNC over 10% of the data and we report the ACC and NMI of CNC over the 90% test set.
Revised: For example, in 10%-90%, we train CNC over 10% of the data and we report the ACC and NMI of CNC over the 90% test set.

Original: Figure 3 shows how the ACC and NMI of CNC over the test data change as the size of the training data is varied.
Revised: Figure 3 shows how the ACC and NMI of CNC over the test data change as the size of the training data is varied.

Original: For example, when the size of the training data is 90%, the ACC of CNC over the test data is 0.824.
Revised: For example, when the size of the training data is 90%, the ACC of CNC over the test data is 0.824.

Original: As we expected and shown in Figure 3 the ACC and NMI of CNC increased as the size of the training data is increased.
Revised: As we expected and shown in Figure 3 the ACC and NMI of CNC increased as the size of the training data is increased.

Original: Interestingly, we observed that with only 10% training data the ACC of CNC is 0.68 which is only 14% lower than the ACC with 90% training data.
Revised: Interestingly, we observed that with only 10% training data the ACC of CNC is 0.68 which is only 14% lower than the ACC with 90% training data.

Original: Similarly the NMI of CNC with 10% training data is only 18% lower than the NMI with 90% training data. 
Revised: Similarly the NMI of CNC with 10% training data is only 18% lower than the NMI with 90% training data. ACC NMI0. min_start sec_start min_end sec_end

Revised: Total secend total min    785 13.08333333     7.8    331 5.516666667    221 3.

Original: 5. M ODEL A RCHITECTURE AND H YPER - PARAMETERS :
Revised: 5. M ODEL A RCHITECTURE AND H YPER - PARAMETERS :

Original: Here are the details of the CNC model for each dataset.
Revised: Here are the details of the CNC model for each dataset.

Original: • MNIST:
Revised: • MNIST:

Original: The Siamese network has 4 layers sized [1024, 1024, 512, 10] with ReLU.
Revised: The Siamese network has 4 layers sized [1024, 1024, 512, 10] with ReLU (Embedding size d is 10).

Original: The clustering module has 2 layers sized [512, 512] with a ﬁnal gumbel softmax layer.
Revised: The clustering module has 2 layers sized [512, 512] with a ﬁnal gumbel softmax layer.

Original: Batch sized is 

Revised: Batch sized is

Original: and we only consider 3 nearest neighbors to ﬁnd the embeddings and constructing the afﬁnity graph for each batch.
Revised: and we only consider 3 nearest neighbors to ﬁnd the embeddings and constructing the afﬁnity graph for each batch.

Original: We use Adam with lr = 0.005 with decay 0.5.
Revised: We use Adam with lr = 0.005 with decay 0.5.

Original: Temperature starts at 1.5 and the minimum is set to 0.5.
Revised: Temperature starts at 1.5 and the minimum is set to 0.5.

Original: • Reuters:
Revised: • Reuters:

Original: The Siamese network has 3 layers sized [512, 256, 128] with ReLU.
Revised: The Siamese network has 3 layers sized [512, 256, 128] with ReLU (Embedding size d is 128).

Original: The clustering module has 3 layers sized [512, 512, 512] with tanh activation and a ﬁnal gumbel softmax layer.
Revised: The clustering module has 3 layers sized [512, 512, 512] with tanh activation and a ﬁnal gumbel softmax layer.

Original: Batch sized is 

Revised: Batch sized is

Original: and we only consider 3 nearest neighbors to ﬁnd the embeddings and constructing the afﬁnity graph for each batch.
Revised: and we only consider 3 nearest neighbors to ﬁnd the embeddings and constructing the afﬁnity graph for each batch.

Original: We use Adam with lr = 1e-4 with decay 0.5.
Revised: We use Adam with lr = 1e-4 with decay 0.5.

Original: Temperature starts at 1. and the minimum is set to 1.0.
Revised: Temperature starts at 1.5 and the minimum is set to 1.0.

Original: • CIFAR-10:
Revised: • CIFAR-10:

Original: The Siamese network has 2 layers sized [512, 256] with ReLU.
Revised: The Siamese network has 2 layers sized [512, 256] with ReLU (Embedding size d is 256).

Original: The clustering module has 2 layers sized [512, 512] with tanh activation and a ﬁnal gumbel softmax layer.
Revised: The clustering module has 2 layers sized [512, 512] with tanh activation and a ﬁnal gumbel softmax layer.

Original: Batch sized is 

Revised: Batch sized is

Original: and we only consider 2 nearest neighbors to ﬁnd the embeddings and constructing the afﬁnity graph for each batch.
Revised: and we only consider 2 nearest neighbors to ﬁnd the embeddings and constructing the afﬁnity graph for each batch.

Original: We use Adam with lr = 1e-4 with decay 0.1.
Revised: We use Adam with lr = 1e-4 with decay 0.1.

Original: Temperature starts at 2.5 and the minimum is set to 0.5.
Revised: Temperature starts at 2.5 and the minimum is set to 0.5.

Original: • CIFAR-100: The Siamese network has 2 layers sized [512, 256] with ReLU.
Revised: • CIFAR-100: The Siamese network has 2 layers sized [512, 256] with ReLU (Embedding size d is 256).

Original: The clustering module has 3 layers sized [512, 512, 512] with tanh activation and a ﬁnal gumbel softmax layer.
Revised: The clustering module has 3 layers sized [512, 512, 512] with tanh activation and a ﬁnal gumbel softmax layer.

Original: Batch sized is 1024 and we only consider 3 nearest neighbors to ﬁnd the embeddings and constructing the afﬁnity graph for each batch.
Revised: Batch sized is 1024 and we only consider 3 nearest neighbors to ﬁnd the embeddings and constructing the afﬁnity graph for each batch.

Original: We use Adam with lr = 1e-3 with decay 0.5.
Revised: We use Adam with lr = 1e-3 with decay 0.5.

Original: Temperature starts at 1.
Revised: Temperature starts at 1.5 and the minimum is set to 1.0.

Revised: Hyper parameter Sensitivity: We train the CNC on the Reuters dataset by ﬁxing some hyperparameters and varying others.

Original:  and the minimum is set to 1.0.
Revised: We noticed that CNC beneﬁts from tuning the number of hidden layers (hl), learning rate (lr), batch size (b), and the number of nearest neighbors (k), but is not particularly sensitive to any of the other hyper-parameters, including decay rate, patience parameter (cadence of epochs where decay is applied), Gumbel-Softmax temperature or minimum temperature (Figure 4). More precisely, we varied decay rate over the range [0.1-1.0], patience from [5-25] epochs, Gumbel-Softmax temperature from [1.0-2.0], and minimum temperature from [0.5-1.0]. When we ﬁx hl=3, lr=5e-5, b=64, and k=3, the average accuracy is 0 . 803 ± 2 e − 3 .

Original: C ONCLUSION
Revised: With hl=3, lr=5e-4, b=512, and k=10, the average accuracy is 0 . 811 ± 2 e − 3 . With hl=3, lr=5e-4, b=128, and k=3, the average accuracy is 0 . 821 ± 4 e − 3 .

Revised: With hl=2, lr=1e-4, b=256, and k=3, the average accuracy is 0 .

Revised: 766 ± 9 e − 4 .

Revised: And ﬁnally with hl=4, lr=1e-5, b=512, and k=3, the average accuracy is 0 .

Revised: 766 ± 7 e − 3 .

Revised: As one can see, the accuracy varied from 0.766 to 0.821.

Original: We propose CNC ( C lustering by learning to optimize N ormalized C uts ), a framework for learning to cluster unlabeled examples.
Revised: We propose CNC ( C lustering by learning to optimize N ormalized C uts ), a framework for learning to cluster unlabeled examples.

Original: We deﬁne a differentiable loss function equivalent to the expected normalized cuts and use it to train CNC model that directly outputs ﬁnal cluster assignments.
Revised: We deﬁne a differentiable loss function equivalent to the expected normalized cuts and use it to train CNC model that directly outputs ﬁnal cluster assignments.

Original: CNC achieves state-of-the-art results on popular unsupervised clustering benchmarks (MNIST, Reuters,
Revised: CNC achieves state-of-the-art results on popular unsupervised clustering benchmarks (MNIST, Reuters,

Original: CIFAR-10, and CIFAR-100 and outperforms the strongest baselines by up to 10.9%.
Revised: CIFAR-10, and CIFAR-100 and outperforms the strongest baselines by up to 10.9%.

Original: CNC also enables generation, yielding up to 21.9% improvement over SpectralNet (Shaham et al., 2018), the previous best-performing generalizable clustering approach.  
Revised: CNC also enables generation, yielding up to 21.9% improvement over SpectralNet (Shaham et al., 2018), the previous best-performing generalizable clustering approach.  A PPENDIX

Revised: Graph partitioning is a well-studied subject in computer science (Karypis & Kumar, 2000; Karypis et al., 1999; Karypis & Kumar, 1998;

Revised: Miettinen et al., 2006; Sanders & Schulz, 2013; Andersen et al., 2006; Chung, 2007) with numerous applications in computer vision, VLSI design, biology, social networks, transportation networks and more.

Revised: In this section we evaluate the performance of the CNC to partition computational graphs.

Revised: In such graphs, nodes represent operations (e g., MatMul, Conv2d, Sum) and edges represent the computational ﬂow.

Revised: Partitioning of the computational graph can be used for efﬁcient mapping of the computation across the underlying hardware (e g., CPUs and GPUs).

Revised: Minimizing the normalized cuts translates to less communication between devices and balanced computation on each device.

Revised: 7. D ATASETS AND B ASELINE M ETHODS

Revised: We conducted experiments on ﬁve widely used TensorFlow computation graphs: ResNet , Inceptionv3 , AlexNet , MNIST-conv and MNIST-conv .

Revised: Clustering of operations can be used for the mapping of the computation across the underlying hardware (e.g., CPUs and GPUs).We use the open source partitioners hMETIS (Karypis & Kumar, 2000) and KaHIP (Sanders & Schulz, 2013), a family of graph partitioning programs based on (Sanders & Schulz, 2012b;a) to ﬁnd high quality ground truth partitions on TensorFlow computation graphs.

Revised: More speciﬁcally, KaHIP includes KaFFPa (Karlsruhe

Revised: Fast Flow Partitioner) and KaFFPaE (KaFFPaEvolutionary).

Revised: KaFFPaE is a parallel evolutionary algorithm that uses KaFFPa’s combine and mutation operations, as well as KaBaPE which extends the evolutionary algorithm.

Revised: We compare the generalization results of CNC against the best results among the partitioners.

Revised: The details of the datasets are as follows:

Revised: • ResNet (He et al., 2016) is a deep convolutional network with residual connections.

Revised: The TensorFlow implementation of ResNet_v1_50 with 50 layers contains 20,586 operations.

Revised: • Inception-v3 (Szegedy et al., 2017) consists of multiple blocks, each composed of several convolutional and pooling layers.

Revised: The TensorFlow graph of this model contains 27,114 operations.

Revised: • AlexNet

Revised: (Krizhevsky et al., 2012) consists of 5 convolutional layers, some of which are followed by maxpool layers, and 3 dense layers with a ﬁnal softmax.

Revised: The TensorFlow graph of this model has 798 operations.

Revised: • MNIST-conv has 3 convolutional layers.

Revised: Its TensorFlow graph contains 414 operations.

Revised: • VGG (Simonyan & Zisserman, 2014) has 16 convolutional layers.

Revised: The TensorFlow graph of VGG contains 1,325 operations.

Revised: 7. P ERFORMANCE M EASURES

Revised: To evaluate the quality of CNC ’s clusterings, we use two commonly used performance metrics in graph partitioning: 1) Edge cut (Cut) : the ratio of the cut to the total number of edges, and 2)

Revised: Balancedness (Bal) : one minus the MSE between the number of nodes in every cluster and the number of nodes in an ideal balanced cluster ( ng ).

Revised: Both Cut and Bal are between 0 and 1.

Revised: A lower Cut is better while a higher Bal is better.

Revised: The CNC loss function in Equation 6 only considers optimizing the expected normalized cut.

Revised: We add a regularizer to improve balancedness between clusters.

Revised: [Table]

Revised: 7. E XPERIMENTAL R ESULTS

Revised: To show that CNC generalizes effectively on unseen graphs, we train CNC on a single TensorFlow graph, VGG , and validate on MNIST-conv .

Revised: During inference, we test the trained model on unseen

Revised: TensorFlow graphs: AlexNet , ResNet , and Inception-v3 .

Revised: We consider the best quality result among hMETIS, KaFFPa, and KaFFPaE as the ground truth.

Revised: The ground truth for AlexNet is Bal = 99%, Cut = 4.6%, for Inception-v3 , is Bal = 99%, Cut = 3.7%, and for ResNet is Bal = 99% and Cut = 3.3%.

Revised: Table 4 shows the result of our experiments, and illustrates the importance of graph embeddings in generalization.

Revised: The operation type (such as Add, Conv2d, and L2loss in TensorFlow) is used as the node feature as a one-hot.

Revised: We leverage GCN (Kipf & Welling, 2017) and GraphSAGE (Hamilton et al., 2017) to capture similarities across graphs.

Revised: In GraphSAGE-on both node embedding and clustering modules are trained jointly, while in GCN and GraphSAGE-off , only the clustering module is trained.

Revised: Table 4 shows that the GraphSAGE-on (last row) achieves the best performance and generalizes better than the other models.

Revised: Note that this model is trained on a single graph, VGG with only 1325 nodes, and is tested on AlexNet , ResNet , and Inception-v3 with 798, 20586, andnodes respectively.

Revised: On the other hand, the ground truth is the result of running different partitioning algorithms on each graph individually.

Revised: In this work, our goal is not to beat the existing graph partitioning algorithms which involve a lot of heuristics on a given graph.

Revised: Our generalization results show promise that rather than using heuristics, CNC is able to learn graph structure for generalizable graph partitioning.

Revised: Model Architecture and Hyper-parameters: The details of the model with the best performance ( GraphSAGE-on ) are as follows: the input feature dimension is 1518.

Revised: GraphSAGE has 5 layers sized 512 with shared pooling, and the graph clustering module has 3 layers sized 64 with a ﬁnal softmax layer.

Revised: We use ReLU, Xavier initialization (Glorot & Bengio, 2010), and Adam with lr = 7.5e-5.