Original: N EURAL C LUSTERING B Y P REDICTING AND C OPYING N OISE
Revised: N EURAL C LUSTERING B Y P REDICTING AND C OPYING N OISE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a neural clustering model that jointly learns both latent features and how they cluster.
Revised: We propose a neural clustering model that jointly learns both latent features and how they cluster.

Original: Unlike similar methods our model does not require a predeﬁned number of clusters.
Revised: Unlike similar methods our model does not require a predeﬁned number of clusters.

Original: Using a supervised approach, we agglomerate latent features towards randomly sampled targets within the same space whilst progressively removing the targets until we are left with only targets which represent cluster centroids.
Revised: Using a supervised approach, we agglomerate latent features towards randomly sampled targets within the same space whilst progressively removing the targets until we are left with only targets which represent cluster centroids.

Original: To show the behavior of our model across different modalities we apply our model on both text and image data and achieve state-of-the-art results on MNIST.
Revised: To show the behavior of our model across different modalities we apply our model on both text and image data and achieve very competitive results on MNIST against methods that require a predeﬁned number of clusters.

Original: Finally, we also provide results against baseline models for fashion-MNIST, the 20 newsgroups dataset, and a Twitter dataset we ourselves create.
Revised: Finally, we also provide results against baseline models for fashion-MNIST, the 20 newsgroups dataset, and a Twitter dataset we ourselves create.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Clustering is one of the fundamental problems of unsupervised learning.
Revised: Clustering is one of the fundamental problems of unsupervised learning.

Original: It involves the grouping of items into clusters such that items within the same cluster are more similar than items in different clusters.
Revised: It involves the grouping of items into clusters such that items within the same cluster are more similar than items in different clusters.

Original: Crucially, the ability to do this often hinges upon learning latent features in the input data which can be used to differentiate items from each other in some feature space.
Revised: Crucially, the ability to do this often hinges upon learning latent features in the input data which can be used to differentiate items from each other in some feature space.

Original: Two key questions thus arise: How do we decide upon cluster membership?
Revised: Two key questions thus arise: How do we decide upon cluster membership?

Original: and How do we learn good representations of data in feature space?
Revised: and How do we learn good representations of data in feature space?

Original: Spurred initially by studies into the division of animals into taxa (Sokol & Sneath, 1963), cluster analysis matured as a ﬁeld in the subsequent decades with the advent of various models.
Revised: Spurred initially by studies into the division of animals into taxa (Sokol & Sneath, 1963), cluster analysis matured as a ﬁeld in the subsequent decades with the advent of various models.

Original: These included distribution-based models, such as Gaussian mixture models (Duda et al., 1973); densitybased models, such as DBSCAN (Ester et al., 1996); centroid-based models, such as k-means.
Revised: These included distribution-based models, such as Gaussian mixture models (Duda et al., 1973); densitybased models, such as DBSCAN (Ester et al., 1996); centroid-based models, such as k-means.

Original: 2 and hierarchical models, including agglomerative (Orloci, 1967) and divisive models (Gower, 1967).
Revised: 2 and hierarchical models, including agglomerative (Orloci, 1967) and divisive models (Gower, 1967).

Original: While the cluster analysis community has focused on the unsupervised learning of cluster membership , the deep learning community has a long history of unsupervised representation learning , yielding models such as variational autoencoders (Kingma & Welling, 2013), generative adversarial networks (Goodfellow et al., 2014), and vector space word models (Mikolov et al., 2013).
Revised: While the cluster analysis community has focused on the unsupervised learning of cluster membership , the deep learning community has a long history of unsupervised representation learning , yielding models such as variational autoencoders (Kingma & Welling, 2013), generative adversarial networks (Goodfellow et al., 2014), and vector space word models (Mikolov et al., 2013).

Original: In this paper, we propose using noise as targets for agglomerative clustering (or NATAC).
Revised: In this paper, we propose using noise as targets for agglomerative clustering (or NATAC).

Original: As in
Revised: As in

Original: Bojanowski & Joulin (2017) we begin by sampling points in features space called noise targets which we match with latent features.
Revised: Bojanowski & Joulin (2017) we begin by sampling points in features space called noise targets which we match with latent features.

Original: During training we progressively remove targets and thus agglomerate latent features around fewer and fewer target centroids.
Revised: During training we progressively remove targets and thus agglomerate latent features around fewer and fewer target centroids using a simple heuristic.

Original: To tackle the instability of such training we augment our objective with an auxiliary loss which prevents the model from collapsing and helps it learn better representations.
Revised: To tackle the instability of such training we augment our objective with an auxiliary loss which prevents the model from collapsing and helps it learn better representations.

Original: We explore the performance of our model across different modalities in Section 3.
Revised: We explore the performance of our model across different modalities in Section 3.

Original: Recently, there have been several attempts at jointly learning both cluster membership and good representations using end-to-end differentiable methods.
Revised: Recently, there have been several attempts at jointly learning both cluster membership and good representations using end-to-end differentiable methods.

Original: Similarly to us, Yang et al.
Revised: Similarly to us, Yang et al.

Original: (2016) use a policy to move points around in feature space but they use a different agglomerative strategy.
Revised: (2016a) use a policy to agglomerate points at each training step but they require a given number of clusters to stop agglomerating at.

Revised: Law et al.

Revised: (2017) propose a form of supervised neural clustering which outperforms classiﬁers in certain datasets.

Original: Liao et al.
Revised: Liao et al.

Original: (2016) propose jointly learning representations and clusters by using a k-means style objective.
Revised: (2016) propose jointly learning representations and clusters by using a k-means style objective.

Original: Xie et al.
Revised: Xie et al.

Original: (2016) introduce deep embedding clustering (DEC) which learns a mapping from data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective [Figure]
Revised: (2016) introduce deep embedding clustering (DEC) which learns a mapping from data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective (however, as opposed to the hard assignment we use, they optimize based on soft assignment).

Original: , as opposed optimize
Revised: Additionally, there have been unsupervised clustering methods that assume a linear mapping from data to a low dimensional clustering space (Yang et al., 2016b; Bresson et al., 2013; Yang et al., 2012) which perform competitively to current neural methods.

Original: Unlikeus, all of the above papers require a predeﬁned number of clusters.
Revised: Unlike all of the above papers, our method does not require a predeﬁned number of clusters.

Original:  M ODEL
Revised:  M ODEL

Original: We begin by discussing the use of noise as targets (NAT) introduced in Bojanowski & Joulin (2017) which is crucial to the understanding of our model.
Revised: We begin by discussing the use of noise as targets (NAT) introduced in Bojanowski & Joulin (2017) which is crucial to the understanding of our model.

Original: We then describe the intuition behind our approach, then proceed to describe the mechanism itself.
Revised: We then describe the intuition behind our approach, then proceed to describe the mechanism itself.

Original: 2. N OISE AS T ARGETS (NAT)
Revised: 2. N OISE AS T ARGETS (NAT)

Original: Bojanowski & Joulin (2017) proposed a new form of unsupervised learning re-posited as a supervised problem where the task is to learn an encoder function f θ : X → Z from input space to a latent representation space.
Revised: Bojanowski & Joulin (2017) proposed a new form of unsupervised learning re-posited as a supervised problem where the task is to learn an encoder function f θ : X → Z from input space to a latent representation space.

Original: The objective is to minimize the L 2 loss between representations z i ∈ Z , where points in Z are unit normalized, and corresponding targets y k ∈ Y where the targets are uniformly sampled from the L 2 unit sphere (thus inhabiting the same space as the z i ).
Revised: The objective is to minimize the L 2 loss between representations z i ∈ Z , where points in Z are unit normalized, and corresponding targets y k ∈ Y where the targets are uniformly sampled from the L 2 unit sphere (thus inhabiting the same space as the z i ).

Original: Instead of being tied to corresponding representations as in classic regression, targets are one-to-one re-assigned to different representations after every training step so as to minimize the total sum of distances between matched pairs (this is done using the Hungarian method (Kuhn, 1955)).
Revised: Instead of being tied to corresponding representations as in classic regression, targets are one-to-one reassigned to different representations after every training step so as to minimize the total sum of distances between matched pairs (this is done using the Hungarian method (Kuhn, 1955)).

Original: This pushes the representations
Revised: This pushes the representations

Original: z i of similar inputs x i ∈
Revised: z i of similar inputs x i ∈

Original: X to neighborhoods of similar targets y k .
Revised: X to neighborhoods of similar targets y k .

Revised: As every example must be paired to a single example in the noise targets, it also forces a model to learn a mapping to latent space that very closely matches the distribution of the noise targets.

Original: The motivation behind using NAT was to learn unsupervised features from input data.
Revised: The motivation behind using NAT was to learn unsupervised features from input data.

Original: They show their method performs on par with state-of-the-art unsupervised representation learning methods.
Revised: They show their method performs on par with state-of-the-art unsupervised representation learning methods.

Original: 2. N OISE AS T ARGETS FOR A GGLOMERATIVE C LUSTERING (NATAC)
Revised: 2. N OISE AS T ARGETS FOR A GGLOMERATIVE C LUSTERING (NATAC)

Original: Viewed from a clustering perspective we can think of the targets
Revised: Viewed from a clustering perspective we can think of the targets

Original: y i as cluster centroids to which the latent representations
Revised: y i as cluster centroids to which the latent representations

Original: z i are (one-to-one) assigned.
Revised: z i are (one-to-one) assigned.

Original: Note that although the method introduced in
Revised: Note that although the method introduced in

Original: Bojanowski & Joulin (2017) brings the representations of similar x i closer together (by matching and moving them closer to neighborhoods of similar targets) it cannot produce many-to-one matchings or match multiple similar z i with a single centroid thus forming a cluster.
Revised: Bojanowski & Joulin (2017) brings the representations of similar x i closer together (by matching and moving them closer to neighborhoods of similar targets) it cannot produce many-to-one matchings or match multiple similar z i with a single centroid thus forming a cluster.

Original: Simply changing the re-assignment policy to allow for many-to-one matchings is difﬁcult because it causes the model to collapse in on a single target.
Revised: Simply changing the re-assignment policy to allow for many-to-one matchings is difﬁcult because it causes the model to collapse in on a single target.

Original: In this paper we use the above to propose a new form of neural clus- tering where we progressively delete targets over time and re-assign representations to other nearby targets, all while keeping the model stable using an auxiliary objective.
Revised: In this paper we use the above to propose a new form of neural clustering where we progressively delete targets over time and re-assign representations to other nearby targets, all while keeping the model stable using an auxiliary objective.

Original: Delete-and-copy To be able to cluster latent representations we need a way of assigning them to cluster centroids.
Revised: Delete-and-copy To be able to cluster latent representations we need a way of assigning them to cluster centroids.

Original: We propose doing this via an additional delete-and-copy step which allows for many-to-one matchings.
Revised: We propose doing this via an additional delete-and-copy step which allows for many-to-one matchings.

Original: Similarly to the NAT method, we ﬁrst assign representations z to targets y using the Hungarian method.
Revised: Similarly to the NAT method, we ﬁrst assign representations z to targets y using the Hungarian method.

Revised: In some cases, the optimally assigned target y opt is not the nearest target to a latent representation z .

Original: We then remove the assigned target and reassign a copy of thenext nearest target for each representations with some probability α as in Algorithm 1 (see also Figure 1).
Revised: In this case, we remove the assigned target and reassign a copy of the nearest target for that z with some probability α as in Algorithm 1 (see also Figure 1).

Original: This has the effect of not only reassigning targets so as to minimize the distance between matched pairs, but also to encourage the model to allow similar examples to be assigned to the same target.
Revised: This has the effect of not only reassigning targets so as to minimize the distance between matched pairs, but also to encourage the model to allow similar examples to be assigned to the same target.

Original: The new assignments are denoted as y new i .
Revised: The new assignments are denoted as y new i .

Original: The loss is then deﬁned as:
Revised: The loss is then deﬁned as:

Original: [Equation]
Revised: [Equation]

Original:  Auxiliary objective To prevent the model from collapsing to a single point, we introduce an auxiliary objective in addition to the loss between representations and targets.
Revised: [Figure] Auxiliary objective To prevent the model from collapsing to a single point, we introduce an auxiliary objective in addition to the loss between representations and targets.

Original: In our case we set the auxiliary objective L aux to be the reconstruction loss (cid:80) i (cid:107) x
Revised: In our case we set the auxiliary objective L aux to be the reconstruction loss (cid:80) i (cid:107) x

Original: i − f dec ( z i ) (cid:107) 2 where f dec is some decoder network.
Revised: i − f dec ( z i ) (cid:107) 2 where f dec is some decoder network.

Original: The ﬁnal objective L is then a weighted sum of the NAT loss L NAT (which in our case is the L 2 loss) and the auxiliary objective L aux :
Revised: The ﬁnal objective L is then a weighted sum of the NAT loss L NAT (which in our case is the L 2 loss) and the auxiliary objective L aux :

Original: [Equation]
Revised: [Equation]

Original: Importantly, the auxiliary objective not only prevents the model from collapsing, it also informs how the model clusters.
Revised: Importantly, the auxiliary objective not only prevents the model from collapsing, it also informs how the model clusters.

Original: For example, when clustering images the reconstruction loss encourages the model to cluster on similar pixel values.
Revised: For example, when clustering images the reconstruction loss encourages the model to cluster on similar pixel values.

Original: Alternative forms of auxiliary objectives could allow for a discriminator loss or a classiﬁcation loss for tackling semi-supervised settings.
Revised: Alternative forms of auxiliary objectives could allow for a discriminator loss or a classiﬁcation loss for tackling semi-supervised settings.

Original: As our goal is unsupervised clustering, we only consider the reconstruction loss.
Revised: As our goal is unsupervised clustering, we only consider the reconstruction loss.

Original: Model deﬁnition During initialization each example x i in the dataset is paired with a random target y i , uniformly sampled from a d -dimensional sphere.
Revised: Model deﬁnition During initialization each example x i in the dataset is paired with a random target y i , uniformly sampled from a d -dimensional sphere.

Original: The NATAC model is then trained using minibatches from the dataset.
Revised: The NATAC model is then trained using minibatches from the dataset.

Original: Each training step can be broken down as follows:
Revised: Each training step can be broken down as follows:

Original: Forward step: The examples x from a random batch of example-target pairs (cid:0) ( x 1 , y 1 ) . . .
Revised: Forward step: The examples x from a random batch of example-target pairs (cid:0) ( x 1 , y 1 ) . . .

Original: ( x i , y i ) . . .
Revised: ( x i , y i ) . . .

Original: ( x N , y N ) (cid:1) are embedded using an encoder function f θ : R n → R d into corresponding latent representations z i ∈ R d where (cid:107)
Revised: ( x N , y N ) (cid:1) are embedded using an encoder function f θ : R n → R d into corresponding latent representations z i ∈ R d where (cid:107)

Original: z i (cid:107) =
Revised: z i (cid:107) =

Original: 1 for all i .
Revised: 1 for all i .

Original: Re-assignment step: Using the Hungarian method from Kuhn (1955), the representations z are optimally one-to-one matched with targets y so as to minimize the total sum of distances between matched pairs in the batch.
Revised: Re-assignment step: Using the Hungarian method from Kuhn (1955), the representations z are optimally one-to-one matched with targets y so as to minimize the total sum of distances between matched pairs in the batch.

Original: The newly assigned example-target pairing (cid:0) ( x 1 , y opt 1 ) . . .
Revised: The newly assigned example-target pairing (cid:0) ( x 1 , y opt 1 ) . . .

Original: ( x i , y opt i ) . . .
Revised: ( x i , y opt i ) . . .

Original: ( x N , y opt N ) (cid:1) has the permutation of labels within the batch to minimize the batch-wise loss.
Revised: ( x N , y opt N ) (cid:1) has the permutation of labels within the batch to minimize the batch-wise loss.

Original: Delete-and-copy (DnC): With a probability α , delete the optimal target y opt i and instead assign the nearest target to z i (which may be the same target) for each example-target pair in the batch producing (cid:0) ( x 1 , y new 1 ) . . .
Revised: Delete-and-copy (DnC): With a probability α , delete the optimal target y opt i and instead assign the nearest target to z i (which may be the same target) for each example-target pair in the batch producing (cid:0) ( x 1 , y new 1 ) . . .

Original: ( x i , y new i ) . . .
Revised: ( x i , y new i ) . . .

Original: ( x N , y new N ) (cid:1) as described in Algorithm 1 (see also Figure 1).
Revised: ( x N , y new N ) (cid:1) as described in Algorithm 1 (see also Figure 1).

Original: Train and update step: The L 2 loss between targets and latent representations is taken and combined with the auxiliary loss.
Revised: Train and update step: The L 2 loss between targets and latent representations is taken and combined with the auxiliary loss.

Original: Gradients w.r.t.
Revised: Gradients w.r.t.

Original: θ are then taken and back-propagated along.
Revised: θ are then taken and back-propagated along.

Original: Notice that although the (re)-assignment step follows a non-differentiable policy, the  model is still end-to-end differentiable.
Revised: Notice that although the (re)-assignment step follows a non-differentiable policy, the do y else y end end return model is still end-to-end differentiable.

Original: Finally, the new example-target assignments are kept after the training step, and persist into the next training step where they are reassigned again.
Revised: Finally, the new example-target assignments are kept after the training step, and persist into the next training step where they are reassigned again.

Original:  I MPLEMENTATION D ETAILS
Revised: 2. I MPLEMENTATION D ETAILS

Original: Stopping criterion During training the number of unique targets is tracked.
Revised: Stopping criterion During training the number of unique targets is tracked.

Original: We stop training when the number of unique targets stops decreasing after an epoch of training.
Revised: We stop training when the number of unique targets stops decreasing after an epoch of training.

Original: Multi-stage training We found that an initial period of training where the auxiliary objective is prioritized (i.e. the NAT loss is multiplied by a very small coefﬁcient) and the DnC policy is not used, improved overall performance.
Revised: Multi-stage training We found that an initial period of training where the auxiliary objective is prioritized (i.e. the NAT loss is multiplied by a very small coefﬁcient) and the DnC policy is not used, improved overall performance.

Original: Transitioning to a higher NAT loss and turning on the deleteand-copy policy later on in training increased the stability of our model.
Revised: Transitioning to a higher NAT loss and turning on the deleteand-copy policy later on in training increased the stability of our model.

Original: We therefore propose training NATAC models as follows:
Revised: We therefore propose training NATAC models as follows:

Original: Warm-Up stage : α = 0 and λ are very small.
Revised: Warm-Up stage : α = 0 and λ is very small.

Original: Transition stage : α increases gradually from 0 to 1 , λ also increases gradually to a larger value (approximately 100 × larger than its initial value).
Revised: Transition stage : α increases gradually from 0 to 1 , λ also increases gradually to a larger value (approximately 100 × larger than its initial value).

Original: Clustering stage : α = 1 , λ is large.
Revised: Clustering stage : α = 1 , λ is large.

Original: Dimensionality of the latent space
Revised: Dimensionality of the latent space

Original: In all of our experiments, we found that the best performing models tend to have a latent space dimensionality between 6 and 12 .
Revised: In all of our experiments, we found that the best performing models tend to have a latent space dimensionality between 4 and 12 .

Original: At dimensionalities much larger than this, the model collapses to very few points during the transition stage of training, possibly due to the high expressiveness of the latent space.
Revised: At dimensionalities much larger than this, the model collapses to very few points during the transition stage of training, possibly due to the high expressiveness of the latent space.

Original: On the other hand, using a low dimensional representation results in an information bottleneck too small to sufﬁciently learn the auxiliary objective.
Revised: On the other hand, using a low dimensional representation results in an information bottleneck too small to sufﬁciently learn the auxiliary objective.

Original: For example, when clustering tweets from our Twitter dataset, a latent space of two dimensions was too small for the decoder to reliably reconstruct tweets from latent vectors.
Revised: For example, when clustering tweets from our Twitter dataset, a latent space of two dimensions was too small for the decoder to reliably reconstruct tweets from latent vectors.

Original: With an auxiliary objective that cannot be effectively learned, centroids collapse to a single point.
Revised: With an auxiliary objective that cannot be effectively learned, centroids collapse to a single point.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now describe the datasets and evaluation metrics used in our experiments followed by the presentation and analysis of our results in comparison to others.
Revised: We now describe the datasets and evaluation metrics used in our experiments followed by the presentation and analysis of our results in comparison to others.

Original: The full details regarding the hyperparameters used in our experiments can be found in appendix D.
Revised: The full details regarding the hyperparameters used in our experiments can be found in appendix D.

Original: Datasets We evaluate our models on four different datasets - two image and two text datasets.
Revised: Datasets We evaluate our models on four different datasets - two image and two text datasets.

Original: For images we use MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017).
Revised: For images we use MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017).

Original: For text we use 20 Newsgroups (Joachims, 1996) and a Twitter dataset which we gather ourselves. 
Revised: For text we use 20 Newsgroups (Joachims, 1996) and a Twitter dataset which we gather ourselves. [Table]

Original: 3. E VALUATION
Revised: 3. E VALUATION

Original: Our key evaluation metric is the normalized mutual information (NMI) score (Strehl & Ghosh, 2002) which measures the information gained from knowing cluster membership whilst taking the number of clusters into account.
Revised: Our key evaluation metric is the normalized mutual information (NMI) score (Strehl & Ghosh, 2002) which measures the information gained from knowing cluster membership whilst taking the number of clusters into account.

Original: Values of NMI range from 0 , where clustering gives no information, to 1 , where clustering gives perfect information i.e. cluster membership is identical to class membership.
Revised: Values of NMI range from 0 , where clustering gives no information, to 1 , where clustering gives perfect information i.e. cluster membership is identical to class membership.

Original: In our experiments, we train models on the concatenation of the train and validation sets and evaluate them on the test set.
Revised: In our experiments [Table] we train models on the concatenation of the train and validation sets and evaluate them on the test set.

Original: This is done by computing the latent representations of test examples and then assigning them to their nearest respective centroids, then computing the NMI score.
Revised: This is done by computing the latent representations of test examples and then assigning them to their nearest respective centroids, then computing the NMI score.

Original: Additionally, we provide classiﬁcation error scores on the MNIST dataset to compare ourselves to other related methods.
Revised: Additionally, we provide classiﬁcation error scores on the MNIST dataset to compare ourselves to other related methods.

Revised: We also compare our model to clustering methods trained on the 20 Newsgroups dataset.

Original: The guiding motivation behind our experiments is to analyze how well our models learns cluster membership and latent representations.
Revised: The guiding motivation behind our experiments is to analyze how well our models learns cluster membership and latent representations.

Original: 3. MNIST
Revised: 3. MNIST

Original: Introduced by LeCun et al.
Revised: Introduced by LeCun et al.

Original: (1998), MNIST is canonical dataset for numerous machine learning tasks including clustering.
Revised: (1998), MNIST is canonical dataset for numerous machine learning tasks including clustering.

Original: MNIST has ten classes, corresponding to the ten digits 0-9, and contains 60,000 train and 10,000 test examples.
Revised: MNIST has ten classes, corresponding to the ten digits 0-9, and contains 60,000 train and 10,000 test examples.

Original: We train our model with an auxiliary reconstruction loss and use small convolutional architectures (see Figure 6) for our encoder and decoder networks.
Revised: We train our model with an auxiliary reconstruction loss and use small convolutional architectures (see Figure 5) for our encoder and decoder networks.

Original: As points of comparison we also provide results of using k-means on the latent representations learned by our model (NATAC-k in Table 1) and k-means on representations learned by a simple autoencoder with the same encoder and decoder architecture (AE-k in Table 1).
Revised: As points of comparison we also provide results of using k-means on the latent representations learned by our model (NATAC-k in Table 1) and k-means on representations learned by a simple autoencoder with the same encoder and decoder architecture (AE-k in Table 1).

Original: Table 1 shows our model’s performance is best when d = 10 and worse for much lower or higher values of d .
Revised: Table 1 shows our model’s performance is best when d = 10 and worse for much lower or higher values of d .

Original: This indicates that the dimensionality of the latent space impacts our model’s performance (see Section 2.3 for further discussion).
Revised: This indicates that the dimensionality of the latent space impacts our model’s performance (see Section 2.3 for further discussion).

Original: The ability of our model to cluster MNIST examples well is shown by two keys results.
Revised: The ability of our model to cluster MNIST examples well is shown by two keys results.

Original: First, it beats both NATAC-k and AE-k (Table 1).
Revised: First, it beats both NATAC-k and AE-k (Table 1).

Original: Second, it achieves state-of-the-art results when compared to other methods (see NMI column in Table 2).
Revised: Second, it achieves very competitive results when compared to other methods (see NMI column in Table 2).

Original: Note that the second result reported in Yang et al.

Original: (2016) is achieved using a separate re-clustering step after training.

Original: To re-iterate, other clustering techniques cited in Table 2 required a predeﬁned number of clusters (in the case of MNIST k=10).
Revised: To re-iterate, other clustering techniques cited in Table 2 required a predeﬁned number of clusters (in the case of MNIST k=10).

Original: We note that NATAC-k beats AE-k, indicating that our model learns better representation as compared to a simple autoencoder.
Revised: We note that NATAC-k beats AE-k indicating that our model learns representations that suit k-means clustering more than a simple autoencoder.

Original: Consequently, NATAC also achieves a state-of-the-art NMI on MNIST, as measured by the classiﬁcation error result (see Table 2).
Revised: However, we note that this is not consistent across all modalities in this paper (see results in section 3.4).

Original: Finally, we discuss the number of centroids our model converges on (see Table 1).
Revised: Finally, we discuss the number of centroids our model converges on (see Table 1).

Original: We show that our model is successfully capable of ﬁnding centroids that represent different digits, as shown in the top row of Figure 2.
Revised: We show that our model is successfully capable of ﬁnding centroids that represent different digits, as shown in the top row of Figure 2.

Original: However, the model also learns centroids that contain very few examples for which the decoded images do not represent any handwritten digits, as shown in the second row of Figure 2.
Revised: However, the model also learns centroids that contain very few examples for which the decoded images do not represent any handwritten digits, as shown in the second row of Figure 2.

Original: Even with these “dead centroids”, the model still performs well.
Revised: Even with these “dead centroids”, the model still performs well.

Original: Interestingly, the model also differentiates between ones with different slopes.
Revised: Indeed, the twelve most dense centroids contain 98% of all of the examples in MNIST (out of a total of 61).Interestingly, the model also differentiates between ones with different slopes.

Original: This suggests that the latent representations of these digits are sufﬁciently far apart to warrant splitting them into different clusters.
Revised: This suggests that the latent representations of these digits are sufﬁciently far apart to warrant splitting them into different clusters.

Original: 3. F ASHION -MNIST
Revised: 3. F ASHION -MNIST

Original: Introduced in Xiao et al.
Revised: Introduced in Xiao et al.

Original: (2017), Fashion-MNIST is a convenient swap-in dataset for MNIST.
Revised: (2017), Fashion-MNIST is a convenient swap-in dataset for MNIST.

Original: Instead of digits the dataset consists of ten different types of clothes.
Revised: Instead of digits the dataset consists of ten different types of clothes.

Original: There are 60,000 train and 10,000 test examples just like in MNIST.
Revised: There are 60,000 train and 10,000 test examples just like in MNIST.

Original: Fashion-MNIST is generally considered more difﬁcult than MNIST, with classiﬁers scoring consistently lower on it.
Revised: Fashion-MNIST is generally considered more difﬁcult than MNIST, with classiﬁers scoring consistently lower on it.

Original: The model and analysis from the previous section carry over for fashion-MNIST with a few additional important points.
Revised: The model and analysis from the previous section carry over for fashion-MNIST with a few additional important points.

Original: First, the differences between NATAC-k and AE-k are less pronounced (see
Revised: First, the differences between NATAC-k and AE-k are less pronounced (see

Original: Table 1) in fashion-MNIST which indicates that the representations learned by NATAC in comparison to a simple autoencoder are not as important for k-means clustering.
Revised: Table 1) in fashion-MNIST which indicates that the representations learned by NATAC in comparison to a simple autoencoder are not as important for k-means clustering.

Original: Interestingly, our model still outperforms both NATAC-k and AE-k.
Revised: Interestingly, our model still outperforms both NATAC-k and AE-k, with one exception being when d = 12 .

Original: Qualitatively, Figure 2 shows that the model separates garments into slightly different categories than the labels in the dataset.
Revised: Qualitatively, Figure 2 shows that the model separates garments into slightly different categories than the labels in the dataset.

Original: For example, the most dense cluster seems to be a merging of both “pullovers” and “shirts”, suggesting that the model ﬁnds it difﬁcult to separate the two different garments.
Revised: For example, the most dense cluster seems to be a merging of both “pullovers” and “shirts”, suggesting that the model ﬁnds it difﬁcult to separate the two different garments.

Original: We ourselves ﬁnd it difﬁcult to discriminate between the two categories, as the lowresolution images do not easily show whether or not the garment has buttons.
Revised: We ourselves ﬁnd it difﬁcult to discriminate between the two categories, as the lowresolution images do not easily show whether or not the garment has buttons.

Original: Additionally, the “sandal” class has been split into two separate clusters: ﬂip-ﬂops and high-heeled shoes with straps.
Revised: Additionally, the “sandal” class has been split into two separate clusters: ﬂip-ﬂops and high-heeled shoes with straps.

Original: This indicates that our model has found an important distinction between these two type of shoes, that the original Fashion-MNIST labels ignore.
Revised: This indicates that our model has found an important distinction between these two type of shoes, that the original Fashion-MNIST labels ignore.

Original: Similarly to MNIST, our model also learns “dead[Table] clusters”, which the model does not decode into any discernible garment.
Revised: Similarly to MNIST, our model also learns “dead clusters”, which the model does not decode into any discernible garment.

Original: Further visualizations of these experiments can be found appendix section A. 
Revised: Further visualizations of these experiments can be found appendix section A. [Table]

Original: [Figure]
Revised: [Table]

Original:  C LUSTERING ON 20 N EWSGROUPS
Revised: 3. L ARGE D OCUMENT C LUSTERING ON 20 N EWSGROUPS

Original: Introduced in Joachims (1996) the 20 Newsgroups dataset is a collection of 18,846 documents pertaining to twenty different news categories.
Revised: Introduced in Joachims (1996) the 20 Newsgroups dataset is a collection of 18,846 documents pertaining to twenty different news categories.

Original: We use the commonly used 60:40 temporal train-test 
Revised: We use the commonly used 60:40 temporal train-test split.

Original: [Table] split. Interestingly, because of the temporal split in the data, the test set contains documents which differ considerably from the train set.
Revised: Interestingly, because of the temporal split in the data, the test set contains documents which differ considerably from the train set.

Original: We calculate NMI on the news categories in the dataset.
Revised: We calculate NMI on the news categories in the dataset.

Original: We use an auxiliary reconstruction loss and a two layer fully connected network for both the encoder and decoder, both with hidden layer sizes of 256 and ReLU nonlinearities.
Revised: We use an auxiliary reconstruction loss and a two layer fully connected network for both the encoder and decoder, both with hidden layer sizes of 256 and ReLU nonlinearities.

Original: We represent each article as an L2 normalized term-frequency-inverse-document-frequency (TF-IDF) vector of the 5000 most occurring words in the train set.
Revised: We represent each article as an L2 normalized term-frequency-inverse-document-frequency (TF-IDF) vector of the 5000 most occurring words in the train set.

Original: Two common baselines are used as points of comparison for our model. A TF-IDF k-means model and  a spherical k-means model.
Revised: Along with NATAC-k and AE-k comparisons, we also use a spherical k-means model.

Original: Spherical k-means is a commonly used technique of unsupervised document clustering, a good description of it can be found in Buchta et al.
Revised: Spherical k-means is a commonly used technique of unsupervised document clustering, a good description of it can be found in Buchta et al.

Original: Table 3 shows how the performance of the NATAC model varies with different dimensionalities of the latent space.
Revised: Table 3 shows how the performance of the each model varies with different dimensionalities of the latent space.

Original: The best NATAC models with a latent dimensionality of 3 to 7 centroids outperform a spherical k-means model with 1000 clusters, far more clusters than any of the NATAC models.
Revised: The best NATAC models with a latent dimensionality of 3 to 6 centroids outperform a spherical k-means model with 1000 clusters, far more clusters than any of the NATAC models.

Revised: Although we could not ﬁnd any neural clustering techniques which report performance on 20 Newsgroups, many non-neural methods report NMI on the whole dataset.

Revised: Table 4 shows the NATAC model performs competitively to these methods.

Revised: However, our method does converge on a higher number of clusters (other methods are trained with a pre-deﬁned number of 20 clusters).

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 3. C LUSTERING T WEETS
Revised: 3. C LUSTERING T WEETS

Original: To further explore the performance of our model on text data we build a dataset of 38,309 ASCIIonly tweets of English speaking users containing exactly one hashtag.
Revised: To further explore the performance of our model on text data we build a dataset of 38,309 ASCIIonly tweets of English speaking users containing exactly one hashtag.

Original: The dataset has 647 different hashtags, with each hashtag having at least ten tweets containing it.
Revised: The dataset has 647 different hashtags, with each hashtag having at least ten tweets containing it.

Original: We use 5000 of the tweets as the test set.
Revised: We use 5000 of the tweets as the test set.

Original: As a preprocessing step, URLs and hashtags are replaced with special characters.
Revised: As a preprocessing step, URLs and hashtags are replaced with special characters.

Original: We calculate NMI on the hashtag used in each tweet.
Revised: We calculate NMI on the hashtag used in each tweet.

Original: We train a character-based Sequence-to-Sequence autoencoder on the Twitter dataset.
Revised: We train a character-based Sequence-to-Sequence autoencoder on the Twitter dataset.

Original: Just as before we use an auxiliary reconstruction loss.
Revised: Just as before we use an auxiliary reconstruction loss.

Original: We set the encoder to be a bidirectional GRU with a hidden size of 64 followed by a fully connected layer which takes the GRU’s ﬁnal output and maps it to the latent space.
Revised: We set the encoder to be a bidirectional GRU with a hidden size of 64 followed by a fully connected layer which takes the GRU’s ﬁnal output and maps it to the latent space.

Original: The decoder uses a fully connected layer to map the latent vectors to a 128 dimensional vector which is then used as the initial hidden state for the decoder GRU.
Revised: The decoder uses a fully connected layer to map the latent vectors to a 128 dimensional vector which is then used as the initial hidden state for the decoder GRU.

Original: We also experiment with using a larger encoding for both the encoder and decoder.
Revised: We also experiment with using a larger encoding for both the encoder and decoder.

Original: The encoder produces an encoding with e dimensions which is then mean pooled to produce the latent representation with d dimensions.
Revised: The encoder produces an encoding with e dimensions which is then mean pooled to produce the latent representation with d dimensions.

Original: The decoder then takes the larger encoding as input.
Revised: The decoder then takes the larger encoding as input.

Original: Similarly to section 3.4, we use standard k-means and spherical k-means as baselines.
Revised: Similarly to section 3.4, we compare our approach to using spherical k-means along with the

Original: As withnewsgroups, the NATAC model  outperforms both baselines giving us an indication of its stability across different datasets.
Revised: NATAC-k and AE-k baselines. Both the NATAC and NATAC-k methods outperform the AE-k and spherical k-means methods, suggesting that the latent representations learned in the NATAC model produce more effective latent representations for clustering than a vanilla autoencoder, and that our methods also outperforms a strong text-clustering baseline when clustered with a similar number of centroids.

Original: Rather than varying the number of clusters for the baselines, we instead vary the vocabulary size for the term-frequency vectors.
Revised: Rather than varying the number of clusters spherical k-means models, we instead vary the vocabulary size for the term-frequency vectors.

Original: As the converged number of centroids for the NATAC models are very similar across different values of d (around 5000), we felt that varying the vocabulary size rather than the number of centroids would give a better range of comparable values.
Revised: As the converged number of centroids for the NATAC models are very similar across different values of d (around 5000), we believe that varying the vocabulary size rather than the number of centroids would give a better range of comparable values.

Original: As shown in table 4, spherical clustering does better on the Twitter dataset than standard k-means. The NATAC models with d = 8 , 10 outperform all of the  k-means baselines, while the lower dimensional NATAC models perform on par with  spherical k-means.
Revised: As shown in table 3.5, the NATAC/NATAC-k models with d = 8 , 10 outperform both the AE-k and spherical k-means baselines, while the lower dimensional NATAC models perform on par with [Figure] spherical k-means.

Original: We see a small increase in the test NMI when using a larger encoding for the decoder and mean pooling to the latent representation. 
Revised: We see a small increase in the test NMI when using a larger encoding for the decoder and mean pooling to the latent representation. 3. V ARIABILITY IN T RAINING

Original: [Table]
Revised: In order to empirically evaluate the robustness of the NATAC training method, we train 50 NATAC models with the same hyperparameters (but different random seeds) and measure the variation in NMI and the number of converged clusters. Figure 3 shows the variability of NMI and the converged number of clusters from training the best performing model on the 20 Newsgroups dataset. Although the converged number of clusters does vary in these models, the NMI varies with a small standard deviation of 0.007.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present a novel neural clustering method which does not depend on a predeﬁned number of clusters.
Revised: In this paper, we present a novel neural clustering method which does not depend on a predeﬁned number of clusters.

Original: Our empirical evaluation shows that our model works well across modalities.
Revised: Our empirical evaluation shows that our model works well across modalities.

Original: It improves state-of-the-art clustering performance on MNIST, without needing to receive the required number of clusters as input.
Revised: It improves state-of-the-art clustering performance on MNIST, without needing to receive the required number of clusters as input.

Original: Further, it outperforms powerful baselines on Fashion-MNIST and text datasets (20 Newsgroups and a Twitter hashtag dataset).
Revised: Further, it outperforms powerful baselines on Fashion-MNIST and text datasets (20 Newsgroups and a Twitter hashtag dataset).

Original: However, NATAC does require some hyperparameters to be tuned, namely the dimensionality of the latent space and the function for the loss coefﬁcient λ .
Revised: However, NATAC does require some hyperparameters to be tuned, namely the dimensionality of the latent space and the function for the loss coefﬁcient λ .

Original: Future work Several avenues of investigation could ﬂow from this work.
Revised: Future work Several avenues of investigation could ﬂow from this work.

Original: Firstly, the effectiveness of this method in a semi-supervised setting could be explored using a joint reconstruction and classiﬁcation auxiliary objective.
Revised: Firstly, the effectiveness of this method in a semi-supervised setting could be explored using a joint reconstruction and classiﬁcation auxiliary objective.

Original: Another interesting avenue to explore would be different agglomerative policies other than delete-and-copy.
Revised: Another interesting avenue to explore would be different agglomerative policies other than delete-and-copy.

Original: Different geometries of the latent space could also be considered other than a unit normalized hypersphere.
Revised: Different geometries of the latent space could also be considered other than a unit normalized hypersphere.

Original: To remove the need of setting hyperparameters by hand, work into automatically controlling the coefﬁcients (e.g. using proportional control) could be studied.
Revised: To remove the need of setting hyperparameters by hand, work into automatically controlling the coefﬁcients (e.g. using proportional control) could be studied.

Original: Finally, it would be interesting to see whether clustering jointly across different feature spaces would help with learning better representations.
Revised: Finally, it would be interesting to see whether clustering jointly across different feature spaces would help with learning better representations

Original: Appendix A V ISUALIZATIONS OF MNIST/F ASHION -MNIST L ATENT R EPRESENTATIONS
Revised: Appendix A V ISUALIZATIONS OF MNIST/F ASHION -MNIST L ATENT R EPRESENTATIONS

Original: We have published the visualizations of MNIST/Fashion-MNIST representations on the TensorFlow embedding projector.
Revised: We have published the visualizations of MNIST/Fashion-MNIST representations on the TensorFlow embedding projector.

Original: We recommend selecting the “Spherize data” option, as this gives a much clearer view of the embeddings.
Revised: We recommend selecting the “Spherize data” option, as this gives a much clearer view of the embeddings.

Original: The images used are taken from the test set of MNIST and FashionMNIST respectively.
Revised: The images used are taken from the test set of MNIST and FashionMNIST respectively.

Original: We include the NATAC models with the highest NMI from each dataset, along with the models with d = 3 , which require no dimensionality reduction to visualize.
Revised: We include the NATAC models with the highest NMI from each dataset, along with the models with d = 3 , which require no dimensionality reduction to visualize.

Original: • MNIST d = 3 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnistembeddings/digits3/conﬁg.json • MNIST d = 10 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnist embeddings/digits10/conﬁg.json • Fashion-MNIST d = 3 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnist embeddings/fashion3/conﬁg.json • Fashion-MNIST d = 9 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnist embeddings/fashion9/conﬁg.json B 
Revised: • MNIST d = 3 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnist embeddings/digits3/conﬁg.json • MNIST d = 10 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnist embeddings/digits10/conﬁg.json • Fashion-MNIST d = 3 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnistembeddings/fashion3/conﬁg.json • Fashion-MNIST d = 9 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnistembeddings/fashion9/conﬁg.json B E XAMPLES FROM THE F ASHION -MNIST DATASET .

Original: C A DDITIONAL D ISCUSSION ON T RAINING NATAC M ODELS C. G EOMETRY OF L ATENT S PACE
Revised: C A DDITIONAL D ISCUSSION ON T RAINING NATAC M ODELS C. G EOMETRY OF L ATENT S PACE

Original: We experimented with using polar coordinates early on in our experiments.
Revised: We experimented with using polar coordinates early on in our experiments.

Original: Rather than using euclidean coordinates as the latent representation, z is considered a list of angles θ 1 , θ 2 · · · θ n where θ 1 · · · θ n − 1 ∈ [0 , π ] and θ n ∈ [0 , 2 π ] .
Revised: Rather than using euclidean coordinates as the latent representation, z is considered a list of angles θ 1 , θ 2 · · · θ n where θ 1 · · · θ n − 1 ∈ [0 , π ] and θ n ∈ [0 , 2 π ] .

Original: However, we found that the models using polar geometry performed signiﬁcantly worse than those with euclidean geometry.
Revised: However, we found that the models using polar geometry performed signiﬁcantly worse than those with euclidean geometry.

Original: Additionally, we also experimented with not L2 normalizing the output of the encoder network.
Revised: Additionally, we also experimented with not L2 normalizing the output of the encoder network.

Original: We hypothesized that the model would learn a better representation of the latent space by also “learning” the geometry of the noise targets.
Revised: We hypothesized that the model would learn a better representation of the latent space by also “learning” the geometry of the noise targets.

Original: Unfortunately, the unnormalized representation caused the noise targets to quickly collapse to a single point.
Revised: Unfortunately, the unnormalized representation caused the noise targets to quickly collapse to a single point.

Original: D H YPERPARAMETERS F OR E XPERIMENTS
Revised: D H YPERPARAMETERS F OR E XPERIMENTS

Original: Although each different modality (monochrome images, bag-of-words, sequence of characters) uses a different set of hyperparameters, we follow a similar recipe for determining the values for each one:
Revised: Although each different modality (monochrome images, bag-of-words, sequence of characters) uses a different set of hyperparameters, we follow a similar recipe for determining the values for each one:

Original: • We use a large batch size of 100.
Revised: • We use a large batch size of 100.

Original: This is so each batch has a representative sample of the targets to reassign in each training step.
Revised: This is so each batch has a representative sample of the targets to reassign in each training step.

Original: • The warm-up period is calculated by observing when the auxiliary objective starts to converge.
Revised: • The warm-up period is calculated by observing when the auxiliary objective starts to converge.

Original: • The ﬁnal value for λ in training is set so that the NAT loss was approximately 1% of the total loss.
Revised: • The ﬁnal value for λ in training is set so that the NAT loss was approximately 1% of the total loss.

Original: • The initial value for λ is set as approximately 1% of the ﬁnal value of λ .
Revised: • The initial value for λ is set as approximately 1% of the ﬁnal value of λ .

Original: • The transition phase typically lasts 100 epochs of training.
Revised: • The transition phase typically lasts 100 epochs of training.

Original: • During the transition phase, the value of α is incrementally increased from 0 to 1.
Revised: • During the transition phase, the value of α is incrementally increased from 0 to 1.

Original: We now explicitly list the hyperparameters used for each experiment:
Revised: We now explicitly list the hyperparameters used for each experiment:

Original: D. MNIST AND F ASHION MNIST
Revised: D. MNIST AND F ASHION MNIST

Original: • A batch size of 100.
Revised: • A batch size of 100.

Original: • A warm-up period of 10 × d epochs, during which λ = 0 .
Revised: • A warm-up period of 10 × d epochs, during which λ = 0 .

Original: • A transition period lasts for 250 epochs, where λ is incrementally increased to 0 .
Revised: • A transition period lasts for 250 epochs, where λ is incrementally increased to 0 .

Original: 25 , and α is incremented from 0 to 1 .
Revised: 25 , and α is incremented from 0 to 1 .

Original: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 )
Revised: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 )

Original: Between each subsampling layer in the encoder, a single convolution layer is applied with a ﬁlter shape of 3 × 3 with border padding to keep the same shape before and after the convolution.
Revised: Between each subsampling layer in the encoder, a single convolution layer is applied with a ﬁlter shape of 3 × 3 with border padding to keep the same shape before and after the convolution.

Original: Similarly, in the decoder one transpose convolutional layer is appliedbetween each upsampling layer, 3 × 3 ﬁlter shapeand shape-preserving padding.
Revised: Similarly, in the decoder one transpose convolutional layer is applied each upsampling layer, 3 × 3 ﬁlter shape shape-preserving padding.

Original: [Figure]
Revised: D. 20 N EWSGROUPS

Original: • A batch size of 100.
Revised: • A batch size of 100.

Original: • A warm-up period of 100 epochs, during which  10 − 4 .
Revised: • A warm-up period of 1 , 000 epochs, during which λ = 10 − 4 .

Original: • A transition period lasts for 100 epochs, where λ is incrementally increased to 0 .
Revised: • A transition period lasts for 100 epochs, where λ is incrementally increased to 0 .

Original: 01 , and 
Revised: 01 , and α is incremented from 0 to 1 .

Original: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 ).
Revised: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 ).

Original: • Dropout with a keep-probability of 0.95 in the hidden layers of the encoder and decoder.
Revised: • Dropout with a keep-probability of 0.95 in the hidden layers of the encoder and decoder.

Original: D. T WITTER D ATASET
Revised: D. T WITTER D ATASET

Original: • A batch size of 100.
Revised: • A batch size of 100.

Original: • A warm-up period of 100 epochs, during which λ = 0 .
Revised: • A warm-up period of 100 epochs, during which λ = 0 .

Original: • A transition period lasts for 100 epochs, where λ is incrementally increased to 0 .
Revised: • A transition period lasts for 100 epochs, where λ is incrementally increased to 0 .

Original: 1 , and α is incremented from 0 to 1 .
Revised: 1 , and α is incremented from 0 to 1 .

Original: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 ).
Revised: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 ).