Original: N EURAL C LUSTERING B Y P REDICTING AND C OPYING N OISE
Revised: N EURAL C LUSTERING B Y P REDICTING AND C OPYING N OISE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a neural clustering approach, based on the recently proposed Noise As Targets (NAT) framework.
Revised: We propose a neural clustering model that jointly learns both latent features and how they cluster.

Original: The NAT framework allows uncovering important features of objects by ﬁxing target representations and constraining deep features to align with them.

Original: We extend this framework and show how it can be used to cluster objects. Our approach combines a NAT loss with an auxiliary loss (such as an autoencoder loss), and iteratively applies a policy for perturbing the set of ﬁxed representations by deleting and copying some of the elements. It does not requiredeciding on a speciﬁc desired number of clusters a piori.
Revised: Unlike similar methods our model does not require a predeﬁned number of clusters. Using a supervised approach, we agglomerate latent features towards randomly sampled targets within the same space whilst progressively removing the targets until we are left with only targets which represent cluster centroids.

Original: We empirically evaluate our framework by clustering both  image  and text datasets (including MNIST, Fashion-MNIST, the 20 newsgroups dataset and a Twitter hashtag dataset). Our method outperforms multiple baselines, including autoencoder based clustering and other state-of-the-art neural clustering methods.
Revised: To show the behavior of our model across different modalities we apply our model on both text and image data and achieve state-of-the-art results on MNIST. Finally, we also provide results against baseline models for fashion-MNIST, the 20 newsgroups dataset, and a Twitter dataset we ourselves create.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Clustering is one of the fundamental problems of unsupervised learning.
Revised: Clustering is one of the fundamental problems of unsupervised learning.

Original: It involves the grouping of items into clusters such that items within the same cluster are more similar than items in different clusters.
Revised: It involves the grouping of items into clusters such that items within the same cluster are more similar than items in different clusters.

Original: Crucially, the ability to do this often hinges upon learning latent features in the input data which can be used to differentiate items from each other in some feature space.
Revised: Crucially, the ability to do this often hinges upon learning latent features in the input data which can be used to differentiate items from each other in some feature space.

Original: Two key questions thus arise: How do we decide upon cluster membership?
Revised: Two key questions thus arise: How do we decide upon cluster membership?

Original: and How do we represent data in feature space?
Revised: and How do we learn good representations of data in feature space?

Original: Spurred initially by studies into the division of animals into taxa Sokol & Sneath (1963), cluster analysis matured as aseparate ﬁeld in the following decades with the advent  various models.
Revised: Spurred initially by studies into the division of animals into taxa (Sokol & Sneath, 1963), cluster analysis matured as a ﬁeld in the subsequent decades with the advent of various models.

Original: These included distribution-based models, such as Gaussian mixture models Duda et al. (1973); densitybased models, such as DBSCAN Ester et al.
Revised: These included distribution-based models, such as Gaussian mixture models (Duda et al., 1973); densitybased models, such as DBSCAN (Ester et al., 1996); centroid-based models, such as k-means.

Original: ; centroid-based models, such as k-means 1 and hierarchical models, including agglomerative Orloci (1967) and divisive models Gower (1967).
Revised: 2 and hierarchical models, including agglomerative (Orloci, 1967) and divisive models (Gower, 1967).

Original: Whereas the cluster analysis community mostly concerned itself with the unsupervised learning of cluster membership , the deep learning community has a long history of unsupervised representation learning. Deep neural networks of various designs have recently proven to be an extremely powerful tool for a very wide range of tasks LeCun et al.
Revised: While the cluster analysis community has focused on the unsupervised learning of cluster membership , the deep learning community has a long history of unsupervised representation learning , yielding models such as variational autoencoders (Kingma & Welling, 2013), generative adversarial networks (Goodfellow et al., 2014), and vector space word models (Mikolov et al., 2013).

Original: One key driver of the success of neural networks across many settings is their capability of producing a strong representation, relying on complex features which are automatically identiﬁed when the network is trained to predict labels given the training data. Unfortunately, obtaining labels requires signiﬁcant costly effort. Further, when the network is trained to identify a certain label, it may ignore variability between training instances that does not lead to having a different label. The success of deep neural networks has lead to an explosion of research on applying them to many supervised learning tasks.
Revised: In this paper, we propose using noise as targets for agglomerative clustering (or NATAC).

Original: Neural networks clearly produce very strong object representations insupervised tasks, but less research has gone into examining how they can be used for unsupervised learning. While some work has been done on neural generative models such as GANs Radford et al. ; Chen et al. (2016), applications of neural networks to clustering have received relatively little attention(Yang et al., 2016;
Revised: As in

Original: Dilokthanakul et al., 2016; Xie et al., 2016). Researchers working on applying neural networks to unsupervised learning domains have recently proposed a method called Noise as Targets (NAT) Bojanowski & Joulin (2017), which allows constructing informative features in unsupervised settings.
Revised: Bojanowski & Joulin (2017) we begin by sampling points in features space called noise targets which we match with latent features. During training we progressively remove targets and thus agglomerate latent features around fewer and fewer target centroids.

Original: To evaluate the ability of  the NAT framework to identify key features, the authors have relied on a transfer learning approach; they used a dataset of images, which did include a supervision signal in the form of image labels. The NAT framework was then given access only to the images themselves (with no access to the labels), and applied to extract features and build a compound representation of each object.
Revised: To tackle the instability of such training we augment our objective with an auxiliary loss which prevents the model from collapsing and helps it learn better representations. We explore the performance of our model across different modalities in Section 3. Recently, there have been several attempts at jointly learning both cluster membership and good representations using end-to-end differentiable methods.

Original: The authors then trained another neural network to map these latent representations of images to their labels. This analysis indicates that the NAT framework can detect important features of objects without any labels, as the extracted features can then be used to predict object labels.
Revised: Similarly to us, Yang et al.

Original: Our work more directly examines the question of whether neural networks can be used for object clustering. Earlier work already uses various forms of autoencoders for clustering purposes Song et al.
Revised: (2016) use a policy to move points around in feature space but they use a different agglomerative strategy.

Original: ; Huang et al.
Revised: Liao et al.

Original: ; Makhzani et al. (2015). In contrast, we build on the NAT framework, and propose  a new neural clustering method.
Revised: (2016) propose jointly learning representations and clusters by using a k-means style objective.

Original: The heart of our approach lies in combining a NAT loss with an autoencoder loss (and potentially a discriminator loss), and iteratively applying a policy for perturbing the set of ﬁxed representations by deleting and copying some of the elements.
Revised: Xie et al.

Original: We directly evaluate our approach as a clustering method, comparing its performance with existing clustering baselines on multiple datasets, including both image and text data, such as the MNIST,
Revised: (2016) introduce deep embedding clustering (DEC) which learns a mapping from data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective

Original: FashionMNIST, the 20 newsgroups datasets, and a Twitter hashtag dataset we have sourced 2 .
Revised: [Figure]

Original: We show that our clustering approach outperforms various strong baselines, such as various forms ofautoencoders or various clustering algorithms applied directly on the raw data.
Revised: , as opposed optimize Unlike us, all of the above papers require a predeﬁned number of clusters.

Original:  M ODEL
Revised:  M ODEL

Original: We begin by discussing the use of noise as targets (NAT) introduced in Bojanowski & Joulin (2017) which is crucial to the understanding of our method.
Revised: We begin by discussing the use of noise as targets (NAT) introduced in Bojanowski & Joulin (2017) which is crucial to the understanding of our model.

Original: We then describe the intuition behind our approach, then proceed to describe the mechanism itself
Revised: We then describe the intuition behind our approach, then proceed to describe the mechanism itself.

Original: . 2. N OISE AS T ARGETS (NAT)
Revised: 2. N OISE AS T ARGETS (NAT)

Original: Bojanowski & Joulin (2017)recently proposed  new form of unsupervised learning re-posited as a supervised problem where the task is to learn an encoder function f θ : X → Z from inputs to a latent representation space so as to minimize the L 2 loss between representations z i ∈ Z , where points in Z are unit normalized, and corresponding targets y k ∈ Y where the targets are uniformly sampled from the L 2 unit sphere (thus inhabiting the same space as the z i s).
Revised: Bojanowski & Joulin (2017) proposed a new form of unsupervised learning re-posited as a supervised problem where the task is to learn an encoder function f θ : X → Z from input space to a latent representation space. The objective is to minimize the L 2 loss between representations z i ∈ Z , where points in Z are unit normalized, and corresponding targets y k ∈ Y where the targets are uniformly sampled from the L 2 unit sphere (thus inhabiting the same space as the z i ).

Original: Instead of being tied to corresponding representations as in classic regression,the targets are one-to-one re-assigned to different representations after every training step so as to minimize the total sum of distances between matched pairs (where this optimal assignment is computed usinga fast algorithm known as the “Hungarian method” Kuhn (1955)).
Revised: Instead of being tied to corresponding representations as in classic regression, targets are one-to-one re-assigned to different representations after every training step so as to minimize the total sum of distances between matched pairs (this is done using the Hungarian method (Kuhn, 1955)).

Original: This pushes the representations
Revised: This pushes the representations

Original: z i of similar inputs x i ∈
Revised: z i of similar inputs x i ∈

Original: X to neighborhoods of similar targets y k .
Revised: X to neighborhoods of similar targets y k .

Original: The motivation behind using NAT was to learn unsupervised features from input data.
Revised: The motivation behind using NAT was to learn unsupervised features from input data.

Original: They show their method performs on par with state-of-the-art unsupervised representation learning methods.
Revised: They show their method performs on par with state-of-the-art unsupervised representation learning methods.

Original: 2. N OISE AS T ARGETS FOR A GGLOMERATIVE C LUSTERING (NATAC)
Revised: 2. N OISE AS T ARGETS FOR A GGLOMERATIVE C LUSTERING (NATAC)

Original: Viewed from a clustering perspective we can think of the targets
Revised: Viewed from a clustering perspective we can think of the targets

Original: y i as cluster centroids to which the latent representations
Revised: y i as cluster centroids to which the latent representations

Original: z i are (one-to-one) assigned.
Revised: z i are (one-to-one) assigned.

Original: Note that although the method introduced in
Revised: Note that although the method introduced in

Original: Bojanowski & Joulin (2017) brings the representations of similar x i closer together (by matching and moving them closer to neighborhoods of similar targets) it cannot produce many-to-one matchings or match multiple similar z i with a single centroid thus forming a cluster.
Revised: Bojanowski & Joulin (2017) brings the representations of similar x i closer together (by matching and moving them closer to neighborhoods of similar targets) it cannot produce many-to-one matchings or match multiple similar z i with a single centroid thus forming a cluster.

Original: Simply changing the re-assignment policy to allow for many-to-one matchings is difﬁcult because it causes the model to collapse in on a single target.
Revised: Simply changing the re-assignment policy to allow for many-to-one matchings is difﬁcult because it causes the model to collapse in on a single target.

Original: In this paper we use the above to propose a new form of neural clus-[Figure] tering where we progressively delete targets over time and re-assign representations to other nearby targets, all while keeping the model stable using an auxiliary objective.
Revised: In this paper we use the above to propose a new form of neural clus- tering where we progressively delete targets over time and re-assign representations to other nearby targets, all while keeping the model stable using an auxiliary objective.

Original: Delete-and-copy To be able to cluster latent representations we need a way of assigning them to cluster centroids.
Revised: Delete-and-copy To be able to cluster latent representations we need a way of assigning them to cluster centroids.

Original: We propose doing this via an additional delete-and-copy step which allows for many-to-one matchings.
Revised: We propose doing this via an additional delete-and-copy step which allows for many-to-one matchings.

Original: Similarly to the NAT method, we ﬁrst assign representations z to targets y using the Hungarian method.
Revised: Similarly to the NAT method, we ﬁrst assign representations z to targets y using the Hungarian method.

Original: We then remove the assigned target and reassign a copy of the next nearest target for each representations with some probability α as in Algorithm 1 (see also Figure 1).
Revised: We then remove the assigned target and reassign a copy of the next nearest target for each representations with some probability α as in Algorithm 1 (see also Figure 1).

Original: This has the effect of not only reassigning targets so as to minimize the distance between matched pairs, but also to encourage the model to allow similar examples to be assigned to the same target.
Revised: This has the effect of not only reassigning targets so as to minimize the distance between matched pairs, but also to encourage the model to allow similar examples to be assigned to the same target.

Original: The new assignments are denoted as y new i .
Revised: The new assignments are denoted as y new i .

Revised: The loss is then deﬁned as:

Original:  Auxiliary objective To prevent the model from collapsing to a single point, we introduce an auxiliary objective in addition to the loss between representations and targets.
Revised: [Equation] Auxiliary objective To prevent the model from collapsing to a single point, we introduce an auxiliary objective in addition to the loss between representations and targets.

Original: In our case we set the auxiliary objective L aux to be the reconstruction loss (cid:80) i (cid:107) x
Revised: In our case we set the auxiliary objective L aux to be the reconstruction loss (cid:80) i (cid:107) x

Original: i − f dec ( z i ) (cid:107) 2 where f dec is some decoder network.
Revised: i − f dec ( z i ) (cid:107) 2 where f dec is some decoder network.

Original: The ﬁnal objective L is then a weighted sum of the NAT loss L NAT (which in our case is the L 2 loss) and the auxiliary objective L aux :[Equation] where:
Revised: The ﬁnal objective L is then a weighted sum of the NAT loss L NAT (which in our case is the L 2 loss) and the auxiliary objective L aux :

Original: [Equation]
Revised: [Equation]

Original: Importantly, the auxiliary objective not only prevents the model from collapsing, it also informs how the model clusters.
Revised: Importantly, the auxiliary objective not only prevents the model from collapsing, it also informs how the model clusters.

Original: For example, when clustering images the reconstruction loss would encourage the model to cluster on similar pixel values.
Revised: For example, when clustering images the reconstruction loss encourages the model to cluster on similar pixel values.

Revised: Alternative forms of auxiliary objectives could allow for a discriminator loss or a classiﬁcation loss for tackling semi-supervised settings.

Revised: As our goal is unsupervised clustering, we only consider the reconstruction loss.

Original: Model deﬁnition During initialization each example x i in the dataset is paired with a random target y i , uniformly sampled from a( d − 1) -sphere.
Revised: Model deﬁnition During initialization each example x i in the dataset is paired with a random target y i , uniformly sampled from a d -dimensional sphere.

Original: The NATAC model is then trained using mini-batches from the dataset.
Revised: The NATAC model is then trained using minibatches from the dataset.

Original: Each training step can be broken down as follows:
Revised: Each training step can be broken down as follows:

Original: Forward step: The examples x from a random batch of example-target pairs (cid:0) ( x 1 , y 1 ) . . .
Revised: Forward step: The examples x from a random batch of example-target pairs (cid:0) ( x 1 , y 1 ) . . .

Original: ( x i , y i ) . . .
Revised: ( x i , y i ) . . .

Original: ( x N , y N ) (cid:1) are embedded using an encoder function f θ : R n → R d into corresponding latent representations z i ∈ R d where (cid:107)
Revised: ( x N , y N ) (cid:1) are embedded using an encoder function f θ : R n → R d into corresponding latent representations z i ∈ R d where (cid:107)

Original: z i (cid:107) =
Revised: z i (cid:107) =

Original: 1 for all i .
Revised: 1 for all i .

Original: if y opt i (cid:54) =

Original: y nearest and p <

Original: α then ( y opt i is then deleted, and y nearest is copied and assigned to z i )

Original: y new i := y nearest i else y new end end return y Re-assignment step: Using the Hungarian method from Kuhn (1955), the representations z are optimally one-to-one matched with targets y so as to minimize the total sum of distances between matched pairs in the batch.
Revised: Re-assignment step: Using the Hungarian method from Kuhn (1955), the representations z are optimally one-to-one matched with targets y so as to minimize the total sum of distances between matched pairs in the batch.

Original: The newly assigned example-target pairing (cid:0) ( x 1 , y opt 1 ) . . .
Revised: The newly assigned example-target pairing (cid:0) ( x 1 , y opt 1 ) . . .

Original: ( x i , y opt i ) . . .
Revised: ( x i , y opt i ) . . .

Original: ( x N , y opt N ) (cid:1) has the permutation of labels within the batch to minimize the batch-wise loss.
Revised: ( x N , y opt N ) (cid:1) has the permutation of labels within the batch to minimize the batch-wise loss.

Original: Delete-and-copy: With a probability α , delete the optimal target y opt i and instead assign the nearest target to z i (which may be the same target) for each example-target pair in the batch producing (cid:0) ( x 1 , y new 1 ) . . .
Revised: Delete-and-copy (DnC): With a probability α , delete the optimal target y opt i and instead assign the nearest target to z i (which may be the same target) for each example-target pair in the batch producing (cid:0) ( x 1 , y new 1 ) . . .

Original: ( x i , y new i ) . . .
Revised: ( x i , y new i ) . . .

Original: ( x N , y new N ) (cid:1) as described in Algorithm 1 (see also Figure 1).
Revised: ( x N , y new N ) (cid:1) as described in Algorithm 1 (see also Figure 1).

Original: Train and update step: The L 2 loss between targets and latent representations is taken and combined with the auxiliary loss.
Revised: Train and update step: The L 2 loss between targets and latent representations is taken and combined with the auxiliary loss.

Original: Gradients w.r.t.
Revised: Gradients w.r.t.

Original: θ are then taken and back-propagated along.
Revised: θ are then taken and back-propagated along.

Original: Notice that although the (re)-assignment step follows a non-differentiable policy, the model is still end-to-end differentiable.
Revised: Notice that although the (re)-assignment step follows a non-differentiable policy, the model is still end-to-end differentiable.

Original: Finally, the new example-target assignments are kept after the training step, and persist into the next training step where they are again reassigned.
Revised: Finally, the new example-target assignments are kept after the training step, and persist into the next training step where they are reassigned again.

Original: 2. I MPLEMENTATION D ETAILS
Revised: I MPLEMENTATION D ETAILS

Original: Stopping Criterion During training the number of unique targets is tracked.
Revised: Stopping criterion During training the number of unique targets is tracked.

Original: We stop training when the number of unique targets stops decreasing after an epoch of training.
Revised: We stop training when the number of unique targets stops decreasing after an epoch of training.

Original: Multi-Stage Training We found that an initial period of training where the auxiliary objective is prioritized (i.e. the NAT loss is multiplied by a very small coefﬁcient) and the DnC policy is not used, improved overall performance.
Revised: Multi-stage training We found that an initial period of training where the auxiliary objective is prioritized (i.e. the NAT loss is multiplied by a very small coefﬁcient) and the DnC policy is not used, improved overall performance.

Original: Additionally, using a gradual transition to a higher NAT loss and a more utilized DnC policy  increased the stability of training.
Revised: Transitioning to a higher NAT loss and turning on the deleteand-copy policy later on in training increased the stability of our model.

Original: Therefore, traininga NATAC model has three steps:
Revised: We therefore propose training NATAC models as follows:

Original: Warm-Up Stage : α = 0 and λ are very small.
Revised: Warm-Up stage : α = 0 and λ are very small.

Original: Transition Stage : α increases gradually from 0 to 1 , λ also increases gradually to a larger value (approximately 100 × larger than its initial value).
Revised: Transition stage : α increases gradually from 0 to 1 , λ also increases gradually to a larger value (approximately 100 × larger than its initial value).

Original: Clustering Stage : α = 1 , λ is large.
Revised: Clustering stage : α = 1 , λ is large.

Original: Dimensionality Of The Latent Space
Revised: Dimensionality of the latent space

Original: In all of our experiments, we found that the best performing models tend to have an latent space dimensionality betweend = 6 andd = 12 .
Revised: In all of our experiments, we found that the best performing models tend to have a latent space dimensionality between 6 and 12 .

Original: At dimensionalities much larger than this, the model collapses to very few points during the transition stage of training, possibly due to the high expressiveness of the latent space.
Revised: At dimensionalities much larger than this, the model collapses to very few points during the transition stage of training, possibly due to the high expressiveness of the latent space.

Original: On the contrary, using avery low dimensional representation results in an information bottleneck too small to sufﬁciently learn the auxiliary objective.
Revised: On the other hand, using a low dimensional representation results in an information bottleneck too small to sufﬁciently learn the auxiliary objective.

Original: For example, when trying to cluster a collection of tweets from the twitter dataset, a latent space of two dimensions was too small for a decoder to reliably reconstruct a tweet fromthe latent vector.
Revised: For example, when clustering tweets from our Twitter dataset, a latent space of two dimensions was too small for the decoder to reliably reconstruct tweets from latent vectors.

Original: In the absence of an auxiliary objective that can be effectively learned,the centroids collapse to a single point.
Revised: With an auxiliary objective that cannot be effectively learned, centroids collapse to a single point.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we describe the datasets and evaluation metrics used in our experiments followed by the presentation and analysis ofthe results of our model in comparison to others.
Revised: We now describe the datasets and evaluation metrics used in our experiments followed by the presentation and analysis of our results in comparison to others.

Original: The full details regarding the hyperparameters used in our experiments can be found in appendix D.3. D ATASETS
Revised: The full details regarding the hyperparameters used in our experiments can be found in appendix D.

Original:  We evaluatethe performance of our models on four different datasets - two image and two text datasets.
Revised: Datasets We evaluate our models on four different datasets - two image and two text datasets.

Original: Forexperiments on images we use MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017).
Revised: For images we use MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017).

Original: For text we use 20 Newsgroups (Joachims, 1996) and a Twitter dataset which we built ourselves.
Revised: For text we use 20 Newsgroups (Joachims, 1996) and a Twitter dataset which we gather ourselves.

Original: 3. E VALUATION
Revised: 3. E VALUATION

Original: Our key evaluation metric is the normalized mutual information (NMI) score (Strehl & Ghosh, 2002) which measures the information gained from knowing cluster membership whilst taking the number of clusters into account.
Revised: Our key evaluation metric is the normalized mutual information (NMI) score (Strehl & Ghosh, 2002) which measures the information gained from knowing cluster membership whilst taking the number of clusters into account.

Revised: Values of NMI range from 0 , where clustering gives no information, to 1 , where clustering gives perfect information i.e. cluster membership is identical to class membership.

Original: In our experiments, we train models on the concatenation of the train and validation sets and evaluate them on the test set.
Revised: In our experiments, we train models on the concatenation of the train and validation sets and evaluate them on the test set.

Original: This is done by computing the latent representations of test examples and then assigning them to their nearest respective centroids, then computing the NMI score.
Revised: This is done by computing the latent representations of test examples and then assigning them to their nearest respective centroids, then computing the NMI score.

Original: Additionally, we provide classiﬁcation error scores on the MNIST dataset to compare ourselves to other related methods.
Revised: Additionally, we provide classiﬁcation error scores on the MNIST dataset to compare ourselves to other related methods.

Original: The  motivation behind our experiments is to see how well our model decides upon cluster membership andto see whether it learns useful latent representations.
Revised: The guiding motivation behind our experiments is to analyze how well our models learns cluster membership and latent representations.

Original: 3. MNIST
Revised: 3. MNIST

Original: Introduced by LeCun et al.
Revised: Introduced by LeCun et al.

Original: (1998), MNIST is canonical dataset for numerous machine learning tasks including clustering.
Revised: (1998), MNIST is canonical dataset for numerous machine learning tasks including clustering.

Original: MNIST has ten classes, corresponding to the ten digits 0-9, and contains 60,000 train and 10,000 test examples.
Revised: MNIST has ten classes, corresponding to the ten digits 0-9, and contains 60,000 train and 10,000 test examples.

Original: We train our model with an auxiliary reconstruction loss and use small convolutional architectures (see Figure 6) for our encoder and decoder networks.
Revised: We train our model with an auxiliary reconstruction loss and use small convolutional architectures (see Figure 6) for our encoder and decoder networks.

Original: As points of comparison we also provide results of using k-means on the latent representations learned by our model (NATAC-K in Table 1) and k-means on representations learned by a simple autoencoder with the same encoder and decoder architecture as our model (AE-K in Table 1).
Revised: As points of comparison we also provide results of using k-means on the latent representations learned by our model (NATAC-k in Table 1) and k-means on representations learned by a simple autoencoder with the same encoder and decoder architecture (AE-k in Table 1).

Original: Table 1 showsthat our model’s performance is best when d = 10 and worse for much lower or higher values of d .
Revised: Table 1 shows our model’s performance is best when d = 10 and worse for much lower or higher values of d .

Original: This indicates that the dimensionality of the latent space impacts our model’s performance (see Section 2.3 for further discussion).
Revised: This indicates that the dimensionality of the latent space impacts our model’s performance (see Section 2.3 for further discussion).

Original: The ability of our model to cluster MNIST examples well is shown by two keys results.
Revised: The ability of our model to cluster MNIST examples well is shown by two keys results.

Original: First, it beats both NATAC-K and AE-K (Table 1).
Revised: First, it beats both NATAC-k and AE-k (Table 1).

Original: Second, it achieves state-of-the-art results when compared to other methods (see NMI column in Table 2).
Revised: Second, it achieves state-of-the-art results when compared to other methods (see NMI column in Table 2).

Original: Note that the second result reported in (Yang et al., 2016) is achieved using a separate re-clustering step after training.
Revised: Note that the second result reported in Yang et al. (2016) is achieved using a separate re-clustering step after training.

Revised: To re-iterate, other clustering techniques cited in Table 2 required a predeﬁned number of clusters (in the case of MNIST k=10).

Original: We note that NATAC-K beats AE-K, indicating that our model learns better representation as compared to a simple autoencoder.
Revised: We note that NATAC-k beats AE-k, indicating that our model learns better representation as compared to a simple autoencoder.

Original: Consequently, NATAC  achieves a state-of-the-art clustering quality on MNIST, as measured by the classiﬁcation error result (see Table 2).
Revised: Consequently, NATAC also achieves a state-of-the-art NMI on MNIST, as measured by the classiﬁcation error result (see Table 2).

Original: Finally, we discuss the number of centroids our model converges on (see Table 1).
Revised: Finally, we discuss the number of centroids our model converges on (see Table 1).

Original: We show that our model is successfully capable of ﬁnding centroids that represent different digits, as shown in the top row of Figure 2.
Revised: We show that our model is successfully capable of ﬁnding centroids that represent different digits, as shown in the top row of Figure 2.

Original: However, the model also learns centroids that contain very few examples for which the decoded images do not represent any handwritten digits, as shown in the second row of Figure 2.
Revised: However, the model also learns centroids that contain very few examples for which the decoded images do not represent any handwritten digits, as shown in the second row of Figure 2.

Original: Even with these “dead centroids”, the model still performs well.
Revised: Even with these “dead centroids”, the model still performs well.

Original: Interestingly, the model also differentiates between ones with different slopes, suggesting thatit thinks the latent representations of these digits are sufﬁciently far apart to warrant splitting them into different clusters.[Table] [Table]
Revised: Interestingly, the model also differentiates between ones with different slopes. This suggests that the latent representations of these digits are sufﬁciently far apart to warrant splitting them into different clusters.

Original:  F ASHION -MNIST
Revised: 3. F ASHION -MNIST

Original: Introduced in (Xiao et al., 2017)
Revised: Introduced in Xiao et al.

Original: [Table] Fashion-MNIST is a convenient swap-in dataset for MNIST.
Revised: (2017), Fashion-MNIST is a convenient swap-in dataset for MNIST.

Original: Instead of digits the dataset consists of ten different types of clothes.
Revised: Instead of digits the dataset consists of ten different types of clothes.

Original: There are 60,000 train and 10,000 test examples just like in MNIST.
Revised: There are 60,000 train and 10,000 test examples just like in MNIST.

Original: Fashion-MNIST is generally considered more difﬁcult than MNIST, with classiﬁers scoring consistently lower on it than MNIST.
Revised: Fashion-MNIST is generally considered more difﬁcult than MNIST, with classiﬁers scoring consistently lower on it.

Original: The model and analysis from the previous section carries over for fashion-MNIST with a few additional important points.
Revised: The model and analysis from the previous section carry over for fashion-MNIST with a few additional important points.

Original: First, the differences between NATAC-K and AE-K are less pronounced (see Table 1) in fashion-MNIST which indicates that the representations learned by NATAC in comparison to a simple autoencoder are not as important for k-means clustering.
Revised: First, the differences between NATAC-k and AE-k are less pronounced (see Table 1) in fashion-MNIST which indicates that the representations learned by NATAC in comparison to a simple autoencoder are not as important for k-means clustering.

Original: Interestingly, our model still outperforms both NATAC-K and AE-K.
Revised: Interestingly, our model still outperforms both NATAC-k and AE-k.

Original: Qualitatively, Figure 2 shows that the model has separated garments into slightly different categories than the labels in the dataset.
Revised: Qualitatively, Figure 2 shows that the model separates garments into slightly different categories than the labels in the dataset.

Original: For example, the most dense cluster seems to be a merging of both “pullovers” and “shirts”, suggesting that the model ﬁnds it difﬁcult to separate the two different garments 4 .
Revised: For example, the most dense cluster seems to be a merging of both “pullovers” and “shirts”, suggesting that the model ﬁnds it difﬁcult to separate the two different garments.

Revised: We ourselves ﬁnd it difﬁcult to discriminate between the two categories, as the lowresolution images do not easily show whether or not the garment has buttons.

Original: Additionally, the “sandal” class has been split into two separate clusters: ﬂip-ﬂops and high-heel shoes with straps.
Revised: Additionally, the “sandal” class has been split into two separate clusters: ﬂip-ﬂops and high-heeled shoes with straps.

Original: This indicates that our model has found an important distinction between these two type of shoes, that the original Fashion-MNIST labels ignore.
Revised: This indicates that our model has found an important distinction between these two type of shoes, that the original Fashion-MNIST labels ignore.

Original: Similarly to MNIST, our model also learns “dead  clusters”, which the model does not decode into any discernible garment.
Revised: Similarly to MNIST, our model also learns “dead [Table] clusters”, which the model does not decode into any discernible garment.

Original: Further visualizations of these experiments can be found appendix section A.
Revised: Further visualizations of these experiments can be found appendix section A.

Original: 3. L ARGE D OCUMENT C LUSTERING ON 20 N EWSGROUPS
Revised: [Figure] C LUSTERING ON 20 N EWSGROUPS

Original: Introduced in (Joachims, 1996) the 20 Newsgroups dataset is a collection of 18,846 documents pertaining to twenty different news categories.
Revised: Introduced in Joachims (1996) the 20 Newsgroups dataset is a collection of 18,846 documents pertaining to twenty different news categories.

Original: We use the commonly used 60:40 temporal train-test  split.
Revised: We use the commonly used 60:40 temporal train-test [Table] split.

Original: Interestingly, because of the temporal split in the data the test set contains document which differ considerably from the train set.
Revised: Interestingly, because of the temporal split in the data, the test set contains documents which differ considerably from the train set.

Revised: We calculate NMI on the news categories in the dataset.

Original: We use an auxiliary reconstruction loss and a two layer fully connected network for both the encoder and decoder, both with hidden layer sizes of 256 and ReLU nonlinearities.
Revised: We use an auxiliary reconstruction loss and a two layer fully connected network for both the encoder and decoder, both with hidden layer sizes of 256 and ReLU nonlinearities.

Original: We represent each article as a binary TF vector of the 5000 most occurring words in the train set.[Table]
Revised: We represent each article as an L2 normalized term-frequency-inverse-document-frequency (TF-IDF) vector of the 5000 most occurring words in the train set.

Original: Two  baselines are used as points of comparison for our model.
Revised: Two common baselines are used as points of comparison for our model.

Original: A K-means model and a spherical clustering model, a good description of which can be found in Buchta et al.
Revised: A TF-IDF k-means model and a spherical k-means model. Spherical k-means is a commonly used technique of unsupervised document clustering, a good description of it can be found in Buchta et al.

Original: Table 3 shows how the performance of the NATAC model varies with different dimensionalities of the latent space.
Revised: Table 3 shows how the performance of the NATAC model varies with different dimensionalities of the latent space.

Original: The bestperforming NATAC model with 819 centroids performs marginally better than K-Means clustering with 1000 centroids on TF-IDF vectors of thenews articles. Additionally, NATAC outperforms spherical K-Means by a substantial margin.
Revised: The best NATAC models with a latent dimensionality of 3 to 7 centroids outperform a spherical k-means model with 1000 clusters, far more clusters than any of the NATAC models.

Original: [Table]
Revised: [Table]

Original: 3. C LUSTERING T WEETS
Revised: 3. C LUSTERING T WEETS

Original: To further explore the performance of our model on text data  build a dataset of 38,309 ASCII-only tweets of English speaking users containing exactly one hashtag.
Revised: To further explore the performance of our model on text data we build a dataset of 38,309 ASCIIonly tweets of English speaking users containing exactly one hashtag.

Original: The dataset has 647 different hashtags, with each hashtag having at least ten tweets containing the hashtag.
Revised: The dataset has 647 different hashtags, with each hashtag having at least ten tweets containing it.

Original: We use 5000 of these tweets as the test set.
Revised: We use 5000 of the tweets as the test set.

Original: As a preprocessing step, URLs and hashtags are replaced with special characters.
Revised: As a preprocessing step, URLs and hashtags are replaced with special characters.

Revised: We calculate NMI on the hashtag used in each tweet.

Original: We train a character-based seq2seq autoencoder on the Twitter dataset.
Revised: We train a character-based Sequence-to-Sequence autoencoder on the Twitter dataset.

Original: Just as before we use an auxiliary reconstruction loss.
Revised: Just as before we use an auxiliary reconstruction loss.

Original: We set the encoder to be a bidirectional GRU with a hidden size offollowed by a fully connected layer which takes the GRU’s ﬁnal output(we also experiment with a mean pool across outputs instead of the ﬁnal output) and maps it to the latent space.
Revised: We set the encoder to be a bidirectional GRU with a hidden size of 64 followed by a fully connected layer which takes the GRU’s ﬁnal output and maps it to the latent space.

Original: The decoder uses a fully connected layer to map the latent vectors to a 128 dimensional vector which is then used as the initial hidden state for the decoder GRU.
Revised: The decoder uses a fully connected layer to map the latent vectors to a 128 dimensional vector which is then used as the initial hidden state for the decoder GRU.

Revised: We also experiment with using a larger encoding for both the encoder and decoder.

Revised: The encoder produces an encoding with e dimensions which is then mean pooled to produce the latent representation with d dimensions.

Revised: The decoder then takes the larger encoding as input.

Original: Similarly to section 3.5, we used spherical K-means and TF-IDF + K-means for comparable baselines.
Revised: Similarly to section 3.4, we use standard k-means and spherical k-means as baselines.

Revised: As withnewsgroups, the NATAC model outperforms both baselines giving us an indication of its stability across different datasets.

Original: Rather than varying the number of clusters, we instead vary the vocabulary size for the termfrequency vectors.
Revised: Rather than varying the number of clusters for the baselines, we instead vary the vocabulary size for the term-frequency vectors.

Original: As the converged number of centroids for the NATAC models are very similar across different values of d (around 5000), we felt that varying the vocabulary size rather than the number of centroids would give a better range of comparable values.
Revised: As the converged number of centroids for the NATAC models are very similar across different values of d (around 5000), we felt that varying the vocabulary size rather than the number of centroids would give a better range of comparable values.

Original: As shown in table 4, spherical clustering does better on the Twitter dataset than raw TF/IDF clustering (as opposed to then on the 20 newsgroup dataset).
Revised: As shown in table 4, spherical clustering does better on the Twitter dataset than standard k-means.

Original: The NATAC models with d = 8 ,outperform all of the K-means baselines, while the lower dimensional NATAC models perform on par with spherical K-means.
Revised: The NATAC models with d = 8 , 10 outperform all of the k-means baselines, while the lower dimensional NATAC models perform on par with spherical k-means.

Original: We see a small increase in the test NMI when using a larger encoding for the decoder and mean pooling to the latent representation. 
Revised: We see a small increase in the test NMI when using a larger encoding for the decoder and mean pooling to the latent representation. [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a  neural clustering method which works by automatically identifying key features of objects by applying a Noise As Target (NAT) procedure. NAT is a recently proposed approach which extracts features by ﬁxing target representations and constraining deep features to align with them. We extend NAT into a full clustering framework by combining the NAT loss with an autoencoder loss, and iteratively applying a policy for deleting and copying target representations (DnC).
Revised: In this paper, we present a novel neural clustering method which does not depend on a predeﬁned number of clusters.

Original: Our empirical evaluation shows that our framework works well, across datasets and across modalities.
Revised: Our empirical evaluation shows that our model works well across modalities.

Original: It improves state-of-the-art clustering performance on MNIST, without needing to receive the[Table] required number of clusters as input.
Revised: It improves state-of-the-art clustering performance on MNIST, without needing to receive the required number of clusters as input.

Original: Further,  outperforms powerful baselines on FashionMNIST andon text datasets (20 Newsgroups and a Twitter hashtag dataset).
Revised: Further, it outperforms powerful baselines on Fashion-MNIST and text datasets (20 Newsgroups and a Twitter hashtag dataset).

Original: NATAC achieves a moderate improvement over baselines on the text datasets

Original: [Table]

Original: and its performance is very strong on the image datasets we’ve examined.

Original: However, NATAC does require some hyperparameters to be tuned, namely the dimensionality of the latent space and the function for the loss coefﬁcient λ .
Revised: However, NATAC does require some hyperparameters to be tuned, namely the dimensionality of the latent space and the function for the loss coefﬁcient λ .

Original:  Several questions remain open.
Revised: Future work Several avenues of investigation could ﬂow from this work.

Original: First, how can our framework be extended to semi-supervised scenarios? Could we use a set of labels for a small fraction of the data so as to achieve stronger clustering for a large set of unlabeled instances? Second, could one use a training method that does not require hyperparameters
Revised: Firstly, the effectiveness of this method in a semi-supervised setting could be explored using a joint reconstruction and classiﬁcation auxiliary objective. Another interesting avenue to explore would be different agglomerative policies other than delete-and-copy.

Original: [Table]
Revised: Different geometries of the latent space could also be considered other than a unit normalized hypersphere. To remove the need of setting hyperparameters by hand, work into automatically controlling the coefﬁcients (e.g. using proportional control) could be studied.

Original: and apply better perturbations for the target set than our simple DnC policy? Finally, could our framework be modiﬁed to scale better to a higher latent dimensionality?
Revised: Finally, it would be interesting to see whether clustering jointly across different feature spaces would help with learning better representations.

Original: Appendix A V ISUALIZATIONS OF MNIST/F ASHION -MNIST L ATENT R EPRESENTATIONS
Revised: Appendix A V ISUALIZATIONS OF MNIST/F ASHION -MNIST L ATENT R EPRESENTATIONS

Original: We have published the visualizations of MNIST/Fashion-MNIST representations on the tensorﬂow embedding projector.
Revised: We have published the visualizations of MNIST/Fashion-MNIST representations on the TensorFlow embedding projector.

Original: We recommend selecting the “Spherize data” option, as this gives a much clearer view of the embeddings.
Revised: We recommend selecting the “Spherize data” option, as this gives a much clearer view of the embeddings.

Original: The images used are taken from the test set of MNIST and FashionMNIST respectively.
Revised: The images used are taken from the test set of MNIST and FashionMNIST respectively.

Original: We include the NATAC models with the highest NMI from each dataset, along with the models with d = 3 , which require no dimensionality reduction to visualize.
Revised: We include the NATAC models with the highest NMI from each dataset, along with the models with d = 3 , which require no dimensionality reduction to visualize.

Original: • MNIST d = 3 : http://projector.tensorflow.org/?config=https://raw.

Original: githubusercontent.com/neuralclusteringNAT/paper-resources/ master/mnist_embeddings/digits3/config.json • MNIST d = 10 : http://projector.tensorflow.org/?config=https://raw. githubusercontent.com/neuralclusteringNAT/paper-resources/ master/mnist_embeddings/digits10/config.json • Fashion-MNIST d = 3 : http://projector.tensorflow.org/?config=https://raw. githubusercontent.com/neuralclusteringNAT/paper-resources/ master/mnist_embeddings/fashion3/config.json • Fashion-MNIST d 
Revised: • MNIST d = 3 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnistembeddings/digits3/conﬁg.json • MNIST d = 10 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnist embeddings/digits10/conﬁg.json • Fashion-MNIST d = 3 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnist embeddings/fashion3/conﬁg.json • Fashion-MNIST d = 9 : http://projector.tensorﬂow.org/?conﬁg=https://raw.githubusercontent.com/ neuralclusteringNAT/paper-resources/master/mnist embeddings/fashion9/conﬁg.json

Original: = 9 : http://projector.tensorflow.org/?config=https://raw.

Original: githubusercontent.com/neuralclusteringNAT/paper-resources/ master/mnist_embeddings/fashion9/config.json

Original: B
Revised: B

Original: C A DDITIONAL D ISCUSSION ON T RAINING NATAC M ODELS C. G EOMETRY OF L ATENT S PACE
Revised: C A DDITIONAL D ISCUSSION ON T RAINING NATAC M ODELS C. G EOMETRY OF L ATENT S PACE

Original: We experimented with using polar coordinates early on in experimentation.
Revised: We experimented with using polar coordinates early on in our experiments.

Original: Rather than using euclidean coordinates as the latent representation, z is considered a list of angles θ 1 , θ 2 · · · θ n where θ 1 · · · θ n − 1 ∈ [0 , π ] and θ n ∈ [0 , 2 π ] .
Revised: Rather than using euclidean coordinates as the latent representation, z is considered a list of angles θ 1 , θ 2 · · · θ n where θ 1 · · · θ n − 1 ∈ [0 , π ] and θ n ∈ [0 , 2 π ] .

Original: However, we found that the models using polar geometry performed signiﬁcantly worse than those with euclidean geometry.
Revised: However, we found that the models using polar geometry performed signiﬁcantly worse than those with euclidean geometry.

Original: Additionally, we also experimented with not l2 normalizing the output of the encoder network.
Revised: Additionally, we also experimented with not L2 normalizing the output of the encoder network.

Original: We hypothesized that the model would learn a better representation of the latent space by also ‘learning’ the geometry of the noise targets.
Revised: We hypothesized that the model would learn a better representation of the latent space by also “learning” the geometry of the noise targets.

Original: Unfortunately, the unnormalized representation caused the noise targets to quickly collapse to a single point.
Revised: Unfortunately, the unnormalized representation caused the noise targets to quickly collapse to a single point.

Original: D H YPERPARAMETERS F OR E XPERIMENTS
Revised: D H YPERPARAMETERS F OR E XPERIMENTS

Original: Although each different modality (monochrome images, bag-of-words, sequence of characters) uses a different set of hyperparameters, we followed a similar recipe for determining the values for each one:
Revised: Although each different modality (monochrome images, bag-of-words, sequence of characters) uses a different set of hyperparameters, we follow a similar recipe for determining the values for each one:

Original: • We used a large batch size of 100.
Revised: • We use a large batch size of 100.

Original: This is so each batch has a representative sample of the targets to reassign in each training step.
Revised: This is so each batch has a representative sample of the targets to reassign in each training step.

Original: • The warm-up period was calculated by observing when the auxiliary objective starts to converge.
Revised: • The warm-up period is calculated by observing when the auxiliary objective starts to converge.

Original: • The ﬁnal value for λ in training was set so that the NAT loss was approximately 1% of the total loss.
Revised: • The ﬁnal value for λ in training is set so that the NAT loss was approximately 1% of the total loss.

Original: • The initial value for λ is set as approximately 1% of the ﬁnal value of λ .
Revised: • The initial value for λ is set as approximately 1% of the ﬁnal value of λ .

Original: • The transition phase typically lasts 100 epochs of training.
Revised: • The transition phase typically lasts 100 epochs of training.

Original: • During the transition phase, the value for α is incrementally increased from 0 to 1.
Revised: • During the transition phase, the value of α is incrementally increased from 0 to 1.

Original: We now explicitly list the hyperparameters used for each experiment:
Revised: We now explicitly list the hyperparameters used for each experiment:

Original: D. MNIST AND F ASHION MNIST
Revised: D. MNIST AND F ASHION MNIST

Original: • A batch size of 100.
Revised: • A batch size of 100.

Original: • A warm-up period of 10 × d epochs, during which λ = 0 .
Revised: • A warm-up period of 10 × d epochs, during which λ = 0 .

Original: • A transition period lasts for 250 epochs, where λ is incrementally increased to 0 .
Revised: • A transition period lasts for 250 epochs, where λ is incrementally increased to 0 .

Original: 25 , and α is incremented from 0 to 1 .
Revised: 25 , and α is incremented from 0 to 1 .

Original: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 )
Revised: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 )

Original: Between each subsampling layer in the encoder, a single convolution layer is applied with a ﬁlter shape of 3 × 3 with border padding to keep the same shape before and after the convolution.
Revised: Between each subsampling layer in the encoder, a single convolution layer is applied with a ﬁlter shape of 3 × 3 with border padding to keep the same shape before and after the convolution.

Original: Similarly, in the decoder one transpose convolutional layer is applied between each upsampling layer, 3 × 3 ﬁlter shape and shape-preserving padding.D.
Revised: Similarly, in the decoder one transpose convolutional layer is applied between each upsampling layer, 3 × 3 ﬁlter shape and shape-preserving padding.

Original: [Figure]
Revised: [Figure]

Original: • A batch size of 100.
Revised: • A batch size of 100.

Original: • A warm-up period of 200 epochs, during whichλ = 10  .
Revised: • A warm-up period of 100 epochs, during which 10 − 4 .

Original: • A transition period lasts for 100 epochs, where λ is incrementally increased to100 , and α is incremented from 0to 1 . 
Revised: • A transition period lasts for 100 epochs, where λ is incrementally increased to 0 . 01 , and

Original: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 ).
Revised: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 ).

Original: • Dropout with a keep probability of 0.99 in the hidden layers of the encoder and decoder.
Revised: • Dropout with a keep-probability of 0.95 in the hidden layers of the encoder and decoder.

Original: D. T WITTER D ATASET
Revised: D. T WITTER D ATASET

Original: • A batch size of 100.
Revised: • A batch size of 100.

Original: • A warm-up period of 100 epochs, during which λ = 0 .
Revised: • A warm-up period of 100 epochs, during which λ = 0 .

Original: • A transition period lasts for 100 epochs, where λ is incrementally increased to 0 .
Revised: • A transition period lasts for 100 epochs, where λ is incrementally increased to 0 .

Original: 1 , and α is incremented from 0 to 1 .
Revised: 1 , and α is incremented from 0 to 1 .

Original: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 ).
Revised: • The ADAM optimizer (Kingma & Ba, 2014) with a learning rate ( α = 10 − 5 ).