Original: A P ROBABILISTIC A PPROACH TO C ONSTRAINED D EEP C LUSTERING .
Revised: A P ROBABILISTIC A PPROACH TO C ONSTRAINED D EEP C LUSTERING .

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Clustering with constraints has gained signiﬁcant attention in the ﬁeld of semisupervised machine learning as it can leverage partial prior information on a growing amount of unlabelled data.
Revised: Clustering with constraints has gained signiﬁcant attention in the ﬁeld of semisupervised machine learning as it can leverage partial prior information on a growing amount of unlabelled data.

Original: Following recent advances in deep generative models, we derive a novel probabilistic approach to constrained clustering that can be trained efﬁciently in the framework of stochastic gradient variational Bayes.
Revised: Following recent advances in deep generative models, we derive a novel probabilistic approach to constrained clustering that can be trained efﬁciently in the framework of stochastic gradient variational Bayes.

Original: In contrast to existing approaches, our model (CVaDE) uncovers the underlying distribution of the data conditioned on prior clustering preferences, expressed as pairwise constraints .
Revised: In contrast to existing approaches, our model (CVaDE) uncovers the underlying distribution of the data conditioned on prior clustering preferences, expressed as pairwise constraints .

Original: The inclusion of such constraints allows the user to guide the clustering process towards a desirable partition of the data by indicating which samples should or should not belong to the same class.
Revised: The inclusion of such constraints allows the user to guide the clustering process towards a desirable partition of the data by indicating which samples should or should not belong to the same class.

Original: We provide extensive experiments to demonstrate that CVaDE shows superior clustering performances and robustness compared to state-of-the-art deep constrained clustering methods in a variety of data sets.
Revised: We provide extensive experiments to demonstrate that CVaDE shows superior clustering performances and robustness compared to state-of-the-art deep constrained clustering methods in a variety of data sets.

Original: We further demonstrate the usefulness of our approach on challenging real-world medical applications and face image generation.
Revised: We further demonstrate the usefulness of our approach on challenging real-world medical applications and face image generation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ever-growing amount of data and the time cost associated with its labeling has made clustering a relevant task in the ﬁeld of machine learning.
Revised: The ever-growing amount of data and the time cost associated with its labeling has made clustering a relevant task in the ﬁeld of machine learning.

Original: Yet, in many cases, a fully unsupervised clustering algorithm might naturally ﬁnd a solution which is not consistent with the domain knowledge (Basu et al., 2008).
Revised: Yet, in many cases, a fully unsupervised clustering algorithm might naturally ﬁnd a solution which is not consistent with the domain knowledge (Basu et al., 2008).

Original: In medicine, for example, clustering could be driven by unwanted bias, such as the type of machine used to record the data, rather than more informative features.
Revised: In medicine, for example, clustering could be driven by unwanted bias, such as the type of machine used to record the data, rather than more informative features.

Original: Moreover, practitioners often have access to prior information about the types of clusters that are sought, and a principled method to guide the algorithm towards a desirable conﬁguration is then needed.
Revised: Moreover, practitioners often have access to prior information about the types of clusters that are sought, and a principled method to guide the algorithm towards a desirable conﬁguration is then needed.

Original: Constrained clustering , therefore has a long history in machine learning as it enforces desirable clustering properties by incorporating domain knowledge, in the form of constraints, into the clustering objective.
Revised: Constrained clustering , therefore has a long history in machine learning as it enforces desirable clustering properties by incorporating domain knowledge, in the form of constraints, into the clustering objective.

Original: Following recent advances in deep clustering, constrained clustering algorithms have been recently used in combination with deep neural networks (DNN) to favor a better representation of highdimensional data sets.
Revised: Following recent advances in deep clustering, constrained clustering algorithms have been recently used in combination with deep neural networks (DNN) to favor a better representation of highdimensional data sets.

Original: The methods proposed so far mainly extend some of the most widely used deep clustering algorithms, such as DEC (Xie et al., 2016), to include a variety of loss functions that force the clustering process to be consistent with the given constraints (Ren et al., 2019; Shukla et al., 2018; Zhang et al., 2019b).
Revised: The methods proposed so far mainly extend some of the most widely used deep clustering algorithms, such as DEC (Xie et al., 2016), to include a variety of loss functions that force the clustering process to be consistent with the given constraints (Ren et al., 2019; Shukla et al., 2018; Zhang et al., 2019b).

Original: Although they perform well, none of the above methods model the data generative process.
Revised: Although they perform well, none of the above methods model the data generative process.

Original: As a result, they can neither uncover the underlying structure of the data, nor control the strength of the clustering preferences, nor generate new samples (Min et al., 2018).
Revised: As a result, they can neither uncover the underlying structure of the data, nor control the strength of the clustering preferences, nor generate new samples (Min et al., 2018).

Original: To address the above issues, we propose a novel probabilistic approach to constrained clustering,
Revised: To address the above issues, we propose a novel probabilistic approach to constrained clustering,

Original: the Constrained Variational Deep
Revised: the Constrained Variational Deep

Original: Embedding (CVaDE), that uncovers the underlying data distribution conditioned on domain knowledge, expressed in the form of pairwise constraints.
Revised: Embedding (CVaDE), that uncovers the underlying data distribution conditioned on domain knowledge, expressed in the form of pairwise constraints.

Original: Our method extends previous work in unsupervised variational deep clustering (Jiang et al., 2017; Dilokthanakul et al., 2016) to incorporate clustering preferences as Bayesian prior probabilities with varying degrees of uncertainty.
Revised: Our method extends previous work in unsupervised variational deep clustering (Jiang et al., 2017; Dilokthanakul et al., 2016) to incorporate clustering preferences as Bayesian prior probabilities with varying degrees of uncertainty.

Original: This allows systematical reasoning about parameter uncertainty (Zhang et al., 2019a), thereby enabling the ability to perform Bayesian model validation, outlier detection and data generation.
Revised: This allows systematical reasoning about parameter uncertainty (Zhang et al., 2019a), thereby enabling the ability to perform Bayesian model validation, outlier detection and data generation.

Original: By integrating prior information in the generative process of the data, our model can guide the clustering process towards the conﬁguration sought by the practitioners.
Revised: By integrating prior information in the generative process of the data, our model can guide the clustering process towards the conﬁguration sought by the practitioners.

Original: Our main contributions are as follows: (i)
Revised: Our main contributions are as follows: (i)

Original: We propose a constrained clustering method (CVaDE) to incorporate given clustering preferences, with varying degrees of certainty, within the Variational
Revised: We propose a constrained clustering method (CVaDE) to incorporate given clustering preferences, with varying degrees of certainty, within the Variational

Original: Auto-Encoder (VAE) framework.
Revised: Auto-Encoder (VAE) framework.

Original: (ii)
Revised: (ii)

Original: We provide a thorough empirical assessment of our model.
Revised: We provide a thorough empirical assessment of our model.

Original: In particular, we show that (a) a small fraction of prior information remarkably increases the performance of CVaDE compared to unsupervised variational clustering methods, (b) our model shows superior clustering performance compared to state-of-the-art deep constrained clustering models on a wide range of data sets and, (c) our model proves to be robust against noise as it can easily incorporate the uncertainty of the given constraints.
Revised: In particular, we show that (a) a small fraction of prior information remarkably increases the performance of CVaDE compared to unsupervised variational clustering methods, (b) our model shows superior clustering performance compared to state-of-the-art deep constrained clustering models on a wide range of data sets and, (c) our model proves to be robust against noise as it can easily incorporate the uncertainty of the given constraints.

Original: (iii)
Revised: (iii)

Original: We show that our model can drive the clustering performance towards different desirable conﬁgurations, depending on the constraints used, and it successfully generates new samples on challenging real-world image data.
Revised: We show that our model can drive the clustering performance towards different desirable conﬁgurations, depending on the constraints used, and it successfully generates new samples on challenging real-world image data.

Original:  T HEORETICAL B ACKGROUND & R ELATED W ORK
Revised:  T HEORETICAL B ACKGROUND & R ELATED W ORK

Original: Constrained Clustering.
Revised: Constrained Clustering.

Original: A constrained clustering problem differs from the classical clustering scenario as the user has access to some pre-existing knowledge about the desired partition of the data.
Revised: A constrained clustering problem differs from the classical clustering scenario as the user has access to some pre-existing knowledge about the desired partition of the data.

Original: The constraints are usually expressed as pairwise constraints (Wagstaff & Cardie, 2000), consisting of must-links and cannot-links , which indicate whether two samples are believed to belong to the same cluster or to different clusters.
Revised: The constraints are usually expressed as pairwise constraints (Wagstaff & Cardie, 2000), consisting of must-links and cannot-links , which indicate whether two samples are believed to belong to the same cluster or to different clusters.

Original: Such pairwise relations contain less information than the labels used in classiﬁcation tasks, but are usually easier to obtain.
Revised: Such pairwise relations contain less information than the labels used in classiﬁcation tasks but are usually easier to obtain.

Original: Traditional clustering methods have been then extended to enforce pairwise constraints (Lange et al., 2005).
Revised: Traditional clustering methods have been then extended to enforce pairwise constraints (Lange et al., 2005).

Original: COP-KMEANS (Wagstaff et al., 2001) and MPCK-mean (Bilenko et al., 2004) adapted the well-known K-means algorithm, while several methods proposed a constrained version of the Gaussian Mixture Models (Shental et al., 2003; Law et al., 2004; 2005).
Revised: COP-KMEANS (Wagstaff et al., 2001) and MPCK-mean (Bilenko et al., 2004) adapted the well-known K-means algorithm, while several methods proposed a constrained version of the Gaussian Mixture Models (Shental et al., 2003; Law et al., 2004; 2005).

Original: Among them, penalized probabilistic clustering (PPC, Lu & Leen (2004)) is most related to our work as it expresses the pairwise constraints as Bayesian priors over the assignment of data points to clusters.
Revised: Among them, penalized probabilistic clustering (PPC, Lu & Leen (2004)) is most related to our work as it expresses the pairwise constraints as Bayesian priors over the assignment of data points to clusters.

Original: However, PPC, as well as the previous models, shows poor performance and high computational complexity on high-dimensional and large-scale data sets.
Revised: However, PPC, as well as the previous models, shows poor performance and high computational complexity on high-dimensional and large-scale data sets.

Original: Deep Constrained Clustering.
Revised: Deep Constrained Clustering.

Original: To overcome the limitations of the above models, constrained clustering algorithms have been lately used in combination with DNNs.
Revised: To overcome the limitations of the above models, constrained clustering algorithms have been lately used in combination with DNNs.

Original: Hsu & Kira (2015) train a DNN to minimize the Kullback-Leibler (KL) divergence between similar pairs of samples, while Chen (2015) performs semi-supervised maximum margin clustering of the learned features on a DNN.
Revised: Hsu & Kira (2015) train a DNN to minimize the Kullback-Leibler (KL) divergence between similar pairs of samples, while Chen (2015) performs semi-supervised maximum margin clustering of the learned features on a DNN.

Original: More recently, many extensions of the widely used DEC model (Xie et al., 2016) have been proposed to include a variety of loss functions to enforce pairwise constraints.
Revised: More recently, many extensions of the widely used DEC model (Xie et al., 2016) have been proposed to include a variety of loss functions to enforce pairwise constraints.

Original: Among them, SDEC (Ren et al., 2019) includes a distance loss function which forces the data points with a must-link to be close in the latent space and vice-versa.
Revised: Among them, SDEC (Ren et al., 2019) includes a distance loss function that forces the data points with a must-link to be close in the latent space and vice-versa.

Original: C-IDEC (Zhang et al., 2019b), uses, instead, a KL divergence loss, extending the work of Shukla et al.
Revised: C-IDEC (Zhang et al., 2019b), uses, instead, a KL divergence loss, extending the work of Shukla et al.

Original: Other works have focused on discriminative clustering methods by self-generating pairwise constraints from either Siamese networks or
Revised: Other works have focused on discriminative clustering methods by self-generating pairwise constraints from either Siamese networks or

Original: KNN graphs (Smieja et al., 2020) (Fogel et al., 2019).
Revised: KNN graphs (Smieja et al., 2020) (Fogel et al., 2019).

Original: As none of the approaches proposed so far is based on generative models, the above methods fail to uncover the underlying data distribution.
Revised: As none of the approaches proposed so far is based on generative models, the above methods fail to uncover the underlying data distribution.

Original: Additionally, DEC-based architectures rely on heavy pretraining of the autoencoder, resulting in no theoretical guarantee that the learned latent space is indeed suitable for clustering (Min et al., 2018).
Revised: Additionally, DEC-based architectures rely on heavy pretraining of the autoencoder, resulting in no theoretical guarantee that the learned latent space is indeed suitable for clustering (Min et al., 2018).

Original: VAE-based deep clustering.
Revised: VAE-based deep clustering.

Original: Many models have been proposed in the literature to perform unsupervised clustering through deep generative models (Li et al., 2019; Yang et al., 2019; Manduchi et al., 2019; Kopf et al., 2019).
Revised: Many models have been proposed in the literature to perform unsupervised clustering through deep generative models (Li et al., 2019; Yang et al., 2019; Manduchi et al., 2019; Kopf et al., 2019).

Original: Among them, the Variational Deep Embedding (VaDE, Jiang et al.
Revised: Among them, the Variational Deep Embedding (VaDE, Jiang et al.

Original: and the Gaussian Mixture Variational Autoencoder (GMM-VAE, Dilokthanakul et al.
Revised: and the Gaussian Mixture Variational Autoencoder (GMM-VAE, Dilokthanakul et al.

Original: (2016)) propose a variant of the VAE (Kingma & Welling (2014);
Revised: (2016)) propose a variant of the VAE (Kingma & Welling (2014);

Original: Rezende et al.
Revised: Rezende et al.

Original: (2014)) in which the prior is
Revised: (2014)) in which the prior is

Original: a
Revised: a

Original: Gaussian Mixture distribution.
Revised: Gaussian Mixture distribution.

Original: With this assumption, they construct an inference model that can be directly optimised in the framework of stochastic gradient variational Bayes.
Revised: With this assumption, they construct an inference model that can be directly optimised in the framework of stochastic gradient variational Bayes.

Original: However, variational deep clustering methods, such as the VaDE, cannot incorporate domain knowledge and clustering preferences.
Revised: However, variational deep clustering methods, such as the VaDE, cannot incorporate domain knowledge and clustering preferences.

Original: Even though a semi-supervised version on the VAE has been proposed by Kingma et al.
Revised: Even though a semi-supervised version on the VAE has been proposed by Kingma et al.

Original: (2014), the latter cannot be naturally applied to clustering.
Revised: (2014), the latter cannot be naturally applied to clustering.

Original: For this reason, we aim at extending the above methods to incorporate clustering preferences in the form of constraints, modeled as Bayesian priors, to guide the clustering process towards a desirable conﬁguration.
Revised: For this reason, we aim at extending the above methods to incorporate clustering preferences in the form of constraints, modeled as Bayesian priors, to guide the clustering process towards a desirable conﬁguration.

Original:  C ONSTRAINED V ARIATIONAL D EEP E MBEDDING
Revised:  C ONSTRAINED V ARIATIONAL D EEP E MBEDDING

Original: In the following section, we propose a novel constrained clustering model (CVaDE) to incorporate clustering preferences, with varying degree of certainty, in a VAE-based deep clustering setting.
Revised: In the following section, we propose a novel constrained clustering model (CVaDE) to incorporate clustering preferences, with varying degree of certainty, in a VAE-based deep clustering setting.

Original: In particular, we use the VaDE (Jiang et al., 2017) generative assumptions of the data, conditioned on the domain knowledge.
Revised: In particular, we use the VaDE (Jiang et al., 2017) generative assumptions of the data, conditioned on the domain knowledge.

Original: We then illustrate how our model can be trained efﬁciently in the framework of stochastic gradient variational Bayes by optimizing the Conditional Variational Lower Bound.
Revised: We then illustrate how our model can be trained efﬁciently in the framework of stochastic gradient variational Bayes by optimizing the Conditional Variational Lower Bound.

Original: Additionally, we deﬁne concrete prior formulations to incorporate our preferences, with a focus on pairwise constraints.
Revised: Additionally, we deﬁne concrete prior formulations to incorporate our preferences, with a focus on pairwise constraints.

Original: 3. T HE G ENERATIVE A SSUMPTIONS
Revised: 3. T HE G ENERATIVE A SSUMPTIONS

Original: Let us consider a data set X
Revised: Let us consider a data set X

Original: = { x i } Ni =1 consisting of N samples with x i ∈ R M that we wish to cluster into K groups according to some prior information encoded as G .
Revised: = { x i } Ni =1 consisting of N samples with x i ∈ R M that we wish to cluster into K groups according to some prior information encoded as G .

Original: For example, we may know a priori that certain samples should be clustered together with different degree of certainty.
Revised: For example, we may know a priori that certain samples should be clustered together with different degree of certainty.

Original: Hence G encodes both our prior knowledge on the data set and the degree of conﬁdence.
Revised: Hence G encodes both our prior knowledge on the data set and the degree of conﬁdence.

Original: We assume the data is generated from a random process consisting of three steps.
Revised: We assume the data is generated from a random process consisting of three steps.

Original: First, the cluster assignments c = { c i } Ni =1 , with c i ∈ { 1 , . . .
Revised: First, the cluster assignments c = { c i } Ni =1 , with c i ∈ { 1 , . . .

Original: , K } , are sampled from a distribution conditioned on the prior information, c ∼ p ( c | G ) .
Revised: , K } , are sampled from a distribution conditioned on the prior information, c ∼ p ( c | G ) .

Original: Next, for each cluster assignment c i , a continuous latent embedding, z i ∈ R D , is sampled from a Gaussian distribution, whose mean and variance depend on the selected cluster c i .
Revised: Next, for each cluster assignment c i , a continuous latent embedding, z i ∈ R D , is sampled from a Gaussian distribution, whose mean and variance depend on the selected cluster c i .

Original: Finally, the sample x i is generated from a distribution conditioned on z i .
Revised: Finally, the sample x i is generated from a distribution conditioned on z i .

Original: Given c i , the generative process can be summarized as:
Revised: Given c i , the generative process can be summarized as:

Original: [Equation]
Revised: [Equation]

Original: where µ c i and σ 2 c i are mean and variance of the Gaussian distribution corresponding to cluster c i in the latent space and the function f ( z ; θ ) is a neural network, called decoder , parametrized by θ .
Revised: where µ c i and σ 2 c i are mean and variance of the Gaussian distribution corresponding to cluster c i in the latent space and the function f ( z ; θ ) is a neural network, called decoder , parametrized by θ .

Original: Without prior information, that is when p ( c | G )
Revised: Without prior information, that is when p ( c | G )

Original: =
Revised: =

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: c ) = (cid:81) i p
Revised: c ) = (cid:81) i p

Original: [Equation]
Revised: [Equation]

Original: c i ) =
Revised: c i ) =

Original: (cid:81) i Cat ( c i | π ) , the cluster assignments are independent and identical distributed as they follow a categorical distribution with mixing parameters
Revised: [Equation] i Cat ( c i | π ) , the cluster assignments are independent and identical distributed as they follow a categorical distribution with mixing parameters

Original: π .
Revised: π .

Original: In that case, the generative assumptions described above are equal to those of Jiang et al.
Revised: In that case, the generative assumptions described above are equal to those of Jiang et al.

Original: (2017) and
Revised: (2017) and

Original: the parameters of the model can be learned using the unsupervised VaDE method (see Appendix C).
Revised: the parameters of the model can be learned using the unsupervised VaDE method (see Appendix C).

Original: In the following, we explore the case when p ( c | G ) (cid:54) =
Revised: In the following, we explore the case when p ( c | G ) (cid:54) =

Original: p ( c ) .
Revised: p ( c ) .

Original: 3. C ONDITIONAL V ARIATIONAL L OWER B OUND
Revised: 3. C ONDITIONAL V ARIATIONAL L OWER B OUND

Original: Given the data generative assumptions illustrated in Sec. 3.1, the objective is to infer the parameters π , µ c , σ 2 c and θ which better explain the data X given prior information on the cluster assignments
Revised: Given the data generative assumptions illustrated in Sec. 3.1, the objective is to infer the parameters π , µ c , σ 2 c and θ which better explain the data X given prior information on the cluster assignments

Original: G .
Revised: G .

Original: We achieve this by maximizing the marginal log-likelihood conditioned on G , that is:
Revised: We achieve this by maximizing the marginal log-likelihood conditioned on G , that is:

Original: [Equation]
Revised: [Equation]

Original: where Z = { z i } Ni =1 is the collection of the latent embeddings corresponding to the data set X .
Revised: where Z = { z i } Ni =1 is the collection of the latent embeddings corresponding to the data set X .

Original: The conditional joint probability is derived from Eq 1/2 and can be factorized as:
Revised: The conditional joint probability is derived from Eq 1/2 and can be factorized as:

Original: [Equation]
Revised: [Equation]

Original: Since the conditional log-likelihood is intractable, we derive a lower bound of the log marginal conditional probability of the data, which we call Conditional ELBO (C-ELBO, L C ):
Revised: Since the conditional log-likelihood is intractable, we derive a lower bound of the log marginal conditional probability of the data, which we call Conditional ELBO (C-ELBO, L C ):

Original: [Equation]
Revised: [Equation]

Original: Similarly to Jiang et al.
Revised: Similarly to Jiang et al.

Original: (2017) and Dilokthanakul et al.
Revised: (2017) and Dilokthanakul et al.

Original: , we employ the following amortized mean-ﬁeld variational distribution:
Revised: , we employ the following amortized mean-ﬁeld variational distribution:

Original: [Equation]
Revised: [Equation]

Original: ( Z ,
Revised: ( Z ,

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: N (cid:89) i =
Revised: N (cid:89) i =

Original: q φ ( z i
Revised: q φ ( z i

Original: [Equation]
Revised: [Equation]

Original: x i ) p ( c i
Revised: x i ) p ( c i

Original: [Equation]
Revised: [Equation]

Original: z i ) with p ( c i | z i ) =
Revised: z i ) with p ( c i | z i ) =

Original: p ( z
Revised: p ( z

Original: [Equation]
Revised: [Equation]

Original: c i
Revised: c i

Original: [Equation]
Revised: [Equation]

Original: p ( c i
Revised: p ( c i

Original: [Equation]
Revised: [Equation]

Original: (cid:80) k p ( z i | k ) p ( k ) , (6) where q φ ( z i | x i ) is a Gaussian distribution with mean µ ( x i ) and variance σ 2 ( x i )
Revised: (cid:80) k p ( z i | k ) p ( k ) , (6) where q φ ( z i | x i ) is a Gaussian distribution with mean µ ( x i ) and variance σ 2 ( x i )

Original: I which are the outputs of a neural network, called encoder , parametrized by φ and p ( c i = k ) is denoted as p ( k ) for simplicity.
Revised: I which are the outputs of a neural network, called encoder , parametrized by φ and p ( c i = k ) is denoted as p ( k ) for simplicity.

Original: It is important to note that, in this formulation, the variational distribution does not depend on G .
Revised: It is important to note that, in this formulation, the variational distribution does not depend on G .

Original: This approximation is used to retain a mean-ﬁeld variational distribution if the cluster assignments, conditioned on the prior information, are not independent (Sec 3.4.1), that is when p ( c | G ) (cid:54) =
Revised: This approximation is used to retain a mean-ﬁeld variational distribution if the cluster assignments, conditioned on the prior information, are not independent (Sec 3.4.1), that is when p ( c | G ) (cid:54) =

Original: (cid:81)
Revised: (cid:81)

Original: i p
Revised: i p

Original: ( c i | G ) .
Revised: ( c i | G ) .

Original: 3. R OLE OF THE PRIOR INFORMATION
Revised: 3. R OLE OF THE PRIOR INFORMATION

Original: To highlight how the prior information G inﬂuences the clustering objective, we rewrite Eq.
Revised: To highlight how the prior information G inﬂuences the clustering objective, we rewrite Eq.

Original: 5 as:
Revised: 5 as:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is called the reconstruction term , similarly to the VAE.
Revised: The ﬁrst term is called the reconstruction term , similarly to the VAE.

Original: The second term, on the other hand, is the Kullback-Leibler (KL) divergence between the variational posterior and the Constrained
Revised: The second term, on the other hand, is the Kullback-Leibler (KL) divergence between the variational posterior and the Constrained

Original: Gaussian Mixture prior.
Revised: Gaussian Mixture prior.

Original: By maximizing the C-ELBO, the variational posterior mimics the true conditional probability of the latent embeddings and the cluster assignments.
Revised: By maximizing the C-ELBO, the variational posterior mimics the true conditional probability of the latent embeddings and the cluster assignments.

Original: This results in enforcing the latent embeddings to follow a Gaussian mixture which agrees on the clustering preferences.
Revised: This results in enforcing the latent embeddings to follow a Gaussian mixture which agrees on the clustering preferences.

Original: Using Eq. 6, the Conditional ELBO can be further factorized as:
Revised: Using Eq. 6, the Conditional ELBO can be further factorized as:

Original: [Equation]
Revised: [Equation]

Original: where the last three terms are not affected by G and they can be rewritten using the SGVB estimator and the reparameterization trick (Kingma & Welling, 2014) to be trained efﬁciently using stochastic gradient descent (see Appendix B).
Revised: where the last three terms are not affected by G and they can be rewritten using the SGVB estimator and the reparameterization trick (Kingma & Welling, 2014) to be trained efﬁciently using stochastic gradient descent (see Appendix B).

Original: The ﬁrst term, on the other hand, is investigated in Sec 3.4.
Revised: The ﬁrst term, on the other hand, is investigated in Sec 3.4.

Original:  C ONDITIONAL P RIOR P ROBABILITY
Revised:  C ONDITIONAL P RIOR P ROBABILITY

Original: We incorporate our clustering preference through the conditional probability p ( c | G ) .
Revised: We incorporate our clustering preference through the conditional probability p ( c | G ) .

Original: In particular, we construct the conditional prior probability to be:
Revised: In particular, we construct the conditional prior probability to be:

Original: [Equation]
Revised: [Equation]

Original: where π = { π k } Kk =1 are the weights associated to each cluster,
Revised: where π = { π k } Kk =1 are the weights associated to each cluster,

Original: c i is the cluster assignment of sample x i , Ω( π ) is the normalization factor and g i ( c ) is a weighting function that assumes large values if c i agrees with our belief with respect to c and low values otherwise.
Revised: c i is the cluster assignment of sample x i , Ω( π ) is the normalization factor and g i ( c ) is a weighting function that assumes large values if c i agrees with our belief with respect to c and low values otherwise.

Original: 3.4. P AIRWISE C ONSTRAINTS
Revised: 3.4. P AIRWISE C ONSTRAINTS

Original: We hereby focus on expressing the conditional prior distribution in the context of pairwise constrained clustering and we do so by adapting the work of Lu & Leen (2004) in our variational framework.
Revised: We hereby focus on expressing the conditional prior distribution in the context of pairwise constrained clustering and we do so by adapting the work of Lu & Leen (2004) in our variational framework.

Original: The weighting function g i ( c ) is then deﬁned as:
Revised: The weighting function g i ( c ) is then deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where δ is the Kronecker δ -function and W ∈ R N × N is a symmetric matrix containing the pairwise preferences and conﬁdence.
Revised: where δ is the Kronecker δ -function and W ∈ R N × N is a symmetric matrix containing the pairwise preferences and conﬁdence.

Original: In particular, W i,j = 0 if we have no prior information on samples x i and x j , W i,j > 0 if there is a must-link constraint (the two samples should be clustered together) and
Revised: In particular, W i,j = 0 if we have no prior information on samples x i and x j , W i,j > 0 if there is a must-link constraint (the two samples should be clustered together) and

Original: W i,j < 0 if there is a cannot-link constraint (the two samples should not be clustered together).
Revised: W i,j < 0 if there is a cannot-link constraint (the two samples should not be clustered together).

Original: The value | W i,j | ∈ [0 , ∞ ) reﬂects the degree of certainty in the constraint.
Revised: The value | W i,j | ∈ [0 , ∞ ) reﬂects the degree of certainty in the constraint.

Original: For example, if W i,j −→ −∞ then x i and x j must be assigned to different clusters otherwise p ( c | G ) −→ 0 ( hard constraint).
Revised: For example, if W i,j −→ −∞ then x i and x j must be assigned to different clusters otherwise p ( c | G ) −→ 0 ( hard constraint).

Original: On the other hand, smaller values indicate a soft preference as they admit some degree of freedom in the model.
Revised: On the other hand, smaller values indicate a soft preference as they admit some degree of freedom in the model.

Original: An heuristic to select | W
Revised: An heuristic to select | W

Original: i
Revised: i

Original: ,j | is presented in Sec 4.
Revised: ,j | is presented in Sec 4.

Original: Interestingly, the probability p ( c | G ) with pairwise constraints can be seen as the posterior of the superparamagnetic clustering method (Blatt et al., 1996), with loss function given by a fully connected Potts model (Wu, 1982).
Revised: Interestingly, the probability p ( c | G ) with pairwise constraints can be seen as the posterior of the superparamagnetic clustering method (Blatt et al., 1996), with loss function given by a fully connected Potts model (Wu, 1982).

Original: Differently from our method, Blatt et al.
Revised: Differently from our method, Blatt et al.

Original: (1996) cluster the data according to the pairwise correlation functions E p ( c | G ) δ c
Revised: (1996) cluster the data according to the pairwise correlation functions E p ( c | G ) δ c

Original: i c j that are estimated with MCMC methods.
Revised: i c j that are estimated with MCMC methods.

Original: Finally, we incorporate the conditional prior distribution in Eq. 8.
Revised: Finally, we incorporate the conditional prior distribution in Eq. 8.

Original: The ﬁrst term can be written as:
Revised: The ﬁrst term can be written as:

Original: [Equation]
Revised: [Equation]

Original: Maximizing Eq.
Revised: Maximizing Eq.

Original: 13 w.r.t.
Revised: 13 w.r.t.

Original: π poses computational problems due to the normalization factor Ω( π ) .
Revised: π poses computational problems due to the normalization factor Ω( π ) .

Original: Crude approximations are investigated in (Basu et al., 2008), however we choose to ﬁx the parameter π k = 1 /K to make z uniformly distributed in the latent space, as in previous works (Dilokthanakul et al., 2016).
Revised: Crude approximations are investigated in (Basu et al., 2008), however we choose to ﬁx the parameter π k = 1 /K to make z uniformly distributed in the latent space, as in previous works (Dilokthanakul et al., 2016).

Original: By doing so, the normalization factor can be treated as a constant.
Revised: By doing so, the normalization factor can be treated as a constant.

Original: The analysis of different approaches to learn the weights π is left for future work.
Revised: The analysis of different approaches to learn the weights π is left for future work.

Original: The C-ELBO with a pairwise constrained prior can then be optimized using Monte Carlo sampling and stochastic gradient descent.
Revised: The C-ELBO with a pairwise constrained prior can then be optimized using Monte Carlo sampling and stochastic gradient descent.

Original: 3.4. F URTHER POSSIBLE C ONSTRAINTS
Revised: 3.4. F URTHER POSSIBLE C ONSTRAINTS

Original: Given the ﬂexibility of our general framework, different types of constraints can be included in the formulation of the weighting functions
Revised: Given the ﬂexibility of our general framework, different types of constraints can be included in the formulation of the weighting functions

Original: g i ( c ) .
Revised: g i ( c ) .

Original: In particular, we can perform semi-supervised learning by setting the prior information as:
Revised: In particular, we can perform semi-supervised learning by setting the prior information as:

Original: [Equation]
Revised: [Equation]

Original: where W i,k indicates whether the sample x i should be assigned to cluster k .
Revised: where W i,k indicates whether the sample x i should be assigned to cluster k .

Original: Additionally, one could also include triple-constraints by modifying the weighting function to be:
Revised: Additionally, one could also include triple-constraints by modifying the weighting function to be:

Original: [Equation]
Revised: [Equation]

Original: where W i,j,k = 0 if we do not have any prior information, W i,j,k > 0 indicates that the samples x i , x j and x k should be clustered together and
Revised: where W i,j,k = 0 if we do not have any prior information, W i,j,k > 0 indicates that the samples x i , x j and x k should be clustered together and

Original: W i,j,k < 0 if they should belong to different clusters.
Revised: W i,j,k < 0 if they should belong to different clusters.

Original: The analysis of these different constraints formulation is outside the scope of our work but they may represent interesting directions for future work.
Revised: The analysis of these different constraints formulation is outside the scope of our work but they may represent interesting directions for future work.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In the following, we provide a thorough empirical assessment of our proposed method (CVaDE) with pairwise constraints using a wide range of data sets.
Revised: In the following, we provide a thorough empirical assessment of our proposed method (CVaDE) with pairwise constraints using a wide range of data sets.

Original: First, we evaluate our model’s performance compared to both unsupervised variational deep clustering methods and state-of-the-art constrained clustering methods.
Revised: First, we evaluate our model’s performance compared to both unsupervised variational deep clustering methods and state-of-the-art constrained clustering methods.

Original: Then, we present extensive evidence of the ability of our model to handle noisy constraint information.
Revised: Then, we present extensive evidence of the ability of our model to handle noisy constraint information.

Original: We additionally perform experiments on a medical application to prove that our model can reach different desirable partitions of the data, depending on the constraints used, even with real-world, noisy data.
Revised: We additionally perform experiments on a medical application to prove that our model can reach different desirable partitions of the data, depending on the constraints used, even with real-world, noisy data.

Original: Finally, we show that CVaDE successfully generates new data, using the learned generative process of Sec 3.1, on a challenging face image data set.
Revised: Finally, we show that CVaDE successfully generates new data, using the learned generative process of Sec 3.1, on a challenging face image data set.

Original: Baselines & implementation details.
Revised: Baselines & implementation details.

Original: As baselines, we include the traditional constrained Kmeans (MPCK-means, Bilenko et al.
Revised: As baselines, we include the traditional constrained Kmeans (MPCK-means, Bilenko et al.

Original: and two recent deep constrained clustering methods based on DEC (SDEC, Ren et al.
Revised: and two recent deep constrained clustering methods based on DEC (SDEC, Ren et al.

Original: (2019), and C-IDEC, Zhang et al.
Revised: (2019), and C-IDEC, Zhang et al.

Original: (2019b))
Revised: (2019b))

Original: as they achieve stateof-the-art performance in constrained clustering.
Revised: as they achieve stateof-the-art performance in constrained clustering.

Original: We also compare our model to the unsupervised variational deep clustering method VaDE (Jiang et al., 2017).
Revised: We also compare our model to the unsupervised variational deep clustering method VaDE (Jiang et al., 2017).

Original: To implement our model, we were careful in maintaining a fair comparison with the baselines.
Revised: To implement our model, we were careful in maintaining a fair comparison with the baselines.

Original: In particular, we adopted the same encoder and decoder feed-forward architecture among all methods with four layers of 500 , 500 , 2000 ,
Revised: In particular, we adopted the same encoder and decoder feed-forward architecture among all methods with four layers of 500 , 500 , 2000 ,

Original: D units respectively, where D = 10 unless stated otherwise.
Revised: D units respectively, where D = 10 unless stated otherwise.

Original: The VAE is pretrained for 10 epochs while the DEC-based baselines need a more complex layer-wise pretraining of the autoencoder which involves 50 epochs of pretraining for each layer and 100 epochs of pretraining as ﬁnetuning.
Revised: The VAE is pretrained for 10 epochs while the DEC-based baselines need a more complex layer-wise pretraining of the autoencoder which involves 50 epochs of pretraining for each layer and 100 epochs of pretraining as ﬁnetuning.

Original: The pairwise constraints are chosen randomly within the training set by sampling two data points and assigning a must-link if they have the same label and a cannot-link otherwise.
Revised: The pairwise constraints are chosen randomly within the training set by sampling two data points and assigning a must-link if they have the same label and a cannot-link otherwise.

Original: Unless stated otherwise, the values of | W i,j | are set to 10 4 for all data sets, and N pairwise constraints are used for both our model and the constrained clustering baselines, where N is the number of samples of the considered data set.
Revised: Unless stated otherwise, the values of | W i,j | are set to 10 4 for all data sets, and N pairwise constraints are used for both our model and the constrained clustering baselines, where N is the number of samples of the considered data set.

Original: Note that N pairwise constraints can be obtained by using only √ N labeled data points.
Revised: Note that N pairwise constraints can be obtained by using only √ N labeled data points.

Original: To allow for fast iteration we simplify the last term of Eq. 13 by allowing the search of pairwise constraints to be performed only inside the considered batch.
Revised: To allow for fast iteration we simplify the last term of Eq. 13 by allowing the search of pairwise constraints to be performed only inside the considered batch.

Original: We observed empirically that, with a batch size of 1024 , this approximation does not affect the clustering performance.
Revised: We observed empirically that, with a batch size of 1024 , this approximation does not affect the clustering performance.

Original: Further details on the other hyper-parameters setting can be found in the Appendix E.1.
Revised: Further details on the other hyper-parameters setting can be found in the Appendix E.1.

Original: Constrained clustering.
Revised: Constrained clustering.

Original: We test the clustering performance of our model compared with the baselines on four different data sets: MNIST (LeCun et al., 2010), Fashion MNIST (Xiao et al., 2017),
Revised: We test the clustering performance of our model compared with the baselines on four different data sets: MNIST (LeCun et al., 2010), Fashion MNIST (Xiao et al., 2017),

Original: Reuters (Xie et al., 2016) and HHAR (Stisen et al., 2015) (see Appendix A).
Revised: Reuters (Xie et al., 2016) and HHAR (Stisen et al., 2015) (see Appendix A).

Original: Note that we preprocessed the Reuters data by computing the tf-idf features on the 2000 most frequent words on a random subset of 10000 documents and by selecting 4 root categories (Xie et al., 2016).
Revised: Note that we preprocessed the Reuters data by computing the tf-idf features on the 2000 most frequent words on a random subset of 10000 documents and by selecting 4 root categories (Xie et al., 2016).

Original: Accuracy and Normalized Mutual Information (NMI) are used as evaluation metrics.
Revised: Accuracy and Normalized Mutual Information (NMI) are used as evaluation metrics.

Original: The results are shown in
Revised: The results are shown in

Original: Table 1.
Revised: Table 1.

Original: We observe that our model reaches state-of-the-art performance in the Fashion MNIST and
Revised: We observe that our model reaches state-of-the-art performance in the Fashion MNIST and

Original: Reuters data sets.
Revised: Reuters data sets.

Original: On MNIST and HHAR data sets, on the other hand, it has comparable clustering performance with C-IDEC, however it is generally more stable accross different data sets.
Revised: On MNIST and HHAR data sets, on the other hand, it has comparable clustering performance with C-IDEC, however it is generally more stable accross different data sets.

Original: [Table]
Revised: [Table]

Original: Constrained clustering with noisy labels.
Revised: Constrained clustering with noisy labels.

Original: In real-world applications it is often the case that the additional information comes from different sources with different noise levels.
Revised: In real-world applications it is often the case that the additional information comes from different sources with different noise levels.

Original: As an example, pairwise annotations could be obtained by both very experienced domain experts and by lessexperienced users.
Revised: As an example, pairwise annotations could be obtained by both very experienced domain experts and by lessexperienced users.

Original: Hence, the ability to integrate constraints with different degrees of certainty into the clustering algorithm is of signiﬁcant practical importance.
Revised: Hence, the ability to integrate constraints with different degrees of certainty into the clustering algorithm is of signiﬁcant practical importance.

Original: In this experiment, we consider the case in which the given pairwise constraints have three different noise levels, q ∈ { 0 . 1 , 0 .
Revised: In this experiment, we consider the case in which the given pairwise constraints have three different noise levels, q ∈ { 0 . 1 , 0 .

Original: 2 , 0 . 3 } , where q determines the fraction of pairwise constraints with ﬂipped signs
Revised: 2 , 0 . 3 } , where q determines the fraction of pairwise constraints with ﬂipped signs

Original: (that is, when a mustlink is turned into a cannot-link and vice-versa).
Revised: (that is, when a mustlink is turned into a cannot-link and vice-versa).

Original: In Fig.
Revised: In Fig.

Original: 1 we show the clustering performance of our model compared to the strongest baseline derived from the previous section, C-IDEC.
Revised: 1 we show the clustering performance of our model compared to the strongest baseline derived from the previous section, C-IDEC.

Original: For all data sets, we decrease the value of the pairwise conﬁdence of our method using the heuristic | W i,j | = α log (cid:16) 1 − qq (cid:17) with α = 3500 .
Revised: For all data sets, we decrease the value of the pairwise conﬁdence of our method using the heuristic | W i,j | = α log (cid:16) 1 − qq (cid:17) with α = 3500 .

Original: Also, we use grid search to choose the hyper-parameters of
Revised: Also, we use grid search to choose the hyper-parameters of

Original: C-IDEC for the different noise levels (in particular we set the penalty weight to 0 .
Revised: C-IDEC for the different noise levels (in particular we set the penalty weight to 0 .

Original: 005 , and 0 .respectively).
Revised: 005 , and 0 .respectively).

Original: CVaDE clearly achieves better performance in terms of accuracy and NMI on all three noise levels for all data sets.
Revised: CVaDE clearly achieves better performance in terms of accuracy and NMI on all three noise levels for all data sets.

Original: In particular, the higher the noise level the greater the performance difference.
Revised: In particular, the higher the noise level the greater the performance difference.

Original: We can conclude that our model is more robust than its main competitor.
Revised: We can conclude that our model is more robust than its main competitor.

Original: Additionally,
Revised: Additionally,

Original: C-IDEC cannot model different noise levels within the same data set, while our model can easily include different source of information with different degree of uncertainty.
Revised: C-IDEC cannot model different noise levels within the same data set, while our model can easily include different source of information with different degree of uncertainty.

Original: Heart Echo.
Revised: Heart Echo.

Original: We evaluate the effectiveness of our model in a real-world application by using infant heart echo cardiogram videos.
Revised: We evaluate the effectiveness of our model in a real-world application by using infant heart echo cardiogram videos.

Original: The data set consists of 305 infant echo cardiogram videos frompatient visits obtained from a large University Children’s Hospital.
Revised: The data set consists of 305 infant echo cardiogram videos frompatient visits obtained from a large University Children’s Hospital.

Original: Each visit may consist of videos taken from several different angles (called view), denoted by [LA, KAKL, KAPAP, KAAP, 4CV].
Revised: Each visit may consist of videos taken from several different angles (called view), denoted by [LA, KAKL, KAPAP, KAAP, 4CV].

Original: We cropped the videos by isolating the cone of the echo cardiogram, we resized them to 64 ×pixels and split them into individual frames obtaining a total of N = 20000 images.
Revised: We cropped the videos by isolating the cone of the echo cardiogram, we resized them to 64 ×pixels and split them into individual frames obtaining a total of N = 20000 images.

Original: We focused on investigating two different constrained clustering problems using this data set.
Revised: We focused on investigating two different constrained clustering problems using this data set.

Original: First, we cluster the echo video frames by view.
Revised: First, we cluster the echo video frames by view.

Original: This is a relevant task, as in many datasets, heart echo videos are not explicitly labeled (Zhang et al., 2018).
Revised: This is a relevant task, as in many datasets, heart echo videos are not explicitly labeled (Zhang et al., 2018).

Original: Then, we cluster the echo video frames by infant maturity at birth, following the WHO deﬁnition of premature birth categories (”Preterm”).
Revised: Then, we cluster the echo video frames by infant maturity at birth, following the WHO deﬁnition of premature birth categories (”Preterm”).

Original: We believe that these two different clustering tasks demonstrate that our model admits a degree of control in choosing the underlying structure of the learned clusters.
Revised: We believe that these two different clustering tasks demonstrate that our model admits a degree of control in choosing the underlying structure of the learned clusters.

Original: That is, as the experiments demonstrate, by providing different pairwise constraints, it is possible to guide the clustering process towards a preferred conﬁguration, depending on what the practitioners are seeking in the data.
Revised: That is, as the experiments demonstrate, by providing different pairwise constraints, it is possible to guide the clustering process towards a preferred conﬁguration, depending on what the practitioners are seeking in the data.

Original: For both experiments, we compare the performance of our method, CVaDE, with the unsupervised
Revised: For both experiments, we compare the performance of our method, CVaDE, with the unsupervised

Original: VaDE method and the C-IDEC.
Revised: VaDE method and the C-IDEC.

Original: Additionally, we include a variant of both our method and VaDE in
Revised: Additionally, we include a variant of both our method and VaDE in

Original: [Figure]
Revised: [Figure]

Original: which we use convolutional layers (CNN-CVaDE and CNN-VaDE), for details on the implementation we refer to the Appendix E.2.
Revised: which we use convolutional layers (CNN-CVaDE and CNN-VaDE), for details on the implementation we refer to the Appendix E.2.

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: The CVaDE model outperforms both baselines by a signiﬁcant margin, in both accuracy and NMI in all clustering experiments.
Revised: The CVaDE model outperforms both baselines by a signiﬁcant margin, in both accuracy and NMI in all clustering experiments.

Original: This demonstrates that the addition of domain knowledge is particularly effective for medical purposes.
Revised: This demonstrates that the addition of domain knowledge is particularly effective for medical purposes.

Original: Additionally, we observe that C-IDEC performs poorly on real-world noisy data.
Revised: Additionally, we observe that C-IDEC performs poorly on real-world noisy data.

Original: We believe this is due to the heavy pretraining of the autoencoder, required by DEC-based methods, as it does not always enforce that the learned latent space is suitable for clustering.
Revised: We believe this is due to the heavy pretraining of the autoencoder, required by DEC-based methods, as it does not always enforce that the learned latent space is suitable for clustering.

Original: Additionally, we investigate the relationship between the number of constraints and clustering performance for the view detection task.
Revised: Additionally, we investigate the relationship between the number of constraints and clustering performance for the view detection task.

Original: We observed that with only 5000 pairwise constraints, which could be obtained with less than 80 labels, our model achieves results comparable to those of Table 2 (see Appendix D.2 for details).
Revised: We observed that with only 5000 pairwise constraints, which could be obtained with less than 80 labels, our model achieves results comparable to those of Table 2 (see Appendix D.2 for details).

Original: [Table]
Revised: [Table]

Original: Face Image Generation
Revised: Face Image Generation

Original: We evaluate the generative capabilities of our model using the UTKFace dataset (Zhang et al., 2017).
Revised: We evaluate the generative capabilities of our model using the UTKFace dataset (Zhang et al., 2017).

Original: This dataset contains over 20000 images of male and female faces, aged from 1 to 118 years old, with multiple ethnicities represented.
Revised: This dataset contains over 20000 images of male and female faces, aged from 1 to 118 years old, with multiple ethnicities represented.

Original: We use a convolutional network for the VAE (the implementation details are described in the Appendix E.3).
Revised: We use a convolutional network for the VAE (the implementation details are described in the Appendix E.3).

Original: As a ﬁrst task, we cluster the data using the gender prior information, in the form of 2 N pairwise constraints, which requires labels for 0 .
Revised: As a ﬁrst task, we cluster the data using the gender prior information, in the form of 2 N pairwise constraints, which requires labels for 0 .

Original: 7% of the data set.
Revised: 7% of the data set.

Original: Fig.
Revised: Fig.

Original: 2a/2b shows the PCA decomposition of the embedding space of both CVaDE and the unsupervised VaDE method.
Revised: 2a/2b shows the PCA decomposition of the embedding space of both CVaDE and the unsupervised VaDE method.

Original: As a second task, we select a sub-sample of individuals between 18 and 50 years of age (approx.
Revised: As a second task, we select a sub-sample of individuals between 18 and 50 years of age (approx.

Original: 11000 samples), and cluster by ethnicity (White, Black, Indian, Asian) using 2 N pairwise constraints, Fig. 2c/2d.
Revised: 11000 samples), and cluster by ethnicity (White, Black, Indian, Asian) using 2 N pairwise constraints, Fig. 2c/2d.

Original: We believe that one possible reason for biases in machine learning models currently is due to under-representation of various ethnic groups in training data sets.
Revised: We believe that one possible reason for biases in machine learning models currently is due to under-representation of various ethnic groups in training data sets.

Original: The constrained CVaDE method could be used to identify such imbalances, while requiring only a relatively small number of labeled training samples.
Revised: The constrained CVaDE method could be used to identify such imbalances, while requiring only a relatively small number of labeled training samples.

Original: This would allow easy identiﬁcation of such instances of bias in training data sets at a relatively low cost.
Revised: This would allow easy identiﬁcation of such instances of bias in training data sets at a relatively low cost.

Original: Furthermore, the ability to generate samples from these detected clusters potentially allows automatic data set augmentation.
Revised: Furthermore, the ability to generate samples from these detected clusters potentially allows automatic data set augmentation.

Original: With the inclusion of domain knowledge, we observe a neat division of the
Revised: With the inclusion of domain knowledge, we observe a neat division of the

Original: [Figure]
Revised: [Figure]

Original: selected clusters in the embedding space in both tasks, conversely the unsupervised approach is not able to distinguish any feature of interest.
Revised: selected clusters in the embedding space in both tasks, conversely the unsupervised approach is not able to distinguish any feature of interest.

Original: Finally, using the multivariate Gaussian distributions of each cluster in the learned embedded space, we test the generative capabilities of CVaDE by ﬁrst recovering the mean face of each cluster, and then generating several more faces from each cluster.
Revised: Finally, using the multivariate Gaussian distributions of each cluster in the learned embedded space, we test the generative capabilities of CVaDE by ﬁrst recovering the mean face of each cluster, and then generating several more faces from each cluster.

Original: Figure 3 shows these generated samples.
Revised: Figure 3 shows these generated samples.

Original: As can be observed, the ethnicities present in the data set are represented well by the mean face.
Revised: As can be observed, the ethnicities present in the data set are represented well by the mean face.

Original: Furthermore, the sampled faces all correspond to the respective cluster, and have a good amount of variation.
Revised: Furthermore, the sampled faces all correspond to the respective cluster, and have a good amount of variation.

Original: The quality of generated samples could be improved by using higher resolution training samples or different CNN architectures.
Revised: The quality of generated samples could be improved by using higher resolution training samples or different CNN architectures.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we present a novel deep constrained clustering method, CVaDE, that incorporates given clustering preferences in the form of pairwise constraints, with varying degrees of certainty.
Revised: In this work, we present a novel deep constrained clustering method, CVaDE, that incorporates given clustering preferences in the form of pairwise constraints, with varying degrees of certainty.

Original: In contrast to existing constrained deep clustering approaches, CVaDE uncovers the underlying distribution of the data, resulting in the ability to generate new samples, perform Bayesian model validation and outlier detection.
Revised: In contrast to existing constrained deep clustering approaches, CVaDE uncovers the underlying distribution of the data, resulting in the ability to generate new samples, perform Bayesian model validation and outlier detection.

Original: With the integration of domain knowledge, we show that our model can successfully drive the clustering algorithm towards the partitions of the data sought by the practitioners, achieving state-of-the-art constrained clustering performance in real-world and complex data sets.
Revised: With the integration of domain knowledge, we show that our model can successfully drive the clustering algorithm towards the partitions of the data sought by the practitioners, achieving state-of-the-art constrained clustering performance in real-world and complex data sets.

Original: Additionally, our model proves to be robust to noisy constraints as it can efﬁciently include uncertainty into the clustering preferences.
Revised: Additionally, our model proves to be robust to noisy constraints as it can efﬁciently include uncertainty into the clustering preferences.

Original: As a result, the proposed model can be applied to a variety of applications, where the difﬁculty of obtaining labelled data prevents the use of fully supervised algorithms.
Revised: As a result, the proposed model can be applied to a variety of applications, where the difﬁculty of obtaining labelled data prevents the use of fully supervised algorithms.

Original: A PPENDIX A D ATA SETS
Revised: A PPENDIX A D ATA SETS

Original: The data sets used in the experiments are the followings:
Revised: The data sets used in the experiments are the followings:

Original: • MNIST:
Revised: • MNIST:

Original: It consists of 70000 handwritten digits.
Revised: It consists of 70000 handwritten digits.

Original: The images are centered and of sizeby 28 pixels.
Revised: The images are centered and of sizeby 28 pixels.

Original: We reshaped each image to a 784- dimensional vector (LeCun et al., 2010).
Revised: We reshaped each image to a 784- dimensional vector (LeCun et al., 2010).

Original: • Fashion MNIST:
Revised: • Fashion MNIST:

Original: A data set of Zalando’s article images consisting of a training set of 60,000 examples and a test set of 10,000 examples (Xiao et al., 2017).
Revised: A data set of Zalando’s article images consisting of a training set of 60,000 examples and a test set of 10,000 examples (Xiao et al., 2017).

Original: • HHAR:
Revised: • HHAR:

Original: The Heterogeneity Human Activity Recognition (HHAR) dataset containssensor records from smart phones and smart watches.
Revised: The Heterogeneity Human Activity Recognition (HHAR) dataset containssensor records from smart phones and smart watches.

Original: All samples are partitioned intocategories of human activities and each sample is of 561 dimensions (Stisen et al., 2015).
Revised: All samples are partitioned intocategories of human activities and each sample is of 561 dimensions (Stisen et al., 2015).

Original: • Reuters:
Revised: • Reuters:

Original: It contains 810000 English news stories (Lewis et al., 2004).
Revised: It contains 810000 English news stories (Lewis et al., 2004).

Original: Following the work of Xie et al.
Revised: Following the work of Xie et al.

Original: , we used 4 root categories: corporate/industrial, government/social, markets, and economics as labels and discarded all documents with multiple labels, which results in a 685071-article dataset.
Revised: , we used 4 root categories: corporate/industrial, government/social, markets, and economics as labels and discarded all documents with multiple labels, which results in a 685071-article dataset.

Original: We computed tf-idf features on the 2000 most frequent words to represent all articles.
Revised: We computed tf-idf features on the 2000 most frequent words to represent all articles.

Original: A random subset of 10000 documents is then sampled.
Revised: A random subset of 10000 documents is then sampled.

Original: • Newborn echo cardiograms:
Revised: • Newborn echo cardiograms:

Original: The dataset consists of 305 infant echo cardiogram videos from 65 patient visits obtained from a large children hospital.
Revised: The dataset consists of 305 infant echo cardiogram videos from 65 patient visits obtained from a large children hospital.

Original: Each visit may consist of videos taken from several different angles, denoted by [LA, KAKL, KAPAP, KAAP, 4CV].
Revised: Each visit may consist of videos taken from several different angles, denoted by [LA, KAKL, KAPAP, KAAP, 4CV].

Original: We cropped the videos by isolating the cone of the echo cardiogram, we resized them to 64x64 pixels and split them into individual frames obtaining a total of N = 20000 images.
Revised: We cropped the videos by isolating the cone of the echo cardiogram, we resized them to 64x64 pixels and split them into individual frames obtaining a total of N = 20000 images.

Original: • UTKFace:
Revised: • UTKFace:

Original: This dataset contains over 20000 images of male and female face of individuals from 1 to 118 years old, with multiple ethnicities represented (Zhang et al., 2017).
Revised: This dataset contains over 20000 images of male and female face of individuals from 1 to 118 years old, with multiple ethnicities represented (Zhang et al., 2017).

Original: B C ONDITIONAL ELBO D ERIVATIONS
Revised: B C ONDITIONAL ELBO D ERIVATIONS

Original: In this section we provide the detailed derivation of the Conditional ELBO with pairwise constraints and we describe how it can be trained efﬁciently with Monte Carlo sampling.
Revised: In this section we provide the detailed derivation of the Conditional ELBO with pairwise constraints and we describe how it can be trained efﬁciently with Monte Carlo sampling.

Original: Speciﬁcally, the CELBO is deﬁned as:
Revised: Speciﬁcally, the CELBO, L C ( X | G ) , is the upper bound of the marginal log-likelihood conditioned on G : [Equation]

Original: [Equation]
Revised: [Equation]

Original: By plugging in the marginal log-likelihood of Eq. 3 and the variational distribution of Eq.
Revised: By plugging in the marginal log-likelihood of Eq. 3 and the variational distribution of Eq.

Original: 6, the
Revised: 6, the

Original: C-ELBO can be rewritten as:
Revised: C-ELBO can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that the variational distribution can be factorized as q φ ( Z , c | X )
Revised: where we used the fact that the variational distribution can be factorized as q φ ( Z , c | X )

Original: =
Revised: =

Original: q φ ( Z | X ) p ( c | Z ) .
Revised: q φ ( Z | X ) p ( c | Z ) .

Original: Given that q φ ( Z | X ) p ( c | Z ) =
Revised: Given that q φ ( Z | X ) p ( c | Z ) =

Original: (cid:81)
Revised: [Equation]

Original: i q φ ( z i | x i ) p ( c i | z i ) and using Eq 13, we can further factorize Eq 18:
Revised: i q φ ( z i | x i ) p ( c i | z i ) and using Eq 13, we can further factorize Eq 18:

Original: [Equation]
Revised: [Equation]

Original: As p ( c i | z i ) is discrete, E p ( c i
Revised: As p ( c i | z i ) is discrete, E p ( c i

Original: | z i ) [ · ] = (cid:80) k p
Revised: | z i ) [ · ] = (cid:80) k p

Original: ( c i = k | z i )[ · ] , the above equation can then be written as:
Revised: ( c i = k | z i )[ · ] , the above equation can then be written as:

Original: [Equation]
Revised: [Equation]

Original: Using the SGVB estimator
Revised: Using the SGVB estimator

Original: [Equation]
Revised: [Equation]

Original: we can approximate the above equation as:
Revised: we can approximate the above equation as:

Original: [Equation]
Revised: [Equation]

Original: where L is the number of Monte Carlo samples in the SGVB estimator and it is set to L = 1 in all experiments.
Revised: where L is the number of Monte Carlo samples in the SGVB estimator and it is set to L = 1 in all experiments.

Original: C V ARIATIONAL D EEP E MBEDDING
Revised: C V ARIATIONAL D EEP E MBEDDING

Original: The Variational Deep Embedding method, VaDE (Jiang et al., 2017), assumes an observed sample x i is generated by the following generative process: 
Revised: The Variational Deep Embedding method, VaDE (Jiang et al., 2017), assumes an observed sample x i is generated by the following generative process: [Equation] [Equation] z |

Original: [Equation]
Revised: [Equation]

Original: where K is the predeﬁned number of clusters µ c
Revised: x

Original: [Equation]
Revised: [Equation]

Revised: I ) with [ µ x i , σ 2 x i ] =

Original:  σ 2 c are mean and variance of the Gaussian distribution corresponding to cluster c in the latent space and the function f
Revised: f ( z i ; θ ) if x i is real-valued Ber ( µ x i ) with µ x i = f ( z i ; θ ) if x i is binary (23) where K is the predeﬁned number of clusters µ c , σ 2 c are mean and variance of the Gaussian distribution corresponding to cluster c in the latent space and the function f

Original: ( z ; θ ) is a neural network, called decoder [Equation] parametrized by θ , similarly to a VAE.
Revised: ( z ; θ ) is a neural network, called decoder , parametrized by θ , similarly to a VAE.

Original: To infer both the parameters of the Gaussian mixture model and the decoder [Equation] the VaDE maximises the likelihood of the data X , that is:
Revised: To infer both the parameters of the Gaussian mixture model and the decoder, the VaDE maximises the likelihood of the data X , that is:

Original: [Equation]
Revised: [Equation]

Original: The variational distribution is chosen to be:
Revised: The variational distribution is chosen to be:

Original: [Equation]
Revised: [Equation]

Original: p ( c i | z i ) with p

Original: [Equation]

Original: i

Original: [Equation] z [Equation] where q φ ( z i | x i ) is a Gaussian distribution with mean µ ( x i ) and variance σ 2 ( x i )
Revised: where q φ ( z i | x i ) is a Gaussian distribution with mean µ ( x i ) and variance σ 2 ( x i )

Original: I which are the outputs of a neural network, called encoder , parametrized by φ .
Revised: I which are the outputs of a neural network, called encoder , parametrized by φ .

Original: The ELBO can be then formulated as:
Revised: The ELBO can be then formulated as:

Original: [Equation]
Revised: [Equation]

Original: D F URTHER E XPERIMENTS D. N OISY L ABELS
Revised: D F URTHER E XPERIMENTS D. N OISY L ABELS

Original: In Tables 3/4/5/6, we present the quantitative results in term of Accuracy and Normalized Mutual Information of both our model, CVaDE, and the strongest baseline, C-IDEC with N noisy constraints (presented visually in Fig. 1).
Revised: In Tables 3/4/5/6, we present the quantitative results in term of Accuracy and Normalized Mutual Information of both our model, CVaDE, and the strongest baseline, C-IDEC with N noisy constraints (presented visually in Fig. 1).

Original: In particular, the results are computed for q ∈ { 0 . 1 , 0 . 2 , 0 . 3 } , where q determines the fraction of pairwise constraints with ﬂipped signs (that is, when a must-link is turned into a cannot-link and vice-versa).
Revised: In particular, the results are computed for q ∈ { 0 . 1 , 0 . 2 , 0 . 3 } , where q determines the fraction of pairwise constraints with ﬂipped signs (that is, when a must-link is turned into a cannot-link and vice-versa).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D. H EART E CHO
Revised: D. H EART E CHO

Original: PCA Decompostion
Revised: PCA Decompostion

Original: In Figure 4 we present a PCA decomposition of the embedded space learned by both the CVaDE and VaDE baseline
Revised: In Figure 4 we present a PCA decomposition of the embedded space learned by both the CVaDE and VaDE baseline

Original: [Table]
Revised: [Table]

Original: from the experiment presented in Section 4.
Revised: from the experiment presented in Section 4.

Original: It can be observed that the CVaDE model, using N constraints
Revised: It can be observed that the CVaDE model, using N constraints

Original: [Table]
Revised: [Table]

Original: is able to learn an embedded space which clusters by view much more effectively.
Revised: is able to learn an embedded space which clusters by view much more effectively.

Original: Impact of Constraints In order to try to understand how the number of constraints provided impacts the performance of CVaDE, we performed an experiment using the heart echo dataset where progressively more and more constraints were provided to the model during training.
Revised: Impact of Constraints In order to try to understand how the number of constraints provided impacts the performance of CVaDE [Figure] we performed an experiment using the heart echo dataset where progressively more and more constraints were provided to the model during training.

Original: Figuredemonstrates that the clustering performance of CVaDE improves as more pairwise constraints are provided.
Revised: Figuredemonstrates that the clustering performance of CVaDE improves as more pairwise constraints are provided.

Original: This trend continues until 5000 pairwise constraints are provided (requiring 0 .
Revised: This trend continues until 5000 pairwise constraints are provided (requiring 0 .

Original: 35% of the dataset to be labeled), at which point the performance reaches a plateau.
Revised: 35% of the dataset to be labeled), at which point the performance reaches a plateau.

Original: [Figure]
Revised: [Figure]

Original: E I MPLEMENTATION D ETAILS E. H YPER - PARAMETERS SETTING
Revised: E I MPLEMENTATION D ETAILS E. H YPER - PARAMETERS SETTING

Original: In Table 7 we specify the hyper-parameters setting of both our model, CVaDE, and the unsupervised baseline, VaDE.
Revised: In Table 7 we specify the hyper-parameters setting of both our model, CVaDE, and the unsupervised baseline, VaDE.

Original: Given the semi-supervised setting, we did not focus in ﬁne-tuning the hyperparameters but rather we chose standard conﬁgurations for all data sets.
Revised: Given the semi-supervised setting, we did not focus in ﬁne-tuning the hyperparameters but rather we chose standard conﬁgurations for all data sets.

Original: We observed that our model is robust against changes in the hyper-parameters, except for the batch size.
Revised: We observed that our model is robust against changes in the hyper-parameters, except for the batch size.

Original: The latter requires a high value, as we simplify the last term of Eq. 13 by allowing the search of pairwise constraints to be performed only inside the considered batch.
Revised: The latter requires a high value, as we simplify the last term of Eq. 13 by allowing the search of pairwise constraints to be performed only inside the considered batch.

Original: For the VaDE, we used the same hyper-parameters setting used in their paper (Jiang et al., 2017).
Revised: For the VaDE, we used the same hyper-parameters setting used in their paper (Jiang et al., 2017).

Original: [Table]
Revised: [Table]

Original: In addition to the model described in Section
Revised: In addition to the model described in Section

Original: [Table]
Revised: [Table]

Original: we also used a VGG-like convolutional neural network.
Revised: we also used a VGG-like convolutional neural network.

Original: This model is implemented in Tensorﬂow, using two VGG blocks (using a 3 × 3 kernel size) of 32 and 64 ﬁlters for the encoder, followed by a single fully-connected layer reducing down to an embedding of dimension 10 .
Revised: This model is implemented in Tensorﬂow, using two VGG blocks (using a 3 × 3 kernel size) of 32 and 64 ﬁlters for the encoder [Table] followed by a single fully-connected layer reducing down to an embedding of dimension 10 .

Original: The decoder has a symmetric architecture.
Revised: The decoder has a symmetric architecture.

Original: The VAE is pretrained for 10 epochs, following which our model is trained for 300 epochs using a learning rate of 0 .
Revised: The VAE is pretrained for 10 epochs, following which our model is trained for 300 epochs using a learning rate of 0 .

Original: 001 (with an exponential decay of 0 . 00001 ), and a batch size of 1024 .
Revised: 001 (with an exponential decay of 0 . 00001 ), and a batch size of 1024 .

Original: Refer to the accompanying code for further details.
Revised: Refer to the accompanying code for further details.

Original: E. F ACE I MAGE G ENERATION
Revised: E. F ACE I MAGE G ENERATION

Original: The face image generation experiments using the UTK Face dataset described in Section 4 were carried out using VGG-like convolutional neural networks implemented in Tensorﬂow.
Revised: The face image generation experiments using the UTK Face dataset described in Section 4 were carried out using VGG-like convolutional neural networks implemented in Tensorﬂow.

Original: In particular, the input image size of 64 × 64 × 3 allowed two VGG blocks (using a 3 × 3 kernel size) of 64 andﬁlters for the encoder, followed by a single fully-connected layer reducing down to an embedding of dimension 50 .
Revised: In particular, the input image size of 64 × 64 × 3 allowed two VGG blocks (using a 3 × 3 kernel size) of 64 andﬁlters for the encoder, followed by a single fully-connected layer reducing down to an embedding of dimension 50 .

Original: The decoder has a symmetric architecture.
Revised: The decoder has a symmetric architecture.

Original: The VAE is pretrained for 100 epochs, following which our model is trained for 1000 epochs using a learning rate of 0 .
Revised: The VAE is pretrained for 100 epochs, following which our model is trained for 1000 epochs using a learning rate of 0 .

Original: 001 (with a decay of 0 . 00001 ), and a batch size of 1024 .
Revised: 001 (with a decay of 0 . 00001 ), and a batch size of 1024 .

Original: Refer to the accompanying code for further details.
Revised: Refer to the accompanying code for further details.