Amortized Mixing Coupling Processes for Clustering
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Considering the ever-increasing scale of data, which may contain tens of thousands of data points or complicated latent structures, the issue of scalability andalgorithmic efﬁciency becomes of vital importance for clustering.
In this paper,we propose cluster-wise amortized mixing coupling processes (AMCP), whichis able to achieve efﬁcient amortized clustering in a well-deﬁned non-parametric
Bayesian posterior.
Speciﬁcally, AMCP learns clusters sequentially with the aid ofthe proposed intra-cluster mixing (IntraCM) and inter-cluster coupling (InterCC)strategies, which investigate the relationship between data points and reference distribution in a linear optimal transport mixing view, and coupling the unassignedset and assigned set to generate new cluster.
IntraCM and InterCC avoid pairwisecalculation of distances between clusters and reduce the computational complexityfrom quadratic to linear in the current number of clusters.
Furthermore, clusterwise sequential process is able to improve the quick adaptation ability for thenext cluster generation.
In this case, AMCP simultaneously learns what makes acluster, how to group data points into clusters, and how to adaptively control thenumber of clusters.
To illustrate the superiority of the proposed method, we perform experiments on both synthetic data and real-world data in terms of clusteringperformance and computational efﬁciency.
Introduction
Clustering aims to group samples into a certain number of clusters, such that similar samples belongto the same cluster, while dissimilar ones to different clusters.
Aside from its usefulness in manydownstream tasks, clustering is an important but challenging topic which has seen applications invarious settings including social-media recommendation [30], customer partitioning [5], discoveringsocial networks [7] and partitioning protein-protein interaction networks [3].
Among various clustering algorithms [35], probabilistic generative clustering models [37] have been widely concernedbecause of its ﬂexibility and interpretability.
Probabilistic generative clustering models (as known as, mixture models) are a staple of statisticalmodeling in which a discrete latent variable is introduced for each observation, indicating its mixturecomponent identity.
These generative clustering models can be roughly divided into two categories:ﬁnite mixture model [6] and inﬁnite mixture model [29].
In recent years, ﬁnite mixture models havebeen increasingly applied in unsupervised learning problems with the aid of deep neural networks.
Instead of using an arbitrary prior for the latent variable, these methods adopted ﬁnite mixture prior,such as Gaussian mixture model (GMM) [13, 36].
A ﬁnite mixture model with a ﬁxed number ofclusters may ﬁt the given dataset well, while it may be sub-optimal to use the same number of clusters if more data comes under a slightly changed distribution [29].
It would be ideal if the clusteringmodels can ﬁgure out the unknown number of clusters automatically.
Alternatively, inﬁnite mixturemodel is the application of nonparametric Bayesian techniques to mixture modeling, which allowsfor the automatic determination of an appropriate number of mixture components.
The prior dis- tribution can be speciﬁed in terms of a data point sequential process called Dirichlet process (e.g.,
Chinese restaurant process (CRP)[27]), where the number of clusters can arbitrarily grow to better accommodate data as needed [11].
However, the inference of nonparametric posterior largelyrely on the sequential data point modeling (e.g., Markov Chain Monte Carlo (MCMC)), which istime-consuming, with convergence that is difﬁcult to assess.
While above probabilistic clustering is generally approached as an unsupervised learning problem,the amortization power of non-linear transformation (e.g., neural networks) has recently spurredprogress on supervised formulations, based on novel tractable objective functions for generic dataand an arbitrary number of clusters [18, 19, 25].
In exchange for the additional cost of procuringlabeled training data and training the model, amortized clustering offers several beneﬁts on top oftime efﬁciency at test time.
Traditional unsupervised clustering, especially probabilistic clustering,deﬁnes speciﬁc likelihood and prior on the data geometry, even without any prior knowledge of thedata, which are not needed with amortized clustering, since the model learns based on the statisticsof the labeled training data [19].
Moreover, to achieve the best result (e.g., agreement with humanlabeling or with a well-deﬁned posterior), unsupervised methods usually need parameter tuningand post-processing (e.g., cluster merging)
[14], which are obviated with amortized inference byimplicitly learning the deﬁnition of clusters underlying the training datasets.
In a sense this sharesa similar philosophy as Neural Processes [10, 9, 15], which learns from multiple datasets to learn aprior over functions in a meta learning manner.
In this paper, motivated by these prior work, we propose cluster-wise Amortized Mixing Coupling
Processes (AMCP), which is able to provides an efﬁcient clustering architecture by grouping datapoints in a cluster-wise view rather than point-wise view.
AMCP simultaneously learns what makesa cluster, how to group data points into clusters, and how to adaptively control the number of clusters.
In each cluster-wise step, AMCP generates cluster sequentially and ﬁnds one cluster a time.
In thiscase, two key strategies, Intra-Cluster Mixing ( IntraCM ) and Inter-Cluster Coupling ( InterCC ), areproposed to measure the relations between different data points and generate cluster.
Speciﬁcally,
IntraCM focus on investigating the relationship between data points and reference distribution in alinear optimal transport view [31], which avoid pairwise calculation of distances between clustersand reduce the computational complexity from quadratic to linear in the current number of clusters.
Then, InterCC leverages new insights on deﬁning dissimilarity between assigned data points andthe unassigned data points based on the couplings obtained in IntraCM , which is able to select thedissimilar data points to form the next cluster and update existing clusters.
Extensive experimentshave been conducted on both synthetic and real-world datasets, which demonstrate that AMCP cancluster data efﬁciently and accurately.
Related Work
In this section, we brieﬂy introduce some recent developments in several related topics, namelyprobabilistic clustering and amortized clustering.
Probabilistic Clustering Probabilistic clustering models (or equivalently, mixture models) are a staple of statistical modeling in which a discrete latent variable is introduced for each observation, indicating its mixture component identity.
There is a growing interest in developing probabilistic clustering methods using parametric or nonparametric prior for complex data modeling [6, 29, 13, 36],and ﬁnite and inﬁnite mixture models were developed.
The main focus of ﬁnite model introducesﬁnite mixture prior with a ﬁxed number of cluster to clustering.
In this case, Gaussian distributionare widely used to model data distribution [6].
For non-normal settings, skew normal and skewt-distribution have received increasing attention and showed the advantages in modeling data withnon-normality and heavy tails [22, 21].
One of the advantages of ﬁnite mixture models is that bothmaximum likelihood method and Bayesian approach can be applied to not only estimate modelparameters, but also evaluate probabilities of subgroup membership simultaneously.
Alternately, inﬁnite mixture models [2] adopt non-parametric prior to control the number of clusters dynamically,such as Dirichlet process and its variants [27].
Recently, researchers focus on deep generative clustering by combining probabilistic clustering modeling and neural networks [13, 20, 25, 36, 19, 34, 4].
The main focus of these methods is to learn a representation of input data amenable to clustering viadeep neural networks.
However, since such methods typically learn a data representation using deepneural networks, the representation is prone to overﬁtting when applied to small datasets.
Amortized Clustering Clustering is generally formulated as an unsupervised learning manner andwhile amortized clustering [25, 18, 19] investigates the amortized power steam from a predeﬁnedsupervised formulation to achieve an arbitrary number of clusters in new task.
Pakman et al.
[25]presented Neural Clustering Process (NCP), which sequentially computes the conditional probabilityof assigning the current data point to one of already constructed clusters or a new one, similar inspirit to the popular Gibbs sampling algorithm for Dirichlet process mixture models, but withoutpositing particular priors on partitions.
Later, Lee et al.
[18, 19] adopt attention networks to outputinstead the cluster labels of each data point, for general mixture model, with an arbitrary number
K of clusters and a cost of O ( K ) forward passes.
Although existing amortized clustering is ableto achieve arbitrary number of clusters in a sequential manner (point-wise or cluster-wise) [23],the previous build clusters are ﬁxed when new statistical is obtained, which may result in poorperformance if early data points have wrong assignments.
Yet, our work directly models the clustergenerative process, which is much more efﬁcient than the previous work and has ability to exploitthe structure among the previous generated clusters during learning process.
Method
3.1 Notations and Problem Formulation
Given N data points D = { x 1 , x 2 , · · · , x N } , where x i ∈ R
d indicates feature representation
forthe i -th data point represented in d -dim latent space.
Probabilistic models for clustering is usuallypresented by sequential sampling procedure to generate clusters.
Usually, the generative process isc i ∼ p
( c i | α 1 ) , µ k ∼ p ( µ k | α 2 ) , x i ∼ p ( x i | µ c i ) , here c i ∈ { 1 , 2 , · · · , K } encodes the cluster assignment of the data point x i .
α 1 and α 2 are hyperparameters.
The integer random variable K indicatesthe number of distinct values among the sampled c i , and µ k denotes a parameter controlling thedistribution of the k -th cluster.
Speciﬁcally, these models include Gaussian mixture model [13, 36],
Dirichlet process mixture model [8] and etc.
Point-wise Clustering Posterior Probabilistic model for clustering is usually presented by sequential sampling procedure to generate clusters.
One of the prototypical tools for nonparametric clustering modeling is the Dirichlet Process.
It allows for a discrete distribution of observations drawn froman arbitrary base measure over the domain in such a way as that the marginals match draws from,while simultaneously obtaining a countable set of distinct data points.
A useful view of the Dirichletprocess mixture model is the Chinese restaurant metaphor [11], which sequentially computes theconditional probability of assigning the current data point to one of already constructed clusters ora new one.
The cluster assignment for each data point can be sampled by c i ∼ p
( c i | c 1: i − 1 , D ) .
Herec 1: i − 1 = { c 1 , c 2 , · · · , c i − 1 } indicates the existing clusters formed by already assigned data points 125 { x 1 , x 2 , · · · , x i − 1 } .
Based on existing clusters c 1: i − 1 and observations D , the cluster assignment forthe i -th data point is sampled.
Given N data points X
= { x i } Ni =1 , it is natural to draw independent samples to form clusters usingdecomposition
[Equation]
After all data points are assigned, the ﬁnal clusters are formed.
Inference proceeds by traversingdata points and re-sampling their cluster assignments in sequence.
Challenges Above process sequentially computes the conditional probability of assigning the current data point to one of already constructed clusters or a new one, and does not posit any particular prior on partitions.
The sequential sampling procedure makes it process data points one byone, which limits its scalability on large datasets (the computational cost for full i.i.d.
sampling 135 { c 1 , c 2 , · · · , c N } is high) [12].
Furthermore, the clustering result is sensitive to the sequential processing order.
It needs a sufﬁcient number of random samples to obtain stable result, which istime-consuming.
3.2 Amortized Mixing Coupling Processes
For the seek of effective cluster generative process and preventing the effect of sequential processingorder, we focus on an alternative of clustering generative process from the view of cluster ratherthan the view of data point [19].
Let C
=
{C 1 , C 2 , · · · , C K } indicate K clusters, we deﬁne C k as 𝝓 !
𝝓 #’!
(#,%) 𝝓 #(#,%) 𝐶 !
𝝓 !(#(!,%) ...
𝝓 #(#(!,%) 𝝓 #(!
(#(!,%) 𝝓
(#,𝑻) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 !:
,𝑆 # ) 𝝓
(#(!,𝑻) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 !:
# ,𝑆 #(! ) ...
𝐶 !"
# 𝝓 !(#’!
𝝓 #’+(#’!
,%) 𝝓 #’!
,%) 𝐶 !
$# 𝝓 (#’!
,𝑻) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 !:
#’+ ,𝑆 #’! )
𝑸 !
𝑸 #’+(#’!
,%) 𝑸 #’!
,%) 𝑸 !
𝑸 #’!
(#,%) 𝑸 #(#,%) 𝑸 !
𝑸 #(#(!,%) 𝑸 #(!
(#(!,%) 𝐶 #’!
← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (#’!,𝑻) ) 𝐶 # 𝐶 !
$% ...
𝑆 !
$# 𝐶 # 𝐶 !
$# ...
𝑆 !
𝐶 # 𝐶 !
𝑆 !"
𝐶 # ← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (#,𝑻) )
𝐶 #(!
← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (#(!,𝑻) )
𝑇 𝑇 𝑇 ...
... E:
M:
E:
M:
E:
M:
C k =
( C (1) k ,
[Equation]
, C
[Equation]
) k ) , where C ( i ) k is the i -th data point of the k -th cluster. N k is thenumber of data points belonging to the k -th cluster. Based on above deﬁnition, we are interestedin sampling each cluster assignment by C k ∼ p ( C k |C 1: k − 1 , S k ) , k = 1 , · · · , K , where S k indicates theconditional information containing M k available unassigned data points for constructing the k -thcluster.
M k is the number of left data points after generating the previous k − 1 clusters, deﬁned by
M k = N − P k − 1 j =1 N j ( N j is the number of data points belonging to the j -th cluster).
We formulate cluster assignment process in the form of cluster-wise ( C 1 → C 2 → · · · → C K ) ratherthan point-wise ( c 1 → c 2 → · · · → c N ), which enables an efﬁcient clustering process.
We areinterested in sampling {C 1 , C 2 , · · · , C K } via a decomposition as follows:
[Equation]
which is easy to model cluster sequentially, and the arbitrary number of clusters can be adaptivelydetermined until there is no remaining point in S k .
The sampling process deﬁned in Eq.(2) indicatesa clustering posterior using K sets of indices.
To model above cluster-wise sequential posterior, we propose a cluster-wise amortized mixingcoupling processes (AMCP), which is able to provides an efﬁcient amortized clustering architecture by grouping data points in a cluster-wise view rather than point-wise view.
As shown in
Figure 1, AMCP generates clusters sequentially and two key strategies, Intra-Cluster Mixing ( IntraCM ) and Inter-Cluster Coupling ( InterCC ), are proposed.
Speciﬁcally, as shown in Figure 2,
IntraCM focus on investigating the relationship between data points and reference distributionϕ ( k,T ) =
[ ϕ ( k,T ), ϕ ( k,T ), · · · , ϕ ( k,T ) k ] with ϕ ( k,T )
j ∈ R d in a linear optimal transport view [31] with
T inner update steps.
Then, InterCC leverages new insights on deﬁning dissimilarity between unassigned data points in S k and assigned data points in {C 1 , C 2 , · · · , C k − 1 } based on the learned optimaltransport plan Q ( k,T ) obtained in IntraCM in a coupling manner, which is able to select the dissimilardata points to form the next cluster C k and update existing clusters {C 1 , C 2 , · · · , C k − 1 } .
3.2.1 Intra-Cluster Mixing
We present intra-cluster mixing ( IntraCM ) from generative mixing view, a novel and fast frameworkfor embedding each cluster in a vector space and deﬁning the relations of each data point from thereference distribution, which give a clear evidence or constraint that the learned distributions coulddescribe the cluster’s summary statistics well, as shown in Figure 2(b).
Speciﬁcally, in the k -thcluster generation step, data points are partitioned into several subsets, i.e., D ( k ) has structures with 171 {C 1 , C 2 , · · · , C k − 1 , S k } .
We model each data point x i ∈ D ( k ) as a Bayesian mixture model with kcomponent parametrized by reference vectors ϕ ( k ) =
[ ϕ ( k ) 1 , ϕ ( k ) 2 , · · · , ϕ ( k ) k ] ∈ R k × d and non-ﬁxedmixing proportions over k − 1 clusters {C 1 , C 2 , · · · , C k − 1 } and unassigned data points S k , as shownin Figure 2(a).
A differentiable non-linear function f θ ( · ) (a neural network) is used to transformthese representations ϕ ( k ) j into parameters
f θ ( ϕ ( k ) j ) .
By introducing latent variable b ( k ) i,j indicatingthe data point x i comes from the j -th component, we can get the full mixing likelihood related tothe k -th cluster generation as follows:p
( D ( k ) | ϕ
( k ) )
=
Y
D ( k )
X b ( k ) i p (cid:16) x i , b ( k ) i
| f θ ( ϕ ( k ) ) (cid:17) =
Y
D ( k ) k X j = p ( b ( k ) i,j ) | {z } π ( k ) j p (cid:16) x i | f θ ( ϕ ( k ) j ) , b ( k ) i,j (cid:17) , (3) where b ( k ) i =
[ b ( k ) i, 1 , b ( k ) i, 2 , · · · , b ( k ) i,k ] encodes the data point assignments.
π ( k ) =
[ π ( k ) 1 , π ( k ) 2 , · · · , π ( k ) k ]are the mixing coefﬁcients.
𝑸 (",$) 𝐶 $ 𝐶 "&$ 𝑆 " 𝐶 " ...
𝝓 (",$) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 &:"(& ,𝑆 " ) 𝝓 (",’)
𝝓 (",$) 𝑸 (",() 𝝓 (",() Es t
e p M - s t e p ...
𝒎 𝟏(") 𝝓 $(",() 𝝓 "(",() 𝝓 "&$(",() ...
𝒎 𝑴 𝒌 (") ...
𝐶 " ← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (",$) )
M - s t e p (a) Intra-cluster mixing and inter-cluster coupling 𝑆 !
𝐶 !"
# 𝝓 (!,&) 𝑸 !!,& ...
𝑸 #!,& 𝑸 !"
#!,& 𝝓 (",$) 𝝓 (",$) 𝐶 # 𝐶 !"
# 𝝓 (",$) 𝑆 !
𝑴 (!,#) 𝐶 # 𝑆 !
𝑆 !
𝑴 (!,!"
#) 𝐶 !"
# 𝐶 !
Intra-ClusterMixing
Inter-Cluster Coupling ...
𝐶 # [Figure] However, directly optimizing log Q D ( k ) p ( x i | f θ ( ϕ ( k ) )) with respect to ϕ ( k ) is difﬁcult due tomarginalization over b ( k ) i , while for many distributions optimizing log Q D ( k ) p ( x i , b ( k ) i | f θ ( ϕ ( k ) j )) ismuch easier. In order to reﬂect the relationship of data in different sets, we introduce a variationaldistribution q ( k ) i,j = q ( b ( k ) i,j | x i ) indicating a variational probability that the data point x i belongs tothe j -th component.
In this case, we introduce a constrained variational lower bound to satisfy theoptimal transport measure among different components and focus on performing the expected loglikelihood by optimizing a lower bound: [Equation] Optimal Transport View of IntraCM Here we give a detailed analysis to connect the proposedmixing strategy with optimal transport measure [31].
We aim to formulate an optimization problemfor q ( b ( k ) i,j | x i ) given in Eq.(4).
By replacing the variational distribution to a joint distribution, i.e.,q ( k ) i,j = q ( b ( k ) i,j | x i ) q
( x i ) , the optimization problem of maximizing the lower bound in Eq.(4) withrespect to the transport plan matrix Q ( k ) =
( q ( k ) i,j
) ∈ R N × k is equivalent to solving following entropicregularized Kantorovich relaxation of optimal transport [26]:
[Equation]
where − log p ( x i , b ( k ) i,j | f θ ( ϕ ( k ) )) indicates the pairwise transport cost between data point and referencevector.
If Gaussian distribution is used as data likelihood, the transport cost − log p ( x i , b ( k ) [Equation] ϕ ( k ) ))is equivalent to a negative kernel −K
x i , f θ ( ϕ ( k ) j )) .
The ϵ impacted entropic regularized term
H ( Q ( k ) )
= − P i,j q ( k ) i,j log q ( k ) i,j controls the sparsity of Q ( k ) .
In original entropic regularized Kantorovich relaxation of optimal transport [26], there are someconstraints on transport plan Q ( k ) , i.e., P Ni =
1 q ( k ) i,j = 1 / k and P kj =1 q ( k ) i,j = 1 / N .
If we set the factorizedvariational distribution to q ( x i )
= 1 / N , the constraint P j q ( k ) i,j = 1 / N is automatically met.
Optimaltransport view induces various advantages of our proposed mixing strategy.
First, it is is computationally efﬁcient, scalable to vary large dataset.
Unlike the quadratic cost of traditional similarityweight computation (e.g., the attention score in self-attention), our optimal transport mixing uses aﬁxed number ( k ) of references serving as queries and the number of references is set depends ondata without manual setting.
Secondly, it allows unsupervised learning where the model parameters{ ϕ ( k ) j } kj =1 can be learned without target (class) labels.
Expectation Maximization for IntraCM
Considering the latent variable b ( k ) is introduced, naturally the expectation maximization (EM) method can be utilized.
Iterative optimization of this boundalternates between two steps:
• E-step : The E-step focuses on maximizing above bound with respect to the variational probabilityq ( b ( k,t ) i,j | x i ) .
We compute a new estimate of the posterior probability distribution over the latent
variables from the previous iteration, which yields a soft-assignment of the data points to the com-
ponents q ( b ( k,t ) i,j | x i )
=
π ( k,t − 1) j p
( x i | b ( k, t − 1) i,j ,f
[Equation] ∑ kj = [Equation]
( k,
[Equation]
[Equation]
p ( x i | b ( k,t − 1) i,j ,f θ ( ϕ ( k,t − 1) ) ) , where π ( k,t − 1) j and ϕ ( k,t − 1) indicates model parameters from previous iteration.
This soft-assignment is a regular E-step without thebalanced assignment constraints.
Furthermore, the variational lower bound can be formulated asthe optimal transport setting (as shown in Eq.(5)), yielding an Sinkhorn-based optimal transportsolution [28] as follows:
[Equation]
where u i =
[Equation]
Note that above solution usually converges quickly after a few iterations.
Hence, the E-step coincides with the optimal transport based similarity weight computation, which leads to the variousadvantages of our proposed mixing strategy and forms the transport plan Q ( k,t ) for all data points.
• M-step : Once the E-step is done (i
.e.
, Q ( k,t ) is found), we perform the M -step
, that
is,
maximizing the lower
bound in Eq.(4) with respect to ϕ ( k,t ) when Q ( k,t ) is ﬁxed.
Due to the non-linearityof f θ ( · ) there exists no analytical solution to arg max ϕ ( k ) J ( ϕ ( k ) , Q ( k,t ) ) .
With the beneﬁt of differentiable f θ ( · ) , we can solve it with the aid of gradient-based strategy.
Assume that likelihoodp ( x i | f θ ( ϕ ( k ) j ) , b ( k ) i,j ) follows a Gaussian distribution with ﬁxed covariance, we can obtain followingupdate rule:
[Equation]
For simplicity, if we assume the parameters ϕ ( k ) in likelihood with a linear transformationf θ ( ϕ ( k ) )
=
W ϕ ( k ) , it can yield an optimal transport kernel embedding solution [24] when weset it as Gaussian distribution with ﬁxed variance, e.g., π ( k,t )
j
[Equation]
j
[Equation]
N and ϕ ( k,t ) j
[Equation]
[Equation]
i , which is equivalent to induced self-attention introduced in Leeet al.
[18] providing an efﬁcient solution for pairwise weights measuring.
Usually, we can take multiple EM steps for further optimization (e.g., ϕ ( k, 0) → ϕ ( k, 1) → · · · →ϕ ( k,T ) ).
Here T is the number of EM steps.
IntraCM belongs to the class of generalized EMalgorithms and is guaranteed to converge to a local optimum of the data log likelihood [33].
3.2.2 Inter-Cluster Coupling
After T EM steps, the variational probability q ( k,T ) i,j = q ( b ( k,T ) i,j | x i ) can be regarded as the optimaltransport plan, which carries the information on how to distribute the mass of data points x i toreference vectors ϕ
( k,T ) j .
By regarding ϕ ( k,T ) j as couplings between unassigned set and assignedset, we propose inter-cluster coupling ( InterCC ) to generate the next cluster, as shown in Figure 2.
Considering the current data points have structures with k different sets, C 1 , C 2 , · · · , C k − 1 , S k , we canget k − 1 transport plan matrices { U ( k ) 1 , U ( k ) 2 , · · · , U ( k ) k − 1 } related to k − 1 existing clusters and V ( k )related to the unassigned set S k ,
[Equation]
We can measure the relations between unsigned data points in S k and existing assigned data pointsin k − 1 existing clusters C 1 , C 2 , · · · , C k − 1 with the aid of couplings ϕ
( k ) as follows
[Equation]
where ⊕ is a column-wise concatenation operation and v ( k ) i is the i -th column of V ( k ) .
It is reasonable to select dissimilar data point in unassigned data points set S k to form a new cluster C k , i.e.,
[Equation]
Here Multinomial distribution is used to model the generation of C k .
I ∈ R N − M k is a vector of whoseelements are all 1s.
m ( k ) j is the i -th column of M ( k ) .
Thus, the data points related to dissimilarindices in each matrix are selected to generate the new cluster C k .
Furthermore, previous existingclusters C 1 , C 2 , · · · , C k − 1 can be updated based on dissimilarity matrix.
For a given cluster C l , it canbe updated by C l := C l ∪C new l , where C new l := M ULT ( S OFTMAX [1 − I ( l ) ⊤ M ( k,l ) ]) .
I ( l ) ∈ R
|C l | is a vectorof all 1s and M ( k,l )
=
U ( k ) l ϕ ( k,T ) V ( k ) ⊤ .
3.2.3 Iterative Unrolling Inference
IntraCM and InterCC are both differentiable procedures for optimal transport measuring and nextcluster constructing, whose outcome relies on the current existing clusters and the statistical modelparameters ϕ
( k ) .
As shown in Eq.(2), the whole cluster-wise clustering procedure can be regardedas a recurrent IntraCM with InterCC until there are no available data point left.
In this case, theinformation about statistical regularities in our model that is required for generating the sequentialclusters is encoded in a series of parameters { ϕ (1 ,T ) , ϕ (2 ,T ) , · · · , ϕ ( K,T ) } .
By unrolling the iterations of the presented mixing process, we obtain an end-to-end differentiable clustering procedurebased on the statistical model parameterized by { ϕ (1 ,T ) , ϕ (2 ,T ) , · · · , ϕ ( K,T ) } .
We can therefore use(stochastic) gradient descent and ﬁt the statistical model to capture the regularities corresponding toclusters for a given dataset.
[Equation]
The likelihood is derived from approximating the p θ ( C 1: K |D ) to true posterior via KL divergence.
Note that samples from p ( C 1: K , D ) are obtained from the explicit generative model, such as Gaussianmixture model, Dirichlet process mixture model.
With inﬁnite generated samples, we can train amodel to approximate p θ ( C 1: K |D ) accurately.
The whole inference procedure is given in Algorithm 1.
Note that the complexity of traditional methods measuring weights between data points acquires
O ( KN 2 ) (e.g., self-attention) which may be too expensive for large sets, and our AMCP onlyacquires O ( KNk ) in the EM-step for the k -th iteration, which is more efﬁcient.
Experiments
In this section, we evaluate the proposed AMCP in terms of synthetic and real-world datasets bycomparing with the state-of-the-art methods.
4.1 Experiments on Synthetic Datasets
The ﬁrst experiment was conducted on synthetic data with arbitrary number of clusters.
Thus, ﬁveexisting methods handling variable number of clusters, MCMC[1], VI[2], DAC[19], NCP[25] and
ST-ACT[18] are used as baselines.
Among them, MCMC and VI are traditional approximationmethods, and DAC, NCP and ST-ACT are deep amortized clustering methods.
The synthetic datais generated via a 2D Gaussian mixture model (GMM) which is deﬁned by the following process [Table] α ∼ E XP (1) , c 1: N ∼ CRP ( α ) , µ k ∼ N (0 , σ 2 µ 1 ) , x i ∼ N ( µ c i , σ 2 I ) .
We set σ µ = σ = 10and α = 1 .
In this case, the training set contains 200 samples belonging to 4 clusters.
Two testingscenarios ( Scenario 1 and Scenario 2 ) are constructed to evaluate the effectiveness of the proposedmethod.
The testing set in Scenario 1 has the same conﬁguration (200 samples and 4 clusters) astraining set, while Scenario 2 contains different numbers of samples and clusters (400 samples and6 clusters) in order to verify whether the clustering method can generalize to the unseen clusters.
Table 1 summarizes the results in terms of two testing scenarios.
Here we report log-likelihood ( LL ),clustering accuracy ( ACC ), Normalized Mutual Information ( NMI ) and running time (seconds).
Thebest and second results are marked in bold and underlined.
As expected, the proposed AMCP consistently outperforms baselines on both scenarios.
Although Scenario 2 is more challenging, amortized clustering methods (DAC, NCP, ST-ACT and our AMCP) can capture cluster uncertainty andobtain competitive results.
For testing time, amortized methods are obviously faster than traditionalsampling based method (MCMC).
Although VI is much fast than MCMC, it is still lower than amortized methods.
In cluster-wise amortized clustering methods, DAC, ST-ACT and our ACMP achievecompetitive running time since cluster-wise sequential modeling is more efﬁcient than point-wisesequential modeling.
4.2 Experiments on Real-world Datasets
For real-world datasets, MNIST [17], Tiny-ImageNet [16] and CIFAR-10 [32] are used to validate theperformance.
Among them, Tiny-ImageNet and CIFAR-10 are two more large-scale and challengingdatasets.
To formulate the amortized mechanism, we generate training data via following Dirichletprocess mixture model: α ∼ E XP (1) , c 1: N ∼ CRP ( α ) , l 1 ,K ∼ U (0 , 9) , x i ∼ U
[ D , l c
i ] .
Here x i ∼
U [ D , l c i ] indicates sampling x i from original dataset D only with label l c i uniformly.
We set α = 1 ,and adopt truncated CRP to generate cluster index for each data point in each dataset, e.g., 10 for
MNIST and CIFAR-10 , 200 for Tiny-ImageNet .
More information is given in Appendix.
In Table 2, we depict the quantitative clustering results in real-world datasets.
The best and second results are marked in bold and underlined.
The average clustering results are recorded over 5runs with different random parameter initialization.
It can be seen that our AMCP outperforms allbaselines in terms of ACC and NMI .
The main reason, we believe, is that AMCP not only exploitsamortized properties between data points in both inter- and intra-clusters, but also has ﬂexible clustering process, which simultaneously generate the new cluster and update the summary statistics forprevious clusters.
In model efﬁciency evaluation, the DAC, ST-ACT and AMCP consume a morecompetitive running time, which outperform other methods by a large margin.
The reason is that DAC and ST-ACT identify each cluster after one forward pass, and AMCP generates clusters incluster-wise manner rather points-wise manner like NCP.
However, DAC, ST-ACT are pretty worsethan AMCP on clustering performance.
Among amortized clustering methods, DAC, NCP, ST-ACT and AMCP are able to process arbitrarynumber of clusters.
For better visualization of cluster generative process, a digit subset from MNISTtesting set is randomly sampled.
To illustrate how the clustering models capture the shape ambiguityof some of the digits, we plot the ground-truth digit clusters and clustering results in Table 3.
Comparing with DAC, NCP and ST-ACT, the proposed AMCP can get the most accurate results.
Forexample, DAC assigns the digit 7 (with similar appearance to 9) to cluster 9, and NCP generates anew cluster for it.
Fortunately, AMCP correctly assigns it to cluster 7.
All baselines are unable toassign the digit 4 written in a strange way to the right cluster, and our AMCP correctly assigns it tocluster 4.
Results on Tiny-ImageNet and CIFAR-10 can be obtained in Appendix. [Table] [Table]
4.3 Model Analysis
We empirically analyze the proposed model from the impact of various hyper-parameters.
They aresummarized as follows. The Number of EM Steps T : AMCP can apply multiple EM steps to achieve more accurateϕ ( K,T ) .
Here we conduct experiments on varying T in { 1 , 2 , 3 , 4 , 5 } to investigate the effect of T .
Figure 3(a) and (b) show the effect of T on synthetic data ( Scenario 1 ) and real-world data ( CIFAR10 ).
We can see that, the performance ( LL and ACC ) become better and better with the increasing of T , while the running time is linearly scalable to the number of steps. OT E-step vs. Regular E-step We introduce two different ways to conduct E-step in IntraCM:regular soft-assignment E-step and Sinkhorn-based optimal transport E-step.
Figure 3(c) and (d)show the comparison between regular E-step and OT E-step on synthetic data ( Scenario 1 ) and realworld data ( CIFAR-10 ).
The maximum number of Sinkhorn-Knopp iterations for solving the OTproblem is set to 10 .
While the regular E-step is consistently faster than the OT E-step since it’sclosed-form solution, OT E-step can achieve more accurate likelihood and accuracy.
Furthermore, we compare the difference between the strategy of updating the existing clusters andthe strategy of not updating.
More empirical results are given in Appendix.
[Figure]
Conclusion
In this paper, we studied the problem of amortized clustering, and presented our approach thatperform cluster-wise amortized mixing coupling processes via intra-cluster mixing and inter-clustercoupling.
An interesting direction for future research is to explore more complicated and realistichidden structure (e.g., hierarchical clusters) that is not possible with our approach.
Furthermore, it isinterested to explore novel applications that can be enabled by the interpretability and controllabilitybrought by the amortized clustering.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] See Abstract and Introduction.
(b) Did you describe the limitations of your work?
[Yes] See Conclusion
(c) Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[No] All assets used in this paper are open source
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] Does not contain personally identiﬁable
information or offensive content
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]