Amortized Mixing Coupling Processes for Clustering
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Considering the ever-increasing scale of data, which may contain tens of thousands of data points or complicated latent structures, the issue of scalability andalgorithmic efﬁciency becomes of vital importance for clustering.
In this paper,we propose cluster-wise amortized mixing coupling processes (AMCP), whichis able to achieve efﬁcient amortized clustering in a well-deﬁned non-parametric
Bayesian posterior.
Speciﬁcally, AMCP learns clusters sequentially with the aid ofthe proposed intra-cluster mixing (IntraCM) and inter-cluster coupling (InterCC)strategies, which investigate the relationship between data points and reference distribution in a linear optimal transport mixing view, and coupling the unassignedset and assigned set to generate new cluster.
IntraCM and InterCC avoid pairwisecalculation of distances between clusters and reduce the computational complexityfrom quadratic to linear in the current number of clusters.
Furthermore, clusterwise sequential process is able to improve the quick adaptation ability for thenext cluster generation.
In this case, AMCP simultaneously learns what makes acluster, how to group data points into clusters, and how to adaptively control thenumber of clusters.
To illustrate the superiority of the proposed method, we perform experiments on both synthetic data and real-world data in terms of clusteringperformance and computational efﬁciency.
Introduction
Clustering aims to group samples into a certain number of clusters, such that similar samples belongto the same cluster, while dissimilar ones to different clusters.
Aside from its usefulness in manydownstream tasks, clustering is an important but challenging topic which has seen applications invarious settings including social-media recommendation [30], customer partitioning [5], discoveringsocial networks [7] and partitioning protein-protein interaction networks [3].
Among various clustering algorithms [35], probabilistic generative clustering models [37] have been widely concernedbecause of its ﬂexibility and interpretability.
Probabilistic generative clustering models (as known as, mixture models) are a staple of statisticalmodeling in which a discrete latent variable is introduced for each observation, indicating its mixturecomponent identity.
These generative clustering models can be roughly divided into two categories:ﬁnite mixture model [6] and inﬁnite mixture model [29].
In recent years, ﬁnite mixture models havebeen increasingly applied in unsupervised learning problems with the aid of deep neural networks.
Instead of using an arbitrary prior for the latent variable, these methods adopted ﬁnite mixture prior,such as Gaussian mixture model (GMM) [13, 36].
A ﬁnite mixture model with a ﬁxed number ofclusters may ﬁt the given dataset well, while it may be sub-optimal to use the same number of clusters if more data comes under a slightly changed distribution [29].
It would be ideal if the clusteringmodels can ﬁgure out the unknown number of clusters automatically.
Alternatively, inﬁnite mixturemodel is the application of nonparametric Bayesian techniques to mixture modeling, which allowsfor the automatic determination of an appropriate number of mixture components.
The prior distribution can be speciﬁed in terms of a data point sequential process called Dirichlet process (e.g.,
Chinese restaurant process (CRP)[27]), where the number of clusters can arbitrarily grow to better accommodate data as needed [11].
However, the inference of nonparametric posterior largelyrely on the sequential data point modeling (e.g., Markov Chain Monte Carlo (MCMC)), which istime-consuming, with convergence that is difﬁcult to assess.
While above probabilistic clustering is generally approached as an unsupervised learning problem,the amortization power of non-linear transformation (e.g., neural networks) has recently spurredprogress on supervised formulations, based on novel tractable objective functions for generic dataand an arbitrary number of clusters [18, 19, 25].
In exchange for the additional cost of procuringlabeled training data and training the model, amortized clustering offers several beneﬁts on top oftime efﬁciency at test time.
Traditional unsupervised clustering, especially probabilistic clustering,deﬁnes speciﬁc likelihood and prior on the data geometry, even without any prior knowledge of thedata, which are not needed with amortized clustering, since the model learns based on the statisticsof the labeled training data [19].
Moreover, to achieve the best result (e.g., agreement with humanlabeling or with a well-deﬁned posterior), unsupervised methods usually need parameter tuningand post-processing (e.g., cluster merging)
[14], which are obviated with amortized inference byimplicitly learning the deﬁnition of clusters underlying the training datasets.
In a sense this sharesa similar philosophy as Neural Processes [10, 9, 15], which learns from multiple datasets to learn aprior over functions in a meta learning manner.
In this paper, motivated by these prior work, we propose cluster-wise Amortized Mixing Coupling
Processes (AMCP), which is able to provides an efﬁcient clustering architecture by grouping datapoints in an optimal transport cluster-wise view.
In each cluster-wise step, AMCP generates cluster sequentially and ﬁnds one cluster a time.
In this case, two key strategies, Intra-Cluster Mixing( IntraCM ) and Inter-Cluster Coupling ( InterCC ), are proposed to measure the relations betweendifferent data points and generate cluster.
Speciﬁcally, IntraCM focus on investigating the relationship between data points and reference distribution in a linear optimal transport view [31], whichavoid pairwise calculation of distances between clusters and reduce the computational complexityfrom quadratic to linear in the current number of clusters.
Then, InterCC leverages new insightson deﬁning dissimilarity between assigned data points and the unassigned data points based on thecouplings obtained in IntraCM , which is able to select the dissimilar data points to form the nextcluster and update existing clusters.
The key contributions of our work are as follows: (1) Efﬁciency .
The proposed AMCP not only inherent the efﬁcient cluster-wise learning manner, but also reducethe quadratic cost of traditional similarity weight computation with the aid of optimal transport mixing.
(2) Novelty .
To the best of our knowledge, AMCP is the ﬁrst to introduce optimal transport foramortized clustering and effectively correlate the relationship between optimal transport and mixturemodels. (3) Adaptivity . Non-parametric learning manners exist not only in model parameter learningbut also in cluster generation, which allows unsupervised parameter learning and non-handcraftedintervention. (4) Flexibility .
Different from existing amortized clustering methods without changingthe already-generated clusters, AMCP is able to dynamically adjust the previous generated clustersto avoid the accumulated errors with the aid of mixing and coupling manners.
(5) Effectiveness .
Extensive experiments conducted on both synthetic and real-world datasets demonstrate that AMCPcan cluster data effectively and efﬁciently.
Related Work
In this section, we brieﬂy introduce some recent developments in several related topics, namelyprobabilistic clustering and amortized clustering.
Probabilistic Clustering Probabilistic clustering models (or equivalently, mixture models) are a staple of statistical modeling in which a discrete latent variable is introduced for each observation, indicating its mixture component identity.
There is a growing interest in developing probabilistic clustering methods using parametric or nonparametric prior for complex data modeling [6, 29, 13, 36], andﬁnite and inﬁnite mixture models were developed.
The main focus of ﬁnite model introduces ﬁnitemixture prior with a ﬁxed number of cluster to clustering.
In this case, Gaussian distribution or non Gaussian distributions [22, 21] are widely used to model data distribution [6].
Alternately, inﬁnitemixture models [2] adopt non-parametric prior to control the number of clusters dynamically, suchas Dirichlet process and its variants [27].
Recently, researchers focus on deep generative clusteringby combining probabilistic clustering modeling and neural networks [13, 20, 25, 36, 19, 34, 4].
Themain focus of these methods is to learn a representation of input data amenable to clustering viadeep neural networks.
Amortized Clustering Clustering is generally formulated as an unsupervised learning manner andwhile amortized clustering [25, 18, 19] investigates the amortized power steam from a predeﬁnedsupervised formulation to achieve an arbitrary number of clusters in new task.
Lee et al.
[18, 19]adopt attention networks to output instead the cluster labels of each data point, for general mixturemodel, with an arbitrary number K of clusters and a cost of O ( K ) forward passes.
Later, Pakmanet al.
[25] presented Neural Clustering Process (NCP), which sequentially computes the conditionalprobability of assigning the current data point to one of already constructed clusters or a new one.
The most similar work to our methods is NCP [25] and CHiGac [23].
Both of them explain clustering from the perspective of generative models, which similar in spirit to the popular Gibbs samplingalgorithm for Dirichlet process mixture models, but without positing particular priors on partitions. Besides, NCP and CHiGac rely heavily on the selecting of anchor points and the processing orderingand often exhibits unstable properties.
Different from previous work, AMCP connects the relationship of mixture model and optimal transport to describe the cluster’s summary statistics well, andseamlessly combines mixing and coupling to achieve efﬁcient amortized clustering.
Method
3.1 Notations and Problem Formulation
Given N data points D = { x 1 , x 2 , · · · , x N } , where x i ∈ R
d indicates feature representation
forthe i -th data point represented in d -dim latent space.
Probabilistic models for clustering is usuallypresented by sequential sampling procedure to generate clusters.
Usually, the generative process isc i ∼ p
( c i | α 1 ) , µ k ∼ p ( µ k | α 2 ) , x i ∼ p ( x i | µ c i ) , here c i ∈ { 1 , 2 , · · · , K } encodes the cluster assignment of the data point x i .
α 1 and α 2 are hyperparameters.
The integer random variable K indicatesthe number of distinct values among the sampled c i , and µ k denotes a parameter controlling thedistribution of the k -th cluster.
Speciﬁcally, these models include Gaussian mixture model [13, 36],
Dirichlet process mixture model [8] and etc.
Point-wise Clustering Posterior Probabilistic model for clustering is usually presented by sequential sampling procedure to generate clusters.
One of the prototypical tools for nonparametric clustering modeling is the Dirichlet Process.
It allows for a discrete distribution of observations drawn froman arbitrary base measure over the domain in such a way as that the marginals match draws from,while simultaneously obtaining a countable set of distinct data points.
A useful view of the Dirichletprocess mixture model is the Chinese restaurant metaphor [11], which sequentially computes theconditional probability of assigning the current data point to one of already constructed clusters ora new one.
The cluster assignment for each data point can be sampled by c i ∼ p
( c i | c 1: i − 1 , D ) .
Herec 1: i − 1 = { c 1 , c 2 , · · · , c i − 1 } indicates the existing clusters formed by already assigned data points 128 { x 1 , x 2 , · · · , x i − 1 } .
Based on existing clusters c 1: i − 1 and observations D , the cluster assignment forthe i -th data point is sampled.
Given N data points X
= { x i } Ni =1 , it is natural to draw independent samples to form clusters usingdecomposition
[Equation]
After all data points are assigned, the ﬁnal clusters are formed.
Inference proceeds by traversingdata points and re-sampling their cluster assignments in sequence.
Challenges Above process sequentially computes the conditional probability of assigning the current data point to one of already constructed clusters or a new one, and does not posit any particular prior on partitions.
The sequential sampling procedure makes it process data points one byone, which limits its scalability on large datasets (the computational cost for full i.i.d.
sampling 138 { c 1 , c 2 , · · · , c N } is high) [12].
Furthermore, the clustering result is sensitive to the sequential processing order.
It needs a sufﬁcient number of random samples to obtain stable result, which istime-consuming.
3.2 Amortized Mixing Coupling Processes
For the seek of effective cluster generative process and preventing the effect of sequential processingorder, we focus on an alternative of clustering generative process from the view of cluster ratherthan the view of data point [19].
Let C
=
{C 1 , C 2 , · · · , C K } indicate K clusters, we deﬁne C k as
C k =
( C (1) k , · · · , C ( i ) k , · · · , C ( N k ) k ) , where C ( i ) k is the i -th data point of the k -th cluster.
N k is thenumber of data points belonging to the k -th cluster.
Based on above deﬁnition, we are interested in sampling each cluster assignment by C k ∼ p ( C k |C 1: k − 1 , S k ) , k
= 1 , · · · , K , where S k indicates theconditional information containing M k available unassigned data points for constructing the k -thcluster.
M k is the number of left data points after generating the previous k − 1 clusters, deﬁned by
M k = N − P k − 1 j =1 N j ( N j is the number of data points belonging to the j -th cluster).
We formulate cluster assignment process in the form of cluster-wise ( C 1 → C 2 → · · · → C K ) ratherthan point-wise ( c 1 → c 2 → · · · → c N ), which enables an efﬁcient clustering process.
We areinterested in sampling {C 1 , C 2 , · · · , C K } via a decomposition as follows:
[Equation]
which is easy to model cluster sequentially, and the arbitrary number of clusters can be adaptivelydetermined until there is no remaining point in S k .
The sampling process deﬁned in Eq.(2) indicatesa clustering posterior using K sets of indices.
To model above cluster-wise sequential posterior, we propose a cluster-wise amortized mixing coupling processes (AMCP), which is able to provides an efﬁcient amortized clustering architectureby grouping data points in a cluster-wise view rather than point-wise view.
As shown in Figure 1(a), AMCP generates clusters sequentially and two key strategies, Intra-Cluster Mixing ( IntraCM ) and Inter-Cluster Coupling ( InterCC ), are proposed.
Speciﬁcally, as shown in Figure 1,
IntraCM focus on investigating the relationship between data points and reference distributionϕ ( k,T ) =
[ ϕ ( k,T ), ϕ ( k,T ), · · · , ϕ ( k,T ) k ] with ϕ ( k,T )
j ∈ R d in a linear optimal transport view [31] with
T inner update steps.
Then, InterCC leverages new insights on deﬁning dissimilarity between unassigned data points in S k and assigned data points in {C 1 , C 2 , · · · , C k − 1 } based on the learned optimaltransport plan Q ( k,T ) obtained in IntraCM in a coupling manner, which is able to select the dissimilardata points to form the next cluster C k and update existing clusters {C 1 , C 2 , · · · , C k − 1 } .
3.2.1 Intra-Cluster Mixing
We present intra-cluster mixing ( IntraCM ) from generative mixing view, a novel and fast frameworkfor embedding each cluster in a vector space and deﬁning the relations of each data point from thereference distribution, which give a clear evidence or constraint that the learned distributions coulddescribe the cluster’s summary statistics well, as shown in Figure 1(b).
Speciﬁcally, in the k -thcluster generation step, data points are partitioned into several subsets, i.e., D ( k ) has structures with 174 {C 1 , C 2 , · · · , C k − 1 , S k } .
We model each data point x i ∈ D ( k ) as a Bayesian mixture model with kcomponent parametrized by reference vectors ϕ ( k ) =
[ ϕ ( k ) 1 , ϕ ( k ) 2 , · · · , ϕ ( k ) k ] ∈ R k × d and non-ﬁxedmixing proportions over k − 1 clusters {C 1 , C 2 , · · · , C k − 1 } and unassigned data points S k , as shownin Figure 1(a).
A differentiable non-linear function f θ ( · ) (a neural network) is used to transformthese representations ϕ ( k ) j into parameters
f θ ( ϕ ( k ) j ) .
By introducing latent variable b ( k ) i,j indicatingthe data point x i comes from the j -th component, we can get the full mixing likelihood related tothe k -th cluster generation as follows:
[Equation]
) |
[Equation]
Y
D ( k ) X b ( [Equation] i p (cid:16) x i , b ( k ) i | f θ ( ϕ ( k ) ) (cid:17) =
Y
D ( k ) k
X j = p ( b ( k ) i,j ) | {z } π ( k ) j p (cid:16) x i |
f θ ( ϕ ( k ) j ) , b ( k ) i,j (cid:17) , (3) where b ( k ) i
=
[ b ( k ) i, 1 , b ( k ) i, 2 , · · · , b ( k ) i,k ] encodes the data point assignments.
π ( k ) =
[ π ( k ) 1 , π ( k ) 2 , · · · , π ( k ) k ]are the mixing coefﬁcients.
However, directly optimizing log Q D ( k ) p ( x i | f θ ( ϕ ( k ) )) with respect to ϕ ( k ) is difﬁcult due tomarginalization over b ( k ) i , while for many distributions optimizing log Q D ( k ) p ( x i , b ( k ) i | f θ ( ϕ ( k ) j )) ismuch easier.
In order to reﬂect the relationship of data in different sets, we introduce a variationaldistribution q ( k ) i,j = q ( b ( k ) i,j | x i ) indicating a variational probability that the data point x i belongs tothe j -th component.
In this case, we introduce a constrained variational lower bound to satisfy theoptimal transport measure among different components and focus on performing the expected loglikelihood by optimizing a lower bound:
[Equation]
Optimal Transport View of IntraCM
Here we give a detailed analysis to connect the proposedmixing strategy with optimal transport measure [31], which deﬁnes the transport plan between data 𝝓 !(#,%) ...
𝝓 #’!
(#,%) 𝝓 #(#,%) 𝐶 !
𝝓 !(#(!,%) ...
𝝓 #(#(!,%) 𝝓 #(!
(#(!,%) 𝝓
(#,𝑻) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 !:
,𝑆 # ) 𝝓
(#(!,𝑻) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 !:
# ,𝑆 #(! ) ...
𝐶 !"
# 𝝓 !(#’!
𝝓 #’+(#’!
,%) 𝝓 #’!
,%) 𝐶 !
$# 𝝓 (#’!
,𝑻) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 !:
#’+ ,𝑆 #’! )
𝑸 !
𝑸 #’+(#’!
,%) 𝑸 #’!
,%) 𝑸 !
𝑸 #’!
(#,%) 𝑸 #(#,%) 𝑸 !
𝑸 #(#(!,%) 𝑸 #(!
(#(!,%) 𝐶 #’!
← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (#’!,𝑻) ) 𝐶 # 𝐶 !
$% ...
𝑆 !
$# 𝐶 # 𝐶 !
$# ...
𝑆 !
𝐶 # 𝐶 !
𝑆 !"
𝐶 # ← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (#,𝑻) )
𝐶 #(!
← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (#(!,𝑻) )
𝑇 𝑇 𝑇 ...
... E:
M:
E:
M:
E:
M:
$ 𝐶 "&$ 𝑆 " 𝐶 " ...
𝝓 (",$) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 &:"(& ,𝑆 " ) 𝝓 (",’)
𝝓 (",$) 𝑸 (",() 𝝓 (",() Es t
e p M - s t e p ...
𝒎 𝟏(") 𝝓 $(",() 𝝓 "(",() 𝝓 "&$(",() ...
𝒎 𝑴 𝒌 (") ...
𝐶 " ← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (",$) )
M - s t e p (b Then 𝑆 !
𝐶 !"
# 𝝓 (!,&) 𝑸 !!,& ...
𝑸 #!,& 𝑸 !"
#!,& 𝝓 (",$) 𝝓 (",$) 𝐶 # 𝐶 !"
# 𝝓 (",$) 𝑆 !
𝑴 (!,#) 𝐶 # 𝑆 !
𝑆 !
𝑴 (!,!"
#) 𝐶 !"
# 𝐶 !
Intra-ClusterMixing
Inter-Cluster Coupling ...
𝐶 # (c) Optimal transport view of intra-cluster mixing and inter-cluster coupling, which focus on embedding each cluster in a vector space and deﬁning the rela- tions of each data points from the reference distribution.
points as weighted points clouds or discrete measures.
We aim to formulate an optimization problemfor q ( b ( k ) i,j | x i ) given in Eq.(4).
By replacing the variational distribution to a joint distribution, i.e.,q ( k ) i,j = q ( b ( k ) i,j | x i ) q
( x i ) , the optimization problem of maximizing the lower bound in Eq.(4) withrespect to the transport plan matrix Q ( k ) =
( q ( k ) i,j
) ∈ R N × k is equivalent to solving following entropicregularized Kantorovich relaxation of optimal transport [26]:
[Equation]
) , Q ( k ) ) =
min Q ( k )
X D ( k ) k X j = − log p (cid:16) x i , b ( k ) i,j | f θ ( ϕ ( k ) ) (cid:17) q ( k ) i,j − ϵH ( Q ( k ) ) , (5) where − log p ( x i , b ( k ) i,j | f θ ( ϕ ( k ) )) indicates the pairwise transport cost between data point and referencevector.
If Gaussian distribution is used as data likelihood, the transport cost − log p ( x i , b ( k ) i,j | f θ ( ϕ ( k ) ))is equivalent to a negative kernel −K
( x i , f θ ( ϕ ( k ) j )) .
The ϵ impacted entropic regularized term
H ( Q ( k ) )
= − P i,j q ( k ) i,j log q ( k ) i,j controls the sparsity of Q ( k ) .
In original entropic regularized Kantorovich relaxation of optimal transport [26], there are someconstraints on transport plan Q ( k ) , i.e., P N i =
1 q ( k ) i,j = 1 / k and P k j =1 q ( k ) i,j = 1 / N .
If we set the factorizedvariational distribution to q ( x i )
= 1 / N , the constraint P j q ( k ) i,j = 1 / N is automatically met.
Optimaltransport view induces various advantages of our proposed mixing strategy.
First, it is is computationally efﬁcient, scalable to vary large dataset.
Unlike the quadratic cost of traditional similarityweight computation (e.g., the attention score in self-attention), our optimal transport mixing uses aﬁxed number ( k ) of references serving as queries and the number of references is set depends ondata without manual setting.
Secondly, it allows unsupervised learning where the model parameters{ ϕ ( k ) j } kj =1 can be learned without target (class) labels.
Third, it is able to sufﬁciently exploit pairwiseinteractions between data points in intra-cluster level and beneﬁt for the next inter-cluster coupling,which is useful to sufﬁciently mine the hidden structure among data.
Expectation Maximization for IntraCM
Considering the latent variable b ( k ) is introduced, naturally the expectation maximization (EM) method can be utilized.
Iterative optimization of this boundalternates between two steps:
• E-step : The E-step focuses on maximizing above bound with respect to the variational probability q ( b ( k,t ) i,j | x i ) .
We compute a new estimate of the posterior probability distribution over the latent
variables from the previous iteration, which yields a soft-assignment of the data points to the com-
ponents q ( b ( k,t ) i,j | x i )
=
π ( k,t − 1) j p
x i | b ( k,t − 1) i,j ,f θ ( ϕ ( k,t − 1) ) )
∑ kj =1 π ( k,t − 1) j p ( x i | b ( k,t − 1) i,j ,f θ ( ϕ ( k,t − 1) ) ) , where π ( k,t − 1) j and ϕ ( k,t − 1) indicatesmodel parameters from previous iteration.
This soft-assignment is a regular E-step without thebalanced assignment constraints.
Furthermore, the variational lower bound can be formulated asthe optimal transport setting (as shown in Eq.(5)), yielding an Sinkhorn-based optimal transportsolution [28] as follows: [Equation] p [Equation]
i |
[Equation]
j
[Equation]
f θ
[Equation]
( k
[Equation]
t
[Equation]
[Equation] (cid:17) v [Equation] (6) where u i =
[Equation]
ϵ ) v j and v j = 1 k
∑ [Equation] [Equation] Note that above solution usually converges quickly after a few iterations.
Hence, the E-step coincides with the optimal transport based similarity weight computation, which leads to the variousadvantages of our proposed mixing strategy and forms the transport plan Q ( k,t ) for all data points.
• M-step : Once the E-step is done (i
e., Q ( k,t ) is found), we perform the M-step, that is, maximizing the lower bound in Eq.
) with respect to ϕ ( k,t ) when Q ( k,t ) is ﬁxed.
Due to the non-linearityof f θ ( · ) there exists no analytical solution to arg max ϕ ( k ) J ( ϕ ( k ) , Q ( k,t ) ) .
With the beneﬁt of differentiable f θ ( · ) , we can solve it with the aid of gradient-based strategy.
Assume that likelihoodp ( x i | f θ ( ϕ ( k ) j ) , b ( k ) i,j ) follows a Gaussian distribution with ﬁxed covariance, we can obtain followingupdate rule: [Equation] t
[Equation]
For simplicity, if we assume the parameters ϕ ( k ) in likelihood with a linear transformationf θ ( ϕ ( k ) )
=
W ϕ ( k ) , it can yield an optimal transport kernel embedding solution [24] when weset it as Gaussian distribution with ﬁxed variance, e.g., π ( k,t )
j
=
∑ i q ( b ( k,t
[Equation]
1) i,j | x i ) N and ϕ ( k,t )
[Equation]
=
[Equation]
Wx i , which is equivalent to induced self-attention introduced in Leeet al.
[18] providing an efﬁcient solution for pairwise weights measuring.
Usually, we can take multiple EM steps for further optimization (e.g., ϕ ( k, 0) → ϕ ( k, 1) → · · · →ϕ ( k,T ) ).
Here T is the number of EM steps.
IntraCM belongs to the class of generalized EMalgorithms and is guaranteed to converge to a local optimum of the data log likelihood [33].
3.2.2 Inter-Cluster Coupling
After T EM steps, the variational probability q ( k,T ) i,j = q ( b ( k,T ) i,j | x i ) can be regarded as the optimaltransport plan, which carries the information on how to distribute the mass of data points x i toreference vectors ϕ
( k,T ) j .
By regarding ϕ ( k,T ) j as couplings between unassigned set and assignedset, we propose inter-cluster coupling ( InterCC ) to generate the next cluster, as shown in Figure 1.
Considering the current data points have structures with k different sets, C 1 , C 2 , · · · , C k − 1 , S k , we canget k − 1 transport plan matrices { U ( k ) 1 , U ( k ) 2 , · · · , U ( k ) k − 1 } related to k − 1 existing clusters and V ( k )related to the unassigned set S k ,
[Equation]
We can measure the relations between unsigned data points in S k and existing assigned data pointsin k − 1 existing clusters C 1 , C 2 , · · · , C k − 1 with the aid of couplings ϕ
( k ) as follows
[Equation]
where ⊕ is a column-wise concatenation operation and v ( k ) i is the i -th column of V ( k ) .
It is reasonable to select dissimilar data point in unassigned data points set S k to form a new cluster C k , i.e.,
[Equation]
Here Multinomial distribution is used to model the generation of C k .
I ∈ R N − M k is a vector of whoseelements are all 1s.
m ( k ) j is the i -th column of M ( k ) .
Thus, the data points related to dissimilar indices in each matrix are selected to generate the new cluster C k .
Furthermore, previous existingclusters C 1 , C 2 , · · · , C k − 1 can be updated based on dissimilarity matrix.
For a given cluster C l , it canbe updated by C l := C l ∪C new l , where C new l := M ULT ( S OFTMAX [1 − I ( l ) ⊤ M ( k,l ) ]) .
I ( l ) ∈ R
|C l | is a vectorof all 1s and M ( k,l )
=
U ( k ) l ϕ ( k,T ) V ( k ) ⊤ .
3.2.3 Iterative Unrolling Inference
IntraCM and InterCC are both differentiable procedures for optimal transport measuring and nextcluster constructing, whose outcome relies on the current existing clusters and the statistical modelparameters ϕ
( k ) .
As shown in Eq.(2), the whole cluster-wise clustering procedure can be regardedas a recurrent IntraCM with InterCC until there are no available data point left.
In this case, theinformation about statistical regularities in our model that is required for generating the sequentialclusters is encoded in a series of parameters { ϕ (1 ,T ) , ϕ (2 ,T ) , · · · , ϕ ( K,T ) } .
By unrolling the iterations of the presented mixing process, we obtain an end-to-end differentiable clustering procedurebased on the statistical model parameterized by { ϕ (1 ,T ) , ϕ (2 ,T ) , · · · , ϕ ( K,T ) } .
We can therefore use(stochastic) gradient descent and ﬁt the statistical model to capture the regularities corresponding toclusters for a given dataset.
[Equation]
The likelihood is derived from approximating the p θ ( C 1: K |D ) to true posterior via KL divergence.
Note that samples from p ( C 1: K , D ) are obtained from the explicit generative model, such as Gaussianmixture model, Dirichlet process mixture model.
With inﬁnite generated samples, we can train amodel to approximate p θ ( C 1: K |D ) accurately.
The whole inference procedure is given in Algorithm 1.
By repeating the cluster generative process until there are no data points left, IntraCM and InterCCare processed alternatively, and the clusters are iteratively reﬁned by deriving the explicit supervisorysignal from the already formed clusters.
Note that the complexity of traditional methods measuringweights between data points acquires O ( KN 2 ) (e.g., self-attention) which may be too expensive forlarge sets, and our AMCP only acquires O ( KNk ) in the EM-step for the k -th iteration, which ismore efﬁcient. Algorithm 1 Iterative Unrolling Inference for for AMCP EM steps end for 8: //
Inter-Cluster Coupling 9:
Establish k transport plan matrices { U ( k ) l } k − 1 l =1 and V ( k ) , and similarity matrix M ( k ) 10:
Generate C k 11: Update existing clusters C 1 , C 2 , · · · , C k −▷ Optional 12: if S k ̸ = ∅ then 13: Establish unassigned set S k +1 14: end if 15: end while Output: updated parameters ϕ ( K,T )
, generated clusters C 1 , C 2 , · · · , C K
In this section, we evaluate the proposed AMCP in terms of synthetic and real-world datasets bycomparing with the state-of-the-art methods.
4.1 Experiments on Synthetic Datasets
The ﬁrst experiment was conducted on synthetic data with arbitrary number of clusters.
Thus, ﬁveexisting methods handling variable number of clusters, MCMC[1], VI[2], DAC[19], NCP[25] and [Table]
ST-ACT[18] are used as baselines.
Among them, MCMC and VI are traditional approximationmethods, and DAC, NCP and ST-ACT are deep amortized clustering methods.
The synthetic datais generated via a 2D Gaussian mixture model (GMM) which is deﬁned by the following processα ∼ E XP (1) , c 1: N ∼ CRP ( α ) , µ k ∼ N (0 , σ 2 µ 1 ) , x i ∼ N ( µ c i , σ 2 I ) .
We set σ µ = σ = 10 .
At each training step, we generate 10 random datasets according to the above generative process. Each dataset contains 200 points on a 2D plane, and each sampled from one of 4 Gaussians.
Twotesting scenarios ( Scenario 1 and Scenario 2 ) are constructed to evaluate the effectiveness of theproposed method.
The test set in Scenario 1 has the same conﬁguration (200 samples and 4 clusters)as training set, while Scenario 2 contains different numbers of samples and clusters (400 samplesand 6 clusters) in order to verify whether the clustering method can generalize to the unseen clusters.
Table 1 summarizes the results in terms of two testing scenarios.
Here we report log-likelihood ( LL ),clustering accuracy ( ACC ), Normalized Mutual Information ( NMI ) and running time (seconds).
Thebest and second results are marked in bold and underlined.
As expected, the proposed AMCP consistently outperforms baselines on both scenarios.
Although Scenario 2 is more challenging, amortized clustering methods (DAC, NCP, ST-ACT and our AMCP) can capture cluster uncertainty andobtain competitive results.
For testing time, amortized methods are obviously faster than traditionalsampling based method (MCMC).
Although VI is much fast than MCMC, it is still lower than amortized methods.
In cluster-wise amortized clustering methods, DAC, ST-ACT and our ACMP achievecompetitive running time since cluster-wise sequential modeling is more efﬁcient than point-wisesequential modeling.
4.2 Experiments on Real-world Datasets
Real-world datasets, MNIST [17], Tiny-ImageNet [16] and CIFAR-10 [32], are used to validate theperformance.
To formulate the amortized mechanism, we generate training and test data via following Dirichlet process mixture model: α ∼ E XP (1) , c 1: N ∼ CRP ( α ) , K − 1 ∼ B INOMIAL ( K g −1 , 0 . 5) , l k ∼ U (0 , K − 1) , x i ∼ U
[ D , l c
i ] .
Here x i ∼ U
[ D , l c i ] indicates sampling x i with label l c i uniformly from dataset D , which can be training set or test set.
K g indicates the number of clusters wecan sample. For all original dataset, data containing half of the classes is used to generate the trainingsets, and the remaining half is used to generate the test sets, with no overlap between training classesand test classes.
We sample multiple clustered datasets to form training sets with N ∈ { 1000 , 10000 } . For test set, we generate 1000 randomly clustered datasets with N ∈ { 1000 , 5000 } .
More informationis given in Appendix.
In Table 2, we depict the quantitative clustering results in real-world datasets.
The best and second results are marked in bold and underlined.
The average clustering results are recorded over5 runs with different random parameter initialization.
It can be seen that our AMCP outperformsall baselines.
The main reason, we believe, is that AMCP not only exploits amortized propertiesbetween data points in both inter- and intra-clusters, but also has ﬂexible clustering process, whichsimultaneously generate the new cluster and update the summary statistics for previous clusters.
Inmodel efﬁciency evaluation, the DAC, ST-ACT and AMCP consume a more competitive runningtime, which outperform other methods by a large margin.
The reason is that DAC and ST-ACT identify each cluster after one forward pass, and AMCP generates clusters in cluster-wise manner andexploits data relations via the efﬁcient optimal transport.
However, DAC, ST-ACT are pretty worsethan AMCP on clustering performance.
Among amortized clustering methods, DAC, NCP, ST-ACT and AMCP are able to process arbitrarynumber of clusters.
For better visualization of cluster generative process, a digit subset from MNISTtest set is randomly sampled.
To illustrate how the clustering models capture the shape ambiguity [Table] [Table] of some of the digits, we plot the ground-truth digit clusters and clustering results in Table 3.
Comparing with DAC, NCP and ST-ACT, the proposed AMCP can get the most accurate results.
Forexample, DAC assigns the digit 7 (with similar appearance to 9) to cluster 9, and NCP generates anew cluster for it.
Fortunately, AMCP correctly assigns it to cluster 7.
All baselines are unable toassign the digit 4 written in a strange way to the right cluster, and our AMCP correctly assigns it tocluster 4.
Results on Tiny-ImageNet and CIFAR-10 can be obtained in Appendix.
4.3 Model Analysis
We empirically analyze the proposed model from the impact of various hyper-parameters.
They aresummarized as follows: (1) The Number of EM Steps T : AMCP can apply multiple EM stepsto achieve more accurate ϕ ( K,T ) .
Here we conduct experiments on varying T in { 1 , 2 , 3 , 4 , 5 } toinvestigate the effect of T .
Figure 2(a) and (b) show the effect of T on synthetic data ( Scenario 1 )and real-world data ( CIFAR-10 ).
We can see that, the performance ( LL and ACC ) become better andbetter with the increasing of T , while the running time is linearly scalable to the number of steps.(2) OT E-step vs. Regular E-step We introduce two different ways to conduct E-step in IntraCM:regular soft-assignment E-step and Sinkhorn-based optimal transport E-step.
Figure 2(c) and (d)show the comparison between regular E-step and OT E-step on synthetic data ( Scenario 1 ) and realworld data ( CIFAR-10 ).
The maximum number of Sinkhorn-Knopp iterations for solving the OTproblem is set to 10 .
While the regular E-step is consistently faster than the OT E-step since it’sclosed-form solution, OT E-step can achieve more accurate likelihood and accuracy.
Furthermore,we compare the difference between the strategy of updating the existing clusters and the strategy ofnot updating.
More empirical results are given in Appendix.
[Figure]
[Figure]
Conclusion
In this paper, we studied the problem of amortized clustering, and presented our approach thatperform cluster-wise amortized mixing coupling processes via intra-cluster mixing and inter-clustercoupling.
An interesting direction for future research is to explore more complicated and realistichidden structure (e.g., hierarchical clusters) that is not possible with our approach.
Furthermore, it isinterested to explore novel applications that can be enabled by the interpretability and controllabilitybrought by the amortized clustering.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] See Abstract and Introduction.
(b) Did you describe the limitations of your work?
[Yes] See Conclusion
(c) Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[No] All assets used in this paper are open source
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ- able information or offensive content?
[No] Does not contain personally identiﬁable
information or offensive content
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]