Amortized Mixing Coupling Processes for Clustering
Abstract
Considering the ever-increasing scale of data, which may contain tens of thousands of data points or complicated latent structures, the issue of scalability and algorithmic efﬁciency becomes of vital importance for clustering.
In this paper, we propose cluster-wise amortized mixing coupling processes (AMCP), which is able to achieve efﬁcient amortized clustering in a well-deﬁned non-parametric
Bayesian posterior.
Speciﬁcally, AMCP learns clusters sequentially with the aid of the proposed intra-cluster mixing (IntraCM) and inter-cluster coupling (InterCC) strategies, which investigate the relationship between data points and reference distribution in a linear optimal transport mixing view, and coupling the unassigned set and assigned set to generate new cluster.
IntraCM and InterCC avoid pairwise calculation of distances between clusters and reduce the computational complexity from quadratic to linear in the current number of clusters.
Furthermore, clusterwise sequential process is able to improve the quick adaptation ability for the next cluster generation.
In this case, AMCP simultaneously learns what makes a cluster, how to group data points into clusters, and how to adaptively control the number of clusters.
To illustrate the superiority of the proposed method, we perform experiments on both synthetic data and real-world data in terms of clustering performance and computational efﬁciency.
Introduction
Clustering aims to group samples into a certain number of clusters, such that similar samples belong to the same cluster, while dissimilar ones to different clusters.
Aside from its usefulness in many downstream tasks, clustering is an important but challenging topic which has seen applications in various settings including social-media recommendation [32], customer partitioning [5], discovering social networks [7] and partitioning protein-protein interaction networks [3].
Among various clustering algorithms [37], probabilistic generative clustering models [39] have been widely concerned because of its ﬂexibility and interpretability.
Probabilistic generative clustering models (as known as, mixture models) are a staple of statistical modeling in which a discrete latent variable is introduced for each observation, indicating its mixture component identity.
These generative clustering models can be roughly divided into two categories: ﬁnite mixture model [6] and inﬁnite mixture model [31].
In recent years, ﬁnite mixture models have been increasingly applied in unsupervised learning problems with the aid of deep neural networks.
Instead of using an arbitrary prior for the latent variable, these methods adopted ﬁnite mixture prior, such as Gaussian mixture model (GMM) [14, 38].
A ﬁnite mixture model with a ﬁxed number of clusters may ﬁt the given dataset well, while it may be sub-optimal to use the same number of clusters if more data comes under a slightly changed distribution [31].
It would be ideal if the clustering models can ﬁgure out the unknown number of clusters automatically.
Alternatively, inﬁnite mixture model is the application of nonparametric Bayesian techniques to mixture modeling, which allows for the automatic determination of an appropriate number of mixture components.
The prior dis36th on Systems ( tribution can be speciﬁed in terms of a data point sequential process called Dirichlet process (e.g., Chinese restaurant process (CRP)[29]), where the number of clusters can arbitrarily grow to better accommodate data as needed [12].
However, the inference of nonparametric posterior largely rely on the sequential data point modeling (e.g., Markov Chain Monte Carlo (MCMC)), which is time-consuming, with convergence that is difﬁcult to assess.
While above probabilistic clustering is generally approached as an unsupervised learning problem, the amortization power of non-linear transformation (e.g., neural networks) has recently spurred progress on supervised formulations, based on novel tractable objective functions for generic data and an arbitrary number of clusters [20, 21, 27].
In exchange for the additional cost of procuring labeled training data and training the model, amortized clustering offers several beneﬁts on top of time efﬁciency at test time.
Traditional unsupervised clustering, especially probabilistic clustering, deﬁnes speciﬁc likelihood and prior on the data geometry, even without any prior knowledge of the data, which are not needed with amortized clustering, since the model learns based on the statistics of the labeled training data [21].
Moreover, to achieve the best result (e.g., agreement with human labeling or with a well-deﬁned posterior), unsupervised methods usually need parameter tuning and post-processing (e.g., cluster merging)
[15], which are obviated with amortized inference by implicitly learning the deﬁnition of clusters underlying the training datasets.
In a sense this shares a similar philosophy as Neural Processes [10, 9, 16], which learns from multiple datasets to learn a prior over functions in a meta learning manner.
In this paper, motivated by these prior work, we propose cluster-wise Amortized Mixing Coupling
Processes (AMCP), which is able to provides an efﬁcient clustering architecture by grouping data points in an optimal transport cluster-wise view.
In each cluster-wise step, AMCP generates cluster sequentially and ﬁnds one cluster a time.
In this case, two key strategies, Intra-Cluster Mixing ( IntraCM ) and Inter-Cluster Coupling ( InterCC ), are proposed to measure the relations between different data points and generate cluster.
Speciﬁcally, IntraCM focus on investigating the relationship between data points and reference distribution in a linear optimal transport view [33], which avoid pairwise calculation of distances between clusters and reduce the computational complexity from quadratic to linear in the current number of clusters.
Then, InterCC leverages new insights on deﬁning dissimilarity between assigned data points and the unassigned data points based on the couplings obtained in IntraCM , which is able to select the dissimilar data points to form the next cluster and update existing clusters.
The key contributions of our work are as follows: (1) Efﬁciency .
The proposed AMCP not only inherent the efﬁcient cluster-wise learning manner, but also reduce the quadratic cost of traditional similarity weight computation with the aid of optimal transport mixing.
(2) Novelty .
To the best of our knowledge, AMCP is the ﬁrst to introduce optimal transport for amortized clustering and effectively correlate the relationship between optimal transport and mixture models.
(3) Adaptivity .
Non-parametric learning manners exist not only in model parameter learning but also in cluster generation, which allows unsupervised parameter learning and non-handcrafted intervention.
(4) Flexibility .
Different from existing amortized clustering methods without changing the already-generated clusters, AMCP is able to dynamically adjust the previous generated clusters to avoid the accumulated errors with the aid of mixing and coupling manners.
(5) Effectiveness .
Extensive experiments conducted on both synthetic and real-world datasets demonstrate that AMCP can cluster data effectively and efﬁciently.
Related Work
In this section, we brieﬂy introduce some recent developments in several related topics, namely probabilistic clustering and amortized clustering.
Probabilistic Clustering Probabilistic clustering models (or equivalently, mixture models) are a staple of statistical modeling in which a discrete latent variable is introduced for each observation, indicating its mixture component identity.
There is a growing interest in developing probabilistic clustering methods using parametric or nonparametric prior for complex data modeling [6, 31, 14, 38], and ﬁnite and inﬁnite mixture models were developed.
The main focus of ﬁnite model introduces ﬁnite mixture prior with a ﬁxed number of cluster to clustering.
In this case, Gaussian distribution or non Gaussian distributions [24, 23] are widely used to model data distribution [6].
Alternately, inﬁnite mixture models [2] adopt non-parametric prior to control the number of clusters dynamically, such as Dirichlet process and its variants [29].
Recently, researchers focus on deep generative clustering by combining probabilistic clustering modeling and neural networks [14, 22, 27, 38, 21, 36, 4].
The main focus of these methods is to learn a representation of input data amenable to clustering via deep neural networks.
Amortized Clustering Clustering is generally formulated as an unsupervised learning manner and while amortized clustering [27, 20, 21] investigates the amortized power steam from a predeﬁned supervised formulation to achieve an arbitrary number of clusters in new task.
Lee et al.
[20, 21] adopt attention networks to output instead the cluster labels of each data point, for general mixture model, with an arbitrary number K of clusters and a cost of O ( K ) forward passes.
Later, Pakman et al.
[27] presented Neural Clustering Process (NCP), which sequentially computes the conditional probability of assigning the current data point to one of already constructed clusters or a new one.
The most similar work to our methods is NCP [27] and CHiGac [25].
Both of them explain clustering from the perspective of generative models, which similar in spirit to the popular Gibbs sampling algorithm for Dirichlet process mixture models, but without positing particular priors on partitions.
Besides, NCP and CHiGac rely heavily on the selecting of anchor points and the processing ordering and often exhibits unstable properties.
Different from previous work, AMCP connects the relationship of mixture model and optimal transport to describe the cluster’s summary statistics well, and seamlessly combines mixing and coupling to achieve efﬁcient amortized clustering.
Method 3.1 Notations and Problem Formulation
Given N data points D = { x 1 , x 2 , · · · , x N } , where x i ∈ R
d indicates feature representation for the i -th data point represented in d -dim latent space.
Probabilistic models for clustering is usually presented by sequential sampling procedure to generate clusters.
Usually, the generative process is c i ∼ p
( c i | α 1 ) , µ k ∼ p ( µ k | α 2 ) , x i ∼ p ( x i | µ c i ) , here c i ∈ { 1 , 2 , · · · , K } encodes the cluster assignment of the data point x i .
α 1 and α 2 are hyperparameters.
The integer random variable K indicates the number of distinct values among the sampled c i , and µ k denotes a parameter controlling the distribution of the k -th cluster.
Speciﬁcally, these models include Gaussian mixture model [14, 38],
Dirichlet process mixture model [8] and etc.
Point-wise Clustering Posterior Probabilistic model for clustering is usually presented by sequential sampling procedure to generate clusters.
One of the prototypical tools for nonparametric clustering modeling is the Dirichlet Process.
It allows for a discrete distribution of observations drawn from an arbitrary base measure over the domain in such a way as that the marginals match draws from, while simultaneously obtaining a countable set of distinct data points.
A useful view of the Dirichlet process mixture model is the Chinese restaurant metaphor [12], which sequentially computes the conditional probability of assigning the current data point to one of already constructed clusters or a new one.
The cluster assignment for each data point can be sampled by c i ∼ p
( c i | c 1: i − 1 , D ) .
Here c 1: i − 1 = { c 1 , c 2 , · · · , c i − 1 } indicates the existing clusters formed by already assigned data points { x 1 , x 2 , · · · , x i − 1 } .
Based on existing clusters c 1: i − 1 and observations D , the cluster assignment for the i -th data point is sampled.
Given N data points X
= { x i } Ni =1 , it is natural to draw independent samples to form clusters using decomposition
[Equation]
After all data points are assigned, the ﬁnal clusters are formed.
Inference proceeds by traversing data points and re-sampling their cluster assignments in sequence.
Challenges The above process sequentially computes the conditional probability of assigning the current data point to one of already constructed clusters or a new one, and does not posit any particular prior on partitions.
The sequential sampling procedure makes it process data points one by one, which limits its scalability on large datasets (the computational cost for full i.i.d.
sampling { c 1 , c 2 , · · · , c N } is high)
Furthermore, the clustering result is sensitive to the sequential processing order.
It needs a sufﬁcient number of random samples to obtain stable result, which is time-consuming.
3.2 Amortized Mixing Coupling Processes
For the seek of effective cluster generative process and preventing the effect of sequential processing order, we focus on an alternative of clustering generative process from the view of cluster rather than the view of data point [27, 25].
Let C
=
{C 1 , C 2 , · · · , C K } indicate K clusters, we deﬁne C k as C k =
( C (1) k , · · · , C ( i ) k , · · · , C ( N k ) k ) , where C ( i ) k is the i -th data point of the k -th cluster.
N k is the number of data points belonging to the k -th cluster.
Based on the above deﬁnition, we are interested in sampling each cluster assignment by C k ∼ p ( C k |C 1: k − 1 , S k ) , k
= 1 , · · · , K , where S k indicates the conditional information containing M k available unassigned data points for constructing the k -th cluster.
M k is the number of left data points after generating the previous k − 1 clusters, deﬁned by
M k = N − P k − 1 j =1 N j ( N j is the number of data points belonging to the j -th cluster).
We formulate cluster assignment process in the form of cluster-wise ( C 1 → C 2 → · · · → C K ) rather than point-wise ( c 1 → c 2 → · · · → c N ), which enables an efﬁcient clustering process.
We are interested in sampling {C 1 , C 2 , · · · , C K } via a decomposition as follows:
[Equation]
which is easy to model cluster sequentially, and the arbitrary number of clusters can be adaptively determined until there is no remaining point in S k .
The sampling process deﬁned in Eq.(2) indicates a clustering posterior using K sets of indices.
To model the above cluster-wise sequential posterior, we propose a cluster-wise amortized mixing coupling processes (AMCP), which is able to provides an efﬁcient amortized clustering architecture by grouping data points in a cluster-wise view rather than point-wise view.
As shown in Figure 1(a), AMCP generates clusters sequentially and two key strategies, Intra-Cluster Mixing ( IntraCM ) and Inter-Cluster Coupling ( InterCC ), are proposed.
Speciﬁcally, as shown in Figure 1, IntraCM focus on investigating the relationship between data points and reference distribution ϕ
( k,T ) = [ ϕ ( k,T ), ϕ ( k,T ), · · · , ϕ ( k,T ) k ] with ϕ ( k,T ) j ∈ R d in a linear optimal transport view [33] with T inner update steps.
Then, InterCC leverages new insights on deﬁning dissimilarity between unassigned data points in S k and assigned data points in {C 1 , C 2 , · · · , C k − 1 } based on the learned optimal transport plan Q ( k,T ) obtained in IntraCM in a coupling manner, which is able to select the dissimilar data points to form the next cluster C k and update existing clusters {C 1 , C 2 , · · · , C k − 1 } .
3.2.1 Intra-Cluster Mixing
We present intra-cluster mixing ( IntraCM ) from generative mixing view, a novel and fast framework for embedding each cluster in a vector space and deﬁning the relations of each data point from the reference distribution, which give a clear evidence or constraint that the learned distributions could describe the cluster’s summary statistics well, as shown in Figure 1(b).
Speciﬁcally, in the k -th cluster generation step, data points are partitioned into several subsets, i.e., D ( k ) has structures with {C 1 , C 2 , · · · , C k − 1 , S k } .
We model each data point x i ∈ D ( k ) as a Bayesian mixture model with k component parametrized by reference vectors ϕ ( k ) =
[ ϕ ( k ) 1 , ϕ ( k ) 2 , · · · , ϕ ( k ) k ] ∈ R k × d and non-ﬁxed mixing proportions over k − 1 clusters {C 1 , C 2 , · · · , C k − 1 } and unassigned data points S k , as shown in Figure 1(a).
A differentiable non-linear function f θ ( · ) (a neural network) is used to transform these representations ϕ ( k ) j into parameters f θ ( ϕ ( k ) j ) .
By introducing latent variable b ( k ) i,j indicating the data point x i comes from the j -th component, we can get the full mixing likelihood related to the k -th cluster generation as follows:
[Equation]
where b ( k ) i =
[ b ( k ) i, 1 , b ( k ) i, 2 , · · · , b ( k ) i,k ] encodes the data point assignments.
π ( k ) =
[ π ( k ) 1 , π ( k ) 2 , · · · , π ( k ) k ] are the mixing coefﬁcients.
However, directly optimizing log Q D ( k ) p ( x i | f θ ( ϕ ( k ) )) with respect to ϕ ( k ) is difﬁcult due to marginalization over b ( k ) i , while for many distributions optimizing log Q D ( k ) p ( x i , b ( k ) i | f θ ( ϕ ( k ) j )) is much easier.
In order to reﬂect the relationship of data in different sets, we introduce a variational distribution q ( k ) i,j = q ( b ( k ) i,j | x i ) indicating a variational probability that the data point x i belongs to the j -th component.
In this case, we introduce a constrained variational lower bound to satisfy the optimal transport measure among different components and focus on performing the expected log likelihood by optimizing a lower bound:
[Equation]
𝝓 !(#,%) ...
𝝓 #’!
(#,%) 𝝓 #(#,%) 𝐶 !
𝝓 !(#(!,%) ...
𝝓 #(#(!,%) 𝝓 #(!
(#(!,%) 𝝓
(#,𝑻) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 !:
,𝑆 # ) 𝝓
(#(!,𝑻) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 !:
# ,𝑆 #(! ) ...
𝐶 !"
# 𝝓 !(#’!
𝝓 #’+(#’!
,%) 𝝓 #’!
,%) 𝐶 !
$# 𝝓 (#’!
,𝑻) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 !:
#’+ ,𝑆 #’! )
𝑸 !
𝑸 #’+(#’!
,%) 𝑸 #’!
,%) 𝑸 !
𝑸 #’!
(#,%) 𝑸 #(#,%) 𝑸 !
𝑸 #(#(!,%) 𝑸 #(!
(#(!,%) 𝐶 #’!
← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (#’!,𝑻) ) 𝐶 # 𝐶 !
$% ...
𝑆 !
$# 𝐶 # 𝐶 !
$# ...
𝑆 !
𝐶 # 𝐶 !
𝑆 !"
𝐶 # ← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (#,𝑻) )
𝐶 #(!
← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (#(!,𝑻) )
𝑇 𝑇 𝑇 ...
... E:
M:
E:
M:
E:
M:
$ 𝐶 "&$ 𝑆 " 𝐶 " ...
𝝓 (",$) ← 𝑰𝒏𝒕𝒓𝒂𝑪𝑴(𝐶 &:"(& ,𝑆 " ) 𝝓 (",’)
𝝓 (",$) 𝑸 (",() 𝝓 (",() Es t
e p M - s t e p ...
𝒎 𝟏(") 𝝓 $(",() 𝝓 "(",() 𝝓 "&$(",() ...
𝒎 𝑴 𝒌 (") ...
𝐶 " ← 𝑰𝒏𝒕𝒆𝒓𝑪𝑪(𝝓 (",$) )
M - s t e p (b
𝑆 !
𝐶 !"
# 𝝓 (!,&) 𝑸 !!,& ...
𝑸 #!,& 𝑸 !"
#!,& 𝝓 (",$) 𝝓 (",$) 𝐶 # 𝐶 !"
# 𝝓 (",$) 𝑆 !
𝑴 (!,#) 𝐶 # 𝑆 !
𝑆 !
𝑴 (!,!"
#) 𝐶 !"
# 𝐶 !
Intra-ClusterMixing
Inter-Cluster Coupling ...
𝐶 # (c) Optimal transport view of intra-cluster mixing and inter-cluster coupling, which focus on embedding each cluster in a vector space and deﬁning the rela- tions of each data points from the reference distribution.
Optimal Transport View of IntraCM
Here we give a detailed analysis to connect the proposed mixing strategy with optimal transport measure [33, 11, 17], which deﬁnes the transport plan between data points as weighted points clouds or discrete measures.
We aim to formulate an optimization problem for q ( b ( k ) i,j | x i ) given in Eq.(4).
By replacing the variational distribution to a joint distribution, i.e., q ( k ) i,j = q ( b ( k ) i,j | x i ) q
( x i ) , the optimization problem of maximizing the lower bound in Eq.(4) with respect to the transport plan matrix Q ( k ) =
( q ( k ) i,j
) ∈ R N × k is equivalent to solving following entropic regularized Kantorovich relaxation of optimal transport [28]:
[Equation]
where − log p ( x i , b ( k ) i,j | f θ ( ϕ ( k ) )) indicates the pairwise transport cost between data point and reference vector.
If Gaussian distribution is used as data likelihood, the transport cost − log p ( x i , b ( k ) i,j | f θ ( ϕ ( k ) )) is equivalent to a negative kernel −K
( x i , f θ ( ϕ ( k ) j )) .
The ϵ impacted entropic regularized term H ( Q ( k ) )
= − P i,j q ( k ) i,j log q ( k ) i,j controls the sparsity of Q ( k ) .
In original entropic regularized Kantorovich relaxation of optimal transport [28], there are some constraints on transport plan Q ( k ) , i.e., P Ni =
1 q ( k ) i,j = 1 / k and P kj =1 q ( k ) i,j = 1 / N .
If we set the factorized variational distribution to q ( x i )
= 1 / N , the constraint P j q ( k ) i,j = 1 / N is automatically met.
Optimal transport view induces various advantages of our proposed mixing strategy.
First, it is is computationally efﬁcient, scalable to vary large dataset.
Unlike the quadratic cost of traditional similarity weight computation (e.g., the attention score in self-attention), our optimal transport mixing uses a ﬁxed number ( k ) of references serving as queries and the number of references is set depends on data without manual setting.
Secondly, it allows unsupervised learning where the model parameters { ϕ ( k ) j } kj =1 can be learned without target (class) labels.
Third, it is able to sufﬁciently exploit pairwise interactions between data points in intra-cluster level and beneﬁt for the next inter-cluster coupling, which is useful to sufﬁciently mine the hidden structure among data.
Expectation Maximization for IntraCM
Considering the latent variable b ( k ) is introduced, naturally the expectation maximization (EM) method can be utilized [11].
Iterative optimization of this bound alternates between two steps:
• E-step : The E-step focuses on maximizing the above bound with respect to the variational probability q ( b ( k,t ) i,j | x i ) .
We compute a new estimate of the posterior probability distribution over the
latent variables from the previous iteration, which yields a soft-assignment of the data points to the components q ( b ( k,t ) i,j | x i ) = [Equation] k [Equation]
[Equation]
i
[Equation]
b ( k [Equation] t [Equation]
[Equation]
i
[Equation]
j
[Equation]
f
[Equation]
( k
[Equation]
t
[Equation]
[Equation]
kj
[Equation]
( k
[Equation]
t
[Equation]
1) j p ( x i | b ( k
[Equation]
t
[Equation]
i
[Equation]
j ,f
[Equation]
( ϕ ( k
[Equation] t [Equation] 1) ) ) , where π ( k,t − 1) j and ϕ ( k,t − 1) indicates model parameters from previous iteration.
This soft-assignment is a regular E-step without the balanced assignment constraints.
Furthermore, the variational lower bound can be formulated as the optimal transport setting (as shown in Eq.(5)), yielding an Sinkhorn-based optimal transport solution [30] as follows:
[Equation]
6) where u i =
[Equation]
Note that above solution usually converges quickly after a few iterations.
Hence, the E-step coincides with the optimal transport based similarity weight computation, which leads to the various advantages of our proposed mixing strategy and forms the transport plan Q ( k,t ) for all data points.
• M-step : Once the E-step is done (i.e., Q ( k,t ) is found), we perform the M-step, that is, maximiz - ing the lower bound in Eq.(4) with respect to ϕ ( k,t ) when Q ( k,t ) is ﬁxed.
Due to the non-linearity of f θ ( · ) there exists no analytical solution to arg max ϕ ( k ) J ( ϕ ( k ) , Q ( k,t ) ) .
With the beneﬁt of differentiable f θ ( · ) , we can solve it with the aid of gradient-based strategy.
Assume that likelihood p ( x i | f θ ( ϕ ( k ) j ) , b ( k ) i,j ) follows a Gaussian distribution with ﬁxed covariance, we can obtain following update rule:
[Equation]
For simplicity, if we assume the parameters ϕ ( k ) in likelihood with a linear transformation f θ ( ϕ ( k ) )
=
W ϕ ( k ) , it can yield an optimal transport kernel embedding solution [26] when we set it as Gaussian distribution with ﬁxed variance, e.g., π ( k,t )
j
[Equation]
and ϕ [Equation] k
[Equation]
Wx i , which is equivalent to induced self-attention introduced in Lee et al.
[20] providing an efﬁcient solution for pairwise weights measuring.
Usually, we can take multiple EM steps for further optimization (e.g., ϕ ( k, 0) → ϕ ( k, 1) → · · · → ϕ ( k,T ) ).
Here T is the number of EM steps.
IntraCM belongs to the class of generalized EM algorithms and is guaranteed to converge to a local optimum of the data log likelihood [35].
3.2.2 Inter-Cluster Coupling
After T EM steps, the variational probability q ( k,T ) i,j = q ( b ( k,T ) i,j | x i ) can be regarded as the optimal transport plan, which carries the information on how to distribute the mass of data points x i to reference vectors ϕ ( k,T ) j .
By regarding ϕ ( k,T ) j as couplings between unassigned set and assigned set, we propose inter-cluster coupling ( InterCC ) to generate the next cluster, as shown in Figure 1.
Considering the current data points have structures with k different sets, C 1 , C 2 , · · · , C k − 1 , S k , we can get k − 1 transport plan matrices { U ( k ) 1 , U ( k ) 2 , · · · , U ( k ) k − 1 } related to k − 1 existing clusters and V ( k ) related to the unassigned set S k ,
[Equation]
We can measure the relations between unsigned data points in S k and existing assigned data points in k − 1 existing clusters C 1 , C 2 , · · · , C k − 1 with the aid of couplings ϕ ( k ) as follows
[Equation]
where ⊕ is a column-wise concatenation operation and v ( k ) i is the i -th column of V ( k ) .
It is reasonable to select dissimilar data point in unassigned data points set S k to form a new cluster C k , i.e.,
[Equation]
Here Multinomial distribution is used to model the generation of C k .
I ∈ R N − M k is a vector of whose elements are all 1s.
m ( k ) j is the i -th column of M ( k ) .
Thus, the data points related to dissimilar indices in each matrix are selected to generate the new cluster C k .
Furthermore, previous existing clusters C 1 , C 2 , · · · , C k − 1 can be updated based on dissimilarity matrix.
For a given cluster C l , it can be updated by C l :=
C l ∪C new l , where C new l := M ULT ( S OFTMAX [1 − I ( l ) ⊤ M ( k,l ) ]) .
I ( l ) ∈ R
|C l | is a vector of all 1s and M ( k,l )
=
U ( k ) l ϕ ( k,T ) V ( k ) ⊤ .
3.2.3 Iterative Unrolling Inference
IntraCM and InterCC are both differentiable procedures for optimal transport measuring and next cluster constructing, whose outcome relies on the current existing clusters and the statistical model parameters ϕ ( k ) .
As shown in Eq.(2), the whole cluster-wise clustering procedure can be regarded as a recurrent IntraCM with InterCC until there are no available data point left.
In this case, the information about statistical regularities in our model that is required for generating the sequential clusters is encoded in a series of parameters { ϕ (1 ,T ) , ϕ (2 ,T ) , · · · , ϕ ( K,T ) } .
By unrolling the iterations of the presented mixing process, we obtain an end-to-end differentiable clustering procedure based on the statistical model parameterized by { ϕ (1 ,T ) , ϕ (2 ,T ) , · · · , ϕ ( K,T ) } .
We can therefore use (stochastic) gradient descent and ﬁt the statistical model to capture the regularities corresponding to clusters for a given dataset.
[Equation]
IntraCM
+ p
( C k | f θ ( ϕ ( k ) ))
| {z } InterCC # (11)
The likelihood is derived from approximating the p θ ( C 1: K |D ) to true posterior via KL divergence.
Note that samples from p ( C 1: K , D ) are obtained from the explicit generative model, such as Gaussian mixture model, Dirichlet process mixture model.
With inﬁnite generated samples, we can train a model to approximate p θ ( C 1: K |D ) accurately.
The whole inference procedure is given in Algorithm 1.
By repeating the cluster generative process until there are no data points left, IntraCM and InterCC are processed alternatively, and the clusters are iteratively reﬁned by deriving the explicit supervisory signal from the already formed clusters.
Note that the complexity of traditional methods measuring weights between data points acquires O ( KN 2 ) (e.g., self-attention) which may be too expensive for large sets, and our AMCP only acquires O ( KNk ) in the EM-step for the k -th iteration, which is more efﬁcient.
Establish
Experiments
In this section, we evaluate the proposed AMCP in terms of synthetic and real-world datasets by comparing with the state-of-the-art methods.
4.1 Experiments on Synthetic Datasets
The ﬁrst experiment was conducted on synthetic data with arbitrary number of clusters.
Thus, ﬁve existing methods handling variable number of clusters, MCMC[1], VI[2], DAC[21], NCP[27] and
[Table]
ST-ACT[20] are used as baselines.
Among them, MCMC and VI are traditional approximation methods, and DAC, NCP and ST-ACT are deep amortized clustering methods.
The synthetic data is generated via a 2D Gaussian mixture model (GMM) which is deﬁned by the following process α ∼ E XP (1) , c 1: N ∼ CRP ( α ) , µ k ∼ N (0 , σ 2 µ 1 ) , x i ∼ N ( µ c i , σ 2 I ) .
We set σ µ = σ = 10 .
At each training step, we generate 10 random datasets according to the above generative process.
Each dataset contains 200 points on a 2D plane, and each sampled from one of 4 Gaussians.
Two testing scenarios ( Scenario 1 and Scenario 2 ) are constructed to evaluate the effectiveness of the proposed method.
The test set in Scenario 1 has the same conﬁguration (200 samples and 4 clusters) as training set, while Scenario 2 contains different numbers of samples and clusters (400 samples and 6 clusters) in order to verify whether the clustering method can generalize to the unseen clusters.
Table 1 summarizes the results in terms of two testing scenarios.
Here we report log-likelihood ( LL ), clustering accuracy ( ACC ), Normalized Mutual Information ( NMI ) and running time (seconds).
The best and second results are marked in bold and underlined.
As expected, the proposed AMCP consistently outperforms baselines on both scenarios.
Although Scenario 2 is more challenging, amortized clustering methods (DAC, NCP, ST-ACT and our AMCP) can capture cluster uncertainty and obtain competitive results.
For testing time, amortized methods are obviously faster than traditional sampling based method (MCMC).
Although VI is much fast than MCMC, it is still lower than amortized methods.
In cluster-wise amortized clustering methods, DAC, ST-ACT and our ACMP achieve competitive running time since cluster-wise sequential modeling is more efﬁcient than point-wise sequential modeling.
4.2 Experiments on Real-world Datasets
Real-world datasets, MNIST [19], Tiny-ImageNet [18] and CIFAR-10 [34], are used to validate the performance.
To formulate the amortized mechanism, we generate training and test data via following Dirichlet process mixture model: α ∼ E XP (1) , c 1: N ∼ CRP ( α ) , K − 1 ∼ B INOMIAL ( K g − 1 , 0 . 5) , l k ∼ U (0 , K − 1) , x i ∼ U
[ D , l c
i ] .
Here x i ∼ U
[ D , l c
i ] indicates sampling x i with label l c i uniformly from dataset D , which can be training set or test set.
K g indicates the number of clusters we can sample.
For all original dataset, data containing half of the classes is used to generate the training sets, and the remaining half is used to generate the test sets, with no overlap between training classes and test classes.
We sample multiple clustered datasets to form training sets with N ∈ { 1000 , 10000 } .
For test set, we generate 1000 randomly clustered datasets with N ∈ { 1000 , 5000 } .
More information is given in Appendix.
In Table 2, we depict the quantitative clustering results in real-world datasets.
The best and second results are marked in bold and underlined.
The average clustering results are recorded over 5 runs with different random parameter initialization.
It can be seen that our AMCP outperforms all baselines.
The main reason, we believe, is that AMCP not only exploits amortized properties between data points in both inter- and intra-clusters, but also has ﬂexible clustering process, which simultaneously generate the new cluster and update the summary statistics for previous clusters.
In model efﬁciency evaluation, the DAC, ST-ACT and AMCP consume a more competitive running time, which outperform other methods by a large margin.
The reason is that DAC and ST-ACT identify each cluster after one forward pass, and AMCP generates clusters in cluster-wise manner and exploits data relations via the efﬁcient optimal transport.
However, DAC, ST-ACT are pretty worse than AMCP on clustering performance.
Among amortized clustering methods, DAC, NCP, ST-ACT and AMCP are able to process arbitrary number of clusters.
For better visualization of cluster generative process, a digit subset from MNIST test set is randomly sampled.
To illustrate how the clustering models capture the shape ambiguity
[Table] [Table]
of some of the digits, we plot the ground-truth digit clusters and clustering results in Table 3.
Comparing with DAC, NCP and ST-ACT, the proposed AMCP can get the most accurate results.
For example, DAC assigns the digit 7 (with similar appearance to 9) to cluster 9, and NCP generates a new cluster for it.
Fortunately, AMCP correctly assigns it to cluster 7.
All baselines are unable to assign the digit 4 written in a strange way to the right cluster, and our AMCP correctly assigns it to cluster 4.
Results on Tiny-ImageNet and CIFAR-10 can be obtained in Appendix.
4.3 Model Analysis
We empirically analyze the proposed model from the impact of various hyper-parameters.
They are summarized as follows: (1) The Number of EM Steps T : AMCP can apply multiple EM steps to achieve more accurate ϕ ( K,T ) .
Here we conduct experiments on varying T in { 1 , 2 , 3 , 4 , 5 } to investigate the effect of T .
Figure 2(a) and (b) show the effect of T on synthetic data ( Scenario 1 ) and real-world data ( CIFAR-10 ).
We can see that, the performance ( LL and ACC ) become better and better with the increasing of T , while the running time is linearly scalable to the number of steps.
(2) OT E-step vs. Regular E-step We introduce two different ways to conduct E-step in IntraCM: regular soft-assignment E-step and Sinkhorn-based optimal transport E-step.
Figure 2(c) and (d) show the comparison between regular E-step and OT E-step on synthetic data ( Scenario 1 ) and realworld data ( CIFAR-10 ).
The maximum number of Sinkhorn-Knopp iterations for solving the OT problem is set to 10 .
While the regular E-step is consistently faster than the OT E-step since it’s closed-form solution, OT E-step can achieve more accurate likelihood and accuracy.
Furthermore, we compare the difference between the strategy of updating the existing clusters and the strategy of not updating.
More empirical results are given in Appendix.
[Figure]
Conclusion
In this paper
[Figure] we studied the problem of amortized clustering [Figure] and presented our approach that perform cluster-wise amortized mixing coupling processes via intra-cluster mixing and inter-cluster coupling.
An interesting direction for future research is to explore more complicated and realistic hidden structure (e.g., hierarchical clusters) that is not possible with our approach.
Furthermore, it is interested to explore novel applications that can be enabled by the interpretability and controllability brought by the amortized clustering.
Acknowledgement
This work was partly supported by the Beijing Natural Science Foundation under Grant (Z180006,
L211016); the National Key Research and Development Program (2020AAA0106800); the National
Natural Science Foundation of China under Grant 62176020; CAAI-Huawei MindSpore Open Fund; and Chinese Academy of Sciences(OEIP-O-202004)
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Abstract and Introduction.
(b) Did you describe the limitations of your work?
[Yes] See Conclusion (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[No] All assets used in this paper are open source (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] Does not contain personally identiﬁable information or offensive content 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]