S TATISTICAL G UARANTEES FOR C ONSENSUS C LUSTERING
A BSTRACT
Consider the problem of clustering n objects.
One can apply multiple algorithms to produce N potentially different clustersings of the same objects, that is, partitions of the n objects into K groups.
Even a single randomized algorithm can output different clusterings.
This often happens when one samples from the posterior of a
Bayesian model, or runs multiple MCMC chains from random initializations.
A natural task is then to form a consensus among these different clusterings.
The challenge in an unsupervised setting is that the optimal matching between clusters of different inputs is unknown.
We model this problem as finding a barycenter (also known as Fréchet mean) relative to the misclassification rate.
We show that by lifting the problem to the space of association matrices, one can derive aggregation algorithms that circumvent the knowledge of the optimal matchings.
We analyze the statistical performance of aggregation algorithms under a stochastic label perturbation model, and show that a K -means type algorithm followed by a local refinement step can achieve near optimal performance, with a rate that decays exponentially fast in N .
Numerical experiments show the effectiveness of the proposed methods.
I NTRODUCTION
Clustering is a fundamental task in machine learning and data analysis.
Given data on each of the n objects in a set, there are numerous algorithms to produce a clustering of these n objects, which is formally a partitioning of { 1 , . . . , n } into K disjoint sets.
A natural problem that arises in practice is how to form a consensus among these clusterings.
This is especially important if the different clusterings are produced by a single randomized algorithm.
This situation often arises in Bayesian modeling, where the posterior naturally encodes the variability of the clustering problem.
Finding a consensus clustering then corresponds to finding the center of the posterior, from which we can also obtain estimates of the variability of the posterior.
A clustering of n objects can be viewed as a label vector in [ K ] n where [ K ] = { 1 , . . .
, K } .
We assume that we are given N label vectors z j ∈ [ K j ] n for j = 1 , . . .
, N , with potentially different number of clusters each.
Let K = max j K j and note that we can view all z j as vectors in [ K ] n .
The task is to obtain a consensus K -clustering, that is, a label vector z ∈
[ K ] n which is close to all z 1 , . . .
, z N at the same time.
We also refer to this task as the label aggregation problem.
In the context of clustering, there is no meaning to the label of each cluster, that is, the label aggregation problem is unsupervised , in the sense that there is no natural correspondence between labels of different clusterings.
This is in contrast to label aggregation in classification in which the labels have a common meaning among different input classifications.
We refer to the latter task as supervised label aggregation.
In the unsupervised setting, forming a consensus label is a nontrivial task due the label-switching problem.
Consider for example, the case n = 5 and the two label vectors z 1 = (1 , 1 , 1 , 2 , 2) and z 2 = (2 , 2 , 2 , 1 , 1) .
These two vectors are different in all 5 positions but they define the same clusterings of the objects.
In this case, the consensus label (cid:98) z can be taken to be either z 1 or z 2 .
More generally, for every z j , there could be a permutation π j on [ K ] , such that the permuted vectors π j ◦ z j :=
( π j ( z ji ))
n i =1 , are closer to each other than the original z j s.
To formalize the above idea, we recall the definition of the misclassification rate between two label vectors, z, y ∈
[ K ] n :
[Equation]
where the minimum is taken over all the permutations π : [ K ] → [ K ] .
Mis( · , · ) is a proper metric on the space of K -clusterings of n objects.
It is also a metric on [ K ] n if we identify vectors that are obtained from each other by label-switching.
We can now define the consensus label as the barycenter of z 1 , . . .
, z N in Mis( · , · ) metric, that is,
[Equation]
where w j ≥ 0 are a given set of weights.
We often assume uniform weights: w j = 1 for all j .
The barycenter (cid:98) z is also known as the Frechét mean.
Solving (2) is complicated by the presence of the permutation in the definition of Mis function.
More explicitly, we need to solve
[Equation]
showing that in addition to z , we have to optimize over N permutations π j , j = 1 , . .
N .
In this paper, we provide alternative solutions that avoid optimizing over these permutations.
Our contributions The unsupervised version of the label aggregation problem is the realistic and practical one when dealing with aggregating labels from Bayesian clustering algorithms, since the posterior has K !
modes corresponding to all possible label permutations, and the output will be near an arbitrary mode in each run of the algorithm.
The main contributions of this paper to unsupervised aggregation are the following:
We show that by lifting the barycenter problem to the space of association matrices, one can derive algorithms that avoid optimizing over the unknown permutations (Section 2.1).
In particular, we propose both a basic and a spectral K -means type aggregation algorithm.
We propose a random perturbation model (RPM) under which we can study the theoretical performance of both supervised and unsupervised aggregation algorithms.
We prove the statistical consistency of the basic aggregation algorithm under RPM (Section 2.2).
Under RPM, the supervised setting corresponds to an oracle that knows the true matching permutations.
By studying this oracle, we derive the optimal statistical misclassification rate for supervised aggregation (Section 3.1).
We propose an efficient local refinement step on the output of any consistent aggregation algorithm in the unsupervised setting, and show that the updated labels achieve nearly the same misclassification rate as the above oracle (Section 3.2).
Our theoretical analysis illustrates how different parameters affect the difficulty of the label aggregation problem.
In Section 4, we provide numerical experiments comparing the performance of the proposed algorithms against each other and existing methods.
Related work In the supervised setting, the problem of label aggregation is to combine multiple annotated dataset.
The label inferred for each item from those produced by multiple annotators acts as the ground truth for the classification task.
Various probabilistic models have been proposed for aggregating annotations, with parameters to account for the expertise of the annotators and the noise in the labeling process (47; 37).
The unsupervised setting is more challenging as there is no meaning to the cluster labels (the label-switching issue) and the clusterings can have potentially different number of clusters.
The idea of passing to association matrices to get around the label-switching issue, has been leveraged in several existing approaches (13; 24; 43; 13; 21; 29), although the connection we make to the lifted barycenter problem and the resulting spectral methods is new to the best of our knowledge.
In (24; 43), the authors employ an Expectation-Maximization strategy to obtain a nonnegative matrix factorization of the combined association matrix.
The authors of (41) provide several approaches, using the hypergraph representation of the clusterings, that have shown promising results in the context of image segmentation (22).
A set of fuzzy consensus algorithms is proposed in (40) that generate soft consensus partitions by combining a collection of fuzzy clusterings.
What we referred to as unsupervised label aggregation problem has appeared under many different names in the literature, including but not limited to, cluster ensembles (41), clustering/cluster ensemble problem (44; 10), ensemble clustering (1), clustering aggregation (17), combining clusterings (42; 14; 27), consensus clustering (48; 18) and the median partition problem (12; 46; 18).
As can be surmised from the variety of names, there is a copious literature on the subject, spanning over multiple fields, with many ideas rediscovered time after time.
We refer to the excellent surveys (48; 44; 18) for a more exhaustive list of references and historical discussions.
There is also a parallel line of work in the Bayesian clustering literature on aggregating the posterior clusterings (31; 7; 9; 25; 15; 45; 8).
The barycentric view to aggregation that we take in (2) has appeared in many previous work, but often with a different distance in place of Mis , including but not limited to the symmetric difference distance (SDD), a.k.a.
the Mirkin metric (up to a constant), in the median partition problem (39; 23; 12; 17; 18), the Binder loss and variational information (VI) in (45; 15), the normalized mutual information (NMI) in the pioneering work of (41), the adjusted Rand index (15) and the category utility function (42; 35).
After introducing our methods, in Section 2.3, we give a more detailed comparison with the literature.
We choose Mis as the distance in the present work for a better comparison with the oracle problem in
Section 3.1.
We also show in Appendix B.1 that consistency in Mis implies the consistency in other distances.
Despite the voluminous literature on the subject, statistical analysis of the methods under a statistical model for the input clusterings has not been undertaken before.
This is the gap that we fill in this paper, by providing the first consistency and optimality results under a statistical model (the RPM) for a method of clustering aggregation that we propose.
To the best of our knowledge, the question of consistency, let alone optimality, has not been considered for any method of aggregation before.
We also shed more light on the relation between the barycenteric approach and those based on association matrices (Section 2.3), and how convex relaxation leading to a spectral method can be used to approximate the median partition.
To illustrate the importance of statistical analysis, we also show that a simple common approach to the median partition problem, known as the BestOfK (18), is in general inconsistent under RPM, despite being shown to be a 2-factor approximation of the median partition problem (12).
This further highlights the key insights that statistical analysis under a model can provide which is not possible to obtain by CS-type theory on approximation algorithms.
L IFTED A GGREGATION A LGORITHMS
We start by introducing some notation.
Let E K
= { e k } Kk =1 be the set of standard basis vectors of R K .
The elements of E K can be considered one-hot encodings of the labels from [ K ] .
From now on, instead of encoding labels as element of [ K ] , we encode them as element of E k .
We can then view labelings of n objects as elements of the following set
[Equation]
Each z i is viewed as a K × 1 vector and each element of E nK as K × n matrices, which we refer to as label matrices.
For Z ∈ E K , permuting the cluster labels is equivalent to pre-multiplication by a
K × K permutation matrix P , that is, PZ .
The label aggregation problem can be restated as follows:
Given label matrices Z 1 , . . .
, Z N ∈ E nK , find a consensus label matrix Z ∈ E nK , by solving the barycenter problem:
[Equation]
where P 1 , . .
. , P N are K × K permutation matrices and ∥ X ∥ F := (cid:0) (cid:80) i,j X 2 ij (cid:1) 1 / 2 is the Frobenius norm of matrix X .
One can verify that problem (5) is equivalent to (3).
The following result shows that if we know the optimal permutations P j s, we can easily find the barycenter (cid:98) Z :
Proposition 1.
Let { (cid:98) P 1 , . . .
, (cid:98) P N } be an optimal set of permutation matrices in (5) .
Then, the optimal solution (cid:98) Z of (5) is the columnwise “argmax” of (cid:80) j w j (cid:98) P j Z j .
Algorithm 2 Spectral label aggregation algorithm
K × K
K eigenvalues
of U
n
2. L IFTING TO ASSOCIATION MATRICES
The difficulty in the unsupervised setting is that the optimal permutations { (cid:98) P j } are unknown.
To get around this issue, we lift the barycenter problem to the space of association matrices.
For a label matrix Z ∈ E n K , we define the corresponding association matrix as X =
Z T Z ∈ { 0 , 1 } n × n .
Note that X ij = 1 iff i and j are in the same cluster according to Z , otherwise X ij = 0 .
The advantage of
X is that it is invariant to label switching: X = Z T Z = ( PZ ) T PZ for any permutation matrix P .
This suggests solving the following lifted barycenter problem instead of (3):
[Equation]
where X j = Z Tj Z j and X K =
{ Z T Z : Z ∈ E nK } , that is, the set of (binary) association matrices with at most K clusters.
Semidefinite relaxation Problem (6) is still hard to solve due to the combinatorial nature of X K .
An approach to solving problems over X K is to relax to a semidefinite program, an idea that has been applied before to community detection in networks (4).
In particular, X K is inside the doubly nonnegative cone { X : X ⪰ 0 , X ≥ 0 } , where X ⪰ 0 and X ≥ 0 mean X is positive semidefinite and elementwise nonnegative.
We note that X ii
= 1 for all i .
This suggests relaxing to the following problem
[Equation]
Problem (7) has a simple solution.
The solution of the unconstrained version of (7) over R n × n is
X ′ :
= (cid:80) nj =1 w j X j / (cid:80) nj
=1 w j .
Since { X j } belong to the constraint set of (7) and this set is convex,
X ′ too belongs to the constraint set.
Hence, X ′ is the solution of (7), that is, (cid:98) X
= X ′ .
It remains to translate (cid:98) X back to labels, for which we can preform rowwise K -means, leading to Algorithm 1.
Since elements of X K are of rank at most K , to get a solution which is closer to that of the lifted barycenter problem (6), we can perform a spectral truncation of (cid:98) X to its K top eigenvectors, before applying the rowwise K -means.
This leads to the spectral aggregation Algorithm 1.
Other variants of spectral clustering on (cid:98) X are also possible, e.g., using the normalized Laplacian, etc.
In the K -means step of Algorithm 1 and 2, any constant-factor approximation to the K -means problem can be used.
2. C ONSISTENCY
In order to study the statistical performance of different aggregation algorithms, we propose a random perturbation model (RPM) , where both the clusters and the labels of the clusters can undergo random perturbations, allowing us to study the difficulty of the unsupervised aggregation problem.
Let
Z ∗ ∈
E nK be the “true” label matrix with columns z ∗ i ∈ E K , i = 1 , . . . , n .
Definition 1 (RPM) .
We write Z ∼ L ( Z ∗ , p ) if Z = ( z 1 , . . .
, z n ) ∈ E nK with columns
z i drawn independently as follows:
[Equation]
where δ z ∗ i is the point mass at z ∗ i and P is an independent K × K permutation matrix.
Under RPM, the observed label matrices Z 1 , . . .
, Z N are i.i.d.
, so it is reasonable to let w 1 = · · · = w N = 1 in Algorithms 1 and 2.
We will discuss the algorithm for weighted samples in Section 5.
Let n k ( Z ∗ ) be the number of objects in cluster k according to Z ∗ .
We make the following assumption:
[Equation]
for some β ∈ [1 , K ] .
Here, β measures how much the true clustering deviates from being balanced .
For β = 1 , we have n k ( Z ∗ ) =
n/K for all k , while β = K corresponds to no restriction on the sizes of the true clusters.
The following result shows that the basic aggregation algorithm is statistically consistent under the RPM:
Theorem 1 (Consistency) .
Let Z ∗
∈
E nK be a label matrix satisfying (9) .
Assume that Z 1 , . . .
, Z N are i.i.d.
draws from L ( Z ∗ , p ) and let ξ := p (2 − p ) .
Let Mis be the misclassification rate between the true label matrix Z ∗ , and the output of Algorithm 1 with w j = 1 for all j ∈ [ N ] .
Then, there exists a universal constant C > 0 , such that
[Equation]
Consistency of Algorithm 1 follows from (10) and the Markov inequality: For any δ > 0 , we have P (Mis ≥ δ ) ≤ δ − 1 E
[Mis] → 0 as n, N → ∞ and p is bounded away from 1.
We note that in this and subsequent results all the parameters, such as K and p , are allowed to change as n, N → ∞ , subject to the conditions of the theorems.
The first term inside the parentheses in (10) is the dominant one.
Assume for simplicity that β ≍ 1 .
If the model has low noise, then p is small and so is ξ since ξ ≍ p .
Then, the dominant term is O ( pK/N ) , that is, a smaller number of clusters, K , and a larger sample size, N , improve the performance.
The second term is independent of of the sample size, but vanishes at the rate
O ( p 2 K 2 /n ) in the low noise setting, as the number of objects, n , grows.
2. L ITERATURE COMPARISON
The relation between the metrics on clusterings is discussed in (34; 33).
Let d ′ M
( Z, Z j ) be the Mirkin metric between clusterings Z and Z j (34, Eqn (6)).
As we show in the Appendix B, it turns out that d ′ M
( Z, Z j ) =
∥
X − X j ∥ 2 F .
It is also not hard to see that d ′ M / 2 = SDD =
Binder = (cid:0) n 2 (cid:1) (1 − Rand ) where Binder denotes the Binder loss (5) and Rand, the Rand index (36).
It follows that the lifted barycenter (6) that we derived is equivalent to the median partition (12) and the Binder loss barycenter of (45) as well as the Rand barycenter (15).
This problem is often solved by greedy search starting from a random initialization (45; 8).
Our Algorithm 2 then provides a fast scalable spectral method of obtaining an approximate solution to this ubiquitous problem.
A lot of algorithms proposed for consensus clustering operate on the average association matrix ¯ X , by treating it as a similarity matrix and performing usual clustering on it, for example, by performing agglomerative clustering (31; 18; 14; 28).
Wade and Ghahramani (45) criticize these approaches as being ad-hoc compared to the decision-theoretic approach of finding a barycenter.
However, we show ¯ X is indeed a solution to the relaxed version (7) of the barycenter (6) which is the same as the Binder barycenter in (45), hence clustering ¯ X is effectively solving the same problem with a different method.
A k -means based aggregation algorithm, called KCC, has been proposed in (48).
In our notation, this is equivalent to concatenating Z j s row-wise to form an NK × n matrix and running k -means on the columns.
This is different from our Algorithm (1) that operates on ¯ X .
We compare with
KCC in Section 4.
Unlike KCC, Algorithm (1) comes with a consistency guarantee under our model assumptions.
The BestOfK (12) essentially solves (6) by restricting the feasible region to { X 1 , . . . , X N } , hence picking the lowest-scoring input clustering.
The approach proposed in (7; 9) (see also (15)) is to find the input clustering that minimizes the cost X (cid:55)→ ∥
X − ¯ X ∥ F .
It is not hard to see that the barycenter cost (6) is equal to ∥ X − ¯ X ∥ 2 F plus a constant (essentially a bias-variance decomposition; see (26)).
Hence, the approach of (7; 9) is equivalent to the BestOfK. As we show in
Appendix A, BestOfK is, in general, inconsistent under RPM unless N grows exponentially in n , a very strong condition not needed by our algorithms.
The K -means step 3 in Algorithm (1) can be replaced by other clustering algorithms, e.g. averagelinkage clustering, BestOfK and CC pivot algorithm of (18; 12).
Besides global clustering algorithms, many authors, including (41; 17; 18; 45; 8) also propose local search, a.k.a.
greedy, algorithms which
1 p 0.
I ( p )
Chernoff divergence
K = 2 K =
K =0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 p0.
-( l og (
P N ))
/ N
Bayesian Error Probability
N = 2 N =
N = 30 N =
(Chernoff div.) update one label at a time to minimize the barycenter loss, based on any number of metrics discussed earlier, including the information-theoretic ones (NMI and VI (32)).
In (26), after reducing (6) to minimizing X (cid:55)→ ∥
X − ¯ X ∥ 2 F , they write X =
Z T Z (in our notation) and then relax Z to a general nonnegative matrix and solve the problem with nonnegative matrix factorization.
Their work has the flavor of our Algorithm 2, although our approach, being based on regular spectral decomposition, is highly robust and scalable.
The RPM, with P set equal to the identity, is closely related to the artificial data model considered in (18), with the difference that RPM does not potentially create a new cluster after perturbation, and introduces a random permutation of the clusters labels after perturbation (via P ).
In Appendix E, we argue that RPM is a good model of a concentrated posterior, hence consistency under RPM is relevant to Bayesian aggregation problems for which posterior consistency has been shown.
O PTIMAL R ATE
Theorem 1 guarantees an O ( N − 1 ) rate of misclassification for Algorithm 1.
A natural question is whether we can do better.
To answer this question, we first consider what is the best an oracle, with the knowledge of the random permutations in (8), can do.
This oracle is effectively solving the supervised version of the problem.
We then show that a refinement step allows us to achieve nearly the same as the optimal oracle rate, without knowing the matching permutations.
3. S UPERVISED ORACLE
Let Z ′ be a label matrix with the i th column z
′ i ∼ (1 − p )
δ z ∗
i + p Unif ( E K ) , and let Z 1 , . . .
, Z N be independent copies of Z ′ .
We would like to recover the true label matrix Z ∗ .
This is the oracle version of model (8), since Z j s are label matrices without random permutations.
In this case, the problem decouples to n independent label recovery problems.
We further simplify the problem to that of deciding between z ∗ 1 = e 1 and z ∗ 1 = e 2 .
This problem is equivalent the hypothesis testing:
H 0 :
Multinomial ( N, (1 − ˜ p, q, . .
. , q )) versus H 1 :
Multinomial ( N, ( q, 1 − ˜ p, . .
. , q )) , where q := p/K and ˜
p :
=
( K − 1) q := p − q .
A classical result from information theory (6, Theorem 11.9.1) allows us to determine the optimal performance in this case:
Proposition 2.
The Bayesian error probability, P N , for testing H 0 against H 1 , with positive prior probabilities, is bounded by e − NI , where
[Equation]
(11) is the best achievable error exponent in the sense
that − 1 N log P N → I as N → ∞ .
The left panel in Figure 1 shows plots of I as a function of p , for various K , and the right panel shows the convergence of the exponent of P N for K = 2 .
The Bayesian error probability P N can be achieved by performing a likelihood ratio test between the two hypotheses.
We can generalize this result to testing between K hypotheses, in which case the Bayesian error probability is bounded by
Let
X be
Replace the i th column of (cid:101) Z by zeros and denote this matrix by (cid:101) Z − i .
Let ( n 1 , . . .
, n K ) be the row sums of (cid:101)
Z − i .
Let ( b 1 , . . .
, b K
= (cid:101)
Z − i ¯ X i .
Update the i th label arg k b k k
9: end for ( K − 1) e − NI .
The oracle algorithm that achieves this bound is the one that finds the columnwise “argmax” of the average of Z j s.
In light of Proposition 2, the bound in Theorem 1 is far from optimal since it guarantees a linear decay of the error in N , that is O
( N − 1 ) , rather than the exponential decay e − NI .
The question is whether this gap can be filled by a non-oracle algorithm.
3. L OCAL REFINEMENT
To approach the oracle rate, we propose a fast local refinement on the label of each object, as outlined in Algorithm 3.
This refinement can be performed on the output of any reasonable aggregation algorithm.
The idea of performing a local refinement to boost the performance of clustering algorithms has been employed in various settings, including clustering of sub-Gaussian mixtures (30) and community detection in stochastic block models (SBM) (3; 16; 50; 51).
A more detailed comparison with the SBM appears in Appendix F.
Algorithm 3 requires a good initial label matrix (cid:101)
Z , with a small number of mismatches relative to the true matrix Z ∗ .
The algorithm focuses on the local information of the i th object.
With ¯ X the average of the association matrices, ¯ X ij is the sample proportion of objects i and j appearing in the same cluster.
Viewing ¯ X as the adjacency matrix of a weighted graph, one can verify that b k = (cid:80)
j ̸ = i w j X ji 1 { (cid:101)
Z j = k } is the weighted number of connections between object i and objects in cluster k .
The algorithm then normalizes the number of connections by the cluster size n k .
Thus, b k /n k estimates the probability that object i is connected to another object in cluster k .
The higher this probability is, the higher the chance that object i belongs to cluster k .
The last step in the for loop, updates the i th label according to these statistics.
It is possible to repeat the local refinement procedure, by feeding its output (cid:98) Z back as an initial label matrix.
Given a good initialization, local refinements usually converge in constant or O (log n ) number of steps (30).
3. A CHIEVING THE SUPERVISED RATE WITHOUT SUPERVISION
From the oracle result (Proposition 2), we expect the optimal misclassification rate to be close to e − NI .
This is verified by the next result, showing that a single local refinement step applied to a consistent aggregation algorithm, such as Algorithm 1, can get us nearly to the optimal rate:
Theorem 2.
Assume that Z 1 , . . .
, Z N are i.i.d.
draws from the random perturbation model (8) .
Let (cid:98) Z be the output of Algorithm 3 initialized by some, possibly data-dependent, (cid:101) Z .
Let n min =
min k ∈ [ K ] n k
( Z ∗ ) be the smallest true cluster size and recall the definition of I in (11) .
Assume that
[Equation]
and there exists δ satisfying Knδ/ ( n min pI )
=
o (1) such that one of the followings holds:
[Equation]
Then, for some η = o (1) , the misclassification rate satisfies
[Equation]
The assumptions of Theorem 2 are mild.
Suppose that p is bounded away from 0 or 1, say p ∈ [0 . 01 , 0 . 99] , K
= O (1) and the cluster sizes are similar.
Then, the assumptions can be simplified to n min , N → ∞ and δ
=
o
The theorem is most interesting when p → 1 and K is large.
In this case, I → 0 and the first requirement of assumption (a) becomes n min I/K → ∞ .
This assumption guarantees that the samples provide sufficient information to recover the permutations, although we have not attempted to do so in our algorithm.
The second requirement of assumption (a) provides evidence to distinguish the true labels from the other K − 1 labels.
Under the assumptions of Theorem 2, Algorithm 3 initialized with input satisfying (12), will have an output satisfying (b1).
Hence, Theorem 2 also guarantees rate-optimality of an iterative Algorithm 3.
E XPERIMENTAL R ESULTS
We now present empirical results comparing the performances of the proposed aggregation algorithms, with additional results provided in Appendix C.
The ground truth label matrix Z ∗ is generated by randomly assigning each of the n objects to one of the K labels.
The N input clusterings Z j , j ∈ [ N ] are generated from model (8).
We measure the performance of an algorithm by the adjusted Rand index (ARI) of its output against the ground truth.
We consider seven different aggregation algorithms: (1) Our Algorithm 1, referred to as “Basic” in the plots; (2) Our Algorithm 2, referred to as SC; (3) KCC algorithm (48); (4) CC Pivot algorithm (2; 18) with threshold 0.25; (5) Best One Element
Move (BOEM) algorithm (11; 18); (6) the EM algorithm of (24); and (7) BestOfK algorithm (12; 18).
In addition, we consider variants of algorithms (1)–(5) where we apply our refinement step to their output.
This gives us a total of 12 methods.
In the plots, the refined version is denoted with a solid line and the original version (without refinement) with a dotted line.
We also use the average ARI of the
N input clusterings, denoted by the I NPUT , as a baseline.
Balanced setting with varying n and N .
Figure 2 depicts plots of ARI versus the noise probability p in model (8), for various methods.
The results are averaged over 40 replications.
The settings in
Figure 2 all correspond to balanced cluster sizes.
Generally, our proposed Basic and SC algorithms outperform the EM, KCC, CCPivot and BOEM algorithms, with the failure thresholds occurring at larger values of p (harder problems).
In some settings, the refinement shows some improvement, but in others, the output of the refinement applied to Basic and SC nearly coincides with the original algorithm.
This shows that in some settings the original algorithm implicitly performs the refinement itself.
We note that increasing N shifts the failure thresholds to the right as expected, as does the increasing of n , both consistent with our theory.
Note that BestOfK performs no better than I NPUT .
Moreover, refined versions of KCC and BOEM outperform their original versions.
Unbalanced setting.
Figure 3 depicts the results obtained with disproportionate cluster sizes, specifically with p 1 proportion of the objects in one cluster and the rest distributed uniformly to the remaining clusters.
Local refinement over Basic and SC performs significantly better as we deal with input clusterings of disproportionate cluster sizes, especially at lower noise probabilities.
C ONCLUSION AND D ISCUSSION
In the present paper, we defined the random perturbation model to study the label aggregation problem.
We developed a K -means type algorithm followed by an efficient local refinement step to achieve the optimal misclassification rate under the assumptions of the model.
Numerical experiments also show the effectiveness of our proposed methods.
Let us also discuss possible avenues for future work.
A single-stage algorithm.
Two-stage algorithms have been popular in many clustering problems (3; 30; 16).
Numerical experiments show that, in many cases, a K -means type algorithm performs sufficiently close to the local refinement with good initialization (Section 4).
The K -means type algorithm assigns labels based on the distances between the objects and the centers.
This criterion is different from the likelihood ratio test in many cases, and so the output of the algorithm will not achieve the misclassification rate of the oracle problem.
It will be a novel improvement if the
K -means type algorithm can be generalized to an EM-type algorithm so that the “distance" between the parameter and the object is defined by the likelihood.
Whether such an algorithm exists, and how efficient it is statistically and computationally, are interesting open questions.
A robust algorithm.
We observe i.i.d.
label matrices from the random perturbation model defined in Definition 1.
As long as p < 1 , every label matrix from this model provides the same amount
[Figure]
and I nde
[Figure]
and I nde x
[Figure] [Figure]
of information in expectation, so there is no reason to assign different weights to label matrices.
However, in practice, the samples may not be i.i.d.
A CKNOWLEDGEMENT
Zhixin Zhou is supported by the Research Grants Council of Hong Kong [Project Number 21502921].
A. A. Amini was supported by the NSF grant DMS-1945667.