Average Sensitivity of Euclidean k -Clustering
Abstract
Given a set of n points in R d , the goal of Euclidean ( k, (cid:96) ) -clustering is to ﬁnd k centers that minimize the sum of the (cid:96) -th powers of the Euclidean distance of each point to the closest center.
In practical situations, the clustering result must be stable against points missing in the input data so that we can make trustworthy and consistent decisions.
To address this issue, we consider the average sensitivity of Euclidean ( k, (cid:96) ) -clustering, which measures the stability of the output in total variation distance against deleting a random point from the input data.
We ﬁrst show that a popular algorithm k - MEANS ++ and its variant called D (cid:96) - SAMPLING have low average sensitivity.
Next, we show that any approximation algorithm for
Euclidean ( k, (cid:96) ) -clustering can be transformed to an algorithm with low average sensitivity while almost preserving the approximation guarantee.
As byproducts of our results, we provide several algorithms for consistent ( k, (cid:96) ) -clustering and dynamic ( k, (cid:96) ) -clustering in the random-order model, where the input points are randomly permuted and given in an online manner.
The goal of the consistent setting is to maintain a good solution while minimizing the number of changes to the solution during the process, and that of the dynamic setting is to maintain a good solution while minimizing the (amortized) update time.
Introduction
Euclidean ( k, (cid:96) ) -clustering is a popular problem used to compute clustering of points in the Euclidean space, where k is a positive integer and (cid:96) ≥ 1 .
In this problem, given a set of points X = { x 1 , . . .
, x n } in R d , we are to compute a set Z = { z 1 , . . .
, z k } of k points in R d that minimizes cost (cid:96)Z ( X ) := n (cid:88)
i = cost (cid:96)Z ( x i ) , where cost (cid:96)Z ( x i ) :
=
min
j =1 ,...,k (cid:107) x i − z j (cid:107) (cid:96) 2 is the (cid:96) -th power of the Euclidean distance between x and the closest point in Z .
Let opt (cid:96) ( X ) =
min Z cost (cid:96)Z ( X ) , where the minimum is over sets of k points in R d (not X ).
This problem is called k -means clustering when (cid:96) = 2 [18] and k -medians clustering when (cid:96) = 1 .
Although providing an exact solution to Euclidean ( k, (cid:96) ) -clustering is NP-hard [3, 7] (even when (cid:96) = 2 ), polynomial-time approximation schemes (PTASes) do exist [13].
In practice, an algorithm called D (cid:96) - SAMPLING is often used, which computes O (2 (cid:96) log k ) -approximate solution in polynomial time [4].
When (cid:96) = 2 , this algorithm is often called k - MEANS ++.
In practical situations, it is reasonable to assume that the input contains some noise.
If the clustering results are altered by the slightest change in the input, the interpretability and reliability of the clustering results are compromised, and hence the clustering result should be stable.
Stability is also important when making marketing strategies or product production plans building on the clustering result because changing decisions can be very costly.
In this work, we focus on a situation in which some points are missing from the input, and we consider average sensitivity to measure the stability of clustering algorithms, which is deﬁned below.
For a set of points X = { x 1 , . . .
, x n } and i ∈ [ n ] , let X ( i ) denote the set { x 1 , . . .
, x i − 1 , x i +1 , . . .
, x n } , where [ n ] := { 1 , 2 , . . . , n } .
Then, the average sensitivity of a (randomized) algorithm A on X (with respect to the total variation distance) [23] is deﬁned as
[Equation]
where d TV ( A ( X ) , A ( X ( i ) )) is the total variation distance between the distributions of A ( X ) and
A ( X ( i ) ) .
Average sensitivity has been studied on various problems such as maximum matching [25], spectral clustering [21], dynamic programming [15, 16], and network centralities [20].
It is crucial to take the average in (1) instead of the maximum, which measures the worst-case sensitivity against deleting points.
To see this, let us consider the case k
= 2 and a set of points X consisting of two clusters each having ( n − 1) / 2 points and one distant point.
Then any good solution must contain the distant point and a middle point between the two clusters.
However, for the set of points X (cid:48) obtained from X by deleting the distant point, any good solution must contain a point from each cluster.
Hence, the worst-case sensitivity is Ω(1) .
By contrast, the average sensitivity can be
O (1 /n )
because the contribution of deleting the distant point to the average sensitivity is O (1 /n ) .
1.1 Our contributions
Algorithms with low average sensitivity We ﬁrst show that D (cid:96) - SAMPLING has average sensitivity
O ( k/n ) (Section 2).
This matches the lower bound of Ω( k/n ) on the average sensitivity of any algorithm with a ﬁnite approximation ratio (Appendix A).
We then show that any approximation algorithm can be transformed to an algorithm with low average sensitivity while almost preserving the approximation ratio.
Speciﬁcally, given an α -approximation algorithm A for ( k, (cid:96) ) -clustering and (cid:15) > 0 , we can transform it into a (1 + (cid:15) )
α -approximation algorithm A (cid:48) with average sensitivity poly( d, k, 2 (cid:96) , 1 /(cid:15) ) /n
(Section 3).
For example, if we plug in a
PTAS for ( k, (cid:96) ) -clustering [13] as A , we obtain a PTAS with average sensitivity poly( d, k, 2 (cid:96) , 1 /(cid:15) )
/n as A (cid:48) .
In our transformation, we ﬁrst construct a coreset [2, 11] of the input, a subset of the input that preserves the objective function, and then apply the approximation algorithm A .
We notice that we can construct the coreset with low average sensitivity, and hence any approximation algorithm applied on it has low average sensitivity.
This approach is new to the design of algorithms with low average sensitivity.
Consistent clustering In consistent Euclidean ( k, (cid:96) ) -clustering, n points in R d arrive online, and the goal is to maintain a good solution, while minimizing inconsistency , which is deﬁned as the total number of points changed in the solution, i.e., (cid:80) t ∈ [ n − 1] | Z t (cid:52)
Z t +1 | , where Z t is the output of the algorithm after the t -th point arrived.
We consider the random-order model, in which the input points are randomly permuted.
We observe that an average sensitivity analysis can be used to obtain consistent algorithms.
Speciﬁcally, we show that, given an input X , an index i ∈ [ n ] , and the output Z of an algorithm A for Euclidean ( k, (cid:96) ) - clustering for X , we can compute an output Z (cid:48) so that the probability Z and Z (cid:48) disagree is at most the total variation distance between A ( X ) and A ( X ( i ) ) in expectation over Z .
Using this connection, we show that a variant of D (cid:96) - SAMPLING is an O (2 (cid:96) log k ) -approximation algorithm with inconsistency
O ( k 2 log n ) .
Also, we show that the abovementioned PTAS with low average sensitivity can be used to obtain a (1 + (cid:15) ) -approximation algorithm with inconsistency poly( d, k, 2 (cid:96) , 1 /(cid:15) ) · log n .
Lattanzi and Vassilvitskii [17] considered consistent algorithms for metric ( k, (cid:96) ) -clustering , for which the underlying metric can be arbitrary but the output points should be selected from the input points. They showed an 2 O ( (cid:96) ) -approximation algorithm with inconsistency O ( k 2 log 4 n ) , where n is the number of points.
Subsequently, Fichtenberger et al.
[8] gave an algorithm for metric ( k, 1) -clustering with approximation ratio O (1) and inconsistency O ( k · poly(log( n ∆))) , where ∆ is the aspect ratio of the input, that is, the ratio between the largest and smallest distances.
We note that these results are incomparable to ours because in Euclidean ( k, (cid:96) )
-clustering we can output points that do not appear in the input, which may signiﬁcantly reduces the objective value.
Dynamic clustering In dynamic (or incremental)
Euclidean ( k, (cid:96) ) -clustering, points arrive online, and the goal is to maintain a good solution, while minimizing the (amortized) time required to update the solution.
For this problem, a few heuristics have been proposed [1], but to the best of our knowledge, there is no known dynamic algorithm with a theoretical guarantee on its update time.
As with consistent clustering, we consider the random-order model.
We show that the consistent version of D (cid:96) - SAMPLING can be implemented such that the amortized update time is O ( dk + ( k + log n ) k log n ) .
Also, we show that the abovementioned consistent PTAS can be implemented such that the amortized update time is poly( d, k, (cid:96), 1 /(cid:15) ) + 2 poly( d,k, 2 (cid:96) , 1 /(cid:15) ) log 2 n . We note that several dynamic algorithms that maintain coresets for metric k -clustering have been proposed [10, 12].
However, it is not clear whether these algorithms can be used to solve dynamic
Euclidean ( k, (cid:96) ) -clustering.
1.2 Related work
Differentially private k -means/medians clustering Differential privacy is a popular notion used to measure privacy risks.
Although we do not deﬁne it formally here, we note that, if an algorithm is β -differentially private, then its average sensitivity is at most β (see, e.g., [23]).
In [9], it is shown that any polynomial-time α -approximation algorithm for k -means/medians clustering can be converted into an β -differentially private algorithm that, given a set X of points in a d -dimensional unit ball, outputs a solution Z such that
[Equation]
where O α ( · ) means that α is regarded as a constant and ˜ O ( · ) suppresses polylogarithmic dependence on n , d , k , and β .
We note that the additive error is inevitable while obtaining β -differentially private algorithms for k -means/medians clustering [22], whereas we do not need it to bound average sensitivity.
Stability Let D be a distribution over points in R d , and suppose that we want to ﬁnd a cluster structure in D using an algorithm A .
In many cases, we do not know the parameter k that best describes D .
A popular method for selecting k is based on the following stability arguments.
That is, we select the k that minimizes the instability of A ,
[Equation]
where A ( X, k ) is the output of A on a set of points X and the number of clusters k , and d ( · , · ) is the distance between two clusterings, which can be deﬁned in various ways.
The main focus of the literature on instability is characterizing D such that Instab( k, n ) → 0 ( n → ∞ ) holds when A is the algorithm that outputs an optimal solution, because we can then claim that the k captures the cluster structure in D well.
Refer to [24] for a survey on this topic.
A notable difference of our work is that we focus on designing an algorithm with low average sensitivity for any input set X , and not characterizing X such that a certain algorithm has low average sensitivity.
Average sensitivity of D (cid:96) -sampling
In this section, we analyze the average sensitivity of a popular algorithm for Euclidean ( k, (cid:96) ) - clustering called D (cid:96) -sampling, which has approximation ratio O (2 (cid:96) log k ) [4].
This algorithm starts with a uniformly sampled point in the input X , and then iteratively adds points to the solution Z : at each step, a point x ∈ X is sampled with probability proportional to cost (cid:96)Z ( x ) (See Algorithm 1 for details).
Throughout, we use the symbol n to denote the number of points in the input set X .
Our goal is to show the following:
Theorem 2.1.
The average sensitivity of D (cid:96) - SAMPLING ( X, k, (cid:96) ) is O
( k/n ) .
probability
X x ∈ X is sampled We start by analyzing the average sensitivity of the process of sampling from D X,Z :
Lemma 2.2.
The average sensitivity of the process of sampling from D X,Z is O
(1 /n ) .
Proof.
For notational simplicity, we use p ( x ) and p ( i ) ( x ) to denote p X,Z ( x ) and p X ( i ) ,Z ( x ) , respectively (Recall that X ( i ) = { x 1 , . . .
, x i − 1 , x i +1 , . . .
, x n } ).
Note that p ( x ) = cost (cid:96)Z ( x ) / cost (cid:96)Z ( X ) .
We start by analyzing the change in the sampling probability p ( x ) caused by deleting a point from X .
For any i ∈ [ n ] and x ∈ X ( i ) , we have
[Equation]
where the last equality follows from cost [Equation] −
[Equation] ( x i ) .
Then, we have n [Equation]
=
[Equation]
Noting that x i does not exist in X ( i ) , the average sensitivity of the process of sampling from D X,
Z is
[Equation]
We next analyze the average sensitivity of D (cid:96) - SAMPLING and prove Theorem 2.1.
Proof of Theorem 2.1.
Let z 1 , . .
z k ∈
X be the sequence of points added to Z .
Note that they are random variables.
We use z ( i ) j to denote z j for the input X ( i ) .
First, we have n
[Equation]
n + n (cid:19) =
O (cid:18)n (cid:19) .
For j ∈ [ k ] and q 1 , . . .
, q k − 1 ∈ X , we denote by { z j | z 1 = q 1 , . . .
, z j − 1 =
q j − 1 } the random variable z j conditioned on z 1 = q 1 , . .
z j − 1 =
q j − 1 .
We deﬁne z ( i ) 1 , . . .
, z ( i ) k
and { z ( i ) j | z
( i ) 1 = q 1 , . . .
, z ( i ) j − 1 = q
j − 1 } similarly using X ( i ) .
Then by Lemma 2.2, we have n
[Equation]
(3) for every j ∈ { 2 , . . .
, k } and q 1 , . .
, q k − 1 ∈ X .
Given (2) and (3), by induction, we obtain n
[Equation]
General transformation
In this section, we show that any approximation algorithm for Euclidean ( k, (cid:96) ) -clustering can be converted into an algorithm with low average sensitivity.
Speciﬁcally, we show the following:
Theorem 3.1.
Let A be an α -approximation algorithm for Euclidean ( k, (cid:96) )
-clustering with time complexity T ( n, d, k, (cid:96) ) , where n is the number of points and d is the dimension of the Euclidean space.
Then for any (cid:15), δ > 0 , there exists an algorithm for Euclidean ( k, (cid:96) ) -clustering such that
• it outputs (1 + (cid:15) )
α -approximation with probability at least 1 − δ , • its average sensitivity is
[Equation]
where (cid:101) O ( · ) suppresses a polylogarithmic factor in k and 1 /(cid:15) , and • it runs in poly( n, d, k ) + T (poly( d, k, 1 /(cid:15), log(1 /δ )) , d, k, (cid:96) )
For example, by plugging in a PTAS to Theorem 3.1 [13], we obtain a PTAS with the same average sensitivity as in the theorem.
Our transformation constructs a coreset for the input, which is a small subset that approximately preserves the objective function, and then apply the given algorithm.
We ﬁrst describe known results about coresets in Section 3.1, and then provide an algorithm that constructs coresets with low average sensitivity in Section 3.2.
Finally, we prove Theorem 3.1 in Section 3.3.
3.1 Preliminaries
A weighted set is a pair of a set X and a weight function w : X → R + .
We say that ( X, w ) is a weighted subset of Y if X ⊆ Y .
For a weighted set ( X, w ) and a set of points Z , we deﬁne cost (cid:96)Z (( X, w )) as (cid:80) x ∈ X w ( x )cost (cid:96)Z ( x ) .
Note that cost (cid:96)Z (( X, w ))
= cost (cid:96) Z ( X ) when w ≡ 1 .
A coreset for Euclidean ( k, (cid:96) ) -clustering is deﬁned as follows:
Deﬁnition 3.2 (Coreset for Euclidean ( k, (cid:96) ) -clustering [2, 11]) .
Let X be a set of points in R d .
For (cid:15) > 0 , we say that a weighted subset ( C, w ) of X is an (cid:15) -coreset of X if for any set Z ⊆ X of k points, we have
[Equation]
Importance sampling is a popular approach to obtain a small coreset.
Given a set of points X in R d , an importance function s : X → R + , and a positive integer m > 0 , we sample a point x ∈ X with probability proportional to s ( x ) and then set its weight to be the reciprocal of the sampling probability, and then we repeat this process m times to construct a coreset.
For technical reasons, we slightly perturb the weight of the points.
See Algorithm 2 for details.
A popular choice for the importance is the sensitivity with respect to the input.
In the context of
Euclidean ( k, (cid:96) ) -clustering, for a set X of points in R d , the sensitivity of x ∈ X is deﬁned to be
[Equation]
Indeed, any upper bound on sensitivity can be used to construct a coreset:
Theorem 3.3 (Theorem 5.5 in [5], restated.
See also [13]) .
Let X be a set of points in R d and s : X → R + be an arbitrary upper bound on σ X .
Let ( C, w ) =
C ORESET ( X, k, (cid:96), s, m ) with
[Equation]
where c > 0 is some universal constant and S
=
(cid:80) x ∈
X s ( x ) .
Then, with probability at least 1 − δ , ( C, w ) is an (cid:15) -coreset of X .
We note that the slight modiﬁcation to the assigned weights preserves the objective value to within the factor of 1 / (1 + (cid:15)/ 2) , and hence the original analysis of Theorem 3.3 goes through.
The reason that we consider upper bounds on sensitivity in Theorem 3.3 is that no algorithm for exactly computing sensitivity is known.
The following lemma shows that any approximate solution can be used to obtain an upper bound.
Lemma 3.4 ([13]) .
Let X be a set of points in R d and let Z ⊆
X be an α -approximate solution to Euclidean ( k, (cid:96) ) -clustering on X .
For x ∈ X , let Z x
= { y ∈ X : ∃ z ∈ Z s.t.
(cid:107) x − z (cid:107) (cid:96) 2 = cost (cid:96) Z ( x ) , (cid:107) y − z (cid:107) (cid:96) 2 = cost (cid:96)Z ( y ) } be the set of all points in the same cluster as x .
Then, the sensitivity σ X ( x ) is bounded from above by
[Equation]
Furthermore, we have S =
(cid:80) x ∈ X s ( x ) ≤ 2 2 (cid:96) +
α 2 k .
3.2 Coreset construction with low average sensitivity
In this section, we consider constructing coresets with low average sensitivity.
Speciﬁcally, we show the following:
Lemma 3.5.
For any (cid:15), δ > 0 , there exists a polynomial-time algorithm that, given a set X of points in R d and a positive integer k and (cid:96) ≥ 1 , outputs a weighted subset ( C, w ) such that (i) ( C, w ) is an (cid:15) -coreset of X with probability at least 1 − δ , and (ii) we have
[Equation]
The average sensitivity of the algorithm is at most O ( m(cid:15)n ) .
Our algorithm is as follows.
We ﬁrst compute a set Z ⊆ X by calling D (cid:96) -
SAMPLING ( X, k, (cid:96) ).
Then, we compute the upper bounds on the sensitivities of points in X as in Lemma 3.4, which we denote by s X,Z ( x ) for x ∈ X .
Finally, we apply Theorem 3.3 using s X,Z ( x ) with the smallest m satisfying the condition in the statement, denoted m Z .
Note that s X,Z ( x ) and m Z are random variables because the output Z of D (cid:96) - SAMPLING is a random variable.
It is clear that this algorithm outputs an O ( (cid:15) ) -coreset with probability at least 1 − δ .
The proof of Lemma 3.5 is deferred to Appendix B.
3.3 Proof of Theorem 3.
Our algorithm is as follows: We ﬁrst compute an (cid:15) -coreset ( C, w ) of the input X using Lemma 3.5, and then apply the α -approximation algorithm
A given in the statement to the coreset.
The analysis of approximation ratio and average sensitivity is trivial.
We discuss the time complexity of the algorithm above.
Suppose that the algorithm A runs in
T ( n, d, k, (cid:96) ) time.
Then, we can compute the coreset ( C, w ) in poly( n, d, k ) time.
To run the algorithm A , we need T ( | C | , d, k, (cid:96) ) =
T (poly( d, k, 1 /(cid:15), log(1 /δ )) , d, k, (cid:96) ) time.
Hence, in total, we need poly( n, d, k ) + T (poly( d, k, 1 /(cid:15), log(1 /δ )) , d, k, (cid:96) ) time.
Consistent clustering in the random-order model
In this section, we show that algorithms for Euclidean ( k, (cid:96) ) -clustering with low average sensitivity can be used to design algorithms for consistent Euclidean ( k, (cid:96) ) -clustering in the random-order model.
In Section 4.1, we show that an algorithm with low average sensitivity satisfying a certain condition can be transformed into a consistent algorithm.
Then, we see in Section 4.2 and Appendix C that the algorithms discussed in Sections 2 and 3, respectively, satisfy the condition and can be transformed into consistent algorithms.
4.1 Low average sensitivity to consistency
We start with the following deﬁnition.
Deﬁnition 4.1.
Let X
= { x 1 , . . .
, x n } be a set of points in R d .
For i ∈ [ n ] , let Z and
Z ( i ) be two distributions over subsets of X and X ( i ) , respectively, of size k .
Suppose that (cid:80) ni =1 d TV ( Z ( i ) , Z ) /n ≤ β holds.
Then, we say that a probability transportation for Z of average sensitivity β is computable if the following holds:
For each i ∈ [ n ] , there is a distribution D ( i ) over pairs of sets such that its marginal distributions on the ﬁrst and second coordinates are equal to Z ( i ) and Z , respectively.
(cid:80) ni
=1 Pr ( Z ( i ) ,Z ) ∼D ( i )
[ Z ( i ) (cid:54) =
Z ] /n ≤ β . 3.
For each i ∈ [ n ] , there is a (randomized) algorithm that takes a set Z ( i ) and returns a set Z , such that if the input Z ( i ) is distributed as the distribution
Z ( i ) , then the distribution of the pair ( Z ( i ) , Z ) is equal to D ( i ) .
In particular, Z is distributed as Z .
The following gives a general transformation from algorithms with low average sensitivity to consistent algorithms where a probability transportation is computable:
Lemma 4.2.
Let A be a (randomized) algorithm for Euclidean ( k, (cid:96) ) -clustering with approximation ratio α and average sensitivity at most β ( n ) for inputs of size n .
Suppose that the probability transportation for A ( X ) of average sensitivity β ( n ) is computable for any set X of n points.
Then, there is an algorithm for consistent Euclidean ( k, (cid:96) ) -clustering in the random-order model such that
• the expected approximation ratio is α for each step, where the expectation is taken over the randomness of the algorithm, and • the expected inconsistency is at most k · (cid:80) ni
=1 β ( i ) , where the expectation is taken over both the randomness of the algorithm and the arrival order of the input.
Proof.
Let X
= { x 1 , . . .
, x n } be a set of points in R d , and let σ be a random permutation of [ n ] .
The points are given in the order x σ (1) , . . .
, x σ ( n ) .
We ﬁrst compute Z σ, 1 by running A on x σ (1) .
For t ≥ 2 , we compute Z σ,t from Z σ,t − 1 using the probability transportation for A ( { x σ (1) , . . .
, x σ ( t ) } )
given in the statement.
Let D σ,t be the joint distribution given by the probability transportation such that the marginal on the ﬁrst and second coordinates are Z σ,t and Z σ,t − 1 , respectively.
Our output sequence is Z σ, 1 , . . .
, Z σ,n .
We observe that the expected approximation ratio is α because the distribution of Z σ,t is the same as the output distribution of A ( { x σ (1) , . . .
, x σ ( t ) } ) .
The expected number of times that the output changes is bounded as
[Equation]
t =
E σ (cid:20)
Pr ( Z,Z (cid:48) ) ∼D σ,t [ Z (cid:54) = Z (cid:48) ] (cid:21) = n (cid:88) t =  n n (cid:88) i =
E σ (cid:20)
Pr ( Z,Z (cid:48) )
∼D σ,t [ Z (cid:54) =
Z (cid:48) ] | σ t =
i (cid:21) ≤ n (cid:88) t = β ( t ) .
As we have E | Z σ,t − 1 (cid:52) Z σ,t | ≤ k · Pr[ Z σ,t − 1 (cid:54) =
Z σ,t ] , the claim holds.
4.2 Consistent D (cid:96) -sampling
In this section, we propose a consistent algorithm that is a slight modiﬁcation of D (cid:96) - SAMPLING , and show the following:
Theorem 4.3.
There exists a polynomial-time O (log k )
-approximation algorithm for consistent
Euclidean ( k, (cid:96) ) -clustering in the random-order model with inconsistency O ( k 2 log n ) .
For a probability distribution Z , we say that we have sampling access to Z if we can sample a value from Z .
For z in the support of Z , let Z ( z ) denote the probability that z is sampled from Z .
Then, we say that we have query access to Z if we can obtain the value of Z ( z ) for a speciﬁed z .
We use the following lemma.
Lemma 4.4 ([14]) .
Let Z , Z (cid:48) be probability distributions, and suppose that we have sample and query accesses to Z and Z (cid:48) .
Then, there is an algorithm called L AZY S
AMPLING that, given z sampled from Z , outputs z (cid:48) such that
• the distribution of z (cid:48) is equal to Z (cid:48) , • Pr z,z (cid:48)
[ z (cid:54) = z (cid:48) ] ≤ d TV ( Z , Z (cid:48) ) , • the expected number of samples drawn from Z and Z (cid:48) is O (1) .
• the expected number of queries to Z and Z (cid:48) is O (1) .
The following lemma provides the probability transportation for D (cid:96) - SAMPLING .
Lemma 4.5.
The probability transportation for D (cid:96) - SAMPLING ( X, k, (cid:96) ) with average sensitivity
O ( k/n ) is computable.
Proof.
Fix i ∈ [ n ] .
We provide an algorithm that takes a set Z ( i ) =
{ z ( i ) 1 , . . .
, z ( i ) k } of k points distributed as D (cid:96) - SAMPLING ( X ( i ) , k, (cid:96) ) and outputs a set Z = { z 1 , . . .
, z k } of k points such that Z is distributed as D (cid:96) - SAMPLING ( X, k, (cid:96) ) and the probability that Z ( i ) (cid:54)
= Z is at most d TV ( D (cid:96) - SAMPLING ( X ( i ) , k, (cid:96) ) , D (cid:96) -
SAMPLING ( X, k, (cid:96) )) .
We ﬁrst set z 1 =
L AZY S
AMPLING ( z ( i ) 1 , U X ( i ) , U X ) , where U X ( i ) and U X are the uniform distributions over X ( i − 1) and X , respectively.
If z 1 (cid:54) = z ( i ) 1 , then we compute z 2 , . .
z k by following the process of D (cid:96) - SAMPLING ( X, k, (cid:96) ) conditioned on having chosen z 1 as the ﬁrst point.
If z 1 =
z
( i ) 1 , then we compute z 2 =
L AZY S
AMPLING ( z ( i ) 2 , D X ( i ) , { z 1 } , D X, { z 1 } ) (see Algorithm 1 for the deﬁnition of D (cid:96) -sampling distribution D X,Z ).
We repeat this process until we compute z k .
We can observe that the distribution of Z is equal to that of A ( X ) and that the total variation distance between Z ( i ) and Z is at most that between D (cid:96) - SAMPLING ( X ( i ) , k, (cid:96) ) and D (cid:96) - SAMPLING ( X, k, (cid:96) ) .
Hence, the claim holds.
Proof of Theorem 4.3.
Consider an algorithm obtained by combining Lemmas 4.2 and 4.5.
It is clearly an O (log k ) -approximation algorithm, and inconsistency is k · (cid:80) nt
=1 O ( k/t )
=
O ( k 2 log n ) .
Dynamic clustering in the random-order model
In this section, we consider algorithms for dynamic Euclidean ( k, (cid:96) ) -clustering in the random-order model.
Our algorithms are obtained by carefully implementing the consistent algorithms discussed in Section 4 so that the amortized update time is small.
We provide efﬁcient implementations for algorithms discussed in Section 4.2 in Section 5.1.
We also show that the consistent transformation discussed in C can be efﬁciently implemented in D.
5.1 Dynamic D (cid:96) -sampling
In this section, we show the following:
Theorem 5.1.
There exists an O (2 (cid:96) log k ) -approximation algorithm for dynamic Euclidean ( k, (cid:96) ) - clustering in the random-order model with amortized update time
O ( dk + ( k + log n ) k log n ) .
Proof.
Let x 1 , . . .
, x n ∈ R
d be the input points (that arrive in this order).
Throughout the process, we maintain points z 1 , . . .
, z k and sequences S 1 , . .
, S k of real values.
When x t ( t ∈ [ n ]) arrive, for each j ∈ [ k ] , we update z j so that its distribution is equal to that of the j -th point selected in D (cid:96) - SAMPLING ( X t , k, (cid:96) ), where X t = { x 1 , . . .
, x t } .
Also, we update S j to be the sequence cost (cid:96)Z j − 1 ( x 1 ) , . .
, cost (cid:96)Z j − 1 ( x i ) , where Z j = { z 1 , . . . ,
z j } .
When j = 1 , we regard that cost (cid:96)Z j − 1 ( x i )
= 1 for every i ∈ [ t ] .
For each j ∈ [ k ] , we maintain a binary tree on S j so that we can append a value and compute the sum of values in an arbitrary (consecutive) subsequence of S j in O (log n ) time.
We can support those queries using, say, a segment tree.
Note that by querying consecutive sums on S j , we can compute p X t ,Z j − 1 ( x ) (see Algorithm 1 for the deﬁnition), which is required to simulate D (cid:96) - SAMPLING in O (log n ) time.
Also, we can draw a sample from the distribution D X t ,Z j − 1 in O (log 2 n ) time using the consecutive sum queries (e.g., draw a number uniformly from [0 , cost (cid:96)Z j − 1 ( X t )] and then perform binary search to ﬁnd the position that the number belongs to).
Note that for j = 1 , the distribution Z t, 1 is just a uniform distribution over X t .
We initialize z 1 , . . .
, z k and S 1 , . .
, S k using the ﬁrst point x 1 .
That is, z 1 = · · · = z k = x 1 , which is the k points selected by D (cid:96) - SAMPLING ( { x 1 } , k, (cid:96) ).
Then for each j ∈ [ k ] , we initialize S j to be the sequence consisting of a single value cost (cid:96)Z j − 1 ( x 1 ) / cost (cid:96)Z j − 1 ( X 1 )
= 1 .
Below, we explain how we update z 1 , . . .
, z k and S 1 , . .
, S k when x i for t > 1 arrives.
When updating z j using L AZY S
AMPLING as in Section 4.2, we need sampling and query accesses to Z t − 1 ,j and Z t,j .
We can provide sampling and query accesses to the former by using the current binary tree on S j , and those to the latter by using the binary tree obtained by appending x
t
(We can access both by keeping the old one at this step).
If z j is updated to a different point, we rebuild binary trees on
S j , . .
, S k from scratch.
Now we analyze the amortized update time.
At each step, for each j ∈ [ k ] , we need O ( dk ) time to compute cost (cid:96)Z 1 ( x t ) , . . .
, cost (cid:96)Z k ( x t ) , and O ( k log t ) time to append x t to the binary trees on
S 1 , . . .
, S k .
Then, we need O ( k log 2 n ) time to update z 1 , . . .
, z k .
When z j is replaced, we rebuild binary trees on S j , . .
, S k , which takes O ( kt log t ) time.
Recalling that the probability of replacing some of z j ’s at t -th step is O ( k/t ) , we need n (cid:88) t =
( O ( dk ) + O ( k log t )) + O ( k log 2 n ) · n + n (cid:88) t = O ( kt log t ) · O (cid:18) k t (cid:19) =
O ( dkn +( k +log n ) kn log n ) time in total. Hence the amortized update time is O ( dk + ( k + log n ) k log n ) .
Conclusion
We have shown that the average sensitivity of D (cid:96) - SAMPLING is small and that any approximation algorithm can be transformed to one with low average sensitivity.
Then, we show that these algorithms can be used to obtain consistent and dynamic algorithms in the random-order model.
We believe consistent and dynamic algorithms for other problems can also be obtained via average sensitivity analysis.
It is an interesting open question whether we can remove the dependency on d from the sensitivity bound in Theorem 3.1. A natural idea is to apply dimension reduction, e.g., [19]. However, it is not clear whether we can recover a solution for the original space from that for the reduced space with a small average sensitivity.
Another natural idea is to use coresets of size independent of d , e.g., [6, 13]. A challenge here is that their constructions are more complicated than the importance sampling used in this work, and their average sensitivity might be high. Acknowledgments Y.Y. is Supported by JST, PRESTO Grant Number JPMJPR192B.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
In this section, we show the following lower bound: Theorem A.1. Any algorithm for Euclidean ( k, (cid:96) )
-clustering with a ﬁnite approximation ratio has average sensitivity Ω( k/n ) . We note that, for algorithms that select k centroids only from the input X (and not from R d \ X ), there is a trivial lower bound of Ω( k/n ) because when one of the centroids is deleted, which happens with probability Ω( k/n ) , the algorithm must change its output. Theorem A.1 shows that the same lower bound applies even for algorithms that may select centroids from R d \ X . Proof of Theorem A.1.
Let A be an algorithm with a ﬁnite approximation ratio.
Let X = { x 1 , . . .
, x n } be a set of points in R n such that x 1 , . . .
{+, x k +1 are all distinct and x k +
+}
= x k +2 = · · · = x n .
Then for any X ( i ) with 1 ≤ i ≤ k , the set Z i := { x 1 , . . .
, x i − 1 , x i +1 , . . .
, x k +1 } is the unique optimal solution, which gives the objective value zero.
Hence to have a ﬁnite approximation ratio, the algorithm A must output Z i on X ( i ) .
Let p i be the probability that the algorithm A outputs
Z i on X .
Then, the average sensitivity of A on X is
[Equation]
B Proof of Lemma 3.
The following useful lemma is implicit in the proof of Lemma 2.3 of [15].
Lemma B.1.
For (cid:15), B, B (cid:48) > 0 , let X and X (cid:48) be sampled from the uniform distributions over [ B, (1 + (cid:15) ) B ] and [ B (cid:48) ,
(1 + (cid:15) ) B (cid:48) ] , respectively.
Then, we have
[Equation]
Proof of Lemma 3.5.
We now analyze the size of the coreset.
As we mentioned, the approximation ratio of D (cid:96) -
SAMPLING is O
(2 (cid:96) log k ) .
Also, we have E (cid:80) x ∈
X s X,Z ( x ) ≤ 2 2 (cid:96) +3 O (log 2 k )
k
=
O (2 2 (cid:96) k log 2 k ) by Lemma 3.4.
Hence by the choice of m Z , the size of C is at most
[Equation]
Next, we analyze the average sensitivity.
Let X
= { x 1 , . . .
, x n } .
Let Z and Z ( i ) be the outputs of D (cid:96) -
SAMPLING on X and X ( i ) , respectively.
Then by Theorem 2.1, we have (1 /n )
(cid:80) ni
=1 d TV ( Z, Z ( i ) ) =
O ( k/n ) .
Let ( C, w ) and ( C ( i ) , w ( i ) ) be the coresets constructed for X and X ( i ) , respectively.
We have
[Equation]
Now, we bound the second term.
Let p ( x ) and p ( i ) ( x ) denote the probability of sampling x from X and X ( i ) , respectively, in (one iteration of) C ORESET .
Conditioned on that Z = Z ( i ) =
˜ Z , we have
[Equation]
[Equation]
Then, we have
[Equation]
Hence, the second term of (6) is O
( E m Z /n ) .
Now we bound the third term of (6).
By Lemma B.1, it can be bounded by
[Equation]
where the last equality is by (7).
By combining above, the average sensitivity of the algorithm is given as
[Equation]
By combining the above and (5), the claim follows.
C Consistent transformation
In this section, we show that the general transformation discussed in Section 3 can be used to design consistent algorithms in the random-order model.
To this end, we ﬁrst prove the following.
Lemma C.1.
Let A be the algorithm of Lemma 3.5.
Then, the probability transportation for A with average sensitivity as in Lemma 3.5 is computable.
Proof.
Let us ﬁx a set X of n points in R d and i ∈ [ n ] .
Then, given a coreset ( C ( i ) , w ( i ) ) for X ( i ) , we need to compute a coreset ( C, w ) for X .
We apply the probability transportation used in the proof of Theorem 4.3 to compute a set Z of k points for X from a set Z ( i ) of k points for X ( i ) .
If Z (cid:54) =
Z ( i ) , then we compute the coreset ( C, w ) by running C ORESET .
If Z = Z ( i ) , then we recompute points (and weights) added to C by applying L AZY S
AMPLING on each point in C ( i ) .
This provides a probability transportation, and we can observe that all the conditions of Deﬁnition 4.1 are satisﬁed.
Theorem C.2.
Let A be an α -approximation algorithm for Euclidean ( k, (cid:96) )
-clustering.
Then for any (cid:15), δ > 0 , there exists an algorithm for consistent Euclidean ( k, (cid:96) )
-clustering in the random-order model such that (i) it outputs (1 + (cid:15) )
α -approximation with probability at least 1 − δ at each step, and (ii) its inconsistency is
[Equation]
Proof.
We combine Lemma 4.2 and Lemma C.1.
The approximation guarantee is clearly satisﬁed.
The inconsistency of the algorithm is k · (cid:80) nt =1 O ( E | C | /(cid:15)t )
=
k log n · O ( E | C | /(cid:15) ) , and hence the claim holds.
D Dynamic transformation
We show that the consistent transformation discussed in Section C can be implemented in such a way that the amortized update time in the random-order model is small.
Speciﬁcally, we show the following:
Theorem D.1.
Let A be an α -approximation algorithm for Euclidean ( k, (cid:96) )
-clustering with time complexity T ( n, d, k, (cid:96) ) .
Then for any (cid:15), δ > 0 , there exists an algorithm for dynamic Euclidean ( k, (cid:96) )
-clustering in the random-order model that (i) outputs (1 + (cid:15) )
α -approximation with probability at least 1 − δ , and (ii) its amortized update time is
[Equation]
where m = (cid:101) O (cid:16) 2 2 (cid:96) k
(cid:15) 2 (cid:0) dk(cid:96)
+ log 1 δ (cid:1)(cid:17) .
Proof.
The consistent transformation has two components, that is, D (cid:96) - SAMPLING and coreset construction.
We use the dynamic algorithm of Theorem 5.1 to run the D (cid:96) -
SAMPLING part and hence the amortized update time of this part is O
( dk + ( k + log n ) k log n ) .
For the coreset construction part, we maintain a coreset ( C, w ) and a sequence S storing s ( x 1 ) , . . .
, s ( x t ) , where s ( x ) is the upper bound on the sensitivity of x as in the proof of Lemma 3.5.
We maintain a binary tree on S as with dynamic version of D (cid:96) - SAMPLING .
When the output of
D (cid:96) -
SAMPLING changes after x t arrives, we recompute ( C, w ) and the sequence S from scratch.
When the output of D (cid:96) - SAMPLING does not change, we append s ( x t ) to S , and then update the coreset ( C, w ) using L AZY S
AMPLING .
Now we analyze the amortized update time of the coreset construction part.
At each step we need
O ( | C | log n ) time to update ( C, w ) .
Also, when the output of D (cid:96) -sampling changes, we need additional O ( t log t ) time to reconstruct a binary tree over S .
Finally, when ( C, w ) is updated, we need to recompute an optimal solution for C , which takes T ( | C | , d, k, (cid:96) ) time.
Recalling that | C | ≤ m by Lemma 3.5, in expectation, the total computational time is bounded as
[Equation]
Combined with the amortized time of dynamic D (cid:96) - SAMPLING , the claim holds.