Near-Optimal Private and Scalable k -Clustering
Abstract
We study the differentially private (DP) k -means and k -median clustering problems of n points in d -dimensional Euclidean space in the massively parallel computation (MPC) model.
We provide two near-optimal algorithms where the near-optimality is in three aspects: they both achieve (1).
O (1) parallel computation rounds, (2).
near-linear in n and polynomial in k total computational work (i.e., near-linear running time when n is a sufﬁcient polynomial in k ), (3).
O (1) relative approximation and poly ( k, d ) additive error.
Note that Ω(1) relative approximation is provably necessary even for any polynomial-time non-private algorithm, and Ω( k ) additive error is a provable lower bound for any polynomial-time DP k -means/median algorithm.
Our two algorithms provide a tradeoff between the relative approximation and the additive error: the ﬁrst has O (1) relative approximation and ∼ ( k 2 .
5 + k 1 .
√ d ) additive error, and the second one achieves (1 + γ ) relative approximation to the optimal non-private algorithm for an arbitrary small constant γ > 0 and with poly ( k, d ) additive error for a larger polynomial dependence on k and d .
To achieve our result, we develop a general framework which partitions the data and reduces the DP clustering problem for the entire dataset to the DP clustering problem for each part.
To control the blow-up of the additive error introduced by each part, we develop a novel charging argument which might be of independent interest.
Introduction
Over the last decade, the leakage of private information by machine learning and data mining algorithms has had dramatic consequences, from losses of billions of dollars [60] to even costing human lives [8].
Thus, protecting data privacy has become a top priority constraint in many modern machine learning and data mining problems.
This high demand has stimulated an important research effort to design algorithmic techniques enabling privacy-preserving algorithms.
In recent years, the elegant notion of differential privacy (DP) [34] has become the gold standard for privacy-preserving algorithms [38, 67, 33, 1].
Informally, differential privacy requires the output (distribution) of the algorithm to remain almost the same under a small adversarial perturbation of the input.
k -Means and k -median clustering are fundamental and widely-studied problems in unsupervised learning.
They are used to analyze and extract information from massive datasets in machine learning and data mining tasks.
In particular, given a set of n points X ⊆ R d within a ball of radius Λ , the goal is to ﬁnd a set C of k centers such that the clustering cost (cid:80) x ∈X min c ∈C d p ( x, c ) is minimized,
[Table]
Conference on Neural [Table] Processing Systems (NeurIPS 2022)
[Table]
where the power p = 1 , 2 stands for k -median and k -means respectively.
The importance of data privacy has sparked an important research effort on designing accurate and efﬁcient differentially private k -means and k -median algorithms [15, 62, 39, 47, 58, 73, 64, 65, 71, 42, 5, 63, 52, 70, 69, 44, 53, 19, 61, 20, 14, 24].
In the above line of work, one can distinguish two separate efforts: one effort emphasizing the approximation guarantees, namely aiming at best possible approximation guarantee (under privacy constraints) in polynomial time, and a second effort targeting practical and efﬁcient differentially private algorithms.
More concretely, none of the above works achieve optimal accuracy and efﬁciency at the same time, i.e., the algorithms achieving O (1) relative approximation and Λ p · poly( k, log( n )) additive error have running time at least n 1+Ω(1) ·
d (see e.g., [70, 44]), and the algorithms with running time proportional to ˜ O ( nd ) 3 and polynomial in the number of centers, k , have either relative approximation Ω(log n ) or additive error Λ p · n Ω(1) (see e.g., [24, 14]).
In this work, we reconcile these two lines of work and present two algorithms that both achieve near optimal approximation guarantee and near optimal running time simutaneously.
Our two algorithms provide a tradeoff between the relative approximation and the additive error.
In particular, the ﬁrst algorithm runs in ˜ O ( nd ) + poly( k ) time and outputs a solution with O (1) relative approximation and roughly ( k 2 .
5 + k 1 . 01 √ d ) · Λ p additive error (we ignore minor dependencies on log n and the privacy parameters ε, δ in the additive error).
The second algorithm runs in ˜ O
( ndk )
+ poly( k ) time and outputs a solution with (1 + γ ) ρ relative approximation and roughly poly( k, d ) · Λ p additive error for a larger polynomial dependence on k, d , where γ > 0 is an arbitrarily small constant and ρ is the best relative approximation of any polynomial-time non-private algorithm.
Our approximation guarantee is near optimal since the Ω(1) relative approximation is necessary even for any non-private polynomial-time algorithm [28, 27, 26, 55] and Ω( k · Λ p ) is a provable lower bound for additive error achieved by any polynomial-time DP algorithm [47].
In addition, the approximation guarantees of our algorithms match the best previous DP algorithms [70, 44] while our algorithms have faster running time.
In fact, our running time is almost tight when k
= n o (1) because Ω( nd ) time is necessary to read all input points.
Importantly, our algorithms are scalable .
In the era of massive datasets, in-memory sequential algorithms struggle to handle billions of data points.
Algorithms that are suitable for large-scale distributed/parallel computational systems such as MapReduce [32] are more desired.
It motivates the study of algorithms in the massively parallel computation (MPC) model [54, 46, 11].
In the MPC model, there are multiple machines where each has local memory that is sublinear in the data size.
The computation proceeds in rounds.
In each round, each machine sends/receives messages to/from other machines but the size of messages sent/received by a machine cannot exceed its local memory.
As long as the local memory per machine is at least polynomial in k and at least n θ for a small constant θ , our algorithms can be easily implemented in the MPC model with O (1) computation rounds and total space (total memory size across all machines) linear in the input data size.
Note that Ω( k ) local memory is needed for all previous non-private o (log n ) -round MPC O
(1) -approximate k -means and k -median clustering algorithms (see e.g., [36, 40, 4, 6, 10, 18, 41]).
1.1 Other Related Work
k -Means and k -median clustering have seen a large body of work over the past few decades.
There have been numerous works obtaining O (1) -approximation algorithms for both k -means and k -median, using either local search or primal-dual techniques.
The state-of-the-art approximation algorithms are a 5 .
912 -approximation for Euclidean k -means and a 2 .
406 -approximation for Euclidean kmedian, due to [25].
Two other fruitful methods of improving algorithms for these problems are coresets [21, 40, 51, 31, 29], which replace the dataset of points with a smaller set, and dimensionality reduction [22, 56], which reduces the number of dimensions of the ambient Euclidean space the points reside in.
There is a line of work studying k -means and k -median problems under parallel, distributed and streaming computational models.
A popular way to tackle these clustering problems at scale is via coresets.
A coreset is a small weighted subset of input points such that a good clustering solution for coreset yields a good approximate clustering to the original input point set.
k -Means and k -median coresets have been extensively studied in the literature (e.g, [40, 6, 10, 18, 16, 41, 17]), and most of them can be implemented in the MPC model in O (1) rounds as long as the memory per machine is at least poly( k ) , i.e., is large enough to hold the entire coreset.
Other approaches to solve k -means and k -median in parallel are via quad-tree [13] or locality sensitive hashing [12].
These data partitioning based approaches require less local memory size but provide worse approximations or output more than k centers (and so pertain to the line of work sacriﬁcing approximation guarantees to practicality and efﬁciency).
Notice that none of the previous scalable algorithms are differentially private.
Our paper focuses on differentially private k -means and k -median clustering in Euclidean space, which, as described previously, has seen signiﬁcant work over the past several years.
The paper [70] was the ﬁrst paper to provide a differentially private algorithm for k -means clustering with O (1) multiplicative ratio, and additive error polynomial in k, d, log n, ε − 1 , and log δ − 1 .
Later, [44] improved their result by obtaining an algorithm with multiplicative ratio arbitrarily close to the best non-private k -means (or k -median) approximation, at the cost of a much larger polynomial dependence on k in the additive error.
The paper [24] was the ﬁrst paper to study private clustering in the Massively Parallel Computing framework, obtaining a polylogarithmic multiplicative ratio for k -means and k -median with polylogarithmic rounds of communication and computation.
We also remark that [63, 70, 69, 19, 20] also provided private algorithms for k -means clustering in the local differential privacy model, but these algorithms all have additive errors proportional to at least √ n .
1.2 Our Results
In this paper, we provide nearly optimal algorithms for differentially private k -means and k -median, in the MPC setting with O (1) total rounds of communication and computation.
We gave a brief, informal description of our results previously.
We now formally state the two main theorems that we will prove about private clustering in the MPC model.
In our MPC setting, we assume that we have n 1 − θ machines, each of which can store ˜ O ( n θ ) points in R d , where θ > 0 can be an arbitrarily small constant.
(Equivalently, each machine has ˜ O ( n θ · d ) space and the total amount of space is ˜ O ( n · d ) ).
Theorem 1.1.
(Theorem E.4.)
There exists an ( ε, δ ) -DP algorithm for k -means (or k -median) clustering with multiplicative error O (1) and additive error ( k 2 .
5 + k 1 . 01 √ d ) · poly (cid:0) log n, ε − 1 , log δ − 1 (cid:1) .
In addition, assuming each machine can store ˜ O ( n θ ) ≥ ( k 1 .
5 + d 0 . 5 ) · poly(log n, ε − 1 , log δ − 1 ) points, the algorithm can be implemented in MPC with O (1) total rounds of communication and computation, total sequential running time ˜ O ( nd ) + poly( k, ε − 1 , log δ − 1 ) , and total time per machine ˜ O
( n θ d ) + poly( k, ε − 1 , log δ − 1 ) .
Theorem 1.2.
(Theorem E.9)
Suppose that there exists a polynomial-time algorithm that can compute a ρ -approximation to k -means (resp., k -median).
Then, for any constant ρ (cid:48) > ρ , there exists an ( ε, δ ) -DP algorithm for k -means (resp., k -median) with multiplicative error ρ (cid:48) and additive error poly (cid:0) k, d, log n, ε − 1 , log δ − 1 (cid:1) .
In addition, the algorithm can be implemented in MPC with O (1) total rounds of communication and computation, total sequential time ˜ O ( nd ) , and total time per machine ˜
O
( n θ d ) , assuming each machine can store ˜ O ( n θ ) ≥ poly( k, d, log n, ε − 1 , log δ − 1 ) points.
Our two results are similar to the results of [70] and [44], respectively, except that our algorithms are implementable in MPC with O (1) rounds, and our algorithm has near-linear dependence on the size of the dataset n (whereas [70] and [44] have polynomial dependence on n ).
We also remark that our additive error for the ﬁrst algorithm is slightly worse than that of [70] (which had k 1 . 5 + k 1 . 01 d 0 . 51 ), and our additive error for the second algorithm matches that of [44].
Our results improve over the best MPC algorithm for private k -means and k -median [24], as we have an O (1) -approximation whereas their paper had a poly log n -approximation.
In addition, our result only needs O (1) rounds of communication between machines, whereas they require log n rounds for k -median and poly log n rounds for k -means.
1.3 Technical Overview and Roadmap
In this subsection, we assume for simplicity that we are dealing with k -means.
The overall outline for k -median is quite similar.
Some natural approaches and why they don’t work.
A ﬁrst natural approach is to apply a uniform sampling approach.
In other words, we can consider sampling n θ points at random, moving them to a single machine, and then performing a sequential private clustering algorithm on this machine.
This idea of uniform sampling for private clustering was recently used by [14].
While they proved that this method ensures differential privacy, their additive error is quite large: they prove that with T samples one can obtain total additive error of roughly n/ √ T · Λ 2 .
Even if we sampled n/ 2 points at random, this equals √ n · Λ 2 , whereas we wish for a much smaller additive error of poly( k ) · Λ 2 .
One can also construct examples, even when k = 2 , where random sampling must induce large additive error.
For instance, if we co-locate n − √ n points at the origin and co-locate √ n points at a point Λ away from the origin, the optimal cost for 2 -means is 0 , but a random sample of √ n points, with constant probability, consists only of points located at the origin.
Therefore, any algorithm applied to the sampled points will completely fail to recognize the second cluster, and will incur an Ω( √ n · Λ 2 ) error.
Another tempting approach to obtain an efﬁcient and scalable DP algorithm with optimal approximation guarantees is to implement the O (1) -approximate private algorithms of [70] or [44] in the MPC setting.
Unfortunately, both of these algorithms have an intrinsic sequential behavior that makes this approach difﬁcult.
The algorithm of [70] relies on a private local search procedure of [47], which appears very difﬁcult to implement in MPC without a number of rounds of communication of Ω( k ) .
The algorithm of [44] relies on iteratively ﬁnding and peeling off a “privately”-dense ball of points.
This method requires roughly O ( k log n ) iterations, and again appears very difﬁcult to implement in
O (1) parallel rounds.
Our approach: we describe our overall procedure, and give intuition for some of the details.
Overall procedure.
Our overall algorithm can be split into three main steps.
Reduce the dimensionality of the problem d to O (log k ) .
This will remove large dependencies on the dimension for both the runtime and additive error.
Compute in MPC a private and very efﬁcient but weak k -means (or k -median) solution: Namely a solution with an ω (1) -approximation guarantee and that may use up to O ( k log 2 n ) centers instead of just k centers.
Combine the weak but efﬁcient MPC solution with an O (1) -approximate private sequential algorithm to obtain an efﬁcient O (1) -approximate MPC algorithm.
We remark that the ﬁrst and third steps can be done using two different but related methods.
The ﬁrst method obtains an O (1) -approximation with additive error proportional to approximately k 2 .
5 + k 1 . 01 √ d (i.e., Theorem 1.1).
The second method obtains a approximation factor within an arbitrary factor γ of even the best known non-private and sequential algorithm, but with additive error proportional to approximately k ˜ O (1 /γ 2 ) (i.e., Theorem 1.2).
We discuss the second step in Section 3, the ﬁrst method of doing the third step in Section 4, the second method of doing the third step in
Section 5, and both ways of doing the ﬁrst step in Section 6.
We now just focus on a key idea for the third step, and will give more algorithmic description and intuition in the later sections of this paper.
All formal proofs are deferred to the appendix.
Charging Additive Error to Multiplicative Error.
As previously mentioned, a direct sampling approach fails because it incurs a too large additive error.
However, we show that we can still use sampling if we sort the points appropriately and do not sample each point with the same probability.
The ﬁrst idea we use is inspired by Chen [21], which is to start with our weak MPC centers, and map each point x to a bucket ( j, r ) , where j represents the closest of the weak centers to x , and r approximates the distance from x to this center.
Our next idea is to show that if we apply a private clustering procedure on a random sample of a ﬁxed size in each bucket, we can charge the large additive cost obtained to a small constant-factor blowup in the multiplicative cost.
The intuition for this is that if the average cost of each point is roughly R , then most points in our weak approximation are still roughly within O ( R ) of the weak centers, and therefore get mapped to buckets represented by much smaller balls.
So, we can then apply a private sequential algorithm on these buckets with total error depending on R 2 rather than on Λ 2 .
We will show that, by using properties of the initial weak solution, the large additive error can be replaced with a small multiplicative error, even if the sample size T for each bucket is small.
To explain the above intuition further, in our analysis we apply a Chernoff bound and some properties of high-dimensional Euclidean space to show that if there are m points in a bucket all within radius
R of each other, we accumulate roughly R 2 · m/ √ T additive error, as opposed to a more naive Λ 2 · m/ √ T error (which would be obtained by [14]).
We can then evenly distribute the error between each of the m points, to obtain R 2 / √ T error per point.
We charge the error of each point to the squared distance of each point to its center in the weak approximation, which is roughly equal to R 2 .
So, when we add over all points, the error across all points is roughly 1 / √ T times the overall cost of the weak approximation.
So, if the weak solution had cost at most α times the optimal k -means cost, if the sampling size T is much bigger than α 2 then the total error is much smaller than the optimal cost.
Hence, we are able to charge the total additive error into a small multiplicative error!
Preliminaries
We present some basic deﬁnitions and setup that will be sufﬁcient for explaining our algorithms for the main body of the paper.
We defer the full set of preliminaries to Appendix A.
2.1 Differential Privacy
Deﬁnition 2.1 ([34]) .
A (randomized) algorithm
A is said to be ( ε, δ ) - differentially private ( ( ε, δ ) -
DP for short) if for any two “adjacent” datasets X and X (cid:48) and any subset S of the output space of A , we have
[Equation]
In our setting, we say that two datasets X and X (cid:48) are adjacent if we can convert X to X (cid:48) either by adding, removing, or changing a single data point.
We remark that in all of our algorithms, we will implicitly assume that ε, δ ≤ 12 .
A ubiquitous method we use to ensure privacy is the Laplace Mechanism.
Simply, this is a method where we privatize a statistic of the data by adding noise Lap( t ) to the statistic for some t > 0 , where
Lap( t ) has the PDF 12 t · e −| x | /t .
It is well-known that if f ( X ) is a statistic that never changes by more than ∆ between any two adjacent datasets X , X (cid:48) , then f ( X ) + Lap(∆ /ε ) is ( ε, 0) -DP.
2.2 The Massively Parallel Computation (MPC) Model
In the MPC model, there are multiple machines where each machine has sublinear local memory.
The input data points are distributed arbitrarily on the machines before computation starts.
The computation proceeds in rounds.
In each round, each machine reads its local data and performs computations.
At the end of each round, each machine sends/receives messages to/from other machines.
The total size of messages sent/received by a machine in each round does not exceed the local memory size.
The output is distributed on machines at the end of the computation.
The MPC algorithm with small number of rounds and small total space (total memory across all machines) is desired.
Furthermore, scalability is also considered in many applications and thus we want the local memory required by the algorithm as small as possible.
There are many scalable MPC algorithmic primitives.
The most basic one is sorting.
Theorem 2.2 ([46, 45]) .
There is an MPC algorithm which sorts N data items in O (1) rounds using
O ( N ) total space.
The local memory per machine required is at most O ( N θ ) for arbitrary small constant θ > 0 .
Based on the sorting primitive above, other basic subroutines such as indexing, preﬁx sum and set aggregation can be easily implemented in the MPC model with same complexity as sorting.
We refer readers to Appendix E of [3] for more basic MPC algorithmic primitives.
2.3 k -Means and k -Median Clustering
For two points x, y ∈ R d , we deﬁne d ( x, y ) to be the Euclidean distance between x and y .
For a set
C ⊂ R d of points, we deﬁne d ( x, C ) =
d ( C , x ) to be min c ∈C d ( x, c ) .
In both k -means and k -median clustering, we are given a dataset of points X
= { x 1 , . .
, x n } in d -dimensional Euclidean space R d .
We further assume that the points in X are in B (0 , Λ) , which is the ball of radius Λ about the origin in R d .
Our goal in k -means (resp., k -median) clustering is to ﬁnd a subset C of k -points that minimizes the cost of X with respect to C .
Speciﬁcally, for a set of centers C , we deﬁne cost ( X ; C ) :
= (cid:80) x ∈X d ( x, C ) p , where p
= 2 in the setting of k -means and p = 1 in the setting of k -median.
Occasionally, we may assign each point x i ∈
X a positive weight w i , in which case we deﬁne cost ( X ; C ) :
= (cid:80) x
i ∈X w i · d ( x i , C ) p .
We also deﬁne OPT ( X ) to be the minimum value of cost ( X ; C ) for any set of k points C .
Our goal in differentially private clustering is to produce a set of k centers C such that C is ( ε, δ ) -DP with respect to X , and such that cost ( X ; C ) ≤ α · OPT ( X ) + V · Λ p (where p = 2 for k -means and p = 1 for k -median).
If we obtain this guarantee, we say that we have an ( ε, δ ) -DP algorithm for k -means (or k -median) with multiplicative ratio α and additive error V .
In the MPC model, our goal is to make a machine output the set of k centers at the end of the algorithm.
Finally, we brieﬂy discuss coresets.
For a dataset X ⊂ B (0 , Λ) and for some γ ≤ 12 and W ≥ 0 , a ( γ, W ) - coreset is a dataset Y that estimates X with respect to k -means (or k -median) clustering, i.e., (1 − γ ) · cost ( X ; C ) − W · Λ p ≤ cost ( Y ; C ) ≤ (1 + γ ) · cost ( X ; C ) + W · Λ p for all subsets
C ⊂ B (0 , Λ) of size at most k .
We will also want the dataset Y to have much fewer distinct points than X , though each point in Y may have large multiplicity.
In the setting of non-private coresets, we usually have W = 0 , in which case we write γ -coreset to mean ( γ, 0) -coreset.
We note the following theorem regarding MPC algorithms for non-private coresets.
Theorem 2.3 ([17, 40, 41]) .
Consider n points in R d .
There exists a non-private MPC algorithm which outputs a k -means/ k -median γ -coreset with size poly( k, d, log n, 1 /γ ) in O (1) rounds for γ ∈ (0 , 0 . 5) .
The total space needed is O ( nd ) + poly( k, d, log n, 1 /γ ) , and the local memory per machine required is poly( k, d, log n, 1 /γ ) .
In addition, the total running time over all machines is at most O ( nd ) + poly( k, d, log n, 1 /γ ) .
A Preliminary Bicriteria Approximation
A key tool in all of our k -means and k -median algorithms is an initial, crude approximation for k -means (or k -median) clustering.
Speciﬁcally, we start with a bicriteria approximation, which means that the algorithm may output a list of more than k centers.
Our algorithm is very efﬁcient, only needing constant rounds of communication and computation, and essentially optimal time per machine.
Theorem 3.1.
There exists an ( ε, δ ) -DP algorithm that, given a dataset X ⊂ B (0 , Λ) of size n , computes a set of O ( k log 2 n ) points F that provides a d O (1) -approximation to the best k -means (or k -median) clustering with additive error k · poly(log n, d, ε − 1 , log δ − 1 ) .
In addition, the algorithm can be implemented in MPC with O (1) rounds of communication and computation, and ˜ O ( n θ d ) time per machine.
Our techniques for this section heavily rely on a data structure called a Quadtree.
This data structure involves creating a nested series of grids partitioning R d .
The quadtree has varying levels, where the 0 th level is a very coarse grid, and each subsequent level further reﬁnes the previous level with smaller grid pieces.
The Quadtree can be used to embed the input points into a so-called “Hierarchically Separated Tree” (HST) metric, for which computing k -median cost is often much simpler than computing k -median in Euclidean space.
Indeed, [24] uses this embedding, along with a dynamic programming approach, to provide a private approximation to k -median with multiplicative approximation
O ( d 3 / 2 log n ) and additive approximation O ( k ) · poly(log n, d, ε − 1 ) .
Unfortunately, we cannot directly use their approach for two reasons.
The ﬁrst is that their dynamic programming approach requires roughly log n rounds of communication across machines for kmedian (and poly log n rounds for k -means), whereas we wish for O (1) rounds.
The second is that their approach runs into a barrier when the memory per machine is below roughly √ n , whereas we want an MPC algorithm even if each machine can only store n θ points for an arbitrarily small θ > 0 .
Our method of developing a bicriteria approximation is inspired by the Quadtree and embeddings into this HST metric, but we avoid the issues above by using a greedy approach rather than a dynamic programming one.
Speciﬁcally, we will consider each level of the Quadtree grid separately, and map every point x ∈ X to the grid center x lies in.
We then approximately choose the O ( k ) centers that have the most points in them, using a private selection mechanism.
We will have to do this for logarithmically many levels, which causes us to have a bicriteria approximation.
To analyze this procedure, we avoid looking at the HST metric and instead consider the number of points n r that are of distance r away from some center for every choice of radius r .
We then use an integration by partsbased idea to analyze the k -means (or k -median) cost based on summing a weighted combination of n r over r , which we use to establish our accuracy bounds.
Our greedy method can be implemented in O (1) rounds of communication and computation for both k -means and k -median.
The full details of the algorithm and analysis, including pseudocode, are deferred to Appendix B.
A Constant Approximation in MPC
In this section, we describe how to combine our bicriteria approximation from Section 3, along with the sequential private O (1) -approximate algorithm from [70], to obtain a parallel O (1) -approximate algorithm using only O (1) rounds of communication and computation.
Speciﬁcally, we prove the following.
Theorem 4.1.
There exists an ( ε, δ ) -DP algorithm for k -means (or k -median) with multiplicative ratio O (1) and additive error k 2 .
5 · poly (cid:0) d, log n, ε − 1 , log δ − 1 (cid:1) .
In addition, assuming each machine stores ˜ O ( n θ ) ≥ k 1 . 5 · poly( d, log n, ε − 1 , log δ − 1 ) points, the algorithm can be implemented in
MPC with O (1) rounds of communication and computation, total sequential running time ˜ O ( nd ) + poly( k, d, ε − 1 , log δ − 1 ) , and total time per machine
˜ O
( n θ d ) + poly( k, d, ε − 1 , log δ − 1 ) .
We remark that large polynomial dependencies in d are not crucial, as we will use dimensionality reduction in Section 6 to set d = O (log k ) .
Our algorithm is inspired by the coreset framework of Chen [21], who shows how to convert a bicriteria approximation into a coreset for clustering (in the sequential, non-private setting).
The rough method is to start with an approximate set of cluster centers F of β · k points for some β > 1 , and map each point x ∈ X to its nearest neighbor in F .
Then, the points in X are split into “rings”, based on their closest point and their approximate distance to that closest point.
Finally, the points in each ring are replaced by a uniform sample of a few points in each ring, to create a small coreset.
Ordinarily, when performing DP k -means/ k -median over n points, we incur additive cost proportional to Λ p , where Λ is the diameter of the pointset.
Hence, naive application of using private k -means/ kmedian on a random subset (or even a coreset) of T points is quite bad, as we have to scale each point by a factor of n/T for the weights to match, which induces a n/T · Λ p additive error.
In reality, the induced additive error is actually worse, and behaves like n/ √ T · Λ p .
However, an important insight we have, as noted in Subsection 1.3, is that when using the framework of Chen [21], we can charge much of the additive cost to multiplicative cost.
Namely, if we perform a private k -means/ k
-median algorithm on a random sample of T points in each ring separately, if the ring had radius R (cid:28) Λ , we would roughly incur additive error proportional to R p / √ T per point, rather than Λ p / √ T .
We then use the properties of the bicriteria approximation to show that if we add this error across all rings, the additive error is actually only a small multiple of the optimal cost, as opposed to a much worse n/ √ T · Λ p .
Recall that if R i represents the approximate distance of each point x i to its bicriteria center, then (cid:80) i R pi is at most d O (1) times more than the optimal k -means cost (plus some small multiple of Λ p ), by Theorem 3.1.
Hence, if T is roughly d O (1) , we can get the overall additional error induced by each R p / √ T error per point is at most O (1) times the optimal k -means cost.
This is what allows us to get a multiplicative approximation with small additive cost.
We remark that some rings may have fewer than our desired T points, in which case we cannot sample T random points and incur an additional small additive cost.
Finally, is quite simple to sample T points in each ring in parallel, and map each sample to a separate machine.
We can then use a sequential private k -means algorithm for each sample separately, as we can store the T points on a single machine.
One additional issue is that this procedure has to ﬁnd k points for each ring, and there will end up being roughly O ( k · poly log n ) rings.
So this means we need O ( k 2 · poly log n ) centers, whereas we only want k centers.
Our way of ﬁxing this is to convert our k centers into a “semi-coreset” by mapping each point in a ring to its closest point in the corresponding private set of centers, and then adding Laplace noise to the multiplicities of the set.
These semi-coresets will be private, which means we can apply a non-private MPC k -means algorithm on the union of all the semi-coresets to generate a ﬁnal set of points, since the semi-coresets have already been privatized.
Also, while they are not true coresets, we show that any O (1) -approximate k -means algorithm applied on the union of all the semi-coresets still provides an O (1) -approximate solution for the original dataset.
We present the algorithm pseudocode in below in Algorithm 1, and defer the full analysis to Appendix
C. We remark that we replace some instances of mapping points to the nearest center with mapping them to an approximately nearest center, which is useful in speeding up runtime.
Algorithm
1: procedure C ONSTANT A PPROX ( X , ε, δ ) (cid:46) Will be ( O ( ε ) , O ( δ ))
-DP.
Let T = poly( d, log n, ε − 1 , log δ − 1 ) · k 1 .
5 be a sufﬁciently large threshold parameter.
Let F = { f 1 , . . .
, f β · k } be an ( ε, δ ) -DP bicriteria approximation for k -clustering of X , using
Theorem 3.1, and where β
= O (log 2 n ) .
4: for all x i ∈
X do 5:
Assign x i to bucket ( j, r ) if x i has f j as a O (log n ) -approximate nearest neighbor, with d ( x i , f j ) ≈ Λ2 n · 2 r .
6: for all j ≤ β · k , r ≤ O (log n ) do 7:
Let X j,r be set of points x i assigned to ( j, r ) .
Let ˆ N j,r :=
|X j,r | + Lap(1 /ε ) .
9: if ˆ N j,r ≥ 2 T then 10:
Sample T random points Y j,r from X j,r .
Send Y j,r to one machine, and use [70] on each machine to ﬁnd a k -clustering G j,r .
Replace each point in Y j,r with an O (1)
-approximate nearest neighbor in G j,r .
Add Lap(1 /ε ) noise to the multiplicity of each point in the replaced data set, to create dataset ˆ Y j,r .
14: else 15:
Let ˆ Y j,r = ∅ .
Let ˆ Y =
(cid:83) j,r ˆ Y j,r .
Return a non-private MPC k -means clustering algorithm on Y , using, e.g., [17].
An Arbitrarily Good Approximation in MPC
In this section, we present a different method of combining the bicriteria approximation from Section 3 with a sequential private O (1) -approximation from [70] to obtain a parallel approximate algorithm achieving arbitrarily close multiplicative ratio to the best non-private sequential algorithm.
While we obtain an improved multiplicative approximation ratio, we suffer a larger additive error.
Speciﬁcally, we prove the following.
Theorem 5.1.
Suppose that there exists a polynomial-time algorithm that can compute a ρapproximation to k -means (resp., k -median).
Then, for any constant ρ (cid:48) > ρ , exists an ( ε, δ ) -DP algorithm for k -means (resp., k -median) with multiplicative ratio ρ (cid:48) and additive error poly (cid:0) k, e d , log n, ε − 1 , log δ − 1 (cid:1) .
In addition, assuming each machine can store ˜ O ( n θ ) ≥ poly( k, e d , ε − 1 , log δ − 1 ) points, the algorithm is implementable in MPC with O (1) total rounds of communication and computation, and O ( n θ kd ) time per machine.
We remark that while there is an exponential dependence on the dimension d in both the runtime and additive error, in Section 6, we show how to replace the e d with poly( k ) additive error.
Our overall procedure has two steps.
The ﬁrst step is to convert an O (1) -approximate private sequential algorithm into a private sequential coreset: this is somewhat similar to Ghazi et al.
[44], which provides a similar guarantee but has a runtime dependence that is polynomial in n as opposed to near-linear in n .
The second step is to convert a private sequential coreset into a private parallel coreset.
For the ﬁrst step, we start with a private O (1) -approximate clustering, and as in Section 3, we split the dataset X into roughly O ( k log n ) rings based on each point’s closest center and the distance to that center.
For each ring, we use a result of Ghazi et al.
[44] inspired by the theory of error-correcting codes, which forms a “cover” of each ring of size roughly exponential in the dimension, which is a method of selecting points such that every point in the ring is reasonably close to at least 1 point in the cover.
We can create the cover in parallel, and then map every point to its closest point in the cover in e O ( d ) time.
We can use this cover and a Laplace mechanism to construct a private coreset of the data, though the coreset will have size exponential in the dimension.
For the second step, we apply a similar method to Section 4, again using the insight of charging additive error to multiplicative error.
We again start with our weak parallel coreset from Section 3, partition the points into rings, and sample uniformly from each ring which allows each ring’s sample to ﬁt on a single machine.
Now, for each sample, we apply the private coreset procedure from the ﬁrst step, and together we obtain a large private coreset.
We ﬁnally apply a non-private MPC clustering algorithm on the already privatized coreset.
The full details of the algorithm and analysis, including pseudocode, are deferred to Appendix D.
Dimensionality Reduction
Our algorithm for dimensionality reduction relies on [56], which shows that a random projection Π down to O (log k ) dimensions can approximate every k -clustering of n points simultaneously.
While this may seemingly imply that we can automatically assume d = O (log k ) , this is not so obvious, because once we have found a set of centers C (cid:48) in R O (log k ) , it is unclear how to pull this back up to a set of centers in R d .
Ghazi et al.
[44] use dimensionality reduction to improve their exponential dependence on d to a polynomial dependence on k (
as e O (log k )
= poly( k ) ).
They start by constructing a random projection Π down to O (log k ) dimensions and compute a private set C (cid:48)
= { c (cid:48) 1 , . .
c (cid:48) k } in the reduced dimension.
They then map each point x
i to a cluster X j based on which point c (cid:48) j ∈ C (cid:48) is closest to Π x i .
Finally, to obtain a set of centers in high dimensions, they compute a private average (for k -means) or geometric median (for k -median) of the points in each cluster X j .
The main issue in applying this method is that we wish for a very fast scalable algorithm that works in MPC, and it is unknown how to compute a private geometric median that is sufﬁciently scalable, i.e., runs in O (1) MPC rounds and ˜ O ( nd ) total work.
Our ﬁrst observation in obtaining an efﬁcient algorithm is that, rather than computing a geometric median of the points, a coordinate-wise median serves as a reasonable proxy for the geometric median.
Speciﬁcally, it has the important property of being within distance O
( r ) of the geometric median of ˜ X
j if at least 2 / 3 of the points in X j are within r of the geometric median [59].
In addition, the coordinate-wise median can be computed on a small sample without signiﬁcant error, and can be computed privately without much error either.
We can therefore use an approximate coordinate-wise median for each cluster
X j to obtain an
O (1) -approximate private and scalable clustering with minimal dependence on the dimension.
If we want arbitrarily good approximation, such as in Section 5, the coordinate-wise median is unfortunately not strong enough.
To ﬁx this, our rough intuition is to compute a private geometric median of a small uniform subsample of points, but only among those points close to the private coordinate-wise median.
This idea of uniform subsampling to speed up runtime has been used in several previous algorithms for k -means, such as [23, 14, 30, 37].
As we perform the geometric median on a small sample, a polynomial-time algorithm is sufﬁcient.
In addition, by restricting ourselves to points close to the coordinate-wise median, we are able to replace additive error proportional to Λ with additive error proportional to some smaller radius R , which we can charge to multiplicative error in a similar manner as in Sections 4 and 5.
We remark that this algorithm may take ˜ O
( ndk ) total time instead of ˜ O ( nd ) time, but we ﬁx this by performing another uniform subsampling trick at the beginning of the algorithm, which improves the runtime at the cost of a slight blowup in the additive error.
Overall, we obtain our main theorems of this paper, Theorems 1.1 and 1.2, which correspond to dimension reduction applied to Theorems 4.1 and 5.1, respectively.
We provide pseudocode for Theorem 1.1 below, in Algorithm 2.
The full algorithm and analysis for both theorems, as well as pseudocode for Theorem 1.2, are deferred to Appendix E.
1: procedure C ONSTANT A PPROX H IGH D IM ( X ,
ε, δ ) (cid:46) Will be ( O ( ε ) , O ( δ ))
-DP.
Let T
=
O (log( nd ) · ε − 1 (cid:112) d log δ − 1 ) be a threshold parameter.
Let d (cid:48)
=
O (log k ) , and pick a random projection Π ∈ R d (cid:48) × d .
Use Algorithm 1 to ﬁnd a k -means clustering C (cid:48)
= { c (cid:48) 1 , . .
c (cid:48) k } of Π X
= { Π x 1 , . .
Π x n } .
Let S ⊂ [ n ] be a random sample where each i ∈ S with probability k − 0 .
Map each point Π x
i : i ∈ S to a 10 -approximate nearest neighbor in C (cid:48) .
7: for j = 1 to k do 8:
Let X j be the set of points x i such that Π x i is mapped to c (cid:48)
j .
Let ˆ N j = |X
j | + Lap(1 /ε ) .
10: if ˆ N j ≥ 2 T then 11:
Sample T points in X j , and let c j be a private coordinate-wise median of the sampled points, where we use ( ε/ (2 (cid:112) d log δ − 1 , 0)
-DP in each direction.
12: else 13:
Let c j be the origin.
Return C = { c 1 , . . .
, c k }