Sublinear Time Algorithms for Greedy Selection in High Dimensions
Abstract
Greedy selection is a widely used idea for solving many machine learning problems.
But greedy selection algorithms often have high complexities and thus may be prohibitive for large-scale data.
In this paper, we consider two fundamental optimization problems in machine learning: kcenter clustering and convex hull approximation, where they both can be solved via greedy selection.
We propose sublinear time algorithms for them through combining the strategies of randomization and greedy selection.
Our results are similar in spirit to the linear time stochastic greedy selection algorithms for submodular maximization [Mirzasoleiman et al.
, AAAI 2015, Hassidim and
Singer, ICML 2017], but with several important differences.
Our runtimes are independent of the number of input data items n .
In particular, our runtime for k -center clustering significantly improves upon that of the uniform sampling approach, especially when the dimensionality is high.
Our sublinear algorithms can also reduce the computational complexities for various applications, such as data selection and compression, active learning, and topic modeling, etc .
INTRODUCTION
Greedy algorithm is one of the most fundamental tools for algorithm design [Cormen et al., 2009].
In particular, many optimization problems in machine learning can be solved through greedy selection method.
The method iteratively selects a subset of data items from input based on some greedy strategy.
One representative example is the Gonzalez’s algorithm for k -center clustering [Gonzalez, 1985].
Given a set of data items ( e g., a point set in R d ), the algorithm is to iteratively select k items from the input; if one draws k equal-sized balls centered at these k items, the whole input data set can be covered by these balls and the radius is no larger than two times the optimal one (the formal definition for k -center clustering is shown in Section 2).
The algorithm is simple but has many important applications in real world.
One natural application is constructing coreset for compressing a large-scale data, especially when we want to maximize diversity or coverage
[Indyk et al., 2014].
Another closely related application is batch active learning [Sener and Savarese, 2018, Coleman et al., 2020].
Most machine learning models heavily depend on high-quality labeled training datasets.
However, because it is expensive to acquire a large number of labeled data, we may only be able to select a small number of data items (via k -center clustering) to label in each round (as an active learning process).
Another high dimensional optimization problem that depends on greedy selection is convex hull approximation [Blum et al., 2019, Awasthi et al., 2020], where the goal is to find a convex hull so that each data item can be approximately represented by the vertices.
The problem has a number of applications in machine learning, such as topic modeling, sparse approximation, and non-negative matrix factorization [Ge and Moitra, 2020].
Though the convex hull algorithms have been well studied in low dimensions [de Berg et al., 2008], the problem in high dimensions is much more challenging.
Similar with the k -center clustering, a common idea for convex hull approximation is using greedy selection to find the vertices iteratively.
Although these greedy selection methods enjoy promising performances in practice, they often suffer from high complexities when data sizes are extremely large.
For instance, the vanilla Gonzalez’s algorithm needs to run k iterations and each iteration needs to scan the whole dataset in one pass (a detailed introduction on the previous work is shown in Section 1.1).
Similarly, the greedy selection method for convex hull approximation also needs to repeatedly scan the whole dataset and thus yields large runtime.
So a natural question is:
Can we modify these greedy selection algorithms to achieve lower time complexities, e.g., sublinear time complexities that are independent of input data size, and meanwhile preserve their quality guarantees?
1.1 RELATED WORK
We introduce several important existing results related to this paper in this section.
k -center clustering.
As mentioned before, the greedy selection based k -center clustering algorithm
[Gonzalez, 1985] can yield a 2 -approximation result; moreover, it was proved that any approximation ratio lower than 2 implies P =
NP [Hochbaum and Shmoys, 1985].
To speed up the Gonzalez’s algorithm, several improvements have been proposed before [Feder and Greene, 1988, Har-Peled and Mendel, 2006]; however, they usually require some additional assumptions ( e.g., the dimensionality or intrinsic dimensionality should be small).
To deal with large-scale data, a number of streaming algorithms which only need to read the data in one-pass were introduced in Charikar et al.
[2004], McCutchen and Khuller [2008], Guha [2009], Ceccarello et al.
The well known “coreset” technique is also applied to compress data size for k -center clustering B ¯ adoiu et al.
[2002], Aghamolaei and Ghodsi [2019], but their coreset construction algorithms already take at least linear time.
Furthermore, several uniform sampling based ideas were presented for achieving sublinear complexity for k -center clustering (with outliers)
[Charikar et al., 2003, Huang et al., 2018].
Convex hull approximation.
Several elegant convex hull algorithms for low-dimensional space have been introduced in the community of computational geometry before [de Berg et al., 2008].
The high-dimensional convex hull approximation problem is closely related to non-negative matrix factorization and topic modeling [Ge and Moitra, 2020].
Roughly speaking, the vertices of the obtained convex hull can help us to generate the low rank non-negative matrices and discover the hidden topics.
In general, this problem is intractable but it is possible to achieve an efficient solution under the separability assumption [Donoho and Stodden, 2003, Arora et al., 2012].
Recently, several practical algorithms with provable guarantees were also proposed, such as Blum et al.
[2019], Awasthi et al.
[2020], Arora et al.
Other applications of greedy selection in machine learning.
Besides the aforementioned two problems, greedy selection also has several other applications in machine learning.
To name a few: submodular maximization [Nemhauser et al., 1978], column subset selection [Farahat et al., 2015], reinforcement learning [Painter-Wakefield and Parr, 2012], sparse approximation [Tropp, 2004], and SVM Gärtner and
Jaggi [2009].
1.2 OUR CONTRIBUTIONS
In this paper, we aim to develop sublinear time algorithms for the k -center clustering and convex hull approximation problems.
We assume the input data size and the dimensionality are both large.
We combine the strategies of greedy selection and randomization, and show that the randomized greedy selection methods can achieve almost the same approximation guarantees, and meanwhile, the time complexities can be reduced to be sublinear.
Comparison with the algorithms of Mirzasoleiman et al.
[2015], Hassidim and Singer [2017].
Actually, the high complexity issue of greedy selection has been discussed in Mirzasoleiman et al.
[2015], Hassidim and Singer [2017] for the submodular maximization problem.
They showed that if the greedy selection step is replaced by random sampling, a quality guarantee still holds but the complexity ( i.e., the number of function evaluations) can be reduced to be linear.
Our proposed algorithms are inspired by the similar stochastic intuition but with several important differences.
(i)
First, both k -center clustering and convex hull approximation are geometric optimizations in high dimensions which have different objective functions other than submodular maximization.
(ii) Second, our framework yields the sublinear time complexities that are independent of the number of input data items; this property is particularly important when we cannot access the input data and can only take a small sample via an oracle each time ( e.g. , due to privacy preserving or the challenge of data acquisition).
(iii) Finally, we also consider the scenario that the number of iterations for greedy selection is unknown.
For example, the number of clusters “ k ” of the k -center clustering may not be given; instead, we may just run the Gonzalez’s algorithm iteratively until the obtained radius is no larger than a pre-specified threshold r 0 > 0 .
We need to emphasize that designing the sublinear time algorithm becomes much more challenging with such a change, since it will be difficult to set the sample size in each iteration and determine when the algorithm should terminate.
To remedy these issues, we propose a novel stratified sampling method and design a sampling based stopping condition for the greedy selection.
Comparison with the streaming and uniform sampling algorithms.
As mentioned in Section 1.1, the one-pass streaming algorithms [Charikar et al., 2004, McCutchen and Khuller, 2008, Guha, 2009, Ceccarello et al., 2019] can avoid repeatedly reading the input data, however, they still suffer from high time complexities ( e.g., the “doubling algorithm”
[Charikar et al., 2004] takes a total O (cid:0) k
(log k ) nd (cid:1) time that is even higher than the complexity of the vanilla
Gonzalez’s algorithm, where n is the number of input points).
On the other hand, our proposed sublinear time algorithms have the complexities independent of n .
It is also worth to compare our results with the uniform sampling algorithms for k -center clustering [Charikar et al., 2003, Huang et al., 2018].
For example, a simple uniform sample S of size ˜ O ( kdϵ 2 ) 1 can approximately represent the whole input data P based on the theory of VC dimension [Huang et al., 2018], where ϵ ∈ (0 , 1) indicates the small fraction of uncovered points; that is, if one runs the 2 -approximate Gonzalez’s algorithm on the sample S , the obtained k balls still form a 2 -approximate solution in terms of the whole input P but except for ϵn uncovered points of P .
The running time of the Gonzalez’s algorithm on S should be ˜ O
( k | S | d ) =
˜ O
( k 2 d 2 ϵ 2 ) .
In Section 3.1, we show that our algorithm takes ˜ O
( k 3 dϵ ) time (also with ϵn uncovered points).
Usually, k is much smaller than the dimensionality d , and thus our improvement is significant.
In particular, if k is assumed to be constant, we improve their complexity by a factor up to dϵ .
The reader may wonder that whether dimension reduction technique ( e.g. , the JL-transform Dasgupta and Gupta [2003]) can be applied.
Actually the complexities of both our method and the uniform sampling can be reduced by the JL-transform, and our improvement is still significant (just replace the dimension d by the new dimension d ′ for both the two complexities). Also, even we apply the JLtransform, the reduced dimensionality could be still high (which is O (log | S | /µ 2 ) , if supposing | S | is the total sample size and “ µ ” is the pairwise distance distortion error). For example, if we let µ = 0 . 01 , the new dimension is still high.
Not only the runtime, another benefit comparing with the uniform sampling is that we have smaller sample size.
Our algorithm takes ˜ O
( k 2 /ϵ ) samples in total after k iterations, which is much lower than ˜ O
( kd/ϵ 2 ) if assuming k is not large.
In particular, our sample size is independent of the dimension d .
The dimension can be very high or even infinity if using kernel.
The smaller sample size is also important in some specific setting like relational database [Schleich et al., 2019].
It is very expensive to materialize the whole data matrix for a relational database, and a smaller sample size can significantly reduce the total computational complexity
[Zhao et al., 2018].
PRELIMINARIES
In this section, we introduce several important definitions that will be used throughout this paper.
Let c ∈ R d and r ≥ 0 ; we use B ( c, r ) to denote the ball centered at c with radius r .
Also, given a set S of points in R d , we use conv ( S ) to denote the convex hull of S .
We use the function dist ( p, U ) to measure the shortest distance from a point p to a set U , i.e., dist ( q, U ) :
= min q ∈ U || p − q || .
Definition 1 ( k -center clustering ) .
Given a set P of n points in R d and k ∈ Z + , the goal of k -center clustering is to find k balls B ( c 1 , r ) , · · · , B ( c k , r ) with the smallest radius r to cover the set P , that is, P is partitioned into k clusters with each cluster being covered by an individual ball, and the radius r is minimized.
Remark 1.
The k -center clustering problem can be also defined for any abstract metric, where the only difference is that the Euclidean distance is replaced by the distance defined in the metric.
In fact, our proposed sublinear algorithms for k -center clustering in this paper can be applied to any abstract metric with the same quality guarantees.
Let r opt be the radius of the optimal solution for the k -center clustering on P .
For any solution having a radius r ≤ λr opt with some λ ≥ 1 , we call it a “ λ -approximation ”.
Definition 2 ( convex hull approximation ) .
Given a set P of n points in R d and an integer k ≥ 1 , the goal of convex hull approximation is to find a subset P c ⊂ P with | P c | =
k , such that the error, i.e., max p ∈ P dist (cid:0) p, conv ( P c ) (cid:1) is minimized (so if all the points of P are covered by conv ( P c ) , the error is 0 ).
Remark 2.
In general, we can remove the requirement “ P c ⊂ P ”, i.e., P c can contain any points in the space.
But we often want P c to be meaningful or interpretable in practice, and thus it is natural to require it to be a subset of the original input data.
Similar with k -center clustering, we can also define the approximation solution for convex hull approximation.
But since the convex hull approximation is much more challenging, we often obtain bi-criteria approximations.
Suppose α, β ≥ 1 .
If letting δ opt be the optimal error, a bi-criteria ( α, β ) -approximation means that the obtained convex hull has the error δ ≤ αδ opt and the number of vertices k ′ ≤ βk .
The rest of this paper is organized as follows.
In Section 3, we propose our sublinear time algorithm for k -center clustering.
In particular, we also consider the practical case that the number of clusters k is not given (Section 3.2).
Further, in Section 4 we consider developing sublinear time algorithm for convex hull approximation by extending the idea from Section 3.2.
Finally, we present our experiments in
Section 5.
k -CENTER CLUSTERING
In this section, we focus on the k -center clustering problem.
For the sake of completeness, we briefly introduce the aforementioned 2 -approximate Gonzalez’s algorithm
[Gonzalez, 1985] first.
Gonzalez’s algorithm.
It selects an arbitrary point, say c 1 , from the input P and lets C = { c 1 } . In each of the following k − 1 iterations, it selects a new point that has the largest distance to C among the points of P and adds it to C .
Suppose C
= { c 1 , · · · , c k } , and then P is covered by the k balls B ( c 1 , r ) , · · · , B ( c k , r ) with r ≤ min {|| c i − c
j || | 1 ≤ i ̸ =
j ≤ k } .
It is not difficult to prove that the obtained radius r ≤ 2 r opt .
It is also easy to know that the running time of the Gonzalez’s algorithm is O ( knd ) .
As mentioned before, a major drawback of the algorithm is the high complexity, especially when n and d are large.
3.1 OUR SUBLINEAR ALGORITHM
Our proposed algorithm can be viewed as a randomized version of the Gonzalez’s algorithm.
The key change is that we randomly select the next point for C in each round, instead of always picking the furthest one.
Below we prove that this strategy can achieve the same 2 -approximation except for a small error on the number of covering points.
Algorithm 1 S UBLINEAR k -C ENTER C LUSTERING
Input: A set P of n points in R d , k ∈ Z + , and two parameters η, ϵ ∈ (0 , 1) .
Initially, let C = { c 1 } , where c 1 is an arbitrary point picked from P ; i = 1 . 2.
Repeat the following steps k − 1 times: (a) Sample a set Q of kϵ log kη points from P uniformly at random.
(b) Select the furthest point, say q 0 , from Q to C , i.e., q 0 = arg q ∈ Q max dist ( q, C ) .
(c) Let c i +
= q 0 ,
C = C ∪ { c i +1 } , and i
= i
+ 1 . 3.
Return C .
Theorem 1.
Let C
= { c 1 , · · · , c k } be the output from
Algorithm 1.
With probability at least 1 − η , there exists a subset ˜ P ⊂ P with size | ˜ P | ≥ (1 − ϵ ) n , such that ˜ P is covered by ∪ ki =1 B ( c i , 2 r opt ) .
To prove Theorem 1, we need the following claim first.
Claim 1.
Let U be a set of elements and V ⊆
U with | V | | U | = τ > 0 .
Given η ∈ (0 , 1) , we uniformly select a set S of elements from U at random.
Then if | S | ≥ 1 τ log 1 η , with probability at least 1 − η , S contains at least one element from V .
Actually, the above claim is a folklore result that has been presented in several papers before (such as Ding and Xu [2014]).
Since each sampled element falls in V with probability τ , we know that the sample S contains at least one element from V with probability 1 − (1 − τ ) | S | .
If we want to guarantee 1 − (1 − τ ) | S | ≥ 1 − η , | S | should be at least log1 /η log1 / (1 − τ ) ≤ 1 τ log 1 η .
Proof.
(of Theorem 1])
To help our analysis, we define
C 1 := ∅ , and C i := { c 1 , c 2 , · · · , c i − 1 } for each i = 2 , · · · , k of Algorithm 1.
Further, we define
[Equation]
for 2 ≤ i ≤ k . We also define F i to be the farthest ϵk | P | points from P to C i .
Claim 1 implies that the sample Q should contain at least one point from F i with probability at least 1 − ηk .
If this is true, the selected c i should come from F i and thus | P i | ≤ | F i | . Therefore, we have | P i | ≤ ϵk | P | with probability at least 1 − ηk .
Through taking the union bound over all the P i s, we have: with probability at least (1 − ηk ) k − 1 > 1 − η
[Equation]
Let ˜ P :=
P \ ∪ ki =2 P i .
It is easy to know the size
[Equation]
Next, we only need to prove that ˜ P is covered by ∪ ki =1 B ( c i , 2 r opt ) .
We present the following lemma first.
Lemma 1.
For any point p ∈
[Equation]
Let O 1 , O 2 , · · · , O k be the k clusters obtained from the optimal solution, i.e., P = ∪ ki =
O i and each cluster O i can be covered by a ball with radius r opt .
We consider two cases.
Case (i) : { c 1 , · · · , c k } fall into the k clusters
O 1 , O 2 , · · · , O k separately.
Without loss of generality, we assume c i ∈
O i for i = 1 , 2 , · · · , k .
By using the triangle inequality, we know the input set P is covered by ∪ ki =1 B ( c i , 2 r opt ) .
Consequently, ˜ P is also covered by ∪ ki =1 B ( c i , 2 r opt ) .
Case (ii) : there exist two points, say c
i a and c i b , of C that belong to one optimal cluster, say O l .
Thus ||
c i a − c
i b || ≤ 2 r opt .
From Lemma 1, we know
[Equation]
Hence ˜ P is covered by ∪ ki =1 B ( c i , 2 r opt ) .
Proof.
(of Lemma 1])
Suppose Lemma 1 is not true.
Then there exist some p 0 ∈ ˜ P and two points c
i 1 and c i 2 ∈ C , such that
[Equation]
Without loss of generality, we assume i 1 < i 2 .
Since || c i 1 − c i 2 || ≥ dist ( c i 2 , C i 2 ) , the inequality (5) implies dist
[Equation]
So from (1) we know p 0 ∈ P
i 2 , which is in contradiction with the assumption p 0 ∈ ˜ P
= P \ ∪ k i =2 P i .
Time complexity.
It is easy to see that the time complexity of Algorithm 1 is independent of n .
It takes k rounds, and each round needs to compute the distances from the sampled kϵ log kη points to C .
So the total complexity is
O ( k × kϵ log kη × kd )
=
O ( k 3 ϵ d log kη ) .
In some scenarios, we may not be able to access the whole data, e.g., due to privacy preserving or the challenge of data acquisition.
Instead, we may be only allowed to take a small sample each time.
Specifically, we assume the data is a (continuous or discrete) probability distribution with the probability density function f in Ω ⊂ R d , where (cid:82)
p ∈ Ω f ( p ) d p = 1 ; the function f can be hid and we only assume that there is an oracle to sample data based on f .
Obviously, it is prohibitive to directly run the Gonzalez’s algorithm in such scenario.
On the other hand, our proposed Algorithm 1 can be naturally applied to solve this problem because it only takes a random sample in each round.
The following result is a straightforward extension of Theorem 1.
Corollary 1.
We run Algorithm 1 on a (continuous or discrete) probability distribution over Ω ; each sampled point is taken by an oracle based on the probability density function f .
With probability at least 1 − η , there exists a subset ˜Ω ⊂ Ω with the integral (cid:82) p ∈ ˜Ω f ( p ) d p ≥ 1 − ϵ , such that ˜Ω is covered by ∪ ki =1 B ( c i , 2 r opt ) .
3.2 WHEN k IS NOT GIVEN
In many real scenarios, the number of clusters k is often not given.
For instance, we may only have a threshold r 0 >for the radius; so we just try to perform the k -center clustering algorithm for different values of k until the obtained radius is no larger than r 0 .
The reader may realize that this problem is related to the well known geometric set cover problem [Brönnimann and Goodrich, 1995, Agarwal and Pan, 2020]; however, existing geometric set cover algorithms often have large (super linear) running time and can only handle low dimensional case.
Actually, the geometric set cover problem is NP-hard and has only constant factor approximation in 2D plane (the problem is even harder in high dimensions).
In this paper, we simplify the problem and consider a practical approach: using the Gonzalez’s algorithm to achieve our goal.
Suppose the given set P can be covered by ˜ k ∈ Z + balls with radius r 0 / 2 ( i.e., ˜ k is the value that the optimal radius of ˜ k -center clustering on P is no larger than r 0 / 2 ).
Then, if we just run the Gonzalez’s algorithm iteratively, the resulting radius will reach r 0 within at most ˜ k rounds (because it is a 2 -approximation algorithm).
Now we discuss how to implement this procedure in sublinear time.
We cannot directly adapt this procedure to our sublinear Algorithm 1, due to the following two issues.
The sample size kϵ log kη in step 2(a)
depends on a given k (also note that Algorithm 1 is a randomized algorithm and its success probability depends on the sample size); (2) we do not know when to terminate if k is not given.
To resolve these two issues, we introduce a stratified sampling method .
Let k 0 ≥ 1 be any fixed constant.
Imagine we run step 2(a)-2(c) of Algorithm 1 iteratively.
We partition the process into different phases and modify the sample size in step 2(a) for each phase accordingly:
• Phase t = 0 :
for i = 1 , 2 , · · · , k 0 , we set | Q | = 2 kϵ log k 0 η . • Phase t ≥ 1 : for i =
(cid:80) t − 1 s =0
s k 0 + 1 , (cid:80) t − 1 s =
2 s k
+ 2 , · · · , (cid:80) ts =
2 s k 0 , we set | Q | = 2 2 t 2 k 0 ϵ log 2 2 t k 0 η .
So phase t contains 2 t k 0 iterations.
The sample size also increases from phase t to phase t + 1 .
For completeness, we also need to set the stopping condition.
Suppose r 0 > 0 is the given threshold.
At the end of each i -th iteration, we take a sample S from P uniformly at random, and compute the ratio
[Equation]
The following lemma introduces an oracle that can help us to decide when to terminate.
Lemma 2.
Suppose η 0 ∈ (0 , 1) .
We set the sample size | S | ≥ 12 η 0 ϵ log 2 η 0 .
With probability at least 1 − η 0 , the following oracle returns the correct answer: if τ ≤ 32 ϵ , return “ (cid:12)(cid:12)(cid:12) P \ (cid:0) ∪ il =1 B ( c l , r 0 ) (cid:1)(cid:12)(cid:12)(cid:12) ≤ 3 ϵn ”; else, return “ (cid:12)(cid:12)(cid:12) P \ (cid:0) ∪ il =1 B ( c l , r 0 ) (cid:1)(cid:12)(cid:12)(cid:12) > ϵn ”.
Proof.
For convenience, we use ˜ ϵ to denote the ratio (cid:12)(cid:12)
P \
[Equation]
∪
[Equation]
n .
We consider two cases: (i) ˜
ϵ ≤ η 0 ϵ and (ii) ˜ ϵ > η 0 ϵ .
For case (i), (cid:12)(cid:12)(cid:12) P \ (cid:0) ∪ il =1 B ( c l , r 0 ) (cid:1)(cid:12)(cid:12)(cid:12) =
˜ ϵn ≤ η 0 ϵn < 3 ϵn .
Due to the Markov’s inequality, we know that τ ≤ 1 η 0 × η
ϵ
=
ϵ < 3 2 ϵ with probability at least 1 − η 0 .
Thus, it returns “ (cid:12)(cid:12)(cid:12) P \ (cid:0) ∪ il =1 B ( c l , r 0 ) (cid:1)(cid:12)(cid:12)(cid:12) ≤ 3 ϵn ” which is a correct answer, with probability at least 1 − η 0 .
So we focus on the second case below.
We use the Chernoff bound [Alon and Spencer, 2004].
Define | S | random variables { y 1 , · · · , y | S | } : for each 1 ≤ j ≤ | S | , y j = 1 if the j -th sampled element falls in
P \ (cid:0) ∪ il =1 B ( c l , r 0 ) (cid:1) , otherwise, y j = 0 .
So E [ y j ] =
˜
ϵ for each y j .
As a consequence, we have
[Equation]
Since we assume ˜ ϵ > η 0 ϵ , if | S | ≥ 12 η 0 ϵ log 2 η 0 , the above (8) implies that with probability at least 1 − η 0 , (cid:12)(cid:12) (cid:80) | S | j =1 y j − ˜ ϵ | S | (cid:12)(cid:12) ≤
[Equation]
S | , i.e.,
[Equation]
Therefore, if τ ≤ 32 ϵ , we know 12 ˜ ϵ ≤ 32 ϵ from (9), and it implies ˜ ϵ ≤ 3 ϵ .
Otherwise, we know 32 ˜ ϵ > 32 ϵ
and it implies ˜ ϵ
>
ϵ .
Now, we are ready to present our algorithm for the case without knowing ˜ k .
Let i ter be the size of C when Algorithmterminates.
To evaluate the performance of the algorithm, we need to compare i ter with ˜ k and investigate the number of points that are covered by ∪ i ter j =1 B ( c j , r 0 ) .
Input: A set P of n points in R d , a threshold r 0 > 0 , an arbitrary constant integer k 0 ∈
Z + , and two parameters η, ϵ ∈ (0 , 1) .
Initially, let C = { c 1 } , where c 1 is an arbitrary point picked from P ; t = i = 0 .
Repeat the following steps as the stratified sampling procedure: (a) Take a sample S from P uniformly at random, where | S | = 12 η 0 ϵ log 2 η 0 and η 0 = η 2 2 t k 0 .
(b) Repeat the following steps 2 t k 0 times ( i.e., phase t ): i. Randomly pick a set Q from P , where | Q | = 2 2 t 2 k 0 ϵ log 2 2 t k 0 η . ii.
Let q 0 be the furthest point from Q to C ,
i
e., q 0 = arg q ∈ Q max dist ( q, C ) .
iii.
Let c i +
= q 0 ,
C = C ∪{ c i +1 } , and i = i +1 .
iv.
Apply Lemma 2 as the oracle (using the sample S from step 2(a)) to determine whether to terminate: if it returns “ (cid:12)(cid:12)
Theorem 2.
Let C
= { c 1 , · · · , c i ter } be the output from
Algorithm 2.
With probability at least 1 − 4 η , i ter ≤ ˜ k , and there exists a subset ˜ P ⊂ P with size | ˜ P | ≥ (1 − 3 ϵ ) n , such that ˜ P is covered by ∪ i ter j =1 B ( c j , r 0 ) .
Proof.
To prove Theorem 2, we first imagine the “fancied” scenario that ˜ k is given: we just run Algorithm 1 with k = ˜
k and | Q
| = ˜ kϵ log ˜ kη .
Recall the proof of Theorem 1, where we define a sequence of subsets P 2 , P 3 , · · · , P ˜ k and define ˜ P
= P \ ∪ ˜ k
i = 2 P i .
To guarantee | ˜ P | ≥ (1 − ϵ ) n , we prove that each P i contains at most ϵ ˜ k n points.
For Algorithm 2, we also define a sequence of subsets P 2 , P 3 , · · · , P ˜ k
(by using (1)), but we need to modify their sizes.
At each t -th phase, since we have the sample size 2 2 t 2 k 0 ϵ log 2 2 t k 0 η , by using the similar idea from the proof of Theorem 1 we know that the size | P i |
[Equation]
Suppose we run Algorithm 2 until i =
˜
k .
Let t 0 be the total number of phases that the algorithm takes.
Consequently, we have
[Equation]
[Equation]
So we can still guarantee | ˜ P | = | P \ ∪ ˜ ki
=2 P i | ≥ (1 − ϵ ) n .
Furthermore, the total success probability is at least
[Equation]
The remaining issue is that we do not know the value of ˜ k
in reality (in other words, we do not know when to terminate the algorithm).
Therefore, we apply Lemma 2 as an oracle in step 2(b)(iv), where the success probability for each time is
[Equation]
When it returns “ (cid:12)(cid:12)(cid:12) P \ (cid:0) ∪ il =1 B ( c l , r 0 ) (cid:1)(cid:12)(cid:12)(cid:12) > ϵn ”, we know that the algorithm needs to continue.
We stop the algorithm when it returns “ (cid:12)(cid:12)(cid:12) P \ (cid:0)
[Equation]
) (cid:1)(cid:12)(cid:12)(cid:12) ≤ 3 ϵn ”.
Since we relax the error of covering number to be 3 ϵ > ϵ , we know that i ter should be no larger than ˜ k .
By using the similar idea of (12), we can obtain the overall success probability of the oracle that is at least 1 − 2 η.
Combining (12) and (13), the overall success probability of
Algorithm 2 is at least 1 − 4 η .
The time complexity of Algorithm 2.
We analyze the runtime for each phase.
We set k 0 ≥ 2 to be a constant integer.
At the t -th phase, step (b)(i)-(iii) take O ( 2 3 t ϵ log 2 2 t η d ) time; step (b)(iv) takes O ( 2 2 t ηϵ log 2 2 t η d ) time.
Also, the phase repeats step (b)(i)-(iv) O
(2 t ) times.
Thus, the t -th phase takes
O ((2 t + 1 η ) 2 3 t ϵ log 2 2 t η d ) time.
Let t 0 be the total number of phases.
Then we can calculate the bounds for ˜ k :
[Equation]
which implies t 0 ≤ log ˜ kk 0 + 1 ≤ log ˜ k .
So the total time complexity of Algorithm 2 is O
((˜ k + 1 η ) ˜ k
3 ϵ log ˜ kη d ) .
Compared with the case that ˜ k is given, the runtime is increased by only a factor (˜ k + 1 η ) (the runtime of Algorithm 1 is
O ( ˜ k 3 ϵ log ˜ kη d ) ).
We also have the following result for Algorithm 2 which is similar with Corollary 1.
Corollary 2.
We run Algorithm 2 on a (continuous or discrete) probability distribution over Ω ; each sampled point is taken by an oracle based on the probability density function f .
With probability at least 1 − 4 η , i ter ≤ ˜ k , and there exists a subset ˜Ω ⊂ Ω with the integral (cid:82) p ∈ ˜Ω
f
( p ) d p ≥ 1 − 3 ϵ , such that ˜Ω is covered by ∪ i ter i =1 B
( c i , r 0 ) .
CONVEX HULL APPROXIMATION IN HIGH DIMENSIONS
Blum et al.
[2019] introduced a simple greedy convex hull approximation algorithm that is similar in spirit to the Gonzalez’s algorithm for k -center clustering.
Given an instance
P ⊂ R d , it also maintains a set C that contains an arbitrarily selected p ∈ P at the beginning.
In each round, the algorithm always selects the farthest point to conv ( C ) and adds it to C , until some specified stopping condition is satisfied.
For ease of presentation, we assume that P is contained in a unit ball of R d .
The algorithm yields a bi-criteria approximate result: given an error parameter δ ∈ (0 , 1) , suppose k opt = min (cid:8) k
| Q ⊂ P, | Q | = k, max p ∈ P dist ( p, conv ( Q ))
≤ δ (cid:9) ; the algorithm can yield a subset C ⊂ P such that
[Equation]
We consider applying our previous sampling idea to implement this convex hull approximation algorithm in sublinear time.
Here, we have the same issue as Section 3.2, that is, we do not know the exact value of k opt so that we cannot determine the sample size in each iteration and when to terminate.
Thus we apply the same stratified sampling method.
We also use Lemma 2 as the oracle to determine whether the stopping condition is satisfied.
A minor technical issue for implementation is that it is costly to compute the distance from a given point to a convex hull (it needs to solve a quadratic programming for achieving the exact result); instead we can apply the Gilbert’s algorithm
[Gilbert, 1966, Gärtner and Jaggi, 2009] or some other variants like the Triangle algorithm
[Awasthi et al., 2020] to compute an approximate solution efficiently.
Thus, we need another small parameter ξ ∈ (0 , 1) to indicate the approximation error induced by this step.
Compared with the ratio “ τ ” for k -center clustering, we add an extra factor (1 + ξ ) to τ below.
Let r 0 = 8 δ 1 / 3 + δ .
We use C i to denote the set of selected vertices { c 1 , · · · , c i } at the first i rounds.
For convenience, we use conv ( U, r ) to denote the set (cid:8) [Equation] for any given set U and r ≥ 0 .
Then, we compute the ratio
[Equation]
Similar to the case of k -center clustering, the following lemma introduces an oracle that can help us to decide when to terminate (the proof is almost identical to that of
Lemma 2).
Algorithm 3 S UBLINEAR C ONVEX H ULL A PPROXIMATION
R d , a threshold r 0 > 0 , an arbitrary constant integer k 0 ∈
Z + , and three parameters η, ϵ, ξ ∈ (0 , 1) . Initially, let C = { c 1 } , where c 1 is an arbitrary point picked from P ; t = i = 0 . Repeat the following steps as the stratified sampling procedure: (a) Take a sample S from P uniformly at random, where | S | = 12 η 0 ϵ log 2 η 0 and η 0 = 2 2 t k 0 .
(b) Repeat the following steps 2 t k 0 times ( i.e., phase t ): i. Randomly pick a set Q from P , where | Q | =
[Equation]
0 η . ii.
Select the (1 + ξ ) -approximate furthest point, say q 0 , from Q to conv ( C ) via the algorithm of Gärtner and Jaggi [2009].
iii.
{+Let c i +
+}
= q 0 ,
C = C ∪{ c i +1 } , and i = i +1 .
iv.
Apply Lemma 3 as the oracle (using the sample S from step 2(a)) to determine whether
to terminate:
if it returns
“ (cid:12)(cid:12)(cid:12) P \ conv ( C i , (1 + ξ ) r 0 ) (cid:12)(cid:12)(cid:12) ≤ 3 ϵn ”
, stop the algorithm, set i ter =
i ,
and return C .
[Equation]
[Equation]
Lemma 3.
Suppose η 0 ∈ (0 , 1) .
We set the sample size | S | ≥ 12 η 0 ϵ log 2 η 0 .
With probability at least 1 − η 0 , the following oracle returns the correct answer: if τ ≤ 32 ϵ , return “ (cid:12)(cid:12)(cid:12) P \ conv ( C i , (1 + ξ ) r 0 ) (cid:12)(cid:12)(cid:12) ≤ 3 ϵn ”; else, return “ (cid:12)(cid:12) (cid:12) P \ conv ( C i , (
[Equation]
Theorem 3.
Let C
= { c 1 , · · · , c i ter } be the output from Algorithm 3.
Let ˜ k be the number of vertices returned by the greedy selection algorithm
[Blum et al., 2019] (see (15)).
With probability at least 1 − 4 η , i ter ≤ ˜ k , and there exists a subset ˜ P ⊂ P with size | ˜ P | ≥ (1 − 3 ϵ ) n , such that ˜ P is covered by conv ( C, (1 + ξ ) r 0 ) .
Time complexity.
The computation for the time complexity is similar with that for k -center clustering in Section 3.2, where the only difference is that we have to compute the (1 + ξ ) -approximate polytope distance from each sampled point to C in each iteration.
From the analysis of Gärtner and Jaggi [2009], we know it takes O ( 1 ξδ 2 | C | d ) time.
The total complexity of our convex hull algorithm is O
((˜ k + 1 η ) ˜ k 3 ϵξδ 2 log ˜ kη d ) .
EXPERIMENTAL RESULTS
All the experiments were conducted on an Ubuntu workstation with 2.40GHz Intel(R)
Xeon(R) CPU E5-2680 and 256GB main memory.
The algorithms were implemented in MATLAB R2019b.
For each instance, we repeat the experiment 10 times and report the average results with their standard deviations.
We consider several baseline methods including the 2approximate G ONZALEZ [Gonzalez, 1985] and the recently proposed streaming k -center clustering algorithm CPP [Ceccarello et al., 2019]; we also compare our algorithms with the uniform sampling method Huang et al.
[2018] that is denoted as U NIFORM - r , where r denotes the sampling rate ( e.g., U NIFORM - [-
-]0 .
1 means we take 10% points from the input uniformly at random).
We run our proposed Algorithm 1 and Algorithm 2 on the real image dataset CIFAR-10 [Krizhevsky, 2009] which consists of 60 , 000 color images with each image being represented by a 3072 -dimensional vector.
In Figure 1, we can see that our Algorithm 1 runs significantly faster than the other methods.
Also it is worth emphasizing that in our evaluation, we compute the radius for covering all the input points, rather than excluding the farthest ϵn points as the theoretical analysis in Theorem 1.
We can see that our algorithm and G ONZALEZ can achieve very close radii, even though we did not exclude the farthest ϵn points.
In Figure 2, we illustrate the results of Algorithm 2.
Similar with Figure 1, we can see that our algorithm runs much faster than G ONZALEZ .
An interesting observation is that our algorithm returns much less centers than G ONZALEZ for a fixed radius threshold r 0 (Figure 2 (c)).
We believe one possible reason is that our random sampling approach is more likely to select a point closer to the optimal ball center, and thus the obtained radius can decrease faster, while the greedy selection of G ONZALEZ always selects the most “extreme” point which could be far to the optimal ball center.
We also run the algorithms on another real dataset MNIST [LeCun et al., 1998]; it contains n = 60 ,handwritten digit images from 0 to 9 , where each image is represented by a 784 -dimensional vector. To illustrate the scalability of our algorithms for large-scale data, we enlarge
MNIST by 6 times; namely, for each image vector, we generate 5 copies and add small Gaussian noises to them. The results are shown in Figure 3 and 4.
Due to the space limit, we place the experimental results for convex hull approximation to our full version.
CONCLUSION
In this paper, we propose the sublinear algorithms for greedy selection methods.
Following this work, there are also several interesting problems deserving to study in future.
For example, in our experiments we observe that our random sampling based approach can achieve very close radii with the vanilla greedy selection approach G ONZALEZ (even without excluding the farthest ϵn points).
So we expect to have a strict analysis on this phenomenon in theory, e.g., adding some reasonable assumption to the data distribution from the perspective of beyond worst-case analysis [Roughgarden, 2019].
ACKNOWLEDGEMENTS
The authors would like to thank the anonymous reviewers for their helpful discussions and suggestions on improving this paper.
This work was supported in part by National Key
R & D Program of China No. 2021YFA1000900.