A N E FFICIENT P ROTOCOL FOR D ISTRIBUTED C OLUMN S UBSET S ELECTION IN THE E NTRYWISE (cid:96) p N ORM Anonymous authors
Paper under double-blind review
A BSTRACT
We give a distributed protocol with nearly-optimal communication and number of rounds for Column Subset Selection with respect to the entrywise (cid:96) 1 norm ( k -CSS 1 ), and more generally, for the (cid:96) p -norm with 1 ≤ p < 2 .
We study matrix factorization in (cid:96) 1 -norm loss, rather than the more standard Frobenius norm loss, because the (cid:96) 1 norm is more robust to noise, which is observed to lead to improved performance in a wide range of computer vision and robotics problems.
In the distributed setting, we consider s servers in the standard coordinator model of communication, where the columns of the input matrix A ∈ R d × n ( n (cid:29) d ) are distributed across the s servers.
We give a protocol in this model with (cid:101) O
( sdk ) communication, 1 round, and polynomial running time, and which achieves a multiplicative k 1 p − 12 poly (log nd ) -approximation to the best possible column subset.
A key ingredient in our proof is the reduction to the (cid:96) p, 2 -norm, which corresponds to the p -norm of the vector of Euclidean norms of each of the columns of A .
This enables us to use strong coreset constructions for Euclidean norms, which previously had not been used in this context.
This naturally also allows us to implement our algorithm in the popular streaming model of computation.
We further propose a greedy algorithm for selecting columns, which can be used by the coordinator, and show the ﬁrst provable guarantees for a greedy algorithm for the (cid:96) 1 , 2 norm.
Finally, we implement our protocol and give signiﬁcant practical advantages on real-world data analysis tasks.
I NTRODUCTION Column
Subset Selection ( k -CSS) is a widely studied approach for rank- k approximation and feature selection.
In k -CSS, one seeks a small subset U ∈ R d × k of k columns of a data matrix A ∈ R d × n , typically n (cid:29) d , for which there is a right factor V such that | UV − A | is small under some norm | · | .
k -CSS is a special case of low rank approximation for which the left factor is an actual subset of columns.
The main advantage of k -CSS over general low rank approximation is that the resulting factorization is more interpretable, as columns correspond to actual features while general low rank approximation takes linear combinations of such features.
In addition, k -CSS preserves the sparsity of the data matrix A .
k -CSS has been extensively studied in the Frobenius norm (Guruswami & Sinop, 2012; Boutsidis et al., 2014; Boutsidis & Woodruff, 2017; Boutsidis et al., 2008) and operator norms (Halko et al., 2011; Woodruff, 2014).
A number of recent works (Song et al., 2017; Chierichetti et al., 2017; Dan et al., 2019; Ban et al., 2019; Mahankali & Woodruff, 2020) studied this problem in the (cid:96) p norm ( k -CSS p ) for 1 ≤ p < 2 .
The (cid:96) 1 norm is less sensitive to outliers, and better at handling missing data and non-Gaussian noise, than the Frobenius norm (Song et al., 2017).
Speciﬁcally, the (cid:96) 1 norm leads to improved performance in many real-world applications, such as structure-from-motion (Ke & Kanade, 2005) and image denoising (Yu et al., 2012).
Distributed low-rank approximation arises naturally when a dataset is too large to store on one machine, takes prohibitively long time for a single machine to compute a rank- k approximation, or is collected simultaneously on multiple machines.
Despite the ﬂurry of recent work on k -CSS p , this problem remains largely unexplored in the distributed setting.
This should be contrasted to
Frobenius norm column subset selection and low rank approximation, for which a number of results in the distributed model are known, see, e.g., Altschuler et al.
; Balcan et al.
Boutsidis et al.
We consider a widely applicable model in the distributed setting, where s servers communicate to a central coordinator via 2-way channels.
This model can simulate arbitrary point-to-point communication by having the coordinator forward a message from one server to another; this increases the total communication by a factor of 2 and an additive log s bits per message to identify the destination server.
We consider the column partition model, in which each column of A ∈ R d × n is held by exactly one server.
The column partition model is widely-studied and arises naturally in many real world scenarios such as federated learning (Farahat et al., 2013; Altschuler et al., 2016; Liang et al., 2014).
In the column partition model we typically have n (cid:29) d , i e., A has many more columns than rows.
Hence, we desire a protocol for distributed k -CSS p that has a communication cost that is only logarithmic in the large dimension n , as well as fast running time.
In addition, it is important that our protocol only uses a small constant number of communication rounds (meaning back-and-forth exchanges between servers and the coordinator).
Indeed, otherwise, the servers and coordinator would need to interact more, making the protocol sensitive to failures in the machines, e.g., if they go ofﬂine.
Further, a 1 -round protocol can naturally be adapted to an single pass streaming algorithm when we consider applications with limited memory and access to the data.
In fact, our protocol can be easily extended to yield such a streaming algorithm 1 .
In the following, we denote A i ∗ and A ∗ j as the i -th row and j -th column of A respectively, for i ∈ [ d ] , j ∈ [ n ] .
We denote A T as the subset of columns of A with indices in T ⊆ [ n ] .
The entrywise (cid:96) p -norm of A is | A | p =
( (cid:80) di =1 (cid:80) nj
=1 | A ij | p ) 1 p .
The (cid:96) p, 2 norm is deﬁned as | A | p, 2 =
( (cid:80) dj
=1 | A ∗ j | p 2 ) 1 p .
We consider 1 ≤ p < 2 .
We denote the best rank- k approximation error for A in (cid:96) p norm by OPT :=
min rank-k
A k | A − A k | p .
Given an integer k > 0 , we say U ∈ R d × k , V ∈ R k × n are the left and right factors of a rank- k factorization for A in the (cid:96) p norm with approximation factor α if | UV − A | p ≤ α · OPT .
Since general rank- k approximation in (cid:96) 1 norm is NP hard (Gillis & Vavasis, 2015), we follow previous work and consider bi-criteria k -CSS algorithms which obtain polynomial running time.
Instead of outputting exactly k columns, such algorithms return a subset of (cid:101) O ( k ) columns of A , suppressing logarithmic factors in k or n .
It is known that the best approximation factor to OPT that can be obtained through the span of a column subset of size (cid:101) O ( k ) is Ω( k 1 / 2 − γ ) for p =
(Song et al., 2017) and Ω( k 1 /p − 1 / 2 − γ ) for p ∈ (1 , 2) (Mahankali & Woodruff, 2020), where γ is an arbitrarily
small constant.
1. P REVIOUS A PPROACHES TO k -CSS p IN THE D ISTRIBUTED S ETTING
If one only wants to obtain a good left factor U , and not necessarily a column subset of A , in the column partition model, one could simply sketch the columns of A i by applying an oblivious sketching matrix S on each server.
Each server sends A i · S to the coordinator.
The coordinator obtains U =
AS as a column-wise concatenation of A i S ’s. Song et al.
(2017) showed that AS achieves an (cid:101) O ( √ k ) approximation to OPT , and this protocol only requires (cid:101)
O ( sdk ) communication,
O (1) rounds and polynomial running time.
However, while AS is a good left factor, it does not correspond to an actual subset of columns of A .
Obtaining a subset of columns that approximates A well with respect to the p -norm in a distributed setting is non-trivial.
One approach due to Song et al.
(2017) is to take the matrix AS described above, sample rows according to the Lewis weights (Cohen & Peng, 2015) of AS to get a right factor
V , which is in the row span of A , and then use the Lewis weights of V to in turn sample columns of A .
Unfortunately, this protocol only achieves a loose (cid:101) O
( k 3 / 2 ) approximation to OPT (Song et al., 2017).
Moreover, it is not known how to do Lewis weight sampling in a distributed setting.
Alternatively, one could adapt existing single-machine k -CSS p algorithms to the distributed setting under the column partition model.
Existing works on polynomial time k -CSS p (Chierichetti et al., 2017; Song et al., 2019b; Dan et al., 2019; Mahankali & Woodruff, 2020) give bi-criteria algorithms, and are based on a recursive framework with multiple rounds, which is as follows: in each round, (cid:101) O ( k ) columns are selected uniformly at random, and with high probability, the selected columns can provide a good approximation to a constant fraction of all columns of A .
Among the remaining columns that are not well approximated, (cid:101) O ( k ) columns are recursively selected until all columns of
A are well approximated, resulting in a total of O (log n ) rounds.
A na¨ıve extension of this bi-criteria k -CSS p framework to a distributed protocol requires O (log n ) rounds, as in each round, the servers and the coordinator need to communicate with each other in order to ﬁnd the columns that are covered well and select from the remaining unselected columns.
To reduce this to a single round, one might consider running the O (log n ) round selection procedure on the coordinator only.
In order to do this, the coordinator needs to ﬁrst collect all columns of A from the servers, but directly communicating all columns is prohibitive.
Alternatively, one could ﬁrst apply k -CSS p on A i to obtain factors U i and V i on each server, and then send the coordinator all of the U i and V i .
The coordinator then column-wise stacks the U
i V i to obtain U · V and selects (cid:101) O ( k ) columns from U · V .
Even though this protocol applies to all p ≥ 1 , it achieves a loose O ( k 2 ) approximation to OPT and requires a prohibitive O ( n + d ) communication cost 2 .
One could instead try to just communicate the matrices U i to the coordinator, which results in much less communication, but this no longer gives a good approximation.
Indeed, while each U i serves as a good approximation locally, there may be columns that are locally not important, but become globally important when all of the matrices
A i are put together.
What is really needed here is a small coreset C i for each A
i so that if one concatenates all of the C i to obtain C , any good column subset of the coreset C corresponds to a good column subset for A .
Unfortunately, coresets for the entrywise (cid:96) p -norm are not known to exist.
1. O UR C ONTRIBUTIONS
Our Distributed Protocol We overcome these problems and propose the ﬁrst efﬁcient protocol for distributed k -CSS p ( 1 ≤ p < 2 ) in the column partition model that selects (cid:101) O ( k ) columns of A achieving an (cid:101)
O
( k 1 /p − 1 / 2 ) -approximation to the best possible subset of columns and requires only (cid:101)
O ( sdk ) communication cost, 1 round and polynomial time.
Figure 1 gives an overview of the protocol.
We note that our subset of columns does not necessarily achieve an (cid:101)
O ( k 1 /p − 1 / 2 ) -approximation to OPT itself, although it does achieve such an approximation to the best possible subset of columns.
Using the fact that there always exists a subset of columns providing an (cid:101) O ( k 1 /p − 1 / 2 ) -approximation to OPT (Song et al., 2017), we conclude that our subset of columns achieves an (cid:101)
O ( k 2 /p − 1 ) -approximation to
OPT .
Recently, and independently of our work, Mahankali & Woodruff (2020) show how to obtain a subset of columns achieving an (cid:101)
O ( k 1 /p − 1 / 2 ) -approximation to OPT itself; however, such a subset is found by uniformly sampling columns in O (log n ) adaptive rounds using the recursive sampling framework above, and is inherently hard to implement in a distributed setting with fewer rounds.
In contrast, our protocol achieves 1 round of communication, which is optimal.
We make use of a strong coreset , i.e., a sampled and reweighted subset of columns of each A i that approximates the cost of all potential left factors of A i , by ﬁrst embedding all subspaces spanned by any subset of (cid:101) O ( k ) columns of A from (cid:96) p -space to Euclidean space, to bypass the lack of strong coresets for the (cid:96) p norm.
We denote this new norm as (cid:96) p, 2 norm, which is the sum of the p -th powers of the (cid:96) 2 norms of the columns.
To reduce the error incurred by switching to the (cid:96) p, 2 -norm, we reduce the row dimension of A by left-multiplying by an oblivious sketching matrix S shared across servers, resulting in an overall approximation factor of only (cid:101)
O ( k 1 /p − 1 / 2 ) .
Afterwards, each server sends its strong coreset to the coordinator.
The coordinator, upon receiving the coresets from each server, runs an O (1) -approximate bi-criteria k -CSS p, 2 algorithm to select the ﬁnal column subset, giving an overall (cid:101)
O ( k 1 /p − 1 / 2 ) approximation to the best column subset.
We introduce several new technical ideas in the analysis of our protocol.
Our work is the ﬁrst to apply a combination of oblivious sketching in the p -norm via p stable random variables and strong coresets in the (cid:96) p, 2 norm (Sohler & Woodruff, 2018; Huang & Vishnoi, 2020) to distributed k -CSS.
Furthermore, to show that our oblivious sketching step only increases the ﬁnal approximation error by a logarithmic factor, we combine a net argument with a union bound over all possible subspaces spanned by column subsets of A of size (cid:101) O ( k ) .
Previous arguments involving sketching, such as those by Song et al.
; Ban et al.
; Mahankali & Woodruff (2020), only consider a single subspace at a time.
Theoretical Guarantees and Empirical Beneﬁts for Greedy k -CSS 1 ,
We also propose a greedy algorithm to select columns in the k -CSS 1 , 2 step of our protocol, and show the ﬁrst additive error guarantee compared to the best possible subset A S of columns , i.e., our cost is at most (1 − (cid:15) )
min V | A S V − A
| 1 , 2 + (cid:15) |
A | 1 , 2 .
Similar error guarantees were known for the Frobenius norm (Altschuler et al., 2016), though nothing was known for the (cid:96) 1 , 2 norm.
We also implement our protocol and experiment with distributed k -CSS 1 on various real-world datasets.
We compare the
O (1) -approximate bi-criteria k -CSS 1 , 2 and the greedy k -CSS 1 , 2 as different possible subroutines in our protocol, and show that greedy k -CSS 1 , 2 yields an improvement in practice.
P ROBLEM S ETTING 2. T HE C OLUMN P ARTITION M ODEL
We consider a model where there are s servers, the i th of which holds A i ∈ R d × n i , and a coordinator which initially does not hold any data.
Each server talks only to the coordinator, via a 2 -way communication channel.
The communication cost is the total number of words transferred between the servers and the coordinator over the course of the protocol.
Each word is O (log( snd )) bits.
The overall data matrix A ∈ R d × n is A =
[ A 1 , A 2 , . . .
A s ] (the column-wise concatenation of the A
i ’s).
Here, n is deﬁned to be (cid:80) s i =1 n i .
Typically, in the column partition model, n (cid:29) d .
2. p -S TABLE
AND p - STABLE
Let Z, X 1 , . . .
, X d be random variables drawn i.i.d.
from some distribution D .
D is called p -stable if for an arbitrary vector v ∈ R d , (cid:104) v, X (cid:105) = | v | p Z , where X =
[ X 1 , . .
, X d ] T .
Random variables drawn from such a distribution are called p -stable random variables, which exist for p ∈ (0 , 2] .
Though there is no closed form expression for the p -stable distribution in general except for a few values of p , we can efﬁciently generate a single p -stable random variable in O (1) time using the following method due to Chambers et al.
(1976): if θ ∈ [ − π 2 , π 2 ] and r ∈ [0 , 1] are sampled uniformly at random, then, sin( pθ ) cos 1 /p θ ( cos( θ (1 − p ))
ln( 1 r ) ) 1 − pp follows a p -stable distribution.
P RELIMINARIES FOR O UR P ROTOCOL
We ﬁrst note a standard relationship between the (cid:96) p norm and the (cid:96) p, 2 norm.
Lemma 1.
For a matrix A ∈ R d × n and p ∈ [1 , 2) , | A | p, 2 ≤ | A | p ≤ d 1 p − 12 | A | p, 2 .
3. (cid:96) p - NORM O BLIVIOUS S KETCHING
We left-multiply A by an oblivious sketching matrix S with p -stable random variables so that we lose only an (cid:101) O
( k 1 p − 12 ) approximation factor when we switch to the (cid:96) p, 2 norm.
The purpose of the next two lemmas is to show that we can perform oblivious sketching while preserving the costs of all possible column subsets up to logarithmic factors.
We ﬁrst show a lower bound on the approximation error for a sketched subset of columns, | SA T V − SA | p , which holds simultaneously for any arbitrary subset A T of chosen columns, and for any arbitrary right factor V .
Lemma 2 (Sketched Error Lower Bound) .
Let A ∈ R d × n and k ∈ N .
Let t = k · poly (log( nd )) , and let S ∈ R t × d be a matrix whose entries are i.i.d.
standard p -stable random variables, rescaled by Θ(1 /t 1 p ) .
Then, with probability 1 − o (1) , for all T ⊂ [ n ] with | T | = k · poly (log k ) and for all
V ∈ R | T |× n , | A T V − A | p ≤ | SA T V − SA | p .
Next, we show an upper bound on the approximation error of k -CSS p on a sketched subset of columns, | SA T V − SA | p , which holds for a ﬁxed subset of columns A T and a ﬁxed right factor V .
Lemma 3 (Sketched Error Upper Bound) .
Let A ∈ R d × n and k ∈ N .
Let t = k · poly (log( nd )) , and let S ∈ R t × d be a matrix whose entries are i.i.d.
standard p -stable random variables, rescaled by Θ(1 /t 1 p ) .
Then, for a ﬁxed subset T ⊂ [ n ] of columns with | T | = k · poly (log k ) and a ﬁxed V ∈
R | T |× n , with probability 1 − o (1) , we have min V | SA T V − SA | p ≤ min V O
(log 1 /p ( nd ))
| A T V −
A | p .
3. S TRONG C ORESETS IN THE (cid:96) p, 2 N ORM
To enable sub-linear communication cost in the number n of columns, the i -th server sends the coordinator a strong coreset of columns of SA i , which is a reweighted subset of the columns of
SA i .
Such strong coresets preserve the error incurred by any rank- k projection, up to a constant factor, in the (cid:96) p, 2 norm.
The coreset of a matrix A ∈ R d × n is usually denoted as AT , where
T = D · W ∈ R n × t is a sampling and reweighting matrix and t is the number of columns to be included in the coreset.
The sampling matrix D is a matrix with t columns where each column has only one 1 in the index of the column of A to be included in the coreset and 0 everywhere else.
The reweighting matrix W is a diagonal t × t matrix with weights associated with each sample in the coreset.
Lemma 4 (Strong Coreset in (cid:96) p, 2 norm) .
Let A ∈ R d × n , k ∈ N , p ∈ [1 , 2) , and (cid:15), δ ∈ (0 , 1) .
Then, in n · poly ( k log n/(cid:15) ) time, one can ﬁnd a sampling and reweighting matrix T with O ( d log d/(cid:15) 2 ) · log(1 /δ ) columns such that, with probability 1 − δ , for all rank- k matrices U ,
[Equation]
AT is called a strong coreset of A .
3. P OLYNOMIAL T IME , O (1) - APPROXIMATE B I - CRITERIA k -CSS p,
After server i sends a strong coreset to the coordinator, the coordinator does k -CSS on a column-wise concatenation of these coresets, in the (cid:96) p, 2 norm rather than the (cid:96) p norm.
We give a polynomial time,
O (1) -approximate bi-criteria k -CSS p, 2 algorithm for p ∈ [1 , 2) .
Theorem 5 (Bicriteria O (1) -Approximation
Algorithm for k -CSS p, 2 ) .
Let A ∈ R d × n and k ∈ N .
There exists an algorithm that runs in ( nnz ( A )+ d 2 ) · k poly (log k ) time and outputs a rescaled subset of columns U ∈ R d × (cid:101) O ( k ) of A and a right factor V ∈ R (cid:101) O ( k ) × n for which V = min V | UV − A | p, 2 , such that with probability 1 − o (1) ,
[Equation]
Our polynomial time bi-criteria k -CSS p, 2 algorithm is based on that of Clarkson & Woodruff (2015).
The main difference is that the algorithm of Clarkson & Woodruff (2015) outputs a subset with O ( k 2 ) columns due to the usage of (cid:96) p leverage scores — we reduce the number of selected columns to (cid:101) O
( k ) by using (cid:96) p Lewis weights (Cohen & Peng, 2015).
Details are given in Appendix C.
Algorithm
Initial State: Server i holds matrix
A i ∈ R d × n i , ∀ i s ] .
Coordinator:
Generate a dense p stable sketching d .
Send to all servers.
Server i :
Compute SA i .
Let the number of samples in the coreset be t
= O ( k poly (log( nd ))) .
Construct a coreset of SA i under the (cid:96) p, 2 norm by applying a sampling matrix D i of size n i × t and a diagonal reweighting matrix W i of size t × t .
Let T i
=
D i W i .
Send SA i T i along with A i D i to the coordinator.
Coordinator:
Column-wise stack SA i T
i to obtain SAT =
[ SA 1 T 1 , SA 2 T 2 , . . .
, SA s T s ] .
Apply k -CSS p, 2 on SAT to obtain the indices I of the subset of selected columns with size
O ( k · poly (log k )) .
Since D i ’s are sampling matrices, the coordinator can recover the original columns of A by mapping indices I to A i D i ’s.
Denote the ﬁnal selected subset of columns by A I .
Send A I to all servers.
Server i :
Solve min
V i
|
A
I V i − A
i | p to obtain the right factor V i .
A I and V will be factors of a rankk · poly (log k ) factorization of A , where V is the (implicit) column-wise concatenation of the
V i .
A N E FFICIENT P ROTOCOL FOR D ISTRIBUTED k -CSS p
Theorem 6 (A Protocol for Distributed k -CSS p ) .
In the column partition model, let A ∈ R d
×
n be the data matrix whose columns are partitioned across s servers and suppose server i holds a subset of columns
A i ∈ R d × n i , where n = (cid:80) i ∈ [ s ] n i .
Then, given p ∈ [1 , 2) and a desired rank k ∈ N ,
Algorithm 1 outputs a subset of columns
A I ∈ R d × k poly ( log ( k )) in (cid:101)
O ( nnz ( A )
k
+ kd + k 3 ) time, such that with probability 1 − o (1) ,
[Equation]
Algorithm 1 uses 1 round of communication and (cid:101)
O ( sdk ) words of communication.
Proof.
Approximation Factor .
In the following proof, let L ⊂ [ n ] , | L | =
k denote the best possible subset of k columns of A that gives the minimum k -CSS p cost, i.e., the cost min V |
A L V −
A | p achieves minimum.
First, note that
[Equation]
By Lemma
[Equation]
By Lemma 1, and S has k · poly (log( nd )) rows
SA I is the selected columns output from the bi-criteria O (1)
-approximation k -CSS p, 2 algorithm.
Let ( SAT ) ∗ denote the best rank k approximation to SAT .
By Theorem 5,
[Equation]
Note that SAT =
[ SA 1 T 1 , . . .
, SA s T s ] is a column-wise concatenation of all coresets of SA i , ∀ i ∈ [ s ] .
By Lemma 4,
[Equation] [Equation]
Hence,
[Equation] [Equation]
Therefore, we conclude: min V |
A
I
[Equation]
−
[Equation]
p ≤
[Equation]
O
[Equation]
−
[Equation]
p
[Equation]
Communication Cost.
Sharing the dense p -stable sketching matrix S with all servers costs O ( sdk · poly (log( nd )))
communication (this can be removed with a shared random seed).
Sending all coresets SA i T i ( ∀ i ∈ [ s ] ) and the corresponding columns A
i D i to the coordinator costs (cid:101) O
( sdk ) communication, since each coreset contains only (cid:101) O ( k ) columns.
Finally, the coordinator needs (cid:101) O
( sdk ) words of communication to send the (cid:101) O ( k ) selected columns to each server.
Therefore, the overall communication cost is (cid:101) O
( sdk ) , suppressing a logarithmic factor in n, d .
Running time.
Since generating a single p -stable random variable takes O (1) time, generating the dense p -stable sketching matrix S takes O ( dk · poly (log( nd ))) time.
Computing all SA i ’s takes
O ( nnz ( A ) k · poly (log( nd ))) time.
By Lemma 4, computing a single coreset SA i T i on server i takes time n i poly ( k log n i ) and thus computing all coresets across the servers takes at most time n poly ( k log n ) .
By Theorem 5, the k -CSS p, 2 algorithm takes time ( nnz ( SAT ) + k 2 poly (log nd ))
· k poly (log k ) ≤ k 3 poly (log( knd )) to ﬁnd the set of selected columns.
Since the number of selected columns is O ( k poly (log k )) , it then takes the protocol O
( k poly (log k )) time to map the indices of the output columns from k -CSS p, 2 to recover the original columns
A I .
Therefore, the overall running time for the protocol to ﬁnd the subset of columns
A I is (cid:101) O
(( nnz ( A ) k
+ kd + k 3 ) , suppressing a low degree polynomial dependency on log( knd ) .
After the servers receive A I , it is possible to solve min V i
|
A
I V i − A
i | p in (cid:101)
O ( nnz ( A I ))
+ poly ( d log n ) time , ∀ i ∈ [ s ] due to Wang & Woodruff (2019);
Yang et al.
G REEDY k -CSS 1 ,
We propose a greedy algorithm, shown in Algorithm 2, for k -CSS 1 , 2 , which can be used in the place of the algorithm described in Theorem 5.
The basic version of this algorithm, discussed in
Appendix D, performs k -CSS 1 , 2 by simply selecting the additional column, among those of A , that reduces the approximation error the most at each iteration.
Our analysis of that algorithm is inspired by the analysis of Greedy k -CSS 2 for the Frobenius norm in Altschuler et al.
Here we provide the ﬁrst additive error guarantee, compared to the best possible subset of columns, for the greedy k -CSS 1 , 2 algorithm.
For a faster running time, we make use of the Lazier-than-lazy heuristic described in Section 5.2 of Altschuler et al.
(2016), where in each iteration, rather than considering all columns of A as candidate additional columns of A T , we only sample a subset of the columns of
A of size O ( n log(1 /δ ) k ) , and pick the column among those that improves the objective the most.
Theorem 7.
Let A ∈ R d
× n be the data matrix and k ∈ N be the desired rank.
Let A S be the best possible subset of k columns,
i.e., A S = arg min
A S min V | A S V − A
| 1 , 2 .
Let σ be the minimum non-zero singular value of the matrix B of normalized columns of A S , (the j -th column of B is
B ∗ j =
( A S ) ∗ j / | ( A S ) ∗ j | 2 ).
Then, if T ⊂ [ n ] is the subset of columns selected by Greedy k -CSS 1 , 2 , the following holds with | T | =
Ω( kσ 2 (cid:15)
[Equation] [Equation]
Similarly, if T ⊂ [ n ] is the subset of columns selected by Lazier-than-lazy Greedy k -CSS 1 , 2 , the following holds with | T | =
Ω( kσ 2 (cid:15) 2 ) and δ = (cid:15) :
[Equation]
Algorithm 2 Lazier-than-lazy Greedy k -CSS 1 , 2 .
This version of the greedy algorithm is based on
Section 5.2 of Altschuler et al.
for
T ← A subset of n log(1 /δ )
k columns of A , each selected uniformly at random (excluding the columns whose indices are in T )
Column
[Table]
← arg
[Table]
end for
[Table]
If we let | T | =
Ω( kσ 2 (cid:15) 2 ) , then the overall running time of Algorithm 2 is O ( n log(1 /(cid:15) )
σ 2 (cid:15)· F ) , where
F is the running time needed to evaluate min V | A T ∪ j V − A | 1 , 2 for a ﬁxed j ∈ T .
We can get
F = O ( dk 2 σ 4 (cid:15) 4 + ndkσ 2 (cid:15) 2 ) by taking A † T ∪ j .
Since the error upper bound for greedy k -CSS p, 2 depends on | A | 1 , 2 , it is not directly comparable to the error upper bound for the proposed k -CSS p, 2 from Subsection 3.3, which achieves a multiplicative
O (1) -approximation to the best rank- k approximation.
We empirically compare the two versions of k -CSS p, 2 for p = 1 in Section 6.
E XPERIMENTS
We implement our protocol for distributed k -CSS p in Algorithm 1, setting p = 1 , which enables us to compare two subroutines on the coordinator: Regular k -CSS 1 , 2 from Section 3.3 and Greedy k -CSS 1 , 2 from Section 5.
We compare our k -CSS 1 protocol against a commonly applied baseline for (cid:96) p low rank approximation (used by Song et al. (2019a); Chierichetti et al.
rank- k Singular
Value Decomposition (SVD).
Datasets.
We demonstrate the beneﬁts of our k -CSS 1 protocol on one synthetic data and one real-world application.
We present a summary of the datasets, along with the number of servers s , the column distribution across servers and the rank k we consider for each dataset in Table 1.
The synthetic dataset constructs a data matrix M ∈ R ( k + n ) × ( k + n ) such that the top left k × k submatrix is the identity matrix multiplied by n 3 2 , and the bottom right n × n submatrix has all 1 ’s.
The optimal rank- k left factor consists of one of the last n columns along with k − 1 of the ﬁrst k columns, incurring an error of n 32 in the (cid:96) 1 norm and an error n 3 in the squared (cid:96) 2 norm.
SVD, however, will not cover any of the last n columns, and thus will get an error of n 2 in both the (cid:96) 1 and squared (cid:96) 2 norms.
We set n = 2000 and apply i.i.d.
Gaussian noise to each entry with mean 0 and standard deviation 0 .
We consider a real-world application, term-document clustering, where k -CSS 2 algorithm is previously applied (Mahoney & Drineas, 2009).
TechTCcontains 139 documents processed in bag-of-words representation with a dictionary of 18446 words.
Such representation naturally results in a sparse matrix.
k -CSS p is used to select the top k most representative words.
Hyperparameters.
Experiment hyperparameters are summarized in Table 2.
We denote the number of
[Figure]
cauchy
[Figure]
8k 8k
[Figure]
10k min(4 × cauchy size , 6 k ) min(4 × cauchy size , 6 k ) sketch sizek/ - k/3 sparsitymin(5, k/3) - min(5, k/3)
[Figure]
Setup.
We conduct 15 trials for each experiment setting, and report the average (cid:96) 1 error, i.e. | ˆ A k − A | 1 for a (cid:101) O ( k ) -rank
matrix ˆ A k output by the algorithm that approximates the original data matrix A ∈ R d × n , across all trials.
However, we notice that computing the (cid:96) 1 error for our protocol involves computing an (cid:96) 1 regression on all columns of the data matrix, which takes prohibitively long time on the real-world dataset TechTC .
Therefore, for TechTC , we proceed with an approximate computation as follows: for the output subset of columns
A I , ﬁrst i.i.d.
sample 30% columns from A , denoted as A J , with probability proportional to | A ∗ j | 1 , ∀ j ∈ [ n ] ; then compute a lower bound for the (cid:96) 1 error L = (cid:80) j ∈ J min v | A
I v − A ∗ j | 1 ; after that, sample 20 | A | 1 L columns from A , denoted as A K , with probability again proportional to | A ∗ j | 1 , ∀ j ∈ [ n ] ; ﬁnally, the (cid:96) 1 error is computed as L | A min
| A | 1 .
The approximated (cid:96) 1 error can be shown as an unbiased estimation of min V |
A I V − A | 1 with small variance by Hoeffding’s bound.
Results.
We present our empirical results in Figure 2.
The distributed protocol performs better using G REEDY k -CSS 1 , 2 than R EGULAR k -CSS 1 , 2 on both datasets, and in other settings we include in the supplementary material.
We also conducted experiments on three different datasets, bcsstk13 , isolet , and caltech-101 , extensively comparing the approximation error as well as the running time for different algorithms.
A comprehensive reporting of our results is given in Appendix G.
C ONCLUSION
In this work, we give the ﬁrst nearly-optimal communication and number of rounds protocol for distributed k -CSS p ( 1 ≤ p < 2 ) in the column partition model, which achieves (cid:101)
O
( k 1 /p − 1 / 2 ) - approximation to the best possible subset of columns, with (cid:101) O ( sdk ) communication cost, 1 rounds and polynomial time.
To achieve a good approximation factor, we use dense p -stable sketching and work with the (cid:96) p, 2 norm, which enables us to use an efﬁcient construction of strong coresets and an
O (1) -approximation bi-criteria k -CSS p, 2 algorithm.
We further propose a greedy algorithm for kCSS 1 , 2 and show the ﬁrst additive error upper bound compared to the best possible subset of columns.
We implement our distributed protocol using both greedy k -CSS 1 , 2 and regular k -CSS 1 , 2 .
Our results empirically show that greedy k -CSS 1 , 2 gives substantial improvements over regular k -CSS 1 , 2 on real-world datasets.
For future works, it is not known whether a (cid:101) O
( k 1 /p − 1 / 2 ) -approximation factor to the best possible subset of columns, is optimal for distributed k -CSS p ( 1 ≤ p < 2 ).
A A H IGH C OMMUNICATION C OST P ROTOCOL FOR k -CSS p ( p ≥ 1 )
We start by describing a protocol for distributed k -CSS p , which works for all p ≥ 1 , in the column partition model, and which achieves an O ( k 2 ) -approximation to the best rank- k approximation, using O (1) rounds and polynomial time but requiring a communication cost that is linear in n + d .
The inputs are a column-wise partitioned data matrix A ∈ R d × n distributed across s servers and a rank parameter k ∈ N .
Each server i holds part of the data matrix
A i ∈ R d × n i , ∀ i ∈ [ s ] , and such that (cid:80) si =1 n i = n .
We use a single machine, polynomial time bi-criteria k -CSS p algorithm as a subroutine of the protocol, e.g.,
Algorithm 3 in Chierichetti et al.
(2017), which selects a subset of (cid:101) O ( k ) columns A T of the data matrix
A ∈ R d × n in polynomial time, for which min X | A T X − A | p ≤ O ( k ) min rank-k
A k | A − A k | p , ∀ p ≥ 1 .
Algorithm
A
Initial State: Server i holds matrix
A i ∈ R d × n i , ∀ i ∈ [ s ] .
Server i :
Apply polynomial time bi-criteria k -CSS p on A i to obtain a subset B i of columns as the left factor.
Solve for the right factor V i =
arg
min V i | U
i V i − A
i | p .
Send U i and V i to the coordinator.
Coordinator:
Column-wise concatenate the U
i V i to obtain UV =
[ U 1 V 1 , . .
, U s V s ] .
Apply a polynomial time bi-criteria k -CSS p algorithm on UV to obtain a subset C of columns.
Send C to each server.
Server i :
Solve min
X i | CX i − A
i | p to obtain the right factor.
Approximation Factor.
Let UV denote the column-wise concatenation of the U
i V i .
Let X ∗ =
arg min
X | CX − A | p .
Then, | CX ∗ − AS | p ≤ | CX ∗ − UV | p + | UV − A | p
By the triangle inequality ≤ O ( k ) min rank-k ( UV ) k | UV − ( UV ) k | p + | UV − A | p
By the O ( k ) -approximation of k -CSS p ≤ O ( k ) | UV − A | p =
O ( k )( s (cid:88)
i =
| U
i V i − A
i | p ) ≤ O ( k )( s (cid:88) i =
O ( k ) min rank-k
A ∗ i | A
i − A ∗ i | p )
By the O ( k ) -approximation of k -CSS p ≤ O ( k 2 ) s (cid:88)
i =
| A i − ( A ∗ )
i | p
A ∗ = arg min rank-k
A ∗ | A − A ∗ | p =
O ( k 2 )
| A − A ∗ | p
Communication Cost.
Since U i ∈ R
d × (cid:101) O ( k ) and V i ∈ R (cid:101)
O ( k ) × n i , sending U i and V i costs (cid:101) O
( skn ) .
Since
C ∈ R d × (cid:101) O ( k ) , sending C from the coordinator to all servers costs (cid:101) O
( sdk ) .
Thus the overall communication cost is (cid:101) O
( s ( n + d ) k ) .
Running time.
According to Chierichetti et al.
(2017), applying the k -CSS p algorithm and solving (cid:96) p regression can both be done in polynomial time.
Thus the overall running time of the protocol is polynomial.
Problems with this protocol.
Although this protocol works for all p ≥ 1 , a communication cost that linearly depends on the large dimension n is too high, and furthermore, the output C is not a subset of columns of A , because the protocol applies k -CSS p on a concatenation of both the left factor U i and the right factor
V i .
U i is a subset of columns of A i
but V
i is not necessarily a sampling matrix.
One might wonder whether it is possible that each server only sends U i and the coordinator then runs k -CSS p on a concatenation of the
U i .
This will not necessarily give a good approximation to min rank-k A k | A − A k | p because the columns not selected in the U i locally on each server might become globally important.
Finally, although it is possible to improve the approximation factor to (cid:101)
O
( k ) by making use of an (cid:101) O ( √ k
) -approximation algorithm for (cid:96) p -low rank approximation that also selects a subset of columns (Mahankali & Woodruff, 2020)
, this protocol would still suffer from all of the aforementioned problems.
B P RELIMINARIES FOR O UR U PPER B OUND P ROOFS B. N ORMS
Lemma 1.
(Norm Relationships)
For a matrix A ∈ R d × n , | A | p, 2 ≤ | A | p and | A | p ≤ d 1 p − 12 | A | p, 2 , where 1 ≤ p < 2 .
Proof.
Let x ∈ R d .
For 0 < p < r , | x | r ≤ | x | p ≤ d 1 p − 1 r | x | r
Let r = 2 .
Then we have | x | 2 ≤ | x | p ≤ d 1 p − 12 | x |
Note that | A | p, 2 = (cid:16)(cid:80)
j | A ∗ j | p 2 (cid:17) 1 p and | A | p = (cid:16)(cid:80)
j | A ∗ j | pp (cid:17) 1 p .
Therefore, | A | p, 2 = (cid:32)(cid:88)
j | A ∗ j | p(cid:33) 1 p ≤ (cid:32)(cid:88)
j | A ∗ j | pp (cid:33) 1 p = | A | p and | A | p = (cid:32)(cid:88)
j | A ∗ j | pp (cid:33) 1 p ≤ d 1 p − 12 (cid:32)(cid:88)
j | A ∗ j | p(cid:33) 1 p =
d 1 p − 12 | A | p,
B. L OWER B OUND ON THE C OST - N O C ONTRACTION WHEN A PPLYING A p - STABLE S KETCH
We show a lower bound on the approximation error for a sketched subset of columns, | SA T V − SA | p , in terms of | A
T V − A | p .
The lower bound holds simultaneously for any arbitrary subset A T of chosen columns, and for any arbitrary right factor V .
We begin the proof by ﬁrst showing that applying a dense p -stable sketch to a vector will not shrink its p -norm.
This is done in Lemma 2.1.
We further observe that although p -stable random variables are heavy-tailed, we can still bound their tail probabilities by applying Lemma 9 from Meng & Mahoney (2013).
We note this in Lemma 2.2.
Note that the X i ’s do not need to be independent in this lemma.
Equipped with Lemma 2.1, Lemma 2.2 and a net argument, we can now establish a lower bound on | SA T V − SA | p .
We ﬁrst show in Lemma 2.3 that, with high probability, for any arbitrarily selected subset A T of columns and for an arbitrary column A ∗ j , the error incurred to ﬁt SA ∗ j using the columns of SA T is no less than the error incurred to ﬁt
A ∗ j using the columns of A T .
We then apply a union bound over all subsets T ⊂ [ n ] and columns j ∈ [ n ] to conclude our lower bound in Lemma 2.
Lemma 2.1.
(No Contraction of p -stable Sketch)
Given a matrix S ∈ R t × m whose entries are i.i.d.
p-stable random variables rescaled by Θ (cid:18)t 1 p (cid:19) , where 1 ≤ p < 2 , for any ﬁxed y ∈ R m , with probability 1 − 1 e t , the following holds:
[Equation]
Proof.
By p -stability, we have | Sy | pp = (cid:80) ti =1 (cid:16)
| y | p | Z i | t 1 p (cid:17) p , where | Z i | ≥ 0 are half p -stable random variables.
Since Pr [ | Z i | = Ω(1)] > 12 , by applying a Chernoff bound (to the indicators 1 | Z i |≥ C for a sufﬁciently small constant C ), we have (cid:80) ti =
|
Z i | p
=
Ω( t ) with probability 1 − 1 e t .
Therefore, with probability 1 − 1 e t , | Sy | p ≥ | y | p .
Lemma 2.2.
(Upper Tail Inequality for p -stable Distributions) Let p ∈
(1 , 2) , and m > 3 .
For i ∈ [ m ] , let X i be a standard p -stable random variable, and let γ i > 0 and γ = (cid:80) mi =1 γ
i
Let X = (cid:80) mi
=1 γ i |
X i | p .
Then, for any t ≥ 1 , Pr [ X ≥ tα p γ ] ≤ 2log( mt ) t , where α p > 0 is a constant that is at most 2 p − 1 .
Proof.
Lemma 9 from Meng & Mahoney (2013) for p ∈ (1 , 2) .
Lemma 2.3.
(No Contraction for All Sketched Subsets and Columns) Let A ∈ R d × n , and k ∈ N .
Let t = k · poly (log nd ) , and let S ∈ R t × d be a matrix whose entries are i.i.d.
standard p -stable random variables, rescaled by Θ(1 /t 1 p ) .
Finally, let m = k · poly (log k ) .
Then, with probability 1 − o (1) , for all T ⊂ [ n ] with | T | = m , for all j ∈ [ n ] , and for all y ∈ R | T | ,
[Equation]
Proof.
Step 1: We ﬁrst extend Lemma 2.1 and argue that applying a p -stable sketching matrix S ∈ R t × n will not shrink the norm | Sy | p ≥ | y | p ( 1 ≤ p < 2 ) simultaneously for all y in the column span of [ A T , A j ] =:
A T,j , by a net argument.
In order to bound the p -norm of sketched vectors y in a net, we begin by showing that with high probability all entries in S are bounded.
Let D = poly ( mt ) .
Consider the following two cases:
Case 1: p = 1 .
The entries of the 1 -stable sketching matrix S ij are standard Cauchy random variables.
Consider half-Cauchy random variables
X i,j = | S
i,j | .
The cumulative distribution function of half Cauchy random variables x is F ( x ) =
(cid:82) x2 π
( t 2 +1) dt = 1 − Θ( 1 x ) .
Thus, for any i ∈ [ t ] and j ∈ [ m ] , Pr [ | S ij | ≤ D ] = 1 − Θ( 1 D ) .
Case 2: p ∈ (1 , 2) .
We apply the upper tail bound of p -stable random variables in Lemma 2.2.
For any ﬁxed i ∈ [ t ] and j ∈ [ m ] , Pr [ | S ij | p ≤ D p ] ≥ 1 − Θ( log( t ) D p ) , which implies Pr [ | S ij | ≤ D ] ≥ 1 − Θ( 1 D ) .
Therefore, for p ∈ [1 , 2) , by a union bound over all entries in S , if we deﬁne the event E 1 to mean that for all i ∈ [ t ] and j ∈ [ m ] , we simultaneously have | S ij | ≤ D , then Pr[ E 1 ] ≥ 1 − Θ( mtD ) by a union bound.
The event E 1 occurring implies that for any y ∈ R d , since all entries in S are rescaled by O (1 /t 1 /p ) ,
[Equation]
Consider the unit (cid:96) p ball B = { y ∈ R d : | y | p = 1 , ∃ z ∈ R m s.t.
y
=
A T,j z } in the column span of A
T,j .
A subset N ⊂ B is a γ -net for B if for all y ∈ B there exists some u ∈ N such that | y − u | p ≤ γ , for some distance γ > 0 .
There exists such a net N for B of size |N| =
( 1 γ ) O ( m ) by a standard greedy construction, since the column span of A T,j has dimension at most m + 1 .
We choose γ = 1 m 2 D , and thus |N| ≤ ( m 2 D ) O ( m ) =
D O
( m log m ) .
By applying Lemma 2.1, and a union bound over all vectors y ∈ N , we have that event E 2 : for all y ∈ N simultaneously, | Sy | p ≥ | y | p — E 2 has probability
at least 1 − D O ( m log m ) e t .
Consider an arbitrary unit vector x ∈ B .
There exists some y ∈ N such that | x − y | p ≤ γ = 1 m 2 D .
Conditioning on both events E 1 and event E 2 , we have the following with probability 1 − Θ( mtD )
− D O
( m log m ) e t : | Sx | p ≥ | Sy | p − | S ( x − y )
| p
Triangle Inequality ≥ | y | p − | S ( x − y )
| p
By event E≥ | y | p − D | ( x − y )
| p
Implication of event E≥ | y | p − Dγ
By | x − y | p ≤ γ = | y | p − O ( 1 m 2 ) =
| x | p − O ( 1 m 2 ) | x | p = | y | p =
For a sufﬁciently large m , O ( 1 m 2 ) is at most 12 , and thus | x | p 2 = 12 ≥ 1 m 2 .
This implies | Sx | p ≥ | x | p − | x | p 2 = | x | p 2 .
We can rescale S by a factor of 2 so that | Sx | p ≥ | x | p .
We have shown that | Sy | p ≥ | y | p holds simultaneously for all unit vectors y in the column span of A T,j .
By linearity, we conclude that | Sy | p ≥
| y | p ( 1 ≤ p < 2 ) holds simultaneously for all y in the column span of A T,j .
Step 2: Next, we apply a union bound over all possible subsets T ⊂ [ n ] of chosen columns from A and all possible single columns A ∗ j for j ∈ [ n ] , to argue that | S ( A T y − A ∗ j ) | p ≥ | A T y − A ∗ j | p holds for all y ∈ R | T | and all T ⊂ [ n ] , j ⊂ [ n ] with high probability.
Note that | T | = m =
O ( k · poly (log k )) .
From Step 1 , we have shown that event E 2 fails with probability D O
( m log m ) e t .
Thus event E 2 fails over all possible subsets T and all possible single columns A ∗ j with probability at most
D O ( m log m ) e t · (cid:32) d m −(cid:33) · d ≤ D O ( m log m ) e t · d O ( m ) · d
The above failure probability is o (1) as long as
[Equation]
Thus it sufﬁces to have t = k poly (log nd ) to have failure probability at most 1 e O ( t
[Equation]
Now we condition on a single global event E 1 and that event E 2 holds for all possible T and A ∗ j .
We conclude that with probability 1 − Θ( mtD ) −e O ( t ) , the following holds simultaneously for all T ⊂ [ n ] and for all A ∗ j for which j ∈ [ n ] :
[Equation]
Lemma 2.
(Lower Bound for Sketched Error) Let A ∈ R d × n and k ∈ N .
Let t = k · poly (log( nd )) , and let
S ∈ R t × d be a matrix whose entries are i.i.d.
standard p -stable random variables, rescaled by Θ(1 /t 1 p ) .
Then, with probability 1 − o (1) , for all T ⊂ [ n ] with | T | = k · poly (log k ) and for all V ∈ R | T |× n ,
[Equation]
Proof.
Let y j denote the j -th column of V , where j ∈ [ n ] .
By applying Lemma 2.3, and a union bound over all columns of V , for m = | T | =
k · poly (log k ) , t = k · poly (log( nd )) and D = poly ( mt )
=
poly ( k log( nd )) , the following holds with probability 1 − Θ( mtD ) − n e O ( t )
= 1 − o (1) ,
[Equation]
| p = ( n (cid:88)
[Equation]
= |
[Equation]
p ≤ ( n (cid:88)
[Equation]
=
[Equation]
p
[Equation]
B. U PPER B OUND ON THE C OST
We show an upper bound on the approximation error of k -CSS p on a sketched subset of columns, | SA
T V − SA T | p , which holds for a ﬁxed subset A T of columns and for the minimizing right factor V = arg min
V | SA T V − SA | p for that subset of columns.
We ﬁrst adapt Lemma E.17 from Song et al.
(2017) to establish an upper bound on the error | SA T V − SA | p for any ﬁxed V in Lemma 3.1.
We then apply Lemma 3.1 to the minimizer V to conclude the upper bound in
Lemma 3.
Lemma 3.1.
(An Upper Bound on Norm of A Sketched Matrix)
Given A ∈ R n × d and p ∈ [1 , 2) , and U ∈ R n × k and V ∈ R k × d , if S ∈ R t × n is a dense p -stable matrix, whose entries are rescaled by Θ (cid:18) 1 t 1 p (cid:19) , then with probability at least 1 − O (1) ,
[Equation]
Here, the failure probability O (1) can be arbitrarily small.
Proof.
Lemma E.17 from Song et al.
Lemma 3.
(Upper Bound on Sketched Error) Let A ∈ R d × n and k ∈ N .
Let t = k · poly (log( nd )) , and let
S ∈ R t × d be a matrix whose entries are i.i.d.
standard p -stable random variables, rescaled by Θ(1 /t 1 p ) .
Then, for a ﬁxed subset T ⊂ [ n ] of columns with | T | = k · poly (log k ) , with probability 1 − O (1) , we have min
[Table]
Proof.
Let X ∗ 1 = arg min X | SA T X − SA
| p and X ∗ 2 = arg min
X |
A T X − A | p .
By Lemma 3.1, | SA T X ∗ 1 − SA | pp ≤ | SA T X ∗ 2 − SA | pp ≤ O (log( k poly (log n ) d )) | A T X ∗ 2 − A | pp ≤ O (log( nd ))
| A T X ∗ 2 − A | pp
Therefore,
[Equation]
B. S TRONG C ORESETS IN THE (cid:96) p, 2 N ORM
Lemma 4 (Strong Coreset in (cid:96) p, 2 norm) .
Let A ∈ R d × n , k ∈ N , p ∈ [1 , 2) , and (cid:15), δ ∈ (0 , 1) .
Then, in n · poly ( k log n/(cid:15) ) time, one can ﬁnd a sampling and reweighting matrix T with O ( d log d/(cid:15) 2 ) · log(1 /δ ) columns such that, with probability 1 − δ , for all rank- k matrices U ,
[Equation]
AT is called a strong coreset of A .
Proof.
We can obtain T with O ( d (log d ) /(cid:15) 2 ) columns using the strong coreset construction from Lemmain Sohler & Woodruff (2018).
Note that the coreset construction for k -subspace approximation in Sohler & Woodruff (2018) aims at removing a dependence on d in the coreset size.
The algorithm ﬁrst ﬁnds a poly ( k ) - dimensional subspace S by running a dimensionality reduction algorithm and constructs coresets in the lower dimensional subspace S , resulting in a coreset size of poly ( k/(cid:15) ) .
But in our case, we do not want our coreset size to have a polynomial dependency on k while a linear dependency on d sufﬁces.
Thus, instead of running their dimensionality reduction algorithm to ﬁnd such a subspace S to project A to, we directly use the the column span of the input matrix A as the subspace S appended with a row of zeros to construct the B in Lemma 16 of Sohler & Woodruff (2018).
Note that in the original algorithm, the appended column encodes the distances from the input matrix A to the subspace S , but in our case it is just all
’s. Then the guarantees and running time claimed above immediately follow from Lemma 16 of Sohler & Woodruff (2018).
We note that the size of our coreset for k -subspace approximation can be further reduced to (cid:101)
O ( k ) , suppressing a logarithmic dependence on k,(cid:15) , 1 δ , using an additional O ( nd ) time, by combining Corollary 5.16 of Huang & Vishnoi (2020) and the importance sampling scheme from Stage 2 of Algorithm 1 in Huang & Vishnoi (2020).
Furthermore, though not explicitly stated in Lemma 16 of Sohler & Woodruff (2018), the coreset size has a log(δ ) dependence on the failure probability δ due to the importance sampling (Sohler & Woodruff, 2018; Huang & Vishnoi, 2020).
Note that an accuracy of (1 ± (cid:15) ) is desired for our extension to the streaming model, described in Appendix E, and for both Algorithm 1 and Algorithm 7, we perform a union bound over strong coreset constructions, for which a log(1 /δ ) dependence on δ is sufﬁcient.
C P OLYNOMIAL T IME , O (1) - APPROXIMATE B I - CRITERIA k -CSS p,
We give a detailed analysis on the polynomial time, O (1) -approximate k -CSS p, 2 algorithm presented in Algorithm 4, which is based on ideas in Clarkson & Woodruff (2015).
We ﬁrst use a sparse embedding matrix
S to obtain an O (1) -approximate left factor.
We then use (cid:96) p -Lewis weight sampling Cohen & Peng (2015) to select a subset of columns.
1) -approximation for k -CSS p, 2 ( 1 ≤ p < 2 )
Input: The data matrix A ∈ R d × n , rank k ∈ N
Output: The left factor U ∈ R d × (cid:101) O ( k ) , the right factor V ∈ R (cid:101) O ( k ) × n such that | UV − A | p, 2 ≤
O (1) min rank-k
A k | A k − A | p,
S ← (cid:101) O ( k ) ×
d sparse embedding matrix, with sparsity s = poly (log k ) .
S (cid:48)
← n × (cid:101) O
( k ) sampling matrix, each column of which is a standard basis vector chosen randomly according to the (cid:96) p Lewis weights of columns of SA .
return U ← AS (cid:48)
, V ← ( AS (cid:48) ) † A {† denotes the Moore-Penrose pseudoinverse. }
C. S PARSE E MBEDDING M ATRICES
The sparse embedding matrix S ∈ R (cid:101) O
( k ) × d of Nelson & Nguyen (2013), and used by Clarkson & Woodruff (2015), is constructed as follows: each column of S has exactly s non-zero entries chosen in uniformly random locations.
Each non-zero entry is a random value ± 1 √ s with equal probability.
s is also called the sparsity of S .
Let h be the hash function that picks the location of the non-zero entries in each column of S and σ be the hash function that determines the sign ± of each non-zero entry.
Applying the sparse embedding matrix S to A enables us to obtain a rank- k right factor that is at most a factor of O (1) worse than the best rank- k approximation error in the (cid:96) p, 2 norm.
We adapt Theorem 32 from Clarkson & Woodruff (2015) to show this in Theorem 5.5.
Notice that in Theorem 32 of Clarkson & Woodruff (2015), the number of rows required for S is O ( k 2 ) , but this can be reduced to (cid:101)
O
( k ) through a different choice of hyperparameters when constructing the sparse embedding matrix S .
We note two choices of hyperparameters, i.e., the number m of rows and sparsity s , of S in Theorem 5.1 and Theorem 5.2, both of which give the same result.
The proof of Theorem 32 from Clarkson & Woodruff (2015) uses the hyperparameters from Theorem 5.1.
We replace the hyperparameters from Theorem 5.2 and show in
Lemma 5.3 that (cid:101) O ( k ) rows of S sufﬁce to preserve certain desired properties.
We then combine Lemma 5.3 and Lemma 5.4 adapted from Clarkson & Woodruff (2015), to conclude our result in Theorem 5.5, following the analysis from Clarkson & Woodruff (2015).
Theorem 5.1.
(Theorem 3 from Nelson & Nguyen (2013))
For a sparse embedding matrix S ∈ R m × n with sparsity s = 1 and a data matrix U ∈ R n × d , let (cid:15) ∈ (0 , 1) .
With probability at least 1 − δ all singular values of SU are (1 ± (cid:15) ) as long as m ≥ δ − 1 ( d 2 + d ) / (2 (cid:15) − (cid:15) 2 ) 2 .
For the hash functions used to construct S , σ is 4-wise independent and h is pairwise independent.
Theorem 5.2.
(Theorem 9 from Nelson & Nguyen (2013))
For a sparse embedding matrix S ∈ R m × n with sparsity s = Θ(log 3 ( d/δ ) /(cid:15) ) and a data matrix U ∈ R n × d , let (cid:15) ∈ (0 , 1) .
With probability at least 1 − δ all singular values of SU are (1 ± (cid:15) ) as long as m = Ω( d log 8 ( d/δ ) /(cid:15) 2 ) .
For the hash functions used to construct
S , we have that σ, h are both Ω(log( d/δ ))
-wise independent.
Lemma 5.3.
Let C be a constraint set and A ∈ R n × d , B ∈ R n × d (cid:48) be two arbitrary matrices.
For a sparse embedding matrix S ∈ R m × n , there is m =
O ( d log 8 ( d (cid:15)p +1 ) (cid:15) 2( p +1) ) , such that with constant probability, the following holds for all X ∈ R d × d (cid:48)
i ) | S ( AX − B ) | p, 2 ≥ (1 − (cid:15) )
| AX − B | p,ii ) | S ( AX ∗ − B ) | p, 2 ≤ (1 + (cid:15) )
| AX ∗ − B | p, 2 , where X ∗ =
arg min X ∈C | AX − B | p,
Proof.
The proof is the same as the proof of Lemma 29 from Clarkson & Woodruff (2015), except that we use a different choice of hyperparameters in constructing S , i.e., sparsity s and the number m of rows.
In the proof of
Lemma 29 from Clarkson & Woodruff (2015), the construction of S follows Theorem 5.1, where the sparsity s = 1 , but requires m =
O ( d 2 ) rows.
We replace the construction by Theorem 5.2, where we pick δ = (cid:15) p +1 .
Now the sparsity s is larger but this construction reduces the number of rows required to m = (cid:101) O ( d ) .
Since both choices of hyperparameters to construct S result in bounded (1 ± (cid:15) ) singular values of SU for any data matrix
U , the rest of the proof follows.
Lemma 5.4. Consider a data matrix A ∈ R n × d .
Let the best rank- k matrix in the (cid:96) p, 2 norm be
A k = arg min
rank-k A k | A k − A | p, 2 .
For R ∈ R d × m , if R T satisﬁes both of the following two conditions for all
X ∈ R n × n :
i ) | R T (
[Equation]
T ) | p, 2 ≥ (1 − (cid:15) )
| A Tk X − A T | p,ii ) | R T ( A Tk X ∗ − A T )
| p, 2 ≤ (1 + (cid:15) )
| A Tk X ∗ − A T | p, 2 , where X ∗
= arg
min X
[Equation]
p
[Equation]
then
[Equation]
Proof.
Lemma 31 from Clarkson & Woodruff (2015).
Theorem 5.5.
( (cid:96) p, 2 -Low Rank Approximation) Let the data matrix be A ∈ R d × n and k ∈ N be the desired rank.
Let S ∈ R m × d be a sparse embedding matrix with m =
O ( k poly (log k ) poly ( 1 (cid:15) )) rows, and sparsity s = poly (log k ) .
Then, the following holds with constant probability,
[Equation]
[Table]
Proof.
The proof is the same as the proof of Theorem 32 in Clarkson & Woodruff (2015), except that we adapt a different construction of the sparse embedding matrix S , which reduces the number of rows from O ( k 2 ) to (cid:101) O ( k ) with increased sparsity s .
Consider A k = arg min
rank-k A k | A k − A | p, 2 .
Let V k be a basis for the column space of A k .
By applying Lemma 5.3 and Lemma 5.4 on the basis V k , we conclude the above theorem by setting the number m of rows to m =
O (cid:16) k log 8 ( k(cid:15) 2 ) (cid:15)(cid:17) , and sparsity s = poly (log k ) in the sparse embedding matrix S .
C. L EWIS W EIGHTS S AMPLING
The (cid:96) p Lewis weight is an inherent property of a matrix.
By Cohen & Peng (2015), the unique set of (cid:96) p Lewis weights w for a matrix is deﬁned as follows: for the i -th column of a matrix M ∈ R n × d , M ∗ j , is deﬁned as w 2 /pi
[Equation]
j , where W is the diagonal matrix with W ii = w i , ∀ i ∈
[ d ] .
We ﬁrst note that sampling by Lewis weights provides a subspace embedding in the (cid:96) p -norm in Theorem 5.6.
We further apply a randomized version of Dvoretzky’s Theorem in Theorem 5.7, which allows the embedding from the (cid:96) p norm into a low-dimensional Euclidean subspace with very small distortion and thus enables us to switch between the (cid:96) p norm and the (cid:96) p, 2 norm.
Based on Theorem 5.6 and Theorem 5.7, we show in Theorem 5.8 that Lewis weights sampling provides a good subset of columns, on which the analysis of k -CSS p, 2 is based.
Theorem 5.6.
( (cid:96) p -Lewis
Weights
Subspace Embedding)
Let A ∈ R n × d and t = (cid:101)
O ( d ) .
For 1 ≤ p < 2 , there exists a distribution ( λ 1 , λ 2 , . . .
, λ n ) on the rows of A such that if we generate a matrix S with t rows, each chosen independently as the i th standard basis vector times ( rλ i ) 1 p with probability
λ i , then with probability 1 − o (1) , the following holds for all x ∈ R d ,
[Equation]
Proof.
Theorem 7.1 from Cohen & Peng (2015)
Theorem 5.7.
(Randomized Dvoretzky’s Theorem) Let n ∈ N , and (cid:15) ∈ (0 , 1) .
Let r = n(cid:15) 2 .
Let G ∈ R r × n be a random matrix whose entries are i.i.d.
standard Gaussian random variables, rescaled by 1 √ r .
For r = n(cid:15) 2 , the following holds with probability 1 − e − Θ( n ) , for all y ∈ R n ,
[Equation]
Proof.
This follows from Theorem 1.2 from Paouris et al.
Theorem 5.8.
(Subset of Columns by Lewis Weights Sampling) Let A ∈ R d × n .
Let S ∈ R m × d be a sparse embedding matrix, with m =
O ( k · poly (log k ) poly ( 1 (cid:15) )) .
Further, let
S (cid:48) ∈
R n × t be a sampling matrix whose columns are random standard basis vectors generated according to the (cid:96) p Lewis weights of columns of SA , with t = k · poly (log k ) .
Then, for ˆ X = arg min rank-k X | XSAS (cid:48) − AS (cid:48) | p, 2 , the following holds with constant probability,
[Equation]
Proof.
Let X ∗ =
arg min rank-k X ∗ | X
[Equation]
SA − A | p, 2 .
By the triangle inequality,
[Equation]
Let us consider bounding | X ∗ SA − ˆ XSA | p, 2 .
By Lemma D.28 and Lemma D.29 from Song et al.
(2017), for any column sampling matrix S and for any ﬁxed matrix Y , it can be shown that E [ | Y S | pp ] = | Y S | pp .
In our case, since S (cid:48) is a sampling matrix, we have
E [ | Y S (cid:48) | pp ] = | Y S (cid:48) | pp for any ﬁxed matrix Y .
Now let G ∈ R Θ( d ) ×
d be a rescaled random matrix whose entries are i.i.d.
standard Gaussian random variables as in Theorem 5.7.
We apply Theorem 5.7 to transform between the (cid:96) p space and the Euclidean space.
Since transformation of both directions can be done with very small distortion, we obtain a Θ(1) approximation.
With constant probability, we have
[Equation]
By Theorem 5.
[Equation]
By Theorem 5.
[Equation]
By Theorem 5.
[Equation]
Triangle Inequality
[Equation]
Since ˆ X = arg min rank-k X | XSAS (cid:48) − AS (cid:48) | p,
[Equation]
By Theorem 5.
≤ Θ
[Equation] [Equation]
|
[Equation]
By Markov Bound on E [ | Y S (cid:48) | pp ] = | Y S (cid:48) | =
Θ
[Equation]
By Theorem 5.
Therefore,
[Equation]
By Theorem
[Equation]
C. A NALYSIS FOR k -CSS p,
We are now ready to show an O (1) approximation factor and polynomial running time for the bi-criteria k -CSS p, 2 presented in Algorithm 4, given in Theorem 5.
Theorem 5 (Bicriteria O (1) -Approximation
Algorithm for k -CSS p, 2 ) .
Let A ∈ R d × n and k ∈ N .
There exists an algorithm that runs in ( nnz ( A ) + d 2 ) · k poly (log k ) time and outputs a rescaled subset of columns
U ∈ R d × (cid:101) O
( k ) of A and a right factor V ∈ R (cid:101)
O ( k ) × n for which V = min V | UV − A | p, 2 , such that with probability 1 − o (1) ,
[Equation]
Proof.
Approximation Factor First notice that the minimizer ˆ X of | ˆ XSAS (cid:48) − AS (cid:48) | p, 2 has to be in the column span of AS (cid:48) .
Thus we can write ˆ X
=
( AS (cid:48) )
Y for some matrix Y .
By Theorem 5.8,
[Equation]
We denote Y SA = V .
We take the left factor U = AS (cid:48) and solving for min V | UV − A | p, 2 will give us a Θ(1) approximation to min rank-k A k | A − A k | p, 2 .
A good minimizer for the right factor V in the Euclidean space is
V =
( AS (cid:48) )
† A .
This concludes our result.
Notice that since S (cid:48) is a sampling matrix with (cid:101) O ( k ) columns, we get a rank- k left factor U as a subset of columns of A as desired.
Running time
First notice that S is a sparse embedding matrix with poly (log k ) non-zero entries.
Thus computing SA takes time nnz
( A ) · poly (log k ) .
By Cohen & Peng (2015), computing the Lewis weights of SA takes time nnz ( SA )+ poly ( k ) ≤ nnz ( A ) poly (log k )+ poly ( k ) , and computing the output left factor U = AS (cid:48) takes time nnz
( A ) .
Computing ( AS (cid:48) ) † takes time d 2 · k poly (log k ) .
Computing the right factor V =
( AS (cid:48) )
† A takes nnz ( A )
k poly (log k ) .
Therefore, the overall running time is ( nnz ( A ) + d 2 ) · k poly (log k ) .
D A NALYSIS FOR G REEDY k -CSS 1 ,
We propose a greedy algorithm for selecting columns in k -CSS 1 , 2 presented in Algorithm ?? .
We give a detailed analysis on the ﬁrst additive approximation compared to the best possible subset of columns for Greedy k -CSS 1 , 2 .
Our analysis is inspired by the analysis of Greedy k -CSS 2 for the Frobenius norm in Altschuler et al.
We then describe how the running time of Algorithm ??
can be improved to (cid:101) O ( nσ(cid:15) · F ) , where F is the running time required to evaluate min V | A T ∪ j V − A | 1 , 2 for a ﬁxed j ∈ T (note that this running time is O ( nk 2 + knd ) if we evaluate this by computing the pseudo-inverse of A T ∪ j ).
This improvement in the running time is obtained by randomly sampling candidate columns from T and adding the best of these randomly sampled columns to A
T , rather than trying all columns of T — this method was previously used in Altschuler et al.
(2016) for Frobenius norm column subset selection, where it is referred to as “Lazier-than-lazy Greedy,” and the general approach was ﬁrst introduced in Mirzasoleiman et al.
This version of the greedy algorithm is shown in Algorithm 2, and we show both versions of the greedy algorithm below for convenience.
Algorithm 5 Greedy k -CSS 1 , 2 .
Here we denote the set of selected columns T from A by A T and the set of unselected columns by A T .
do
Column j ∗ ← arg min j ∈ A T (min V | A T ∪ j V − A | 1 , 2 )
A T ← A T ∪ j ∗ end for
Algorithm 6 Lazier-than-lazy Greedy k -CSS 1 , 2 .
This version of the greedy algorithm is based on
Section 5.2 of Altschuler et al.
[Equation]
end for
First we analyze Algorithm 4, then show how this analysis can be extended to Algorithm 6.
We ﬁrst show in Lemma 7.2 an improvement of the utility function with one additional column when projecting a single vector, based on Lemma 7.1 from Altschuler et al.
We then show an improvement of the utility function when projecting a matrix in Lemma 7.3, by applying Lemma 7.2 and Jensen’s Inequality, following the analysis in Altschuler et al.
Finally, we conclude our analysis for Greedy k -CSS 1 , 2 in Theorem 7.
Notation Consider the input matrix A ∈ R d × n ( n (cid:29) d ).
Let B be the matrix of normalized columns of A , where the j -th column of B is B ∗ j =
A ∗ j / | A ∗ j | 2 .
Let π T : R d → R d be the projection onto the column span of A
T or equivalently B T .
Let σ min ( M ) denote the minimum singular value of some matrix M .
To aid our analysis, we deﬁne a utility function as follows, inspired by Altschuler et al.
For a subset T ⊂ [ n ] and a matrix M ∈ R d × t (or a vector M ∈ R d ),
[Equation]
Observe that as the number of columns selected and added to T increases, we get a more accurate estimation of M and thus the approximation error | M − π T M | 1 , 2 decreases, which results in an increase in the utility function Φ M ( T ) .
Lemma 7.1.
Let S, T ⊂ [ n ] be two sets of column indices, with | π S u | 2 ≥ | π T u | 2 for some vector u ∈ R d .
Then,
[Equation]
Proof.
Lemma 2 from Altschuler et al.
(2016), except that we replace the condition for S and T , i.e., Φ u ( S ) ≥ Φ u ( T ) in Altschuler et al.
(2016) with | π S u | 2 ≥ | π T u | 2 .
The two conditions are equivalent, since Φ u ( S ) ≥ Φ u ( T ) ⇔ | u − π S u | 2 ≤ | u − π T u |⇔ | u | 22 − | π S u | 22 ≤ | u | 22 − | π T u |⇔ | π S u | 2 ≥ | π T u |
Lemma 7.2.
(Utility Improvement by Projecting a Single Vector) Let S, T ⊂ [ n ] be two sets of column indices, with Φ u ( S ) ≥ Φ u ( T ) for some vector u ∈ R d .
Let k = | S | .
For i ∈ [ k ] , let T (cid:48)
i
=
T ∪ { i } .
Then,
[Equation]
Proof.
We deﬁne a function for convenience in the analysis g : ( −∞ , | u | 22 ]
→ R ≥ 0 by g ( x ) =
(cid:112) | u | 22 − x .
Note that | g (cid:48)
[Equation]
By deﬁnition of Φ
[Equation]
By Pythagorean Theorem
[Equation]
By the Mean Value Theorem
[Equation]
By Pythagorean Theorem
[Equation]
By Lemma 7.
[Equation]
By Pythagorean Theorem
[Equation]
Since | u − π S u | 2 ≥
We now lower bound | u − π T u | 2 | π S u | 22 as follows.
| u − π T u
||
[Equation]
|
[Equation]
By Pythagorean Theorem = |
[Equation]
|
[Equation]
|
[Equation]
| u |
[Equation]
|
[Equation]
= Φ u
[Equation]
· | u
[Equation]
u |
[Equation]
| u |
[Equation]
| u
[Equation]
π
[Equation]
|
By deﬁnition of Φ ≥ 2Φ u ( S ) · | u − π T u || u |
Since | u − π S u | 2 ≤ | u |= 2Φ u ( S ) · | u | 2 − Φ u ( T ) | u |
By deﬁnition of Φ = 2Φ u ( S ) · (cid:16) 1 − Φ u ( T ) | u |(cid:17) ≥ 2Φ u ( S ) · (cid:16) 1 − Φ u ( T )
Φ u ( S ) (cid:17)
Since Φ u ( S ) ≤ | u |
[Equation]
Therefore,
[Equation]
Lemma 7.3.
(Utility Improvement by Projecting a Matrix) Let A ∈ R d × n , and T, S ⊂ [ n ] be two sets of column indices, with Φ A ( S ) ≥ Φ A ( T ) .
Furthermore, let k = | S | .
Then, there exists a column index i ∈ S such that
[Equation]
Proof.
The proof mostly follows the proof of Lemma 1 in Altschuler et al.
We combine Lemma 7.2 with Jensen’s inequality to conclude an improvement of the utility function with one additional column when projecting a matrix instead of a single column.
For j ∈ [ n ] , we deﬁne δ j = min(1 , Φ A ∗ j ( T ) Φ A ∗ j ( S ) ) .
Note that δ j is 1 if the j -th column A ∗ j has a larger projection onto B T than B S , and Φ A ∗ j ( T ) Φ A ∗ j ( S ) otherwise.
Let k = | S | .
For i ∈ [ k ] , let T (cid:48)
i
=
T ∪ { i } .
[Equation]
=
[Equation]
) −
[Equation]
By deﬁnition of Φ ≥
[Equation] [Equation]
S )
By Lemma 7.
[Equation] [Equation]
ni
[Equation]
S )
Note Φ A ( S ) =
n (cid:88)
i =
Φ A
[Equation]
i ( S )
[Equation] [Equation]
(cid:17)
[Equation]
By Jensen’s Inequality =
[Equation]
[Equation]
Since 1 − δ j ≥ 1 − Φ A ∗ j ( T ) Φ A ∗ j ( S ) ⇒ (
[Equation]
δ j ) · Φ A
[Equation]
j ( S ) ≥ Φ A
[Equation]
j ( S )
[Equation]
Φ A ∗ j ( T ) ≥ 16Φ
[Equation]
16Φ
[Equation]
Hence, k
[Equation]
16Φ
[Equation]
This implies there is at least one column of B S , with index i ∈ S , such that when i is added to T , the utility function Φ A ( T ) increases by at least 1 k · σ min ( B S ) 2 (Φ A ( S ) − Φ A ( T ))
16Φ A ( S ).
Theorem 7.
Let A ∈ R d
× n be the data matrix and k ∈ N be the desired rank.
Let A S be the best possible subset of k columns,
i.e., A S = arg min
A S min V | A S V − A
| 1 , 2 .
Let σ be the minimum non-zero singular value of the matrix B of normalized columns of A S , (the j -th column of B is B ∗ j =
( A S ) ∗ j / | ( A S ) ∗ j | 2 ).
Then, if T ⊂ [ n ] is the subset of columns selected by Greedy k -CSS 1 , 2 , the following holds with | T | =
Ω( kσ 2 (cid:15) 2 ) ,
[Equation]
Proof.
The proof follows the one of Theorem 1 in Altschuler et al.
Let T t be the subset of columns of B selected by Greedy k -CSS 1 , 2 after t iterations.
Notice that T 0 = ∅ .
In addition, deﬁne F = Φ A ( S ) =
Φ A ( S ) − Φ A ( T 0 ) , ∆
= F , and ∆ i =
∆ 0 2 i for i ∈ N .
Let ∆ i ≥ Φ A ( S ) − Φ A ( T t ) ≥ ∆ i +1 =
∆
i 2 .
Our goal is to bound the number of iterations needed for the gap between Φ A ( S ) and Φ A ( T t ) to become less than ∆ i 2 .
Consider some iteration s for which Φ A ( S ) − Φ A ( T s ) ≥ ∆ i 2 .
The improvement of the utility function after adding a column of B to T s through greedy selection is at least the improvement of the utility function after adding the best column of B S to T s .
By Lemma 7.3, this is at least
[Equation]
If the gap Φ A ( S ) − Φ A ( T t ) after t iterations is at most ∆ i and at least ∆ i 2 , then after at most 64 kF 2 σ 2 ∆ 2 i iterations, the gap becomes at most ∆ i 2 .
We can use this to bound the number of iterations required for the gap to become at most (cid:15)F .
Take N ∈ N such that ∆ N +1 ≤ (cid:15)F ≤ ∆ N .
Then the number of iterations required for the gap to become at most ∆ N +1 is at most
N (cid:88)
[Equation]
Since ∆ N +1 = ∆ i 2 N +1 − i
[Equation]
Since ∆ N +1 ≥ (cid:15)F 2 and
N (cid:88) i =
[Equation]
Therefore, after | T | =
Ω( k δ 2 (cid:15) 2 ) iterations, we have Φ
[Equation]
S
[Equation]
⇒ | A | 1 ,
[Equation]
|
[Equation]
Since S is the set of indices for the best possible subset of k columns, the above is equivalent to
[Equation]
We now analyze Algorithm 2, based on the analysis of the Lazier-than-lazy greedy heuristic in Altschuler et al.
The ﬁrst step is the following lemma based on Lemma 6 of Altschuler et al.
(2016), which shows that in expectation, the utility improves by a large amount on each iteration.
Lemma 7.4 (Expected Increase in Utility — Based on Lemma 6 of Altschuler et al. (2016)) .
Let A ∈ R d × n , and let T, S ⊂ [ n ] be two sets of column indices, with k := | S | and Φ A ( S ) ≥ Φ A ( T ) .
Let T be a set of n log(1 /δ )
k column indices of A , chosen uniformly at random from [ n ] \ T .
Then,
[Equation]
Proof.
The proof is nearly identical to the proof of Lemma 6 of Altschuler et al.
(2016) — we include the full proof for completeness.
The ﬁrst step in the proof is showing that T ∩ ( S \ T ) is nonempty with high probability.
Then, by conditioning on T ∩ ( S \ T ) being nonempty, we can show that the expected increase in utility is large.
For the purpose of this analysis, we assume that the columns of T are sampled independently with replacement.
At the end of the proof, we discuss sampling the columns of T without replacement.
First, observe that
[Equation]
By 1 − x ≤ e − x
[Equation]
Because n − | T | < n meaning that
[Equation]
Since | S \ T | ≤ k , and 1 − δ x ≥ (1 − δ ) x for x, δ ∈ [0 , 1]
Therefore,
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
(Since it is always better for T ∩ ( S \ T )
[Table]
be larger)
[Equation]
(Since Φ A ( T ∪ { i } ) =
Φ A ( T ) for i ∈ T )
[Equation]
(See the proof of Lemma 7.3.)
This proves the lemma in the case where the columns are sampled with replacement.
Now, we discuss what happens when sampling without replacement.
Note that the expected increase in utility can only be higher if the columns of T are sampled without replacement.
Intuitively, this is because if T has some repeated columns, then it is always better to replace those repeated columns with other columns of A .
Thus, for each instance of T where some columns are sampled multiple times, we can “move” all of the probability mass from this instance of T to other sets T (cid:48)
⊂
[ n ] \ T , which contain T but do not have repeated elements.
This leads to the uniform distribution on subsets of [ n ] \ T with no repeated elements, i.e., the distribution that results from sampling without replacement.
Using this lemma, we analyze the convergence of Algorithm 2:
Theorem 8.
Let A ∈ R d
× n be the data matrix and k ∈ N the desired rank.
Let A S be the best subset of k columns,
i.e., A S = arg min
A S min V | A S V − A
| 1 , 2 .
Let σ be the minimum non-zero singular value of the matrix B of normalized columns of A S (meaning the j -th column of B is B ∗ j = ( A S ) ∗ j / | ( A S ) ∗ j | 2 ).
Then, if
T ⊂ [ n ] is the subset of columns selected by Algorithm 2, the following holds if | T | =
Ω( kσ 2 (cid:15) 2 ) :
[Equation]
Proof.
The proof is uses the same strategy as that of Theorem 5 of Altschuler et al.
(2016) (and Theorem 7 above), with minor modiﬁcations.
Let T t be the subset of columns of B selected by Algorithm 2 after t iterations (in particular, T 0 = ∅ ).
In addition, let F =
Φ A ( S ) =
Φ A ( S ) − Φ A ( T 0 ) , ∆
= F , and ∆ i +1 =
∆
i 2 .
Now, ﬁx a time t such that for some i , ∆ i ≥ Φ A ( S ) − Φ A ( T t ) ≥ ∆ i +1 =
∆
i 2 .
Then, we bound the number of additional iterations t (cid:48) needed so that
[Equation]
For convenience, for each k ≥ 0 , deﬁne E k :=
Φ A ( T t + k ) .
Then, our goal is to ﬁnd t (cid:48) such that
[Equation]
However, observe that from Lemma 7.4 above, we obtain
[Equation]
By E [ E [ X | Y ]] =
E [ X ]
[Equation]
By Lemma 7.
[Equation]
Jensen’s Inequality
[Equation]
Now, suppose that ∆ i ≥ Φ A ( S ) − E s ≥ ∆ i +1 , for s = 0 , . . .
, t (cid:48) − 1 .
Then, for all such s , E s +1 − E s ≥ (1 − δ ) σ
[Equation]
[Equation]
Summing these inequalities for s = 0 , . .
t (cid:48) − 1 , we ﬁnd that
[Equation]
and for the increase from E 0 to E t (cid:48) to be greater than ∆ i +1 , it sufﬁces to have
[Equation]
In summary, if Φ A ( S ) − E [Φ A ( T t )] ≤ ∆ i , then in at most
[Equation]
, Φ
[Equation]
S ) −
E [Φ A ( T t )] ≤ ∆ i +1 .
Thus, if we let N ∈ N such that ∆ N +1 ≤
[Equation]
√ 1 − δ F ≤ ∆ N , then the number of iterations t needed to have Φ A ( S ) − E [Φ A ( T t )] <
∆ N +1 is at most
N (cid:88) i = 32 kF∆
i +1 · (1 − δ ) σ min ( B S ) 2 = 32 kF(1 − δ )
σ min ( B S )
N (cid:88) i =  ∆ 2 i + = 32 kF(1 − δ ) σ min ( B S )
N (cid:88) i =  4 N − i · ∆ 2 N + ≤ 32 kF(1 − δ ) σ min ( B S ) 2 · 4(1 − δ ) (cid:15) 2 F
N (cid:88) i =  4 N − i
Since ∆ N +1 ≥ (cid:15)F 2 √ 1 − δ = 128 k σ min ( B S ) 2 (cid:15)
N (cid:88) i =  4 i ≤ 512 k 3 σ min ( B S ) 2 (cid:15)
Thus, after t = O ( k σ min ( B S ) 2 (cid:15) 2 ) iterations, Φ A ( S ) − E
[Φ A ( T t )] ≤ (cid:15) √ 1 − δ Φ A ( S ) meaning |
[Equation]
| 1 ,and rearranging gives
E
[Equation]
This completes the proof (note that we can select δ = (cid:15) , meaning 1 √ 1 − δ
= O (1) for (cid:15) < 1 2 ).
E E XTENSION TO THE S TREAMING M ODEL
In this section, we describe how our protocol in Algorithm 1 can be made into a 1 -pass streaming algorithm for column subset selection in the (cid:96) p norm.
The algorithm is shown in Algorithm 7, and is analyzed in Theorembelow.
The algorithm and its analysis follow the standard merge-and-reduce framework (see McGregor (2014)).
Theorem 9 (Analysis of Algorithm 7) .
Let A ∈ R d × n and k ∈ N , and assume Algorithm 7 sees the columns of
A one at a time in the stream S .
Then, Algorithm 7 returns U ∈ R d × (cid:101) O ( k ) such that
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
with probability 0 . 9 .
Moreover, the space complexity of Algorithm 7 is (cid:101) O ( dk ) .
Proof.
Let r = (cid:101)
O
( k ) be the bi-criteria rank of Algorithm 4.
Then, we will be repeatedly applying Lemma 4 with k being equal to r , i.e., we will create coresets which preserve the errors when projecting onto all subspaces of dimension r .
Now, at every iteration of Algorithm 7, each element ( L, t ) of L can be thought of as holding a coreset of the columns of A in some interval I in [ n ] .
We prove the following intermediate lemma by induction on t :
Lemma 10.
Let B be the concatenation of all the sketched columns in L (which have been reweighted by multiple applications of Lemma 4).
Then, for all subspaces V ⊂ R (cid:101) O
( k ) of dimension at most r ,
[Equation]
Proof.
We proceed by induction on t .
The lemma is clear when t = 0 , since in that case, the columns of B are simply sketched columns of A which have not been re-weighted.
Now, suppose t > 0 , and suppose the lemma holds for smaller values of t .
Then, the sketched columns in L must have been obtained as follows: there previously existed two elements ( L 1 , t − 1) and ( L 2 , t − 1) of L , such that if B 1 is the concatenation of the sketched columns in L 1 and B 2 is the concatenation of the sketched columns in L 2 , and T is a coreset for the concatenation B 3 of B 1 and B 2 , then B = B 3 T .
The sketched columns in L 1 and L 2 form coresets for two intervals in [ n ] , which we denote by I 1 and I 2 respectively.
Applying Lemma 4, we ﬁnd that
[Equation]
1) where the second and last equalities are because the p th power of the (cid:96) p, 2 norm of a matrix decomposes across the columns of the matrix, and the third equality is by the induction hypothesis.
By taking p th roots, we ﬁnd that
[Equation]
This proves the lemma.
Now, if L is the unique element of L remaining at the end of the for loop in Algorithm 7, let B be the concatenation of the sketched columns in L .
Then, L is a coreset for all the columns of A , and by the above lemma, the distortion of L is at most (1 ± 1 log n ) log n ∈ [ 1 e , e ] (since t ≤ log n — note that, as in all applications of the merge-and-reduce framework, the coresets contained in L over the course of the algorithm form a binary tree, with the leaf nodes being contiguous intervals of length (cid:101) O ( k ) ).
Hence, for all subspaces V of dimension at most r ,
[Equation]
and in particular, if M ∈ R (cid:101) O ( k ) × (cid:101) O ( k ) is the matrix formed by concatenating the columns of B selected by running Algorithm 4, then
[Equation]
where P M on the left-hand side denotes the projection onto the column span of M , and P T on the right-hand side denotes the projection to the column span of SA T .
Hence, by Lemma 1,
[Equation]
meaning that if U ∈ R d × (cid:101) O ( k ) is the matrix whose columns are the unsketched columns corresponding to M , then
[Equation]
and by Lemmas 2 and 3, this means
[Equation]
Now, we analyze the space complexity of Algorithm 7.
Note that at any iteration of the algorithm, L can hold at most log n lists of columns (since each element of L is a coreset corresponding to an interval of column indices in [ n ] of size 2 k for some k ∈ N , and if two adjacent coresets are of the same size then they will have been merged, so all coresets in L are of different sizes).
Each list of columns is of size at most (cid:101) O ( k ) , and each column has d entries, meaning the amount of space used in any iteration is at most (cid:101)
O ( dk ) .
F C OMPARISON OF O UR P ROTOCOL WITH THE D ISTRIBUTED G REEDY P ROTOCOL OF A LTSCHULER ET AL . (2016) FOR THE F ROBENIUS N ORM
In this section, we perform an empirical comparison of our protocol with the distributed greedy protocol for column subset selection in the Frobenius norm due to Altschuler et al.
F. D ISTRIBUTED G REEDY P ROTOCOL OF A LTSCHULER ET AL . (2016)
We ﬁrst recall the distributed protocol of Altschuler et al.
(2016), in Algorithm 8.
Note that it is a bi-criteria algorithm, i.e., more than k columns are selected, and in Altschuler et al.
it is shown that this gives a good approximation relative to the optimal column subset for the Frobenius norm.
Na¨ıvely, this would require a large communication cost, since the entire data matrix A would have to be com- municated between the servers and the coordinator, in order for them to perform the calls G REEDY ( A, T i , 32 kσ ) and G REEDY ( A, T, 12 kσ ) .
Instead, Altschuler et al.
(2016) uses the technique of Projection-Cost Preserving sketches, developed in Cohen et al.
(2015), in which a key result is as follows:
Theorem 11 (Projection-Cost Preserving Sketches - Theorem 4 of Cohen et al.
(2015), as stated in Altschuler et al.
Let R be a random matrix with n rows and n (cid:48)
=
O ( k +log( 1 δ ) (cid:15)) columns, where each entry is independently and uniformly set to ± (cid:113) 1 n (cid:48) .
Then for any matrix A ∈ R d × n , with probability 1 − O ( δ ) , the following holds: for some constant c ≥ 0 , for any k -dimensional subspace U of R d , if P U ∈ R
d × d is the corresponding projection matrix, then
[Equation]
that is, AR is a Projection-Cost Preserving sketch for A .
In other words, R preserves the cost of all projections of A onto rank- k subspaces of R d .
Hence, when each server performs the call G REEDY ( A, T i , 32 kσ ) , in place of A , it can use a projection-cost preserving sketch AR for A (and the coordinator can similarly use AR when it makes the call G REEDY ( A, T, 12 kσ ) ).
This is because the calls to G REEDY repeatedly compute the cost of the projection of the matrix A onto various column subsets of T i or T , and hence all that is needed is a way to efﬁciently compute the cost of the projection of A onto various subsets of size O ( kσ ) .
AR can be computed without communicating the entire matrix A between the servers, as follows:
• First, the coordinator generates R ∈ R n × n (cid:48)
(where n (cid:48) = O ( k ) )
as speciﬁed in Theorem 11.
This is sent to all the servers (which can be done with negligible cost if a random seed is sent, for example).
• Suppose the i th server holds A i , which consists of columns with indices s i through t i of A .
Then, if R i is the ( t i − s i +1)
× n (cid:48) submatrix of R consisting of rows
s i through t i , then the i th server can send A
i R i to the coordinator.
Since A i R i is a d × O ( k ) matrix, this step takes O ( sdk ) communication.
• Finally, by deﬁnition of matrix multiplication, AR = (cid:80) si =1 A
i
R i .
The coordinator performs this computation and sends AR to each server.
This step also takes O ( sdk ) communication.
Algorithm 7 1 -pass streaming algorithm for k -CSS p .
L is a collection of coresets of columns.
Over the course of the algorithm, each element L ∈ L will represent a contiguous subset of 2 t columns, for some t ∈ [log n ] — to determine when two coresets should be merged, we will also keep track of the size of each coreset in L .
Hence, each element of L is of the form ( L, t ) where L is a list of sketched columns (and their corresponding unsketched columns) and t is the number of times this list has been involved in a merging operation.
Input: A stream S in which the columns of the data matrix A ∈ R d × n arrive one at a time, and the target rank k ∈ N
Output: The left factor U ∈ R d × (cid:101) O
( k )
S ← An (cid:101) O ( k ) ×
d random matrix with i.i.d.
standard p -stable entries (cid:15) ← 1 log n δ ← O ( 1 n ) f ← (cid:101) O ( k )
L ← ∅ for Each column A ∗ j that arrives from S do if L is empty.
then
L ← {} , where {} is the empty list.
else if The last element ( J, t ) of L is such that t =
(i.e., it has been merged 0 times).
then
L ← J
Remove ( J, t ) from the end of L .
else
L ← {} end if
L ← L ∪ { ( SA ∗ j , A ∗ j ) }
L ← L ∪ ( L, 0) /
* Now, we merge coresets in L as much as possible.
*/ while True do
Exit this while loop if L has only 1 element.
( L, t ) ← last element of L — remove this from L .
( L (cid:48) , t (cid:48) )
← last element of L — remove this from L .
If t (cid:54) = t (cid:48) , then add ( L (cid:48) , t (cid:48) )
back to L , and add ( L, t ) back to L .
Then, exit this while loop.
C ← A coreset of L ∪ L (cid:48) , computed as speciﬁed in Lemma 4 — the k in the statement of
Lemma 4 will be the bicriteria rank of Algorithm 4, which is (cid:101) O ( k ) .
The parameters δ and (cid:15) will be as speciﬁed at the beginning of this algorithm.
(Only compute the coreset of the sketched columns — for those which are included in the coreset, include the corresponding unsketched columns as well (but without re-weighting them)).
L ← ( C, t + 1) — note that for each of the re-scaled columns that are included in C , we include their original indices in A as well.
end while end for
L ← The unique element of L
L (cid:48) ← The result of running Algorithm 4 on the sketched columns in L — for each of these sketched columns, store the corresponding unsketched column as well.
U ← The d × (cid:101) O ( k ) matrix whose columns are the unsketched columns in L (cid:48) return U
Algorithm 8 Distributed Greedy Column Subset Selection for the Frobenius Norm (Algorithmof Altschuler et al.
Throughout this protocol, G REEDY ( A, T, r ) denotes a single-machine procedure, which does the following: if A ∈ R d × n is a data matrix, and T ∈ R d × t is a set of columns (not necessarily of A ) then a subset S of r columns of T is constructed iteratively over r steps, such that at each step, the new column of T to add to S is greedily chosen — that is, the chosen column increases | π S A | 22 the most, or equivalently, decreases | A − π S A
| 22 the most.
In other words, it is the same as our Algorithm ??
, but for the Frobenius norm rather than the (cid:96) 1 , 2 -norm.
Input: The data matrix A ∈ R d × n , target rank k ∈ N , the number of servers s ∈ N .
The columns of A are assumed to be randomly partitioned among servers T 1 , T 2 , . .
, T s .
Output: A subset of columns A T from A .
| T | =
O ( kσ ) , where if A OPT is the optimal subset of columns of A of size k , and the columns of A OPT are normalized to have unit (cid:96) 2 norm, then σ is the smallest singular value of A OPT .
S i ← G REEDY
( A, T i , 32 kσ ) for all i ∈ [ s ] (The i th server performs this computation.)
Each server sends its S i to the coordinator.
T ← ∪ si =1 S i
(This computation is done by the coordinator.)
S ← G REEDY
( A, T, 12 kσ )
The coordinator returns arg max S (cid:48) ∈{ S,S 1 ,S 2 ,...,S s } | π S (cid:48)
A |
In our implementation of Algorithm 8, we make use of Projection-Cost Preserving sketches (PCPs).
Another optimization described in Altschuler et al.
(2016), which we use in our implementation, is the L AZIERTHAN -L
AZY -G REEDY A LGORITHM .
The difference between the G REEDY ( A, T, r ) algorithm and the L
AZIER - THAN -L
AZY -G
REEDY ( A, T, r ) algorithm is as follows: while at each of the r iterations, G REEDY considers all columns of T and chooses the one that leads to the most improvement in the objective,
L AZIERTHAN -L
AZY -G
REEDY samples | T | log( 1 δ ) r columns uniformly at random for some small δ (which we take in our implementation to be 0 . 005 ), and out of those columns, chooses the one which leads to the greatest improvement (where | T | is the number of columns in T ).
This leads to a signiﬁcant speedup to G REEDY , and it was shown in Altschuler et al.
that this does not signiﬁcantly worsen the approximation guarantees that Altschuler et al.
(2016) shows for G REEDY .
F. S ETUP
We compare our distributed protocol, in the case p = 1 , to Algorithm 8.
For both protocols, at the outset we ﬁx the number of columns which are selected.
Algorithm 8 is rewritten as Algorithm 9 to reﬂect this.
In this section, we use k to denote the number of columns ultimately selected.
Algorithm 9 Distributed Greedy Column Subset Selection for the Frobenius Norm (Algorithm 2 of
Altschuler et al.
For our empirical comparison, we ﬁx the number of columns selected at the outset.
Input: The data matrix A ∈ R d × n , desired number of columns k ∈ N , the number of servers s ∈ N .
The columns of A are assumed to be randomly partitioned among servers T 1 , T 2 , . .
, T s .
Output: A subset of columns A T from A , with | T | = k .
AR ← a PCP for A as discussed in the previous section.
All servers and the coordinator have access to AR .
S i ← G REEDY ( AR, T i , k ) for all i ∈ [ s ] (The i th server performs this computation.)
Each server sends its S i to the coordinator.
T ← ∪ si =1 S i
(This computation is done by the coordinator.)
S ← G REEDY ( AR, T, k )
The coordinator returns arg max S (cid:48) ∈{ S,S 1 ,S 2 ,...,S s } | π S (cid:48)
( AR ) |
Note that in Algorithm 9, the coordinator now chooses as many columns as chosen by the servers, as opposed to Algorithm 8, where it chooses a somewhat smaller number of columns — note that this cannot harm the performance of the algorithm.
We compare our protocol to Algorithm 9, using the following datasets:
• gastro lesions , a 76 × 698 matrix dataset available at https://archive.ics.uci.edu/ ml/datasets/Gastrointestinal+Lesions+in+Regular+Colonoscopy .
• secom ,
a
591 × 1567 matrix dataset.
secom has missing entries, which we replace with 0 s for the purposes of our experiments.
secom is available at https://archive.ics.uci.edu/ml/ datasets/SECOM
F.2. P ARAMETERS
We compare our distributed protocol for k -CSS 1 , using Greedy k -CSS 1 , 2 as a subroutine, with Algorithm 9, for several choices of k — on gastro lesions , we let k ∈ { 10 , 20 , 30 } , while on secom , we let k ∈ { 30 , 60 , 90 , 120 } .
For our protocol, cauchy size is set to 2 k and coreset size is set to 5 k for both datasets (where cauchy size and coreset size have the same meanings as in the main body of this paper.)
For Algorithm 9, the number of columns in the PCPs is set to 8 k on gastro lesions , and 7 k on secom .
The hyperparameters coreset size and cauchy size , and the number of columns in the PCPs, are set this way so that the amount of communication that each algorithm is allowed is roughly equal (Algorithm 9 is allowed slightly more communication).
To see this, let d be the number of rows in the data matrix A .
If c is the number of columns in the PCPs, then the total communication used to transmit the PCPs between servers is 2 scd , since the servers must ﬁrst send their respective A
i R i to the coordinator, and the coordinator then sends AR to all of the servers.
By comparison, if r 1 is the number of rows in the initial Cauchy matrix in our protocol, and r 2 is the number of columns in each coreset sent by the servers to the coordinator, then the total communication required to transmit the Cauchy matrix and the coresets is ( r 1 + r 2 ) sd .
We choose r 1 , r 2 and c so that 2 scd is slightly higher than ( r 1 + r 2 ) sd .
Note that if k is the number of columns ultimately selected, then our protocol uses 2 dk additional bits of communication between the servers and the coordinator to recover the ﬁnal k -subset of columns, while Algorithm 9 will use sdk communication to send the subsets of columns S i (of size k ) from the servers to the coordinators.
This is not included in our hyperparameter calculations.
F.2. H OW T RIALS ARE C ONDUCTED
With these choices of hyperparameters, we conduct 15 trials as follows.
In each trial, if A ∈ R d × n is our data matrix, then we shufﬂe the columns and then divide them equally between 2 servers (since for the theoretical guarantees of Algorithms 8 9 to apply, the columns should be partitioned randomly).
Using this partition across 2 servers, we run our protocol and Algorithm 9.
For each protocol, once the subset of columns is computed, we perform multiple-response (cid:96) 1 regression to evaluate the error.
In the next section, we report the minimum error across the 15 trials for each dataset and for each value of k .
We also report the mean error, along with the standard deviation.
Finally, we compute the work and the span of our protocol and Algorithm 9 using Python’s time.process time() utility.
This does not include the time taken to perform multiple-response (cid:96) 1 regression.
For secom , with k = 30 and k = 60 , the trials were performed on a Late-2016 Macbook Pro with a 2.7 GHz
Quad-Core Intel Core i7 processor and 16GM of memory.
The rest were performed on a 2019 MacBook Pro with a 2.4 GHz Intel Core i5 processor and 8 GB 2133 MHz LPDDR3 memory.
F. R ESULTS
Our results are shown and discussed in Figures 3 (minimum and mean/standard deviation for (cid:96) 1 error) and 4 (average work and span across 15 trials).
G F ULL E XPERIMENTAL R ESULTS
For our protocol, we considered various additional hyperparameter settings on real-world datasets ( bcsstk13 , isolet and 5 images in the Caltech-101 dataset) — these settings are shown in Tables 3, 4, and 5.
As before, cauchy size is the number of rows in our initial Cauchy matrix, sent to the servers at the beginning of the protocol, and coreset size is the size of the strong coreset sent by each server to the coordinator.
For regular k -CSS 1 , 2 , sketch size is the number of rows in the sparse embedding matrix, and sparsity is the number of nonzero entries in each column of the sparse embedding matrix.
For each of these settings, we display the minimum error, as well as the mean error and the standard deviation for each setting and each rank, as shown in Figure 5.
We also display the average work/span of each setting, for each rank, as shown in Figure 6.
We observe that not only does greedy k -CSS 1 , 2 perform better than all settings of regular k -CSS 1 , 2 in minimum and mean errors across multiple trials, it also has a smaller variance in performance.
G. D ETAILS OF W ORK /S PAN C OMPUTATION
In Figure 6, work and span were recorded as the time taken (using Python’s time
.process
time() utility) to compute the column subset in our distributed protocol.
In particular, it does not include the time spent performing (cid:96) 1 regression to obtain the right factor and consequently the entry-wise (cid:96) 1 errors.
[Figure]
[Figure]
entries in any column.
(There is a slight typo in Table 2 in Section 8 of our main paper, where coreset size is given as 10 k .)
[Table]
[Figure] [Figure]
[Figure]
[Table]
Figure 5 shows that we encounter a tradeoff between accuracy and running time when choosing between these two subroutines.
Both lead to the same overall communication cost, and since accuracy is of more interest than running time in the distributed setting, it is (empirically) preferable to use Greedy k -CSS 1 , 2 within the protocol.
G. A DDITIONAL D ETAILS
All experiments on the Caltech-101 dataset were run on a Late-2016 Macbook Pro with a 2.7 GHz QuadCore Intel Core i7 processor and 16GM of memory.
All experiments on bcsstk13 and isolet were run on an AWS z1d.xlarge instance with Deep Learning AMI (Amazon Linux 2) Version 29.0.