Original: A DAPTIVE S INGLE -P ASS S TOCHASTIC G RADIENT D ESCENT IN I NPUT S PARSITY T IME
Revised: A DAPTIVE S INGLE -P ASS S TOCHASTIC G RADIENT D ESCENT IN I NPUT S PARSITY T IME

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study sampling algorithms for variance reduction methods for stochastic optimization.
Revised: We study sampling algorithms for variance reduction methods for stochastic optimization.

Original: Although stochastic gradient descent (SGD) is widely used for large scale machine learning it sometimes experiences slow convergence rates due to the high variance from uniform sampling.
Revised: Although stochastic gradient descent (SGD) is widely used for large scale machine learning it sometimes experiences slow convergence rates due to the high variance from uniform sampling.

Original: In this paper, we introduce an algorithm that approximately samples a gradient from the optimal distribution for a common ﬁnite-sum form with n terms, while just making a single pass over the data, using input sparsity time and
Revised: In this paper, we introduce an algorithm that approximately samples a gradient from the optimal distribution for a common ﬁnite-sum form with n terms, while just making a single pass over the data, using input sparsity time and

Original: ˜ O
Revised: ˜ O

Original: ( Td ) space.
Revised: ( Td ) space.

Original: Our algorithm can be implemented in big data models such as the streaming and distributed models.
Revised: Our algorithm can be implemented in big data models such as the streaming and distributed models.

Original: Moreover, we show that our algorithm can be generalized to approximately sample Hessians and thus provides variance reduction for second-order methods as well.
Revised: Moreover, we show that our algorithm can be generalized to approximately sample Hessians and thus provides variance reduction for second-order methods as well.

Original: We demonstrate the efﬁciency of our algorithm on large-scale datasets
Revised: We demonstrate the efﬁciency of our algorithm on large-scale datasets

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There has recently been tremendous progress in variance reduction methods for stochastic gradient descent (SGD) methods for the standard convex ﬁnite-sum form optimization problem min x ∈ R d F ( x ) :
Revised: There has recently been tremendous progress in variance reduction methods for stochastic gradient descent (SGD) methods for the standard convex ﬁnite-sum form optimization problem min x ∈ R d F ( x ) :

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =

Revised: =

Original: f i ( x ) , where f 1 , . . . ,
Revised: f i ( x ) , where f 1 , . . . ,

Original: f n : R d → R is a set of convex functions that commonly represent loss functions.
Revised: f n : R d → R is a set of convex functions that commonly represent loss functions.

Original: Whereas gradient descent (GD) performs the update rule x t +

Revised: Whereas gradient descent (GD) performs the update rule x t +

Original: = x
Revised: = x

Original: t − η t ∇ F ( x t ) on the iterative solution x t at iterations t = 1 , 2 , . . . , SGD (Robbins & Monro, 1951; Nemirovsky &
Revised: t − η t ∇ F ( x t ) on the iterative solution x t at iterations t = 1 , 2 , . . . , SGD (Robbins & Monro, 1951; Nemirovsky &

Original: Yudin, 1983; Nemirovski et al., 2009) picks i t ∈ [ n ] in iteration t with probability p
Revised: Yudin, 1983; Nemirovski et al., 2009) picks i t ∈ [ n ] in iteration t with probability p

Original: i t and performs the update rule x t
Revised: i t and performs the update rule x t

Original: +1 = x
Revised: +1 = x

Original: t − η t np it ∇ f i t ( x t ) , where ∇ f i t is the gradient (or a subgradient) of f i t and η t is some predetermined learning rate.
Revised: t − η t np it ∇ f i t ( x t ) , where ∇ f i t is the gradient (or a subgradient) of f i t and η t is some predetermined learning rate.

Original: Effectively, training example i t is sampled with probability p i t and the model parameters are updated using the selected example.
Revised: Effectively, training example i t is sampled with probability p i t and the model parameters are updated using the selected example.

Original: The SGD update rule only requires the computation of a single gradient at each iteration and provides an unbiased estimator to the full gradient, compared to GD, which evaluates n gradients at each iteration and is prohibitively expensive for large n .
Revised: The SGD update rule only requires the computation of a single gradient at each iteration and provides an unbiased estimator to the full gradient, compared to GD, which evaluates n gradients at each iteration and is prohibitively expensive for large n .

Original: However, since SGD is often performed with uniform sampling so that the probability p
Revised: However, since SGD is often performed with uniform sampling so that the probability p

Original: i,t of choosing index i ∈ [ n ] at iteration t is p i,t = 1 n at all times, the variance introduced by the randomness of sampling a speciﬁc vector function can be a bottleneck for the convergence rate of the iterative process.
Revised: i,t of choosing index i ∈ [ n ] at iteration t is p i,t = 1 n at all times, the variance introduced by the randomness of sampling a speciﬁc vector function can be a bottleneck for the convergence rate of the iterative process.

Original: Thus the subject of variance reduction beyond uniform sampling has been well-studied in recent years (Roux et al., 2012; Johnson & Zhang, 2013; Defazio et al., 2014; Reddi et al., 2015; Zhao & Zhang, 2015; Daneshmand et al., 2016;
Revised: Thus the subject of variance reduction beyond uniform sampling has been well-studied in recent years (Roux et al., 2012; Johnson & Zhang, 2013; Defazio et al., 2014; Reddi et al., 2015; Zhao & Zhang, 2015; Daneshmand et al., 2016;

Original: Needell et al., 2016; Stich et al., 2017; Johnson & Guestrin, 2018; Katharopoulos & Fleuret, 2018; Salehi et al., 2018;
Revised: Needell et al., 2016; Stich et al., 2017; Johnson & Guestrin, 2018; Katharopoulos & Fleuret, 2018; Salehi et al., 2018;

Original: Qian et al., 2019).
Revised: Qian et al., 2019).

Original: A common technique to reduce variance is importance sampling, where the probabilities p i,t are chosen so that vector functions with larger gradients are more likely to be sampled.
Revised: A common technique to reduce variance is importance sampling, where the probabilities p i,t are chosen so that vector functions with larger gradients are more likely to be sampled.

Original: Thus for
Revised: Thus for

Original: Var( v ) :
Revised: Var( v ) :

Original: = E (cid:104) (cid:107) v (cid:107) 22 (cid:105) − (cid:107) E [ v ] (cid:107) 22 , for a random vector v , then p i,t = 1 n for uniform sampling implies σ 2 t =
Revised: = E (cid:104) (cid:107) v (cid:107) 22 (cid:105) − (cid:107) E [ v ] (cid:107) 22 , for a random vector v , then p i,t = 1 n for uniform sampling implies σ 2 t =

Original: Var (cid:18) np i t ,t ∇ f i t (cid:19) =
Revised: Var (cid:18) np i t ,t ∇ f i t (cid:19) =

Original: n(cid:32) n n (cid:88) i
Revised: n(cid:32) n n (cid:88) i

Original: =
Revised: =

Original: (cid:107)∇ f i ( x t ) (cid:107)2 − (cid:107)∇ F ( x t ) (cid:107)(cid:33) , whereas importance sampling with p i,t = [Table] x [Table]
Revised: (cid:107)∇ f i ( x t ) (cid:107)

Original: gives

Original: [Equation]
Revised: [Equation]

Original: Var
Revised: − n

Original: [Equation]
Revised: [Equation]

Original: (cid:19) [Equation] n
Revised: · (cid:107)∇ F ( x t ) (cid:107)(cid:33) , whereas importance sampling with p i,t =

Original: [Equation]
Revised: [Equation]

Original: (cid:32)

Original: [Equation] (cid:88) i = [Table] (cid:107)∇ f
Revised: gives

Original: [Equation]
Revised: [Equation]

Original: x
Revised: Var

Original: [Equation]
Revised: [Equation]

Original: (cid:107) (cid:33) [Table] [Equation]
Revised: 

Original: (cid:107)∇ [Equation] ( x
Revised: 

Original: [Equation]
Revised: [Equation]

Original: (cid:107)  , which is at most 1 n 2 (cid:16) n (cid:80) (cid:107)∇ f i ( x t ) (cid:107) 2 −  (cid:107)∇ F ( x t ) (cid:107) 2 (cid:17) , by the Root-Mean Square-ArithmeticMean
Revised: which is at most 1 n 2 (cid:16) n (cid:80) (cid:107)∇ f i ( x t ) (cid:107) 2 − n 2 · (cid:107)∇ F ( x t ) (cid:107) 2 (cid:17) , by the Root-Mean Square-Arithmetic

Original:  Inequality, and can be signiﬁcantly less.
Revised: Mean Inequality, and can be signiﬁcantly less.

Original: Hence the variance at each step is reduced, possibly substantially, e.g., Example 1.3 and Example 1.4, by performing importance sampling instead of uniform sampling.
Revised: Hence the variance at each step is reduced, possibly substantially, e.g., Example 1.3 and Example 1.4, by performing importance sampling instead of uniform sampling.

Original: In fact, it follows from the Cauchy-Schwarz inequality that the above importance sampling probability distribution is the optimal distribution for variance reduction.
Revised: In fact, it follows from the Cauchy-Schwarz inequality that the above importance sampling probability distribution is the optimal distribution for variance reduction.

Original: However, computing the probability distribution for importance sampling requires computing the gradients in each round, which is too expensive in the ﬁrst place.
Revised: However, computing the probability distribution for importance sampling requires computing the gradients in each round, which is too expensive in the ﬁrst place.

Original: Second-Order Methods.
Revised: Second-Order Methods.

Original: Although ﬁrst-order methods such as SGD are widely used, they do sometimes have issues such as sensitivity to the choice of hyperparameters, stagnation at high training errors, and difﬁculty in escaping saddle points.
Revised: Although ﬁrst-order methods such as SGD are widely used, they do sometimes have issues such as sensitivity to the choice of hyperparameters, stagnation at high training errors, and difﬁculty in escaping saddle points.

Original: By considering second-order information such as curvature, second-order optimization methods are known to be robust to several of these issues, such as ill-conditioning.
Revised: By considering second-order information such as curvature, second-order optimization methods are known to be robust to several of these issues, such as ill-conditioning.

Original: For example, Newton’s method can achieve a locally super-linear convergence rate under certain conditions, independent of the problem.
Revised: For example, Newton’s method can achieve a locally super-linear convergence rate under certain conditions, independent of the problem.

Original: Although na ¨ ıve second-order methods are generally too slow compared to common ﬁrst-order methods, stochastic Newton-type methods such as
Revised: Although na ¨ ıve second-order methods are generally too slow compared to common ﬁrst-order methods, stochastic Newton-type methods such as

Original: Gauss-Newton have shown to be scalable in the scientiﬁc computing community (Roosta-Khorasani et al., 2014; Roosta-Khorasani & Mahoney, 2016a;b; Xu et al., 2019; 2020).
Revised: Gauss-Newton have shown to be scalable in the scientiﬁc computing community (Roosta-Khorasani et al., 2014; Roosta-Khorasani & Mahoney, 2016a;b; Xu et al., 2019; 2020).

Original: Our Contributions.
Revised: Our Contributions.

Original: We give a time efﬁcient algorithm that provably approximates the optimal importance sampling using a small space data structure.
Revised: We give a time efﬁcient algorithm that provably approximates the optimal importance sampling using a small space data structure.

Original: Remarkably, our data structure can be implemented in big data models such as the streaming model, which just takes a single pass over the data, and the distributed model, which requires just a single round of communication between parties holding each loss function.
Revised: Remarkably, our data structure can be implemented in big data models such as the streaming model, which just takes a single pass over the data, and the distributed model, which requires just a single round of communication between parties holding each loss function.

Original: For ∇ F = 1 n (cid:80) ∇ f i ( x ) , where each ∇ f i = f ( (cid:104) a i , x (cid:105) ) · a i for some polynomial f and vector a i ∈ R d , let nnz( A ) be the number of nonzero entries of A :
Revised: For ∇ F = 1 n (cid:80) ∇ f i ( x ) , where each ∇ f i = f ( (cid:104) a i , x (cid:105) ) · a i for some polynomial f and vector a i ∈ R d , let nnz( A ) be the number of nonzero entries of A :

Original: = a 1 ◦ . .
Revised: = a 1 ◦ . .

Original: ◦ a n 1 .
Revised: ◦ a n 1 .

Original: Thus for T iterations, where d (cid:28) T (cid:28) n , GD has runtime ˜ O
Revised: Thus for T iterations, where d (cid:28) T (cid:28) n , GD has runtime ˜ O

Original: ( T · nnz( A )) while our algorithm has runtime T · poly( d, log n ) + ˜ O
Revised: ( T · nnz( A )) while our algorithm has runtime T · poly( d, log n ) + ˜ O

Original: (nnz( A )) , where we use ˜ O ( · ) to suppress polylogarithmic terms.
Revised: (nnz( A )) , where we use ˜ O ( · ) to suppress polylogarithmic terms.

Original: Theorem 1.1 Let ∇ F = 1 n (cid:80) ∇ f i ( x ) , where each ∇ f i = f
Revised: Theorem 1.1 Let ∇ F = 1 n (cid:80) ∇ f i ( x ) , where each ∇ f i = f

Original: ( (cid:104) a i , x (cid:105) ) · a i for some polynomial f and vector a i ∈ R d and let nnz( A ) be the number of nonzero entries of A :
Revised: ( (cid:104) a i , x (cid:105) ) · a i for some polynomial f and vector a i ∈ R d and let nnz( A ) be the number of nonzero entries of A :

Original: = a 1 ◦ . .
Revised: = a 1 ◦ . .

Original: ◦ a n .
Revised: ◦ a n .

Original: For d (cid:28) T (cid:28) n , there exists an algorithm that performs T steps of SGD and at each step samples a gradient within a constant factor of the optimal probability distribution.
Revised: For d (cid:28) T (cid:28) n , there exists an algorithm that performs T steps of SGD and at each step samples a gradient within a constant factor of the optimal probability distribution.

Original: The algorithm requires a single pass over A and uses ˜ O
Revised: The algorithm requires a single pass over A and uses ˜ O

Original: (nnz( A )) pre-processing time and ˜ O
Revised: (nnz( A )) pre-processing time and ˜ O

Original: ( Td ) space.
Revised: ( Td ) space.

Original: Theorem 1.1 can be used to immediately obtain improved convergence guarantees for a class of functions whose convergence rate depends on the variance σ 2 t , such as µ -smooth functions or strongly convex functions.
Revised: Theorem 1.1 can be used to immediately obtain improved convergence guarantees for a class of functions whose convergence rate depends on the variance σ 2 t , such as µ -smooth functions or strongly convex functions.

Original: Recall that SGD offers the following convergence guarantees for smooth functions:
Revised: Recall that SGD offers the following convergence guarantees for smooth functions:

Original: Theorem 1.2 (Nemirovski et al., 2009; Meka, 2017) Let F be a µ -smooth convex function and x opt = argmin F ( x ) .
Revised: Theorem 1.2 (Nemirovski et al., 2009; Meka, 2017) Let F be a µ -smooth convex function and x opt = argmin F ( x ) .

Original: Let σ 2 be an upper bound for the variance of the unbiased estimator across all iterations and x k = x 1 + ... + x k k .
Revised: Let σ 2 be an upper bound for the variance of the unbiased estimator across all iterations and x k = x 1 + ... + x k k .

Original: Let each step-size η t be η ≤ 1 µ .
Revised: Let each step-size η t be η ≤ 1 µ .

Original: Then for SGD with initial position x 0 ,
Revised: Then for SGD with initial position x 0 ,

Original: [Equation]
Revised: [Equation]

Original: so that k
Revised: so that k

Original: = O (cid:18) 1 (cid:15) 2 (cid:16) σ 2 + µ (cid:107) x 0 − x opt (cid:107) 22 (cid:17) 2 (cid:19) iterations sufﬁces to obtain an (cid:15) -approximate optimal value by setting η = 1 √ k .
Revised: = O (cid:18) 1 (cid:15) 2 (cid:16) σ 2 + µ (cid:107) x 0 − x opt (cid:107) 22 (cid:17) 2 (cid:19) iterations sufﬁces to obtain an (cid:15) -approximate optimal value by setting η = 1 √ k .

Original: In the convergence guarantees of Theorem 1.2, we obtain a constant factor approximation to the variance σ = σ opt from optimal importance sampling, which can be signiﬁcantly better than the variance σ = σ uniform from uniform sampling in standard SGD.
Revised: In the convergence guarantees of Theorem 1.2, we obtain a constant factor approximation to the variance σ = σ opt from optimal importance sampling, which can be signiﬁcantly better than the variance σ = σ uniform from uniform sampling in standard SGD.

Original: We ﬁrst show straightforward examples where uniform sampling an index performs signiﬁcantly worse than importance sampling.
Revised: We ﬁrst show straightforward examples where uniform sampling an index performs signiﬁcantly worse than importance sampling.

Original: For example, if ∇ f i ( x ) = (cid:104) a i , x (cid:105) · a i , then for A = a 1 ◦ . . .
Revised: For example, if ∇ f i ( x ) = (cid:104) a i , x (cid:105) · a i , then for A = a 1 ◦ . . .

Original: ◦ a n :
Revised: ◦ a n :

Original: Example 1.3 When the nonzero entries of the input A are concentrated in a small number of vectors a i , uniform sampling will frequently sample gradients that are small and make little progress, whereas importance sampling will rarely do so.
Revised: Example 1.3 When the nonzero entries of the input A are concentrated in a small number of vectors a i , uniform sampling will frequently sample gradients that are small and make little progress, whereas importance sampling will rarely do so.

Original: In an extreme case, the A can contain exactly one nonzero vector a i and importance sampling will always output the full gradient whereas uniform sampling will only ﬁnd the nonzero row with probability 1 n .
Revised: In an extreme case, the A can contain exactly one nonzero vector a i and importance sampling will always output the full gradient whereas uniform sampling will only ﬁnd the nonzero row with probability 1 n .

Original: Example 1.4 It may be that all rows of A have large magnitude, but x is nearly orthogonal to most of the rows of A and heavily in the direction of row a r .
Revised: Example 1.4 It may be that all rows of A have large magnitude, but x is nearly orthogonal to most of the rows of A and heavily in the direction of row a r .

Original: Then (cid:104) a i , x (cid:105) · a i is small in magnitude for most i , but (cid:104) a r , x (cid:105) · a r is large so uniform sampling will often output small gradients while importance sampling will output (cid:104) a r , x (cid:105) · a r with high probability.
Revised: Then (cid:104) a i , x (cid:105) · a i is small in magnitude for most i , but (cid:104) a r , x (cid:105) · a r is large so uniform sampling will often output small gradients while importance sampling will output (cid:104) a r , x (cid:105) · a r with high probability.

Original: Thus Example 1.3 shows that na¨ıve SGD with uniform sampling can suffer up to a multiplicative n factor loss in the convergence rate of Theorem 1.2 compared to that of SGD with importance sampling whereas Example 1.4 shows a possible additive n factor loss.
Revised: Thus Example 1.3 shows that na¨ıve SGD with uniform sampling can suffer up to a multiplicative n factor loss in the convergence rate of Theorem 1.2 compared to that of SGD with importance sampling whereas Example 1.4 shows a possible additive n factor loss.

Original: Unlike a number of previous variance reduction methods, we do not require distributional assumptions (Bouchard et al., 2015; Frostig et al., 2015; Gopal, 2016; Jothimurugesan et al., 2018) or ofﬂine access to the data (Roux et al., 2012; Johnson & Zhang, 2013; Defazio et al., 2014; Reddi et al., 2015;
Revised: Unlike a number of previous variance reduction methods, we do not require distributional assumptions (Bouchard et al., 2015; Frostig et al., 2015; Gopal, 2016; Jothimurugesan et al., 2018) or ofﬂine access to the data (Roux et al., 2012; Johnson & Zhang, 2013; Defazio et al., 2014; Reddi et al., 2015;

Original: Zhao & Zhang, 2015; Daneshmand et al., 2016;
Revised: Zhao & Zhang, 2015; Daneshmand et al., 2016;

Original: Needell et al., 2016; Stich et al., 2017; Johnson &
Revised: Needell et al., 2016; Stich et al., 2017; Johnson &

Original: Guestrin, 2018; Katharopoulos & Fleuret, 2018; Salehi et al., 2018; Qian et al., 2019).
Revised: Guestrin, 2018; Katharopoulos & Fleuret, 2018; Salehi et al., 2018; Qian et al., 2019).

Original: On the other hand, for applications such as neural nets in which the parameters in the loss function can change, we can use a second-order approximation for a number of iterations, then reread the data to build a new second-order approximation when necessary.
Revised: On the other hand, for applications such as neural nets in which the parameters in the loss function can change, we can use a second-order approximation for a number of iterations, then reread the data to build a new second-order approximation when necessary.

Original: We complement our main theoretical result with empirical evaluations comparing our algorithm to
Revised: We complement our main theoretical result with empirical evaluations comparing our algorithm to

Original: SGD with uniform sampling for logistic regression on the a9a Adult dataset collected by UCI and retrieved from LibSVM (Chang & Lin, 2011).
Revised: SGD with uniform sampling for logistic regression on the a9a Adult dataset collected by UCI and retrieved from LibSVM (Chang & Lin, 2011).

Original: Our evaluations demonstrate that for various step-sizes, our algorithm has signiﬁcantly better performance than uniform sampling across both the number of
Revised: Our evaluations demonstrate that for various step-sizes, our algorithm has signiﬁcantly better performance than uniform sampling across both the number of

Original: SGD iterations and surprisingly, wall-clock time.
Revised: SGD iterations and surprisingly, wall-clock time.

Original: We then show that our same framework can also be reworked to approximate importance sampling for the Hessian, thereby performing variance reduction for second-order optimization methods.
Revised: We then show that our same framework can also be reworked to approximate importance sampling for the Hessian, thereby performing variance reduction for second-order optimization methods.

Original: (Xu et al., 2016) reduce the bottleneck of many second-order optimization methods to the task of sampling s rows of A = a 1 ◦ . . .
Revised: (Xu et al., 2016) reduce the bottleneck of many second-order optimization methods to the task of sampling s rows of A = a 1 ◦ . . .

Original: ◦ a n so that a row a i is sampled with probability
Revised: ◦ a n so that a row a i is sampled with probability

Original: [Equation]
Revised: [Equation]

Original: for some ﬁxed function f so that the Hessian H has the form H := ∇ 2 F = 1 n (cid:80) ∇ f (
Revised: for some ﬁxed function f so that the Hessian H has the form H := ∇ 2 F = 1 n (cid:80) ∇ f (

Original: [Equation]
Revised: [Equation]

Original: a i , x (cid:105) ) a (cid:62) i a i .
Revised: a i , x (cid:105) ) a (cid:62) i a i .

Original: (Xu et al., 2016) show that this ﬁnite-sum form arises frequently in machine learning problems such as logistic regression with least squares loss.
Revised: (Xu et al., 2016) show that this ﬁnite-sum form arises frequently in machine learning problems such as logistic regression with least squares loss.

Original: Theorem 1.5 Let ∇ 2 F = 1 n (cid:80) ∇ f i ( x ) , where each ∇ f i =
Revised: Theorem 1.5 Let ∇ 2 F = 1 n (cid:80) ∇ f i ( x ) , where each ∇ f i =

Original: f
Revised: f

Original: ( (cid:104) a i , x (cid:105) )
Revised: ( (cid:104) a i , x (cid:105) )

Original: · a (cid:62) i a i for some polynomial f and vector a i ∈ R d and let nnz( A ) be the number of nonzero entries of A :
Revised: · a (cid:62) i a i for some polynomial f and vector a i ∈ R d and let nnz( A ) be the number of nonzero entries of A :

Original: = a 1 ◦ . .
Revised: = a 1 ◦ . .

Original: ◦ a n .
Revised: ◦ a n .

Original: For d (cid:28) T (cid:28) n , there exists an algorithm that subsamples T Hessians within a constant factor of the optimal probability distribution.
Revised: For d (cid:28) T (cid:28) n , there exists an algorithm that subsamples T Hessians within a constant factor of the optimal probability distribution.

Original: The algorithm requires a single pass over A and uses ˜ O
Revised: The algorithm requires a single pass over A and uses ˜ O

Original: (nnz( A )) pre-processing time and ˜ O
Revised: (nnz( A )) pre-processing time and ˜ O

Original: ( Td ) space.
Revised: ( Td ) space.

Original:  SGD A LGORITHM
Revised:  SGD A LGORITHM

Original: We ﬁrst introduce a number of algorithms that will be used in our ﬁnal SGD algorithm, along with their guarantees.
Revised: We ﬁrst introduce a number of algorithms that will be used in our ﬁnal SGD algorithm, along with their guarantees.

Original: We defer all formal proofs to the appendix.
Revised: We defer all formal proofs to the appendix.

Original: L 2 polynomial inner product sketch.
Revised: L 2 polynomial inner product sketch.

Original: For a ﬁxed polynomial f , we ﬁrst require a constant-factor approximation to (cid:107) (cid:80) ni =1 f ( (cid:104)
Revised: For a ﬁxed polynomial f , we ﬁrst require a constant-factor approximation to (cid:107) (cid:80) ni =1 f ( (cid:104)

Original: a i , x (cid:105) )
Revised: a i , x (cid:105) )

Original: · a i (cid:107) 2 for any query x ∈ R d ; we call such an algorithm an Lpolynomial inner product sketch and give such an algorithm with the following guarantee:
Revised: · a i (cid:107) 2 for any query x ∈ R d ; we call such an algorithm an Lpolynomial inner product sketch and give such an algorithm with the following guarantee:

Original: Theorem 2.1 For a ﬁxed (cid:15) > 0 and polynomial f , there exists a data structure E STIMATOR that outputs a (1+ (cid:15) ) -approximation to (cid:80) ni
Revised: Theorem 2.1 For a ﬁxed (cid:15) > 0 and polynomial f , there exists a data structure E STIMATOR that outputs a (1+ (cid:15) ) -approximation to (cid:80) ni

Original: =1 (cid:107) f
Revised: =1 (cid:107) f

Original: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 for any query x ∈ R d .
Revised: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 for any query x ∈ R d .

Original: The data structure requires a single pass over A = a 1 ◦ . . .
Revised: The data structure requires a single pass over A = a 1 ◦ . . .

Original: ◦ a n (possibly through turnstile updates 2 ), can be built in ˜ O (cid:0) nnz
Revised: ◦ a n (possibly through turnstile updates 2 ), can be built in ˜ O (cid:0) nnz

Original: ( A ) + d(cid:15) 2 (cid:1) time and ˜ O
Revised: ( A ) + d(cid:15) 2 (cid:1) time and ˜ O

Original: (cid:0) d(cid:15) 2 (cid:1) space, uses query time poly (cid:0) d, 1 (cid:15) , log n (cid:1) , and succeeds with probability 1 − 1 poly( n ) .
Revised: (cid:0) d(cid:15) 2 (cid:1) space, uses query time poly (cid:0) d, 1 (cid:15) , log n (cid:1) , and succeeds with probability 1 − 1 poly( n ) .

Original: The L 2 polynomial inner product sketch E STIMATOR is a generalization of AMS variants Alon et al.
Revised: The L 2 polynomial inner product sketch E STIMATOR is a generalization of AMS variants Alon et al.

Original: ; Mahabadi et al.
Revised: ; Mahabadi et al.

Original: and is simple to implement.
Revised: and is simple to implement.

Original: For intuition, observe that for d = 1 and the identity function f , the matrix A ∈ R n × d reduces to a vector of length n so that estimating (cid:80) ni
Revised: For intuition, observe that for d = 1 and the identity function f , the matrix A ∈ R n × d reduces to a vector of length n so that estimating (cid:80) ni

Original: =1 (cid:107) f
Revised: =1 (cid:107) f

Original: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 is just estimating the squared norm of a vector in sublinear space.
Revised: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 is just estimating the squared norm of a vector in sublinear space.

Original: For a degree p polynomial f , E STIMATOR generates random sign matrices S 0 , S 1 , . .
Revised: For a degree p polynomial f , E STIMATOR generates random sign matrices S 0 , S 1 , . .

Original: . , S p with ˜ O (cid:0) 1 (cid:15) 2 (cid:1) rows and maintains S 0 A , . .
Revised: . , S p with ˜ O (cid:0) 1 (cid:15) 2 (cid:1) rows and maintains S 0 A , . .

Original: S p A .
Revised: S p A .

Original: To estimate (cid:80) ni =1 (cid:107) α q · ( (cid:104) a i , x (cid:105) )
Revised: To estimate (cid:80) ni =1 (cid:107) α q · ( (cid:104) a i , x (cid:105) )

Original: q · a q (cid:107) 22 for an integer q ∈ [0 , p ] and scalar α q on a given query x , E STIMATOR creates the q -fold tensor Y =
Revised: q · a q (cid:107) 22 for an integer q ∈ [0 , p ] and scalar α q on a given query x , E STIMATOR creates the q -fold tensor Y =

Original: y ⊗ q for each row y of S q A and the ( q − 1)
Revised: y ⊗ q for each row y of S q A and the ( q − 1)

Original: -fold tensor X
Revised: -fold tensor X

Original: = x ⊗ ( q − 1) .
Revised: = x ⊗ ( q − 1) .

Original: Note that X and Y can be refolded into dimensions R d q − 1 and R d × d q − 1 so that YX ∈ R d and (cid:107) α q · YX (cid:107) 22 is an unbiased estimator of (cid:80) ni =1 (cid:107) α q · ( (cid:104) a i , x (cid:105) )
Revised: Note that X and Y can be refolded into dimensions R d q − 1 and R d × d q − 1 so that YX ∈ R d and (cid:107) α q · YX (cid:107) 22 is an unbiased estimator of (cid:80) ni =1 (cid:107) α q · ( (cid:104) a i , x (cid:105) )

Original: q · a q (cid:107) 22 .
Revised: q · a q (cid:107) 22 .

Original: We give this algorithm in full in Algorithm 1.
Revised: We give this algorithm in full in Algorithm 1.

Original: Thus, taking the average over O (cid:0) 1 (cid:15) 2 (cid:1) instances of the sums of the tensor products for rows y across the sketches
Revised: Thus, taking the average over O (cid:0) 1 (cid:15) 2 (cid:1) instances of the sums of the tensor products for rows y across the sketches

Original: S 0 A , . . .
Revised: S 0 A , . . .

Original: , S p A gives a (1 + (cid:15) ) -approximation to (cid:80) n i
Revised: , S p A gives a (1 + (cid:15) ) -approximation to (cid:80) n i

Original: =1 (cid:107) f
Revised: =1 (cid:107) f

Original: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 2 2 with constant probability.
Revised: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 2 2 with constant probability.

Original: The success probability of success can then be boosted to 1 − 1 poly( n ) by taking the median of
Revised: The success probability of success can then be boosted to 1 − 1 poly( n ) by taking the median of

Original: O (log n ) such outputs.
Revised: O (log n ) such outputs.

Original: (cid:80) ni =1 (cid:107) ( (cid:104) a
Revised: (cid:80) ni =1 (cid:107) ( (cid:104) a

Original: i , x (cid:105) ) p · a i (cid:107) 22 , where x is a post-processing vector
Revised: i , x (cid:105) ) p · a i (cid:107) 22 , where x is a post-processing vector

Original: Input: Matrix A = a 1 ◦ . . .
Revised: Input: Matrix A = a 1 ◦ . . .

Original: ◦ ∈ R n × d , post-processing vector x ∈ R d , integer p ≥ 0 , constant parameter (cid:15) > 0 .
Revised: ◦ ∈ R n × d , post-processing vector x ∈ R d , integer p ≥ 0 , constant parameter (cid:15) > 0 .

Original: Output: (1 + (cid:15) ) -approximation to (cid:80) ni =1 (cid:107) ( (cid:104) a i , x (cid:105) )
Revised: Output: (1 + (cid:15) ) -approximation to (cid:80) ni =1 (cid:107) ( (cid:104) a i , x (cid:105) )

Original: p · a i (cid:107) 22 .
Revised: p · a i (cid:107) 22 .

Original: 1: r ← Θ(
Revised: 1: r ← Θ(

Original: log
Revised: log

Original: n
Revised: n

Original: ) with a sufﬁciently large constant
Revised: ) with a sufﬁciently large constant

Original: 2: b ← Ω (cid:0)
Revised: 2: b ← Ω (cid:0)

Original: 1 (cid:15)
Revised: 1 (cid:15)

Original: (cid:1)
Revised: (cid:1)

Original: with a sufﬁciently large
Revised: with a sufﬁciently large

Original: constant.
Revised: constant.

Original: 3: Let T be an r × b table of buckets, where each bucket stores an R d vector, initialized to the zeros vector.
Revised: 3: Let T be an r × b table of buckets, where each bucket stores an R d vector, initialized to the zeros vector.

Original: 4: Let s
Revised: 4: Let s

Original: i ∈ {− 1 , +1 } be 4 -wise independent for i ∈ [
Revised: i ∈ {− 1 , +1 } be 4 -wise independent for i ∈ [

Original: n ] .
Revised: n ] .

Original: Let h i : [ n ] → [ b ] be 4 -wise independent for i ∈
Revised: Let h i : [ n ] → [ b ] be 4 -wise independent for i ∈

Original: [ r ] .
Revised: [ r ] .

Original: 6: Let u i,j be the all zeros vector
Revised: 6: Let u i,j be the all zeros vector

Original: 7: for each to do 8: for do 9:
Revised: 7: for each to do 8: for do 9:

Original: Add
Revised: Add

Original: 10: Let v j be the vector
Revised: 10: Let v j be the vector

Original: i
Revised: i

Original: 11: Process x : 12: for i ∈ [ r ] , j ∈ [ b do 13: u i,j ← v ⊗ p
Revised: 11: Process x : 12: for i ∈ [ r ] , j ∈ [ b do 13: u i,j ← v ⊗ p

Original: i,j x ⊗ ( p − 1) 14: return median i ∈ [ r ] 1 b (cid:80) j ∈ [ b ] (cid:107) u i
Revised: i,j x ⊗ ( p − 1) 14: return median i ∈ [ r ] 1 b (cid:80) j ∈ [ b ] (cid:107) u i

Original: [Equation]
Revised: [Equation]

Original: j (cid:107) 22 .
Revised: j (cid:107) 22 .

Original: L 2 polynomial inner product sampler.
Revised: L 2 polynomial inner product sampler.

Original: Given a matrix A = a 1 ◦ . . . ◦ a n ∈ R n × d and a ﬁxed function f , a data structure that takes query x ∈ R d and outputs an index i ∈ [ n ] with probability roughly
Revised: Given a matrix A = a 1 ◦ . . . ◦ a n ∈ R n × d and a ﬁxed function f , a data structure that takes query x ∈ R d and outputs an index i ∈ [ n ] with probability roughly

Original: [Equation]
Revised: [Equation]

Original: is called an L 2 polynomial inner product sampler .
Revised: is called an L 2 polynomial inner product sampler .

Original: We give such a data structure in Section A.1:
Revised: We give such a data structure in Section A.1:

Original: Theorem 2.2 For a ﬁxed (cid:15) > 0 and polynomial f , there exists a data structure S AMPLER that takes any query x ∈ R d and outputs an index i ∈ [ n ] with probability (1 ± (cid:15) ) ·(cid:107) f
Revised: Theorem 2.2 For a ﬁxed (cid:15) > 0 and polynomial f , there exists a data structure S AMPLER that takes any query x ∈ R d and outputs an index i ∈ [ n ] with probability (1 ± (cid:15) ) ·(cid:107) f

Original: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 (cid:80) ni =1 (cid:107) f
Revised: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 (cid:80) ni =1 (cid:107) f

Original: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 + 1 poly( n ) , along with a vector u := f
Revised: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 + 1 poly( n ) , along with a vector u := f

Original: ( (cid:104) a i , x (cid:105) )
Revised: ( (cid:104) a i , x (cid:105) )

Original: · a i + v , where E [ v ] = 0 and (cid:107) v
Revised: · a i + v , where E [ v ] = 0 and (cid:107) v

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: The data structure requires a single pass over A = a 1 ◦ . . .
Revised: The data structure requires a single pass over A = a 1 ◦ . . .

Original: ◦ a n (possibly through turnstile updates), can be built in ˜ O (cid:0) nnz
Revised: ◦ a n (possibly through turnstile updates), can be built in ˜ O (cid:0) nnz

Original: ( A ) + d(cid:15) 2 (cid:1) time and ˜ O
Revised: ( A ) + d(cid:15) 2 (cid:1) time and ˜ O

Original: (cid:0) d(cid:15) 2 (cid:1) space, uses query time poly (cid:0) d, 1 (cid:15) , log n (cid:1) , and succeeds with probability 1 − 1 poly( n ) .
Revised: (cid:0) d(cid:15) 2 (cid:1) space, uses query time poly (cid:0) d, 1 (cid:15) , log n (cid:1) , and succeeds with probability 1 − 1 poly( n ) .

Original: We remark that T independent instances of S AMPLER provide an oracle for T steps of SGD with importance sampling, but the overall runtime would be T · nnz( A ) so it would be just as efﬁcient to run T iterations of GD.
Revised: We remark that T independent instances of S AMPLER provide an oracle for T steps of SGD with importance sampling, but the overall runtime would be T · nnz( A ) so it would be just as efﬁcient to run T iterations of GD.

Original: The subroutine S AMPLER is signiﬁcantly more challenging to describe and analyze, so we defer its discussion to Section A.1, though it can be seen as a combination of E STIMATOR and a generalized CountSketch Charikar et al.
Revised: The subroutine S AMPLER is signiﬁcantly more challenging to describe and analyze, so we defer its discussion to Section A.1, though it can be seen as a combination of E STIMATOR and a generalized CountSketch Charikar et al.

Original: ; Nelson & Nguyen (2013);
Revised: ; Nelson & Nguyen (2013);

Original: Mahabadi et al.
Revised: Mahabadi et al.

Original: (2020) variant and is nevertheless relatively straightforward to implement.
Revised: (2020) variant and is nevertheless relatively straightforward to implement.

Original: Leverage score sampler.
Revised: Leverage score sampler.

Original: Although S AMPLER outputs a (noisy) vector according to the desired probability distribution, we also require an algorithm that automatically does this for indices i ∈ [ n ] that are likely to be sampled multiple times across the T iterations.
Revised: Although S AMPLER outputs a (noisy) vector according to the desired probability distribution, we also require an algorithm that automatically does this for indices i ∈ [ n ] that are likely to be sampled multiple times across the T iterations.

Original: Equivalently, we require explicitly storing the rows with high leverage scores, but we defer the formal discussion and algorithmic presentation to Section A.2.
Revised: Equivalently, we require explicitly storing the rows with high leverage scores, but we defer the formal discussion and algorithmic presentation to Section A.2.

Original: For our purposes, the following sufﬁces:
Revised: For our purposes, the following sufﬁces:

Original: Theorem 2.3 There exists an algorithm L EVERAGE that returns all indices i ∈ [ n ] such that
Revised: Theorem 2.3 There exists an algorithm L EVERAGE that returns all indices i ∈ [ n ] such that

Original: [Equation]
Revised: [Equation]

Original: Td for some x ∈ R n , along with a vector u i := f
Revised: Td for some x ∈ R n , along with a vector u i := f

Original: ( (cid:104) a i , x (cid:105) )
Revised: ( (cid:104) a i , x (cid:105) )

Original: · a i + v i , where (cid:107) v i (cid:107) 2 ≤ (cid:15) · (cid:107)
Revised: · a i + v i , where (cid:107) v i (cid:107) 2 ≤ (cid:15) · (cid:107)

Original: f
Revised: f

Original: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 2 .
Revised: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 2 .

Original: The algorithm requires a single pass over A = a 1 ◦ . . .
Revised: The algorithm requires a single pass over A = a 1 ◦ . . .

Original: ◦ a n (possibly through turnstile updates), uses ˜ O (cid:0) nnz( A ) + d ω (cid:15) 2 (cid:1) runtime (where ω denotes the exponent of square matrix multiplication) and ˜ O (cid:0) d(cid:15) 2 (cid:1) space, and succeeds with probability 1 − 1 poly( n ) .
Revised: ◦ a n (possibly through turnstile updates), uses ˜ O (cid:0) nnz( A ) + d ω (cid:15) 2 (cid:1) runtime (where ω denotes the exponent of square matrix multiplication) and ˜ O (cid:0) d(cid:15) 2 (cid:1) space, and succeeds with probability 1 − 1 poly( n ) .

Original: 2. SGD A LGORITHM AND A NALYSIS
Revised: 2. SGD A LGORITHM AND A NALYSIS

Original: For the ﬁnite-sum optimization problem min x ∈ R d F ( x ) := 1 n (cid:80) ni
Revised: For the ﬁnite-sum optimization problem min x ∈ R d F ( x ) := 1 n (cid:80) ni

Original: =

Revised: =

Original: f i ( x ) , where each ∇ f i = f
Revised: f i ( x ) , where each ∇ f i = f

Original: ( (cid:104) a i , x (cid:105) ) · a i , recall that we could simply an instance of S AMPLER as an oracle for SGD with importance sampling.
Revised: ( (cid:104) a i , x (cid:105) ) · a i , recall that we could simply an instance of S AMPLER as an oracle for SGD with importance sampling.

Original: However, na ¨ ıvely running T SGD steps requires T independent instances, which uses
Revised: However, na ¨ ıvely running T SGD steps requires T independent instances, which uses

Original: T · nnz( A ) runtime by Theorem 2.2.
Revised: T · nnz( A ) runtime by Theorem 2.2.

Original: Thus we use a two level data structure by ﬁrst implicitly partition the rows of matrix A = a 1 ◦ . . .
Revised: Thus we use a two level data structure by ﬁrst implicitly partition the rows of matrix A = a 1 ◦ . . .

Original: ◦ a n into β :
Revised: ◦ a n into β :

Original: = Θ( Td ) buckets B 1 , . . .
Revised: = Θ( Td ) buckets B 1 , . . .

Original: , B β and creating an instance of E STIMATOR and S AMPLER for each bucket.
Revised: , B β and creating an instance of E STIMATOR and S AMPLER for each bucket.

Original: The idea is that for a given query x t in SGD iteration t ∈
Revised: The idea is that for a given query x t in SGD iteration t ∈

Original: [ T ] , we ﬁrst query x t to each of the E STIMATOR data structures to estimate (cid:80) i ∈
Revised: [ T ] , we ﬁrst query x t to each of the E STIMATOR data structures to estimate (cid:80) i ∈

Original: B j (cid:107) f
Revised: B j (cid:107) f

Original: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 for each j ∈ [ β ] .
Revised: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 for each j ∈ [ β ] .

Original: We then sample index j ∈ [ β ] among the buckets
Revised: We then sample index j ∈ [ β ] among the buckets

Original: B 1 , . . .
Revised: B 1 , . . .

Original: , B β with probability roughly (cid:80) i ∈
Revised: , B β with probability roughly (cid:80) i ∈

Original: Bj (cid:107) f
Revised: Bj (cid:107) f

Original: ( (cid:104)
Revised: ( (cid:104)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:105) )
Revised: (cid:105) )

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 22 .
Revised: (cid:107) 22 .

Original: Once we have sampled index j , it would seem that querying the instance S AMPLER corresponding to B j simulates SGD, since S AMPLER now performs importance sampling on the rows in B j , which gives the correct overall probability distribution for each row i ∈ [ n ] .
Revised: Once we have sampled index j , it would seem that querying the instance S AMPLER corresponding to B j simulates SGD, since S AMPLER now performs importance sampling on the rows in B j , which gives the correct overall probability distribution for each row i ∈ [ n ] .

Original: Moreover, S AMPLER has runtime proportional to the sparsity of
Revised: Moreover, S AMPLER has runtime proportional to the sparsity of

Original: B j , so the total runtime across the β instances of S AMPLER is ˜ O
Revised: B j , so the total runtime across the β instances of S AMPLER is ˜ O

Original: (nnz( A )) .
Revised: (nnz( A )) .

Original: However, an issue arises when the same bucket B j is sampled multiple times, as we only create a single instance of S AMPLER for each bucket.
Revised: However, an issue arises when the same bucket B j is sampled multiple times, as we only create a single instance of S AMPLER for each bucket.

Original: We avoid this issue by explicitly accounting for the buckets that are likely to be sampled multiple times.
Revised: We avoid this issue by explicitly accounting for the buckets that are likely to be sampled multiple times.

Original: Namely, we show that if
Revised: Namely, we show that if

Original: [Equation]
Revised: [Equation]

Original: 200 Td for all t ∈
Revised: 200 Td for all t ∈

Original: [ T ] and i ∈ [ n ] , then by Bernstein’s inequality, the probability that no bucket B j is sampled multiple times is at least 99100 .
Revised: [ T ] and i ∈ [ n ] , then by Bernstein’s inequality, the probability that no bucket B j is sampled multiple times is at least 99100 .

Original: Thus we use L EVERAGE to separate all such rows a i that violate this property from their respective buckets and explicitly track the SGD steps in which these rows are sampled.
Revised: Thus we use L EVERAGE to separate all such rows a i that violate this property from their respective buckets and explicitly track the SGD steps in which these rows are sampled.

Original: We give the algorithm in full in Algorithm 2.
Revised: We give the algorithm in full in Algorithm 2.

Original: The key property achieved by Algorithm 2 in partitioning the rows and removing the rows that are likely to be sampled multiple times is that each of the S AMPLER instances are queried at most once.
Revised: The key property achieved by Algorithm 2 in partitioning the rows and removing the rows that are likely to be sampled multiple times is that each of the S AMPLER instances are queried at most once.

Original: Lemma 2.

Revised: Lemma 2.

Original: With probability at least 98100 , each t ∈ [ T ] uses a different instance of S AMPLER j .
Revised: With probability at least 98100 , each t ∈ [ T ] uses a different instance of S AMPLER j .

Original: Proof of Theorem 1.1:
Revised: Proof of Theorem 1.1:

Original: Consider Algorithm 2.
Revised: Consider Algorithm 2.

Original: By Lemma 2.4, each time t ∈ [ T ] uses a fresh instance of S AMPLER j , so that independent randomness is used.
Revised: By Lemma 2.4, each time t ∈ [ T ] uses a fresh instance of S AMPLER j , so that independent randomness is used.

Original: A possible concern is that each instance E STIMATOR j is not using fresh randomness, but we observe that E STIMATOR procedures
Revised: A possible concern is that each instance E STIMATOR j is not using fresh randomness, but we observe that E STIMATOR procedures

Original: Let q i be the output of E STIMATOR j on query x
Revised: Let q i be the output of E STIMATOR j on query x

Original: t − 1 for each i ∈ [ β ] .
Revised: t − 1 for each i ∈ [ β ] .

Original: [Equation]
Revised: [Equation]

Original: 12: if there exists i ∈ L 0 with h ( i ) =
Revised: 12: if there exists i ∈ L 0 with h ( i ) =

Original: j then 13:
Revised: j then 13:

Original: Use E STIMATOR j , L EVERAGE , and S AMPLER j to sample gradient w t = (cid:92) ∇ f i t ( x t ) 14: else 15:
Revised: Use E STIMATOR j , L EVERAGE , and S AMPLER j to sample gradient w t = (cid:92) ∇ f i t ( x t ) 14: else 15:

Original: Use S
Revised: Use S

Original: ) 16: (cid:99)
Revised: ) 16: (cid:99)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (cid:107)
Revised: (cid:107)

Original: [Table]
Revised: [Table]

Original: 17: x
Revised: 17: x

Original: [Table] [Table]
Revised: [Table] [Table]

Original: w
Revised: w

Original: [Table]
Revised: [Table]

Original: is only used in sampling a bucket j ∈ [ β ] as an L 2 polynomial inner product sketch; otherwise the sampling uses fresh randomness whereas the sampling is built into each instance of S AMPLER j .
Revised: is only used in sampling a bucket j ∈ [ β ] as an L 2 polynomial inner product sketch; otherwise the sampling uses fresh randomness whereas the sampling is built into each instance of S AMPLER j .

Original: By Theorem 2.2, each index i is sampled with probability within a factor 2 of the importance sampling probability distribution.
Revised: By Theorem 2.2, each index i is sampled with probability within a factor 2 of the importance sampling probability distribution.

Original: By Theorem 2.1, we have that (cid:99) p i,t is within a factor 4 of the probability p
Revised: By Theorem 2.1, we have that (cid:99) p i,t is within a factor 4 of the probability p

Original: i,t induced by optimal importance sampling SGD.
Revised: i,t induced by optimal importance sampling SGD.

Original: Note that w t = (cid:92) ∇ f i ( x t ) is an unbiased estimator of ∇ f i ( x t ) and (cid:107) w t (cid:107) is a 2 -approximation to (cid:107)∇ f i ( x t ) (cid:107) by Theorem 2.2.
Revised: Note that w t = (cid:92) ∇ f i ( x t ) is an unbiased estimator of ∇ f i ( x t ) and (cid:107) w t (cid:107) is a 2 -approximation to (cid:107)∇ f i ( x t ) (cid:107) by Theorem 2.2.

Original: Hence, the variance at each time t ∈
Revised: Hence, the variance at each time t ∈

Original: [ T ] of Algorithm 2 is within a constant factor of the variance σ 2 =
Revised: [ T ] of Algorithm 2 is within a constant factor of the variance σ 2 =

Original: ( (cid:80) (cid:107)∇ f i ( x t ) (cid:107) ) 2 − (cid:107)∇ F ( x t ) (cid:107) 2 of optimal importance sampling SGD.
Revised: ( (cid:80) (cid:107)∇ f i ( x t ) (cid:107) ) 2 − (cid:107)∇ F ( x t ) (cid:107) 2 of optimal importance sampling SGD.

Original: By Theorem 2.1, Theorem 2.2, and Theorem 2.3, the preprocessing time is ˜ O
Revised: By Theorem 2.1, Theorem 2.2, and Theorem 2.3, the preprocessing time is ˜ O

Original: (nnz( A ))
Revised: (nnz( A ))

Original: + T · poly( d, log n ) due to the choices of (cid:15) =
Revised: + T · poly( d, log n ) due to the choices of (cid:15) =

Original: O (1) and β =
Revised: O (1) and β =

Original: Θ( Td ) , but partitioning the nonzero entries of A across the β buckets.
Revised: Θ( Td ) , but partitioning the nonzero entries of A across the β buckets.

Original: Similarly, the space used by the algorithm is ˜ O
Revised: Similarly, the space used by the algorithm is ˜ O

Original: ( Td ) .
Revised: ( Td ) .

Original: Once the gradient descent stage of Algorithm 2 begins, it takes poly( d ) time in each step t ∈
Revised: Once the gradient descent stage of Algorithm 2 begins, it takes poly( d ) time in each step t ∈

Original: [ T ] to query the β = Θ( Td ) instances of S AMPLER and E STIMATOR , for total time T · poly( d, log n ) .
Revised: [ T ] to query the β = Θ( Td ) instances of S AMPLER and E STIMATOR , for total time T · poly( d, log n ) .

Original: (cid:50)
Revised: (cid:50)

Original:  S ECOND -O RDER O PTIMIZATION
Revised:  S ECOND -O RDER O PTIMIZATION

Original: In this section, we repurpose our data stucture that performs importance sampling for SGD to instead perform importance sampling for second-order optimization.
Revised: In this section, we repurpose our data stucture that performs importance sampling for SGD to instead perform importance sampling for second-order optimization.

Original: Given a second-order optimization algorithm that requires a sampled Hessian H t , possibly along with additional inputs such as the current iterate x t and the gradient g t of F , we model the update rule by an oracle O ( H t ) , suppressing other inputs to the oracle in the notation.
Revised: Given a second-order optimization algorithm that requires a sampled Hessian H t , possibly along with additional inputs such as the current iterate x t and the gradient g t of F , we model the update rule by an oracle O ( H t ) , suppressing other inputs to the oracle in the notation.

Original: For example, the oracle O corresponding to the canonical second-order algorithm Newton’s method can be formulated as
Revised: For example, the oracle O corresponding to the canonical second-order algorithm Newton’s method can be formulated as

Original: [Equation]
Revised: [Equation]

Original: [ H t ] − 1 g t .
Revised: [ H t ] − 1 g t .

Original: By black-boxing the update rule of any second-order optimization algorithm into the oracle, we can focus our attention to the running time of sampling a Hessian with nearly the optimal probability distribution.
Revised: By black-boxing the update rule of any second-order optimization algorithm into the oracle, we can focus our attention to the running time of sampling a Hessian with nearly the optimal probability distribution.

Original: Thus we prove generalizations of the L 2 polynomial inner product sketch, the Lpolynomial inner product sampler, and the leverage score sampler for Hessians.
Revised: Thus we prove generalizations of the L 2 polynomial inner product sketch, the Lpolynomial inner product sampler, and the leverage score sampler for Hessians.

Original: Theorem 3.1 For a ﬁxed (cid:15) > 0 and polynomial f , there exists a data structure HE STIMATOR that outputs a (1 + (cid:15) ) -approximation to (cid:80) ni
Revised: Theorem 3.1 For a ﬁxed (cid:15) > 0 and polynomial f , there exists a data structure HE STIMATOR that outputs a (1 + (cid:15) ) -approximation to (cid:80) ni

Original: =1 (cid:13)(cid:13)
Revised: =1 (cid:13)(cid:13)

Original: f
Revised: f

Original: ( (cid:104) a i , x (cid:105) )
Revised: ( (cid:104) a i , x (cid:105) )

Original: · a (cid:62) i a i (cid:13)(cid:13) 2 F for any query x ∈ R d .
Revised: · a (cid:62) i a i (cid:13)(cid:13) 2 F for any query x ∈ R d .

Original: The data structure requires a single pass over A = a 1 ◦ . . .
Revised: The data structure requires a single pass over A = a 1 ◦ . . .

Original: ◦ a n (possibly through turnstile updates), can be built in ˜ O (cid:0) nnz
Revised: ◦ a n (possibly through turnstile updates), can be built in ˜ O (cid:0) nnz

Original: ( A ) + d(cid:15) 2 (cid:1) time and ˜ O
Revised: ( A ) + d(cid:15) 2 (cid:1) time and ˜ O

Original: (cid:0) d(cid:15) 2 (cid:1) space, uses query time poly (cid:0) d, 1 (cid:15) , log n (cid:1) , and succeeds with probability 1 − 1 poly( n ) .
Revised: (cid:0) d(cid:15) 2 (cid:1) space, uses query time poly (cid:0) d, 1 (cid:15) , log n (cid:1) , and succeeds with probability 1 − 1 poly( n ) .

Original: Theorem 3.2 For a ﬁxed (cid:15) > 0 and polynomial f , there exists a data structure HS AMPLER that takes any query x ∈ R d and outputs an index i ∈ [ n ] with probability (1 ± (cid:15) ) · (cid:107) f
Revised: Theorem 3.2 For a ﬁxed (cid:15) > 0 and polynomial f , there exists a data structure HS AMPLER that takes any query x ∈ R d and outputs an index i ∈ [ n ] with probability (1 ± (cid:15) ) · (cid:107) f

Original: [Table]
Revised: [Table]

Original: poly( n ) , along with a matrix U :
Revised: poly( n ) , along with a matrix U :

Original: = f
Revised: = f

Original: ( (cid:104) a i , x (cid:105) )
Revised: ( (cid:104) a i , x (cid:105) )

Original: · a (cid:62) i a i + V , where E [ V ] = 

Revised: · a (cid:62) i a i + V , where E [ V ] =

Original: and (cid:107) V (cid:107) F ≤ (cid:15) · (cid:13)(cid:13) f
Revised: and (cid:107) V (cid:107) F ≤ (cid:15) · (cid:13)(cid:13) f

Original: ( (cid:104) a i , x (cid:105) )
Revised: ( (cid:104) a i , x (cid:105) )

Original: · a (cid:62) i a i (cid:13)(cid:13) F .
Revised: · a (cid:62) i a i (cid:13)(cid:13) F .

Original: The data structure requires a single pass over A = a 1 ◦ . . .
Revised: The data structure requires a single pass over A = a 1 ◦ . . .

Original: ◦ a n (possibly through turnstile updates), can be built in ˜ O (cid:0) nnz
Revised: ◦ a n (possibly through turnstile updates), can be built in ˜ O (cid:0) nnz

Original: ( A ) + d(cid:15) 2 (cid:1) time and ˜ O
Revised: ( A ) + d(cid:15) 2 (cid:1) time and ˜ O

Original: (cid:0) d(cid:15) 2 (cid:1) space, uses query time poly (cid:0) d, 1 (cid:15) , log n (cid:1) , and succeeds with probability 1 −
Revised: (cid:0) d(cid:15) 2 (cid:1) space, uses query time poly (cid:0) d, 1 (cid:15) , log n (cid:1) , and succeeds with probability 1 −

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.3 There exists an algorithm HL EVERAGE that returns all indices
Revised: Theorem 3.3 There exists an algorithm HL EVERAGE that returns all indices

Original: i ∈ [ n ] such that
Revised: i ∈ [ n ] such that

Original: [Equation]
Revised: [Equation]

Original: ≥ 1 200 Td for some x ∈ R n , along with a matrix U i :
Revised: ≥ 1 200 Td for some x ∈ R n , along with a matrix U i :

Original: = f
Revised: = f

Original: ( (cid:104) a i , x (cid:105) )
Revised: ( (cid:104) a i , x (cid:105) )

Original: · a (cid:62) i a i
Revised: · a (cid:62) i a i

Original: + V i , where (cid:107) V i (cid:107) F ≤ (cid:15) · (cid:13) (cid:13)
Revised: + V i , where (cid:107) V i (cid:107) F ≤ (cid:15) · (cid:13) (cid:13)

Original: f
Revised: f

Original: ( (cid:104) a i , x (cid:105) )
Revised: ( (cid:104) a i , x (cid:105) )

Original: · a (cid:62) i a i (cid:13) (cid:13) F .
Revised: · a (cid:62) i a i (cid:13) (cid:13) F .

Original: The algorithm uses requires a single pass over A = a 1 ◦ . .
Revised: The algorithm uses requires a single pass over A = a 1 ◦ . .

Original: ◦ a n (possibly through turnstile updates), uses ˜ O (cid:0) nnz( A ) + d ω (cid:15) 2 (cid:1) runtime (where ω denotes the exponent of square matrix multiplication) and ˜ O (cid:0) d(cid:15) 2 (cid:1) space, and succeeds with probability 1 − 1 poly( n
Revised: ◦ a n (possibly through turnstile updates), uses ˜ O (cid:0) nnz( A ) + d ω (cid:15) 2 (cid:1) runtime (where ω denotes the exponent of square matrix multiplication) and ˜ O (cid:0) d(cid:15) 2 (cid:1) space, and succeeds with probability 1 − 1 poly( n

Original: [Equation]
Revised: [Equation]

Original: We remark that HS AMPLER and L EVERAGE are generalizations of E STIMATOR and S AMPLER that simply return an outer product of a noisy vector rather than the noisy vector itself.
Revised: We remark that HS AMPLER and L EVERAGE are generalizations of E STIMATOR and S AMPLER that simply return an outer product of a noisy vector rather than the noisy vector itself.

Original: As before, observe that we could simply run an instance of HS AMPLER to sample a Hessian through importance sampling, but sampling T Hessians requires T independent instances, signiﬁcantly increasing the total runtime.
Revised: As before, observe that we could simply run an instance of HS AMPLER to sample a Hessian through importance sampling, but sampling T Hessians requires T independent instances, signiﬁcantly increasing the total runtime.

Original: We thus use the same two level data structure that partitions the rows of matrix A = a 1 ◦ . . .
Revised: We thus use the same two level data structure that partitions the rows of matrix A = a 1 ◦ . . .

Original: ◦ a n into β :
Revised: ◦ a n into β :

Original: = Θ( Td ) buckets B 1 , . . .
Revised: = Θ( Td ) buckets B 1 , . . .

Original: , B β .
Revised: , B β .

Original: We then create an instance of
Revised: We then create an instance of

Original: HE STIMATOR and HS AMPLER for each bucket.
Revised: HE STIMATOR and HS AMPLER for each bucket.

Original: For an iterate x t , we sample j ∈ [ β ] among the buckets B 1 , . . .
Revised: For an iterate x t , we sample j ∈ [ β ] among the buckets B 1 , . . .

Original: , B β with probability roughly (cid:80) i ∈
Revised: , B β with probability roughly (cid:80) i ∈

Original: Bj (cid:107) f
Revised: Bj (cid:107) f

Original: ( (cid:104) a i , x t (cid:105) )
Revised: ( (cid:104) a i , x t (cid:105) )

Original: · a (cid:62) i a i (cid:107) 2 F (cid:80) ni
Revised: · a (cid:62) i a i (cid:107) 2 F (cid:80) ni

Original: =1 (cid:107) f
Revised: =1 (cid:107) f

Original: ( (cid:104) a i , x t (cid:105) )
Revised: ( (cid:104) a i , x t (cid:105) )

Original: · a (cid:62) i a i (cid:107) 2 F using HE STIMATOR and then querying HS AMPLER j at x t to sample a Hessian among the indices partitioned into bucket B j .
Revised: · a (cid:62) i a i (cid:107) 2 F using HE STIMATOR and then querying HS AMPLER j at x t to sample a Hessian among the indices partitioned into bucket B j .

Original: As before, this argument fails when the same bucket B j is sampled multiple times, due to dependencies in randomness, but this issue can be avoided by using HL EVERAGE to decrease the probability that each bucket is sampled.
Revised: As before, this argument fails when the same bucket B j is sampled multiple times, due to dependencies in randomness, but this issue can be avoided by using HL EVERAGE to decrease the probability that each bucket is sampled.

Original: We give the algorithm in full in Algorithm 3.
Revised: We give the algorithm in full in Algorithm 3.

Original: We remark that Algorithm 3 can be generalized to handle oracles
Revised: We remark that Algorithm 3 can be generalized to handle oracles

Original: O corresponding to second-order methods that require batches of subsampled Hessians in each iteration.
Revised: O corresponding to second-order methods that require batches of subsampled Hessians in each iteration.

Original: For example, if we want to run T iterations of a second-order method that requires s subsampled Hessians in each batch, we can simply modify Algorithm 3 to sample s Hessians in each iteration as input to O and thus Ts Hessians in total.
Revised: For example, if we want to run T iterations of a second-order method that requires s subsampled Hessians in each batch, we can simply modify Algorithm 3 to sample s Hessians in each iteration as input to O and thus Ts Hessians in total.

Original:  E MPIRICAL E VALUATIONS
Revised:  E MPIRICAL E VALUATIONS

Original: Our primary contribution is the theoretical design of a nearly input sparsity time algorithm that approximates optimal importance sampling SGD.
Revised: Our primary contribution is the theoretical design of a nearly input sparsity time algorithm that approximates optimal importance sampling SGD.

Original: In this section we implement a scaled-down version of our algorithm and compare its performance on large-scale real world datasets to SGD with uniform sampling on logistic regression.
Revised: In this section we implement a scaled-down version of our algorithm and compare its performance on large-scale real world datasets to SGD with uniform sampling on logistic regression.

Original: We also consider both linear regression and support-vector machines (SVMs) in the supplementary material.
Revised: We also consider both linear regression and support-vector machines (SVMs) in the supplementary material.

Original: Because most rows have roughly uniformly small leverage scores in real-world data, we assume that no bucket contains a row with a signiﬁcantly large leverage score and thus the implementation of our importance sampling algorithm does not create multiple samplers for any buckets.
Revised: Because most rows have roughly uniformly small leverage scores in real-world data, we assume that no bucket contains a row with a signiﬁcantly large leverage score and thus the implementation of our importance sampling algorithm does not create multiple samplers for any buckets.

Original: By similar reasoning, our implementation uniformly samples a number of indices i and estimates (cid:80) ni =1 (cid:107) f
Revised: By similar reasoning, our implementation uniformly samples a number of indices i and estimates (cid:80) ni =1 (cid:107) f

Original: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 by rescaling.
Revised: ( (cid:104) a i , x (cid:105) ) · a i (cid:107) 22 by rescaling.

Original: Observe that although these simpliﬁcations to our algorithm decreases the wall-clock running time and the total space used by our algorithm, they only decrease the quality of our solution for each SGD iteration.
Revised: Observe that although these simpliﬁcations to our algorithm decreases the wall-clock running time and the total space used by our algorithm, they only decrease the quality of our solution for each SGD iteration.

Original: We also consider two hybrid SGD sampling algorithms; the ﬁrst takes the better gradient obtained at each iteration from both uniform sampling and importance sampling while the second performs 25 iterations of
Revised: We also consider two hybrid SGD sampling algorithms; the ﬁrst takes the better gradient obtained at each iteration from both uniform sampling and importance sampling while the second performs 25 iterations of

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e
Revised: A v e r a g e

Original: O b j ec ti v e
Revised: O b j ec ti v e

Original: V a l u e
Revised: V a l u e

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: v
Revised: v

Original: [Figure]
Revised: [Figure]

Original: r a
Revised: r a

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: o t a l T i m
Revised: o t a l T i m

Original: [Figure]
Revised: [Figure]

Original: importance sampling before using uniform sampling for the remaining iterations.
Revised: importance sampling before using uniform sampling for the remaining iterations.

Original: Surprisingly, our
Revised: Surprisingly, our

Original: SGD importance sampling implementation not only signiﬁcantly improves upon SGD with uniform sampling, but are also competitive with the two hybrid algorithms.
Revised: SGD importance sampling implementation not only signiﬁcantly improves upon SGD with uniform sampling, but are also competitive with the two hybrid algorithms.

Original: We do not consider other SGD variants due to either their distributional assumptions or lack of known ﬂexibility to big data models.
Revised: We do not consider other SGD variants due to either their distributional assumptions or lack of known ﬂexibility to big data models.

Original: The experiments were performed in Python 3.6.9 on an Intel Core i7-8700K 3.70 GHz CPU with
Revised: The experiments were performed in Python 3.6.9 on an Intel Core i7-8700K 3.70 GHz CPU with

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e O b j ec ti v e V a l u e
Revised: A v e r a g e O b j ec ti v e V a l u e

Original: 12 cores and 64GB DDR4 memory, using a Nvidia Geforce GTX 1080 Ti 11GB GPU.
Revised: 12 cores and 64GB DDR4 memory, using a Nvidia Geforce GTX 1080 Ti 11GB GPU.

Original: Our code is publicly available at https://github.com/SGD-adaptive-importance/code .
Revised: Our code is publicly available at https://github.com/SGD-adaptive-importance/code .

Original: Logistic Regression.
Revised: Logistic Regression.

Original: We performed logistic regression on the a9a Adult data set collected by UCI and retrieved from LibSVM (Chang & Lin, 2011).
Revised: We performed logistic regression on the a9a Adult data set collected by UCI and retrieved from LibSVM (Chang & Lin, 2011).

Original: The features correspond to responses from the 1994 Census database and the prediction task is to determine whether a person makes over 50K USD a year.
Revised: The features correspond to responses from the 1994 Census database and the prediction task is to determine whether a person makes over 50K USD a year.

Original: We trained using a data batch of 32581 points and 123 features and tested the performance on a separate batch of 16281 data points.
Revised: We trained using a data batch of 32581 points and 123 features and tested the performance on a separate batch of 16281 data points.

Original: For each evaluation, we generated 10 random initial positions shared for importance sampling and uniform sampling.
Revised: For each evaluation, we generated 10 random initial positions shared for importance sampling and uniform sampling.

Original: We then ran 250 iterations of SGD for each of the four algorithms, creating only 250 buckets for the importance sampling algorithm and computed the average performance on each iteration across these 10 separate instances.
Revised: We then ran 250 iterations of SGD for each of the four algorithms, creating only 250 buckets for the importance sampling algorithm and computed the average performance on each iteration across these 10 separate instances.

Original: The relative average performance of all algorithms was relatively robust to the step-size.
Revised: The relative average performance of all algorithms was relatively robust to the step-size.

Original: Although uniform sampling used signiﬁcantly less time overall, our importance sampling SGD algorithm actually had better performance when considering either number of iterations or wall-clock time across all tested step-sizes.
Revised: Although uniform sampling used signiﬁcantly less time overall, our importance sampling SGD algorithm actually had better performance when considering either number of iterations or wall-clock time across all tested step-sizes.

Original: For example, uniform sampling had average objective value 20680 at iteration 250 using 0.0307 seconds with step-size 0.1, but importance sampling had average objective value 12917 at iteration 5 using 0.025 seconds.
Revised: For example, uniform sampling had average objective value 20680 at iteration 250 using 0.0307 seconds with step-size 0.1, but importance sampling had average objective value 12917 at iteration 5 using 0.025 seconds.

Original: We give our results for logistic regression in Figure 1.
Revised: We give our results for logistic regression in Figure 1.

Original: We repeat our experiments in Figure 2 to explicitly compare the objective value of each algorithm with respect to wall-clock time, rather than SGD iterations.
Revised: We repeat our experiments in Figure 2 to explicitly compare the objective value of each algorithm with respect to wall-clock time, rather than SGD iterations.

Original: Thus our results in Figure 2 empirically demonstrate the advantages of our algorithm across the most natural metrics.
Revised: Thus our results in Figure 2 empirically demonstrate the advantages of our algorithm across the most natural metrics.

Original: For additional experiments, see
Revised: For additional experiments, see

Original: Section B.
Revised: Section B.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We have given variance reduction methods for both ﬁrst-order and second-order stochastic optimization.
Revised: We have given variance reduction methods for both ﬁrst-order and second-order stochastic optimization.

Original: Our algorithms require a single pass over the data, which may even arrive implicitly in the form of turnstile updates, and use input sparsity time and ˜ O ( Td ) space.
Revised: Our algorithms require a single pass over the data, which may even arrive implicitly in the form of turnstile updates, and use input sparsity time and ˜ O ( Td ) space.

Original: Our algorithms are also amenable to big data models such as the streaming and distributed models and are supported by empirical evaluations on large-scale datasets.
Revised: Our algorithms are also amenable to big data models such as the streaming and distributed models and are supported by empirical evaluations on large-scale datasets.

Original: We believe there are many interesting future directions to explore.
Revised: We believe there are many interesting future directions to explore.

Original: For example, can we generalize our techniques to show provable guarantees for other SGD variants and accelerated methods?
Revised: For example, can we generalize our techniques to show provable guarantees for other SGD variants and accelerated methods?

Original: A very large-scale empirical study of these methods would also be quite interesting.
Revised: A very large-scale empirical study of these methods would also be quite interesting.

Original: A D ISCUSSION , F ULL A LGORITHMS , AND P ROOFS
Revised: A D ISCUSSION , F ULL A LGORITHMS , AND P ROOFS

Original: For the sake of presentation, we consider the case where p = 2 ; higher dimensions follow from the same approach, using tensor representation instead of matrix representation.
Revised: For the sake of presentation, we consider the case where p = 2 ; higher dimensions follow from the same approach, using tensor representation instead of matrix representation.

Original: Instead of viewing the input matrix A = a 1 ◦ . . .
Revised: Instead of viewing the input matrix A = a 1 ◦ . . .

Original: ◦ a n ∈ R n × d as a number of rows, we instead view the matrix
Revised: ◦ a n ∈ R n × d as a number of rows, we instead view the matrix

Original: A =
Revised: A =

Original: A 1 ◦ . .
Revised: A 1 ◦ . .

Original: . ◦
Revised: . ◦

Original: A n ∈ R nd × d , where each matrix A i = a i ⊗ a i is the outer product of the row a i with itself.
Revised: A n ∈ R nd × d , where each matrix A i = a i ⊗ a i is the outer product of the row a i with itself.

Original: A. L 2 P OLYNOMIAL I NNER P RODUCT S AMPLER
Revised: A. L 2 P OLYNOMIAL I NNER P RODUCT S AMPLER

Original: For ease of discussion, we describe in this section a data structure that allows sampling an index i ∈ [ n ] with probability approximately (cid:107)
Revised: For ease of discussion, we describe in this section a data structure that allows sampling an index i ∈ [ n ] with probability approximately (cid:107)

Original: A i x (cid:107) 1 , 2 ,d (cid:107) Ax (cid:107) 1 , 2 ,d in linear time and sublinear space, where for a matrix A ∈ R nd × d , we use (cid:107) Ax (cid:107) 1 , 2 ,d to denote (cid:80) ni
Revised: A i x (cid:107) 1 , 2 ,d (cid:107) Ax (cid:107) 1 , 2 ,d in linear time and sublinear space, where for a matrix A ∈ R nd × d , we use (cid:107) Ax (cid:107) 1 , 2 ,d to denote (cid:80) ni

Original: =1 (cid:107)
Revised: =1 (cid:107)

Original: A i x (cid:107) 2 , where each A i ∈ R
Revised: A i x (cid:107) 2 , where each A i ∈ R

Original: d × d and
Revised: d × d and

Original: A =
Revised: A =

Original: A 1 ◦ . .
Revised: A 1 ◦ . .

Original: . ◦
Revised: . ◦

Original: A n .
Revised: A n .

Original: The generalization to a L 2 polynomial inner product sampler follows immediately.
Revised: The generalization to a L 2 polynomial inner product sampler follows immediately.

Original: Notably, our data structure can be built simply given access to A , and will still sample from the correct distribution when x is given as a post-processing vector.
Revised: Notably, our data structure can be built simply given access to A , and will still sample from the correct distribution when x is given as a post-processing vector.

Original: We ﬁrst describe in Section A.1.
Revised: We ﬁrst describe in Section A.1.

Original: some necessary subroutines that our sampler requires.
Revised: some necessary subroutines that our sampler requires.

Original: These subroutines are natural generalizations of the well-known frequency moment estimation algorithm of Alon et al.
Revised: These subroutines are natural generalizations of the well-known frequency moment estimation algorithm of Alon et al.

Original: and heavy hitter detection algorithm of Charikar et al.
Revised: and heavy hitter detection algorithm of Charikar et al.

Original: We then give the L 1 , 2 ,d sampler in full in Section A.1.2.
Revised: We then give the L 1 , 2 ,d sampler in full in Section A.1.2.

Original: A.1. F REQUENCY M OMENT AND H EAVY H ITTER G ENERALIZATIONS
Revised: A.1. F REQUENCY M OMENT AND H EAVY H ITTER G ENERALIZATIONS

Original: We ﬁrst recall a generalization to the frequency moment estimation algorithm by Alon et al.
Revised: We ﬁrst recall a generalization to the frequency moment estimation algorithm by Alon et al.

Original: (1999) that also supports post-processing multiplication by any vector x ∈ R d .
Revised: (1999) that also supports post-processing multiplication by any vector x ∈ R d .

Original: Lemma A.1 (Mahabadi et al., 2020)
Revised: Lemma A.1 (Mahabadi et al., 2020)

Original: Given a constant (cid:15) > 0 , there exists a one-pass streaming algorithm AMS that takes updates to entries of a matrix A ∈ R n × d , as well as query access to postprocessing vectors x ∈ R d and v ∈ R d that arrive after the stream, and outputs a quantity ˆ F such that (1 − (cid:15) )
Revised: Given a constant (cid:15) > 0 , there exists a one-pass streaming algorithm AMS that takes updates to entries of a matrix A ∈ R n × d , as well as query access to postprocessing vectors x ∈ R d and v ∈ R d that arrive after the stream, and outputs a quantity ˆ F such that (1 − (cid:15) )

Original: (cid:107)
Revised: (cid:107)

Original: Ax − v (cid:107) 2 ≤ ˆ F ≤ (1 + (cid:15) ) (cid:107)
Revised: Ax − v (cid:107) 2 ≤ ˆ F ≤ (1 + (cid:15) ) (cid:107)

Original: Ax − v (cid:107) 2 .
Revised: Ax − v (cid:107) 2 .

Original: The algorithm uses O (cid:0) d(cid:15) 2 (cid:0) log 2 n + log 1 δ (cid:1)(cid:1) bits of space and succeeds with probability at least 1 − δ .
Revised: The algorithm uses O (cid:0) d(cid:15) 2 (cid:0) log 2 n + log 1 δ (cid:1)(cid:1) bits of space and succeeds with probability at least 1 − δ .

Original: r × b
Revised: r × b

Original: of buckets
Revised: of buckets

Original: bucket stores an R d × d
Revised: bucket stores an R d × d

Original: i
Revised: i

Original: {−
Revised: {−

Original: i ∈ [
Revised: i ∈ [

Original: i
Revised: i

Original: [ n ] → [ b ]
Revised: [ n ] → [ b ]

Original: wise
Revised: wise

Original: i
Revised: i

Original: 12: Process
Revised: 12: Process

Original: Let A 1 , . . .
Revised: Let A 1 , . . .

Original: , A n ∈ R d × d and A =
Revised: , A n ∈ R d × d and A =

Original: A 1 ◦ . .
Revised: A 1 ◦ . .

Original: . ◦
Revised: . ◦

Original: A n ∈ R nd × d .
Revised: A n ∈ R nd × d .

Original: Let x ∈ R
Revised: Let x ∈ R

Original: d
Revised: d

Original: × 1 be a post-processing vector that is revealed only after A has been completely processed.
Revised: × 1 be a post-processing vector that is revealed only after A has been completely processed.

Original: For a given (cid:15) > 0 , we say a block
Revised: For a given (cid:15) > 0 , we say a block

Original: A i with i ∈ [ n ] is heavy if (cid:107)
Revised: A i with i ∈ [ n ] is heavy if (cid:107)

Original: A
Revised: A

Original: i x (cid:107) 2 ≥ (cid:15) (cid:107) Ax (cid:107) 1 , 2 ,d .
Revised: i x (cid:107) 2 ≥ (cid:15) (cid:107) Ax (cid:107) 1 , 2 ,d .

Original: We show in Algorithm 4 an algorithm that processes A into a sublinear space data structure and identiﬁes the heavy blocks of A once x is given.
Revised: We show in Algorithm 4 an algorithm that processes A into a sublinear space data structure and identiﬁes the heavy blocks of A once x is given.

Original: Moreover, for each heavy block A i , the algorithm outputs a vector y that is a good approximation to
Revised: Moreover, for each heavy block A i , the algorithm outputs a vector y that is a good approximation to

Original: A i x .
Revised: A i x .

Original: The algorithm is a natural generalization of the CountSketch heavy-hitter algorithm introduced by Charikar et al.
Revised: The algorithm is a natural generalization of the CountSketch heavy-hitter algorithm introduced by Charikar et al.

Original: For a vector v ∈ R nd × 1 , we use v tail ( b ) to denote v with the b blocks of d rows of v with the largest (cid:96) 2 norm set to zeros.
Revised: For a vector v ∈ R nd × 1 , we use v tail ( b ) to denote v with the b blocks of d rows of v with the largest (cid:96) 2 norm set to zeros.

Original: Lemma A.2 There exists an algorithm that uses O (cid:0) 1 (cid:15) 2 d 2 (cid:0) log 2 n + log 1 δ (cid:1)(cid:1) space that outputs a vector y
Revised: Lemma A.2 There exists an algorithm that uses O (cid:0) 1 (cid:15) 2 d 2 (cid:0) log 2 n + log 1 δ (cid:1)(cid:1) space that outputs a vector y

Original: i for each index i ∈ [ n ] so that | (cid:107)
Revised: i for each index i ∈ [ n ] so that | (cid:107)

Original: y
Revised: y

Original: i (cid:107) 2 − (cid:107) A i x (cid:107) 2 | ≤ (cid:15) (cid:13)(cid:13)(cid:13)
Revised: i (cid:107) 2 − (cid:107) A i x (cid:107) 2 | ≤ (cid:15) (cid:13)(cid:13)(cid:13)

Original: ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:15) (cid:13)(cid:13)(cid:13)
Revised: ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:15) (cid:13)(cid:13)(cid:13)

Original: ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 1 , 2 ,d with probability at least 1 − δ .
Revised: ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 1 , 2 ,d with probability at least 1 − δ .

Original: Moreover if Y = y 1 ◦ . . .
Revised: Moreover if Y = y 1 ◦ . . .

Original: ◦ y n , then (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 )
Revised: ◦ y n , then (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 )

Original: [Table]
Revised: [Table]

Original: 2 ≤ (cid:13)(cid:13)(cid:13)
Revised: 2 ≤ (cid:13)(cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: − (cid:98) Y
Revised: − (cid:98) Y

Original: [Table]
Revised: [Table]

Original: 2 ≤ 2 (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 with probability at least 1 − δ , where (cid:98)
Revised: 2 ≤ 2 (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 with probability at least 1 − δ , where (cid:98)

Original: Y =
Revised: Y =

Original: Y − Y tail ( 2 (cid:15) 2 ) denotes the top 2 (cid:15) 2 blocks of Y by (cid:96) 2 norm.
Revised: Y − Y tail ( 2 (cid:15) 2 ) denotes the top 2 (cid:15) 2 blocks of Y by (cid:96) 2 norm.

Original: Proof : Fix an index i ∈ [ n ] .
Revised: Proof : Fix an index i ∈ [ n ] .

Original: Consider the estimate of (cid:107)
Revised: Consider the estimate of (cid:107)

Original: A i x (cid:107) 2 in row α of the CountSketch table
Revised: A i x (cid:107) 2 in row α of the CountSketch table

Original: T .
Revised: T .

Original: Then h α ( i ) is the bucket of T in row α to which A i x hashes.
Revised: Then h α ( i ) is the bucket of T in row α to which A i x hashes.

Original: Let E 1 be the event that the 2 (cid:15)blocks of size d of Ax with the largest (cid:96) 2 norm are not hashed to h
Revised: Let E 1 be the event that the 2 (cid:15)blocks of size d of Ax with the largest (cid:96) 2 norm are not hashed to h

Original: α ( i ) .
Revised: α ( i ) .

Original: Observe that for b = Ω (cid:0) 1 (cid:15) 2 (cid:1) with sufﬁciently large constant, E 1 occurs with probability at least 112 by a union bound.
Revised: Observe that for b = Ω (cid:0) 1 (cid:15) 2 (cid:1) with sufﬁciently large constant, E 1 occurs with probability at least 112 by a union bound.

Original: Let v be the sum of the vectors representing the blocks that are hashed to bucket h α ( i ) excluding A i x , so that v is the noise for the estimate of A i x in row α .
Revised: Let v be the sum of the vectors representing the blocks that are hashed to bucket h α ( i ) excluding A i x , so that v is the noise for the estimate of A i x in row α .

Original: Conditioned on E 1 , we can bound the expected squared norm of the noise in bucket
Revised: Conditioned on E 1 , we can bound the expected squared norm of the noise in bucket

Original: h α ( i ) for sufﬁciently large b by E (cid:104) (cid:107) v (cid:107) 22 (cid:105) ≤ (cid:15) 2 9 (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 2 .
Revised: h α ( i ) for sufﬁciently large b by E (cid:104) (cid:107) v (cid:107) 22 (cid:105) ≤ (cid:15) 2 9 (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 2 .

Original: Hence we have Var( (cid:107) v i (cid:107) 2 ) ≤ (cid:15) 2 9 (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 2 .
Revised: Hence we have Var( (cid:107) v i (cid:107) 2 ) ≤ (cid:15) 2 9 (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 2 .

Original: Thus from Jensen’s inequality, Chebyshev’s inequality and conditioning on E 1 ,
Revised: Thus from Jensen’s inequality, Chebyshev’s inequality and conditioning on E 1 ,

Original: Pr (cid:104) (cid:107) v
Revised: Pr (cid:104) (cid:107) v

Original: [Table]
Revised: [Table]

Original: 2 ≥
Revised: 2 ≥

Original: [Table]
Revised: [Table]

Original: Ax ) tail (
Revised: Ax ) tail (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 2 (cid:105) ≤ 1 4 + 1 12 = 2 3 .
Revised: 2 (cid:105) ≤ 1 4 + 1 12 = 2 3 .

Original: The ﬁrst claim then follows from the observation that (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)(cid:13)
Revised: The ﬁrst claim then follows from the observation that (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)(cid:13)

Original: ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 1 , 2 ,d and noting that we can boost the probability of success to 1 − 1 poly( n ) by repeating for each of the r = Θ(log n ) rows and taking the median.
Revised: ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 1 , 2 ,d and noting that we can boost the probability of success to 1 − 1 poly( n ) by repeating for each of the r = Θ(log n ) rows and taking the median.

Original: Finally, observe that (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)(cid:13)
Revised: Finally, observe that (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)(cid:13)

Original: Ax − (cid:98) Y (cid:13)(cid:13)(cid:13) 2 , since (cid:98) Y has at most 2 (cid:15)
Revised: Ax − (cid:98) Y (cid:13)(cid:13)(cid:13) 2 , since (cid:98) Y has at most 2 (cid:15)

Original: 2 nonzero blocks, while ( Ax ) tail ( 2 (cid:15) 2 ) has all zeros in the 2 (cid:15) 2 blocks of Ax with the largest (cid:96) 2 norm.
Revised: 2 nonzero blocks, while ( Ax ) tail ( 2 (cid:15) 2 ) has all zeros in the 2 (cid:15) 2 blocks of Ax with the largest (cid:96) 2 norm.

Original: Since Ax − (cid:98) Y alters at most 2 (cid:15) 2 rows of Ax , each by at most (cid:15) (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 , then (cid:13)(cid:13)(cid:13)
Revised: Since Ax − (cid:98) Y alters at most 2 (cid:15) 2 rows of Ax , each by at most (cid:15) (cid:13)(cid:13)(cid:13) ( Ax ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 , then (cid:13)(cid:13)(cid:13)

Original: Ax − (cid:98) Y (cid:13)(cid:13)(cid:13) 2 ≤ (cid:118)(cid:117)(cid:117)(cid:116) 2 /(cid:15) 2 (cid:88) i = (cid:16) (cid:15) (cid:13)(cid:13)(cid:13)
Revised: Ax − (cid:98) Y (cid:13)(cid:13)(cid:13) 2 ≤ (cid:118)(cid:117)(cid:117)(cid:116) 2 /(cid:15) 2 (cid:88) i = (cid:16) (cid:15) (cid:13)(cid:13)(cid:13)

Original: ( Ax ) tail ( 2 (cid:15)
Revised: ( Ax ) tail ( 2 (cid:15)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: A.1. S AMPLING A LGORITHM
Revised: A.1. S AMPLING A LGORITHM

Original: Our approach is similar to (cid:96) p sampling techniques in Andoni et al.
Revised: Our approach is similar to (cid:96) p sampling techniques in Andoni et al.

Original: ; Jowhari et al.
Revised: ; Jowhari et al.

Original: (2011), who consider sampling indices in vectors, and almost verbatim to Mahabadi et al.
Revised: (2011), who consider sampling indices in vectors, and almost verbatim to Mahabadi et al.

Original: (2020), who consider sampling rows in matrices given post-processing multiplication by a vector.
Revised: (2020), who consider sampling rows in matrices given post-processing multiplication by a vector.

Original: The high level idea is to note that if t i ∈ [0 , 1] is chosen uniformly at random, then
Revised: The high level idea is to note that if t i ∈ [0 , 1] is chosen uniformly at random, then

Original: [Equation]
Revised: [Equation]

Original: Thus if B i
Revised: Thus if B i

Original: =
Revised: =

Original: A i t i and there exists exactly one index i
Revised: A i t i and there exists exactly one index i

Original: such that (cid:107) B
Revised: such that (cid:107) B

Original: i (cid:107) 1 , 2 ,d ≥ (cid:107) A (cid:107) 1 , 2 ,d , then the task would reduce to outputting B j that maximizes (cid:107) B j (cid:107) 1 , 2 ,d over all j ∈ [ n ] .
Revised: i (cid:107) 1 , 2 ,d ≥ (cid:107) A (cid:107) 1 , 2 ,d , then the task would reduce to outputting B j that maximizes (cid:107) B j (cid:107) 1 , 2 ,d over all j ∈ [ n ] .

Original: In fact, we can show that
Revised: In fact, we can show that

Original: B i is an O ( (cid:15) )
Revised: B i is an O ( (cid:15) )

Original: -heavy hitter of B with respect to the L 1 , 2 ,d norm.
Revised: -heavy hitter of B with respect to the L 1 , 2 ,d norm.

Original: Hence, we use a generalization of
Revised: Hence, we use a generalization of

Original: C OUNT S KETCH to identify the heavy hitters of B , approximate the maximum index i , and check whether (cid:107) B i (cid:107) 1 , 2 ,d is at least (an estimate of) (cid:107) A (cid:107) 1 , 2 ,d .
Revised: C OUNT S KETCH to identify the heavy hitters of B , approximate the maximum index i , and check whether (cid:107) B i (cid:107) 1 , 2 ,d is at least (an estimate of) (cid:107) A (cid:107) 1 , 2 ,d .

Original: Unfortunately, this argument might fail due to several reasons.
Revised: Unfortunately, this argument might fail due to several reasons.

Original: Firstly, there might exists zero or multiple indices i such that (cid:107) B
Revised: Firstly, there might exists zero or multiple indices i such that (cid:107) B

Original: i (cid:107) 1 , 2 ,d ≥ (cid:107) A (cid:107) 1 , 2 ,d .
Revised: i (cid:107) 1 , 2 ,d ≥ (cid:107) A (cid:107) 1 , 2 ,d .

Original: Then the probability distribution that an index i satisﬁes (cid:107) B i (cid:107) 1 , 2 ,d ≥ (cid:107) A (cid:107) 1 , 2 ,d and that (cid:107) B
Revised: Then the probability distribution that an index i satisﬁes (cid:107) B i (cid:107) 1 , 2 ,d ≥ (cid:107) A (cid:107) 1 , 2 ,d and that (cid:107) B

Original: i (cid:107) 1 , 2 ,d > (cid:107) B j (cid:107) 1 , 2 ,d for all other j ∈ [ n ] is not the same as the desired distribution.
Revised: i (cid:107) 1 , 2 ,d > (cid:107) B j (cid:107) 1 , 2 ,d for all other j ∈ [ n ] is not the same as the desired distribution.

Original: Fortunately, we show that this only happens with small probability, slightly perturbing the probability of returning each i ∈ [ n ] .
Revised: Fortunately, we show that this only happens with small probability, slightly perturbing the probability of returning each i ∈ [ n ] .

Original: Another possibility is that the error in C OUNT S KETCH is large enough to misidentify whether (cid:107) B i (cid:107) 1 , 2 ,d ≥ (cid:107) A (cid:107) 1 , 2 ,d .
Revised: Another possibility is that the error in C OUNT S KETCH is large enough to misidentify whether (cid:107) B i (cid:107) 1 , 2 ,d ≥ (cid:107) A (cid:107) 1 , 2 ,d .

Original: Using a statistical test, this case can usually be identiﬁed
Revised: Using a statistical test, this case can usually be identiﬁed

Original: and so the algorithm will be prevented from outputting a sample in this case.
Revised: and so the algorithm will be prevented from outputting a sample in this case.

Original: Crucially, the probability that the algorithm is aborted by the statistical test is roughly independent of which index achieves the maximum.
Revised: Crucially, the probability that the algorithm is aborted by the statistical test is roughly independent of which index achieves the maximum.

Original: As a result, the probability of returning each i ∈ [ n ] is within a (1 ± (cid:15) ) factor of (cid:107) A i (cid:107) 1 , 2 ,d (cid:107) A (cid:107) 1 , 2 ,d when the algorithm does not abort.
Revised: As a result, the probability of returning each i ∈ [ n ] is within a (1 ± (cid:15) ) factor of (cid:107) A i (cid:107) 1 , 2 ,d (cid:107) A (cid:107) 1 , 2 ,d when the algorithm does not abort.

Original: We show that the probability that algorithm succeeds is Θ( (cid:15) )
Revised: We show that the probability that algorithm succeeds is Θ( (cid:15) )

Original: so then running O (cid:0) log 1 (cid:15) (cid:1) instances of the algorithm sufﬁces to output some index from the desired distribution with constant probability, or abort otherwise.
Revised: so then running O (cid:0) log 1 (cid:15) (cid:1) instances of the algorithm sufﬁces to output some index from the desired distribution with constant probability, or abort otherwise.

Original: Because the underlying data structure is a linear sketch, then it is also robust to post-processing multiplication by any vector x ∈ R d .
Revised: Because the underlying data structure is a linear sketch, then it is also robust to post-processing multiplication by any vector x ∈ R d .

Original: Finally, we note that although our presentation refers to the scaling factors t i as independent random variables, our analysis shows that they only need to be O (1) -wise independent and thus we can generate the scaling factors in small space in the streaming model.
Revised: Finally, we note that although our presentation refers to the scaling factors t i as independent random variables, our analysis shows that they only need to be O (1) -wise independent and thus we can generate the scaling factors in small space in the streaming model.

Original: We give the L 1 , 2 ,d sampler in Algorithm 5.
Revised: We give the L 1 , 2 ,d sampler in Algorithm 5.

Original: M ∈ R nd
Revised: M ∈ R nd

Original: 2 (cid:15) block sparse matrix consisting top (
Revised: 2 (cid:15) block sparse matrix consisting top (

Original: ) block d rows
Revised: ) block d rows

Original: OUNT the largest
Revised: OUNT the largest

Original: 21: else 22: t i r i
Revised: 21: else 22: t i r i

Original: We ﬁrst show that the probability that Algorithm 5 returns FAIL is independent of which index i ∈ [ n ] achieves argmax i ∈ [ n ] 1 t i (cid:107)
Revised: We ﬁrst show that the probability that Algorithm 5 returns FAIL is independent of which index i ∈ [ n ] achieves argmax i ∈ [ n ] 1 t i (cid:107)

Original: A i x (cid:107) 2 .
Revised: A i x (cid:107) 2 .

Original: Lemma A.3 Let i ∈ [ n ] and ﬁx a value of t i ∈ [0 , 1] uniformly at random.
Revised: Lemma A.3 Let i ∈ [ n ] and ﬁx a value of t i ∈ [0 , 1] uniformly at random.

Original: Then conditioned on the value of t i ,
Revised: Then conditioned on the value of t i ,

Original: [Equation]
Revised: [Equation]

Original: Proof :
Revised: Proof :

Original: We ﬁrst observe that if we upper bound (cid:98) S by 4 (cid:13)(cid:13)(cid:13) ( Bx ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 and lower bound (cid:98) F by (cid:107) Ax (cid:107) 1 , 2 ,d , then it sufﬁces to show that the probability of 4 (cid:13)(cid:13)(cid:13) ( Bx ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 > (cid:113) log 1 (cid:15) (cid:107) Ax (cid:107) 1 , 2 ,d is small.
Revised: We ﬁrst observe that if we upper bound (cid:98) S by 4 (cid:13)(cid:13)(cid:13) ( Bx ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 and lower bound (cid:98) F by (cid:107) Ax (cid:107) 1 , 2 ,d , then it sufﬁces to show that the probability of 4 (cid:13)(cid:13)(cid:13) ( Bx ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 > (cid:113) log 1 (cid:15) (cid:107) Ax (cid:107) 1 , 2 ,d is small.

Original: Thus we deﬁne E 1 as the event that:
Revised: Thus we deﬁne E 1 as the event that:

Original: [Equation]
Revised: [Equation]

Original: Note that by Theorem 2.1, Lemma A.2 and Lemma A.1, E 1 holds with high probability.
Revised: Note that by Theorem 2.1, Lemma A.2 and Lemma A.1, E 1 holds with high probability.

Original: Let U
Revised: Let U

Original: =
Revised: =

Original: (cid:107) Ax (cid:107) 1 , 2 ,d .
Revised: (cid:107) Ax (cid:107) 1 , 2 ,d .

Original: For each block A j x , we deﬁne y j to be the indicator variable for whether the scaled block
Revised: For each block A j x , we deﬁne y j to be the indicator variable for whether the scaled block

Original: B j x is heavy, so that y j = 1 if (cid:107) B j x (cid:107) 2 >
Revised: B j x is heavy, so that y j = 1 if (cid:107) B j x (cid:107) 2 >

Original: U and y j
Revised: U and y j

Original: = 0 otherwise.
Revised: = 0 otherwise.

Original: We also deﬁne z j ∈ [0 , 1] as a scaled random variable for whether B j x is light and how much squared mass it contributes, z j = 1 U 2 (cid:107) B j x (cid:107) 22 (1 − y j ) .
Revised: We also deﬁne z j ∈ [0 , 1] as a scaled random variable for whether B j x is light and how much squared mass it contributes, z j = 1 U 2 (cid:107) B j x (cid:107) 22 (1 − y j ) .

Original: Let Y =
Revised: Let Y =

Original: (cid:80) j (cid:54) =
Revised: (cid:80) j (cid:54) =

Original: i
Revised: i

Original: y j be the total number of heavy blocks besides
Revised: y j be the total number of heavy blocks besides

Original: B i x and Z = (cid:80) j (cid:54) =
Revised: B i x and Z = (cid:80) j (cid:54) =

Original: i
Revised: i

Original: z j be the total scaled squared mass of the small rows.
Revised: z j be the total scaled squared mass of the small rows.

Original: Let h ∈ R nd be the vector that contains the heavy blocks so that coordinates ( j − 1)
Revised: Let h ∈ R nd be the vector that contains the heavy blocks so that coordinates ( j − 1)

Original: d
Revised: d

Original: + 1 through jd of h correspond to B j x if y j = 

Revised: + 1 through jd of h correspond to B j x if y j =

Original: and they are all zeros otherwise.
Revised: and they are all zeros otherwise.

Original: Hence, h contains at most Y + 1 nonzero blocks and thus at most ( Y + 1) d nonzero entries.
Revised: Hence, h contains at most Y + 1 nonzero blocks and thus at most ( Y + 1) d nonzero entries.

Original: Moreover, U 2 Z = (cid:107) Bx − h (cid:107) 22 and (cid:13)(cid:13)(cid:13) ( Bx ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ U √ Z unless Y ≥ 2 (cid:15) 2 .
Revised: Moreover, U 2 Z = (cid:107) Bx − h (cid:107) 22 and (cid:13)(cid:13)(cid:13) ( Bx ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ U √ Z unless Y ≥ 2 (cid:15) 2 .

Original: Thus if we deﬁne E 2 to be the event that Y ≥ 2 (cid:15) 2 and E 3 to be the event that Z ≥ 1 16 U 2 log 1 (cid:15) (cid:107) Ax (cid:107) 21 , 2 ,d , then ¬E 2 ∧ ¬E 3 implies 4 (cid:13)(cid:13)(cid:13) ( Bx ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:113) log 1 (cid:15) (cid:107) Ax (cid:107) 1 , 2 ,d , so it sufﬁces to bound the probability of the events E 2 and E 3 by O ( (cid:15) ) .
Revised: Thus if we deﬁne E 2 to be the event that Y ≥ 2 (cid:15) 2 and E 3 to be the event that Z ≥ 1 16 U 2 log 1 (cid:15) (cid:107) Ax (cid:107) 21 , 2 ,d , then ¬E 2 ∧ ¬E 3 implies 4 (cid:13)(cid:13)(cid:13) ( Bx ) tail ( 2 (cid:15) 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:113) log 1 (cid:15) (cid:107) Ax (cid:107) 1 , 2 ,d , so it sufﬁces to bound the probability of the events E 2 and E 3 by O ( (cid:15) ) .

Original: Intuitively, if the number of heavy rows is small ( ¬E 2 ) and the total contribution of the small rows is small ( ¬E 3 ), then the tail estimator is small, so the probability of failure due to the tail estimator is small.
Revised: Intuitively, if the number of heavy rows is small ( ¬E 2 ) and the total contribution of the small rows is small ( ¬E 3 ), then the tail estimator is small, so the probability of failure due to the tail estimator is small.

Original: To analyze E 2 , note that y j
Revised: To analyze E 2 , note that y j

Original: = 1 if and only if 1 t j (cid:107)
Revised: = 1 if and only if 1 t j (cid:107)

Original: A j x (cid:107) 2 >
Revised: A j x (cid:107) 2 >

Original: U , so E
Revised: U , so E

Original: [ y i ]
Revised: [ y i ]

Original: = (cid:107)
Revised: = (cid:107)

Original: A j x (cid:107) 2 U and thus
Revised: A j x (cid:107) 2 U and thus

Original: E [ Y ] ≤ 1 since Y =
Revised: E [ Y ] ≤ 1 since Y =

Original: (cid:80) j (cid:54) =
Revised: (cid:80) j (cid:54) =

Original: i y j and U
Revised: i y j and U

Original: = (cid:107) Ax (cid:107) 1 , 2 ,d
Revised: = (cid:107) Ax (cid:107) 1 , 2 ,d

Original: = (cid:80) j (cid:107)
Revised: = (cid:80) j (cid:107)

Original: A j x (cid:107) 2 .
Revised: A j x (cid:107) 2 .

Original: We also have Var( Y ) ≤ 1 so that Pr [ E 2 ] =
Revised: We also have Var( Y ) ≤ 1 so that Pr [ E 2 ] =

Original: O ( (cid:15) ) for sufﬁciently small (cid:15) , by Chebyshev’s inequality.
Revised: O ( (cid:15) ) for sufﬁciently small (cid:15) , by Chebyshev’s inequality.

Original: To analyze E 3 , recall that z j = 1 U 2 (cid:107) B j x (cid:107) 22 (1 − y j ) .
Revised: To analyze E 3 , recall that z j = 1 U 2 (cid:107) B j x (cid:107) 22 (1 − y j ) .

Original: Thus z j > 0 only if y j = 0 or equivalently, (cid:107) B j x (cid:107) 2 ≤ U .
Revised: Thus z j > 0 only if y j = 0 or equivalently, (cid:107) B j x (cid:107) 2 ≤ U .

Original: Since B j x = 1 t j A j x , then z j > 0 only if t j ≥ (cid:107) A j x (cid:107) 2 (cid:107) Ax (cid:107) 1 , 2 ,d .
Revised: Since B j x = 1 t j A j x , then z j > 0 only if t j ≥ (cid:107) A j x (cid:107) 2 (cid:107) Ax (cid:107) 1 , 2 ,d .

Original: Therefore,
Revised: Therefore,

Original: E [ z j ] ≤ (cid:90) ∞
Revised: E [ z j ] ≤ (cid:90) ∞

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since Z = (cid:80) j (cid:54) =
Revised: Since Z = (cid:80) j (cid:54) =

Original: i
Revised: i

Original: z j , then E [ Z ] ≤ 1 and similarly Var( Z ) ≤ 1 .
Revised: z j , then E [ Z ] ≤ 1 and similarly Var( Z ) ≤ 1 .

Original: Hence by Bernstein’s inequality,
Revised: Hence by Bernstein’s inequality,

Original: Pr (cid:2)
Revised: Pr (cid:2)

Original: Z > 116 log 1 (cid:15) (cid:3) =
Revised: Z > 116 log 1 (cid:15) (cid:3) =

Original: O ( (cid:15) ) , so then Pr [ E 3 ] =
Revised: O ( (cid:15) ) , so then Pr [ E 3 ] =

Original: O ( (cid:15) ) .
Revised: O ( (cid:15) ) .

Original: Thus Pr [ ¬E 1 ∨ E 2 ∨ E 3 ] =
Revised: Thus Pr [ ¬E 1 ∨ E 2 ∨ E 3 ] =

Original: O ( (cid:15) )
Revised: O ( (cid:15) )

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: , as desired.
Revised: , as desired.

Original: [Equation]
Revised: [Equation]

Original: We now show that Algorithm 5 outputs a noisy approximation to A i x , where i ∈ [ n ] is drawn from approximately the correct distribution, i.e., the probability of failure does not correlate with the index that achieves the maximum value.
Revised: We now show that Algorithm 5 outputs a noisy approximation to A i x , where i ∈ [ n ] is drawn from approximately the correct distribution, i.e., the probability of failure does not correlate with the index that achieves the maximum value.

Original: Lemma
Revised: Lemma

Original: A.

Revised: A.

Original: For a ﬁxed value of (cid:98) F , the probability that Algorithm 5 outputs (noisy) submatrix
Revised: For a ﬁxed value of (cid:98) F , the probability that Algorithm 5 outputs (noisy) submatrix

Original: A i x is (1 ± O ( (cid:15) ))
Revised: A i x is (1 ± O ( (cid:15) ))

Original: (cid:107) A
Revised: (cid:107) A

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:98)
Revised: (cid:98)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Proof : Let E be the event that t
Revised: Proof : Let E be the event that t

Original: i < (cid:15) (cid:107) A i P (cid:107) 2 (cid:98) F so that Pr [ E ] = (cid:15) (cid:107) A i x (cid:107) 2 (cid:98) F .
Revised: i < (cid:15) (cid:107) A i P (cid:107) 2 (cid:98) F so that Pr [ E ] = (cid:15) (cid:107) A i x (cid:107) 2 (cid:98) F .

Original: Let E 1 be the event that
Revised: Let E 1 be the event that

Original: C OUNT S KETCH , AMS, or E STIMATOR fails so that Pr [ E 1 ] = 1 poly( n ) by Lemma A.2, Lemma A.1, and Theorem 2.1.
Revised: C OUNT S KETCH , AMS, or E STIMATOR fails so that Pr [ E 1 ] = 1 poly( n ) by Lemma A.2, Lemma A.1, and Theorem 2.1.

Original: Let E 2 be the event that (cid:98) S > (cid:98) F (cid:113) log 1 (cid:15) so that Pr [ E 2 ]
Revised: Let E 2 be the event that (cid:98) S > (cid:98) F (cid:113) log 1 (cid:15) so that Pr [ E 2 ]

Original: =
Revised: =

Original: O ( (cid:15) ) by Lemma A.3.
Revised: O ( (cid:15) ) by Lemma A.3.

Original: Let E 3 be the event that multiple rows B j x exceeding the threshold are observed in the CountSketch data structure and E 4 be the event that (cid:107) B i x (cid:107) 2 exceeds the threshold but is not reported due to noise in the CountSketch data structure.
Revised: Let E 3 be the event that multiple rows B j x exceeding the threshold are observed in the CountSketch data structure and E 4 be the event that (cid:107) B i x (cid:107) 2 exceeds the threshold but is not reported due to noise in the CountSketch data structure.

Original: Observe that E 3 and E 4 are essentially two sides of the same coin, where error is incurred due to the inaccuracies of CountSketch.
Revised: Observe that E 3 and E 4 are essentially two sides of the same coin, where error is incurred due to the inaccuracies of CountSketch.

Original: To analyze E 3 , note that row j (cid:54)
Revised: To analyze E 3 , note that row j (cid:54)

Original: = i can be reported as exceeding the threshold if (cid:107) B j x (cid:107) 2 ≥ 1 (cid:15) (cid:98) F − (cid:98) F (cid:113) log 1 (cid:15) , which occurs with probability at most O (cid:16) (cid:15) (cid:107)
Revised: = i can be reported as exceeding the threshold if (cid:107) B j x (cid:107) 2 ≥ 1 (cid:15) (cid:98) F − (cid:98) F (cid:113) log 1 (cid:15) , which occurs with probability at most O (cid:16) (cid:15) (cid:107)

Original: A j x (cid:107) 2 (cid:98) F (cid:17) .
Revised: A j x (cid:107) 2 (cid:98) F (cid:17) .

Original: By a union bound over all rows j ∈ [ n ] with j (cid:54) =
Revised: By a union bound over all rows j ∈ [ n ] with j (cid:54) =

Original: i , then Pr [ E 3 ]
Revised: i , then Pr [ E 3 ]

Original: = O ( (cid:15) ) .
Revised: = O ( (cid:15) ) .

Original: To analyze E 4 , we ﬁrst condition on ¬E 1 and ¬E 2 , so that (cid:107) Bx − M (cid:107) 2 ≤ (cid:98) S ≤ (cid:98) F (cid:113) log 1 (cid:15) .
Revised: To analyze E 4 , we ﬁrst condition on ¬E 1 and ¬E 2 , so that (cid:107) Bx − M (cid:107) 2 ≤ (cid:98) S ≤ (cid:98) F (cid:113) log 1 (cid:15) .

Original: Then by
Revised: Then by

Original: Lemma A.2, the estimate (cid:100) B i x for B
Revised: Lemma A.2, the estimate (cid:100) B i x for B

Original: i x output by the sampler satisﬁes
Revised: i x output by the sampler satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Hence, E 4 can only occur for
Revised: Hence, E 4 can only occur for

Original: [Equation]
Revised: [Equation]

Original: which occurs with probability at most O (cid:0) (cid:15) 2 (cid:1) .
Revised: which occurs with probability at most O (cid:0) (cid:15) 2 (cid:1) .

Original: To put things together, E occurs with probability (cid:15)
Revised: To put things together, E occurs with probability (cid:15)

Original: [Equation]
Revised: [Equation]

Original: , in which case the L 1 , 2 ,d sampler should output A i x .
Revised: , in which case the L 1 , 2 ,d sampler should output A i x .

Original: However, this may not happen due to any of the events E 1 , E 2 , E 3 , or E 4 .
Revised: However, this may not happen due to any of the events E 1 , E 2 , E 3 , or E 4 .

Original: Since
Revised: Since

Original: Pr [ E 2 ∨ E 3 | E ] =
Revised: Pr [ E 2 ∨ E 3 | E ] =

Original: O ( (cid:15) ) and Pr [ E 4 ] =
Revised: O ( (cid:15) ) and Pr [ E 4 ] =

Original: O (cid:0) (cid:15) 2 (cid:1) , then we have Pr [ E 4 | E ]
Revised: O (cid:0) (cid:15) 2 (cid:1) , then we have Pr [ E 4 | E ]

Original: =
Revised: =

Original: O ( (cid:15) ) .
Revised: O ( (cid:15) ) .

Original: Moreover,
Revised: Moreover,

Original: Pr [ E 1 ] = 1 poly( n ) so that index i is sampled with probability (1 + O ( (cid:15) )) (cid:107)
Revised: Pr [ E 1 ] = 1 poly( n ) so that index i is sampled with probability (1 + O ( (cid:15) )) (cid:107)

Original: A i x (cid:107) 2 (cid:98) F .
Revised: A i x (cid:107) 2 (cid:98) F .

Original: Finally by
Revised: Finally by

Original: Lemma A.2, (cid:12)(cid:12)(cid:12) (cid:107) B
Revised: Lemma A.2, (cid:12)(cid:12)(cid:12) (cid:107) B

Original: i x (cid:107) 2 − (cid:13)(cid:13)(cid:13) (cid:100) B i x (cid:13)(cid:13)(cid:13) 2 (cid:12)(cid:12)(cid:12) ≤ (cid:15) (cid:98) F (cid:113) log 1 (cid:15) and (cid:13)(cid:13)(cid:13) (cid:100)
Revised: i x (cid:107) 2 − (cid:13)(cid:13)(cid:13) (cid:100) B i x (cid:13)(cid:13)(cid:13) 2 (cid:12)(cid:12)(cid:12) ≤ (cid:15) (cid:98) F (cid:113) log 1 (cid:15) and (cid:13)(cid:13)(cid:13) (cid:100)

Original: B i x (cid:13)(cid:13)(cid:13) 2 ≥ 1 (cid:15) (cid:98) F .
Revised: B i x (cid:13)(cid:13)(cid:13) 2 ≥ 1 (cid:15) (cid:98) F .

Original: Hence (cid:13)(cid:13)(cid:13) (cid:100) B
Revised: Hence (cid:13)(cid:13)(cid:13) (cid:100) B

Original: i x (cid:13)(cid:13)(cid:13) 2 is a (1 + (cid:15) ) approximation to (cid:107) B
Revised: i x (cid:13)(cid:13)(cid:13) 2 is a (1 + (cid:15) ) approximation to (cid:107) B

Original: i x (cid:107) 2 and therefore, t i (cid:13)(cid:13)(cid:13) (cid:100) B i x (cid:13)(cid:13)(cid:13) 2 is a (1 + (cid:15) ) approximation to (cid:107)
Revised: i x (cid:107) 2 and therefore, t i (cid:13)(cid:13)(cid:13) (cid:100) B i x (cid:13)(cid:13)(cid:13) 2 is a (1 + (cid:15) ) approximation to (cid:107)

Original: A
Revised: A

Original: i x (cid:107) 2 .
Revised: i x (cid:107) 2 .

Original: (cid:50)
Revised: (cid:50)

Original: Thus we have the following full guarantees for our L 1 , 2 ,d sampler.
Revised: Thus we have the following full guarantees for our L 1 , 2 ,d sampler.

Original: Theorem A.5 Given (cid:15) > 0 , there exists an algorithm that takes a matrix A ∈ R nd × d , which can be written as A =
Revised: Theorem A.5 Given (cid:15) > 0 , there exists an algorithm that takes a matrix A ∈ R nd × d , which can be written as A =

Original: A 1 ◦ . .
Revised: A 1 ◦ . .

Original: ◦
Revised: ◦

Original: A n , where each A i ∈ R d × d .
Revised: A n , where each A i ∈ R d × d .

Original: After A is processed, the algorithm is given a query vector x ∈ R d and outputs a (noisy) vector
Revised: After A is processed, the algorithm is given a query vector x ∈ R d and outputs a (noisy) vector

Original: A i x with probability (1 ± O ( (cid:15) ))
Revised: A i x with probability (1 ± O ( (cid:15) ))

Original: (cid:107)
Revised: (cid:107)

Original: A
Revised: A

Original: i x (cid:107) 2 (cid:107) Ax (cid:107) 1 , 2 ,d + 1 poly( n ) .
Revised: i x (cid:107) 2 (cid:107) Ax (cid:107) 1 , 2 ,d + 1 poly( n ) .

Original: The algorithm uses log 1 δ · nnz( A )+poly (cid:0) d, 1 (cid:15) , log n (cid:1) time, O (cid:0) d (cid:0) poly (cid:0) 1 (cid:15) , log n (cid:1)
Revised: The algorithm uses log 1 δ · nnz( A )+poly (cid:0) d, 1 (cid:15) , log n (cid:1) time, O (cid:0) d (cid:0) poly (cid:0) 1 (cid:15) , log n (cid:1)

Original: + log 1 δ (cid:1)(cid:1) bits of space, and succeeds with probability at least 1 − δ .
Revised: + log 1 δ (cid:1)(cid:1) bits of space, and succeeds with probability at least 1 − δ .

Original: Proof : By Lemma A.4 and Theorem 2.1, then (cid:107) AP (cid:107) 1 , 2 ,d ≤ (cid:98) F ≤ 2 (cid:107) AP (cid:107) 1 , 2 ,d with high probability and so each vector A i x is sampled with probability (1 + (cid:15) )
Revised: Proof : By Lemma A.4 and Theorem 2.1, then (cid:107) AP (cid:107) 1 , 2 ,d ≤ (cid:98) F ≤ 2 (cid:107) AP (cid:107) 1 , 2 ,d with high probability and so each vector A i x is sampled with probability (1 + (cid:15) )

Original: (cid:107)
Revised: (cid:107)

Original: A
Revised: A

Original: i x (cid:107) 2 (cid:107) Ax (cid:107) 1 , 2 ,d + 1 poly( n ) , conditioned on the sampler succeeding.
Revised: i x (cid:107) 2 (cid:107) Ax (cid:107) 1 , 2 ,d + 1 poly( n ) , conditioned on the sampler succeeding.

Original: The probability that the sampler succeeds is Θ( (cid:15) ) , so the sampler can be repeated O (cid:0) 1 (cid:15) log n (cid:1) times to obtain probability of success at least 1 − 1 poly( n ) .
Revised: The probability that the sampler succeeds is Θ( (cid:15) ) , so the sampler can be repeated O (cid:0) 1 (cid:15) log n (cid:1) times to obtain probability of success at least 1 − 1 poly( n ) .

Original: Since each instance of AMS, E STIMATOR , and C OUNT S KETCH use log 1 δ · nnz( A ) + poly (cid:0) d, 1 (cid:15) , log n (cid:1) time and O (cid:0) d (cid:0) poly (cid:0) 1 (cid:15) , log n (cid:1)
Revised: Since each instance of AMS, E STIMATOR , and C OUNT S KETCH use log 1 δ · nnz( A ) + poly (cid:0) d, 1 (cid:15) , log n (cid:1) time and O (cid:0) d (cid:0) poly (cid:0) 1 (cid:15) , log n (cid:1)

Original: + log 1 δ (cid:1)(cid:1) bits of space, then the total time and space complexity follow.
Revised: + log 1 δ (cid:1)(cid:1) bits of space, then the total time and space complexity follow.

Original: (cid:50)
Revised: (cid:50)

Original: By adding sketches corresponding to different polynomial degrees, Theorem A.5 implies Theorem 2. and Theorem 3.2.
Revised: By adding sketches corresponding to different polynomial degrees, Theorem A.5 implies Theorem 2. and Theorem 3.2.

Original: A. L EVERAGE S CORE S AMPLER
Revised: A. L EVERAGE S CORE S AMPLER

Original: Our starting point is the input sparsity time algorithm of (Nelson & Nguyen, 2013) for approximating the leverage scores, which is in turn a modiﬁcation of (Drineas et al., 2012; Clarkson & Woodruff,
Revised: Our starting point is the input sparsity time algorithm of (Nelson & Nguyen, 2013) for approximating the leverage scores, which is in turn a modiﬁcation of (Drineas et al., 2012; Clarkson & Woodruff,

Original: Given an input matrix A , (Nelson & Nguyen, 2013) randomly samples a sparse matrix Πwith ˜ O (cid:0) d(cid:15) 2 (cid:1) rows and ˜ O (cid:0) 1 (cid:15) (cid:1) signs per column, setting the remaining entries to be zero.
Revised: Given an input matrix A , (Nelson & Nguyen, 2013) randomly samples a sparse matrix Πwith ˜ O (cid:0) d(cid:15) 2 (cid:1) rows and ˜ O (cid:0) 1 (cid:15) (cid:1) signs per column, setting the remaining entries to be zero.

Original: (Nelson &
Revised: (Nelson &

Original: Nguyen, 2013) maintains Π 1 A and post-processing, computes R − 1 so that Π 1 AR − 1 has orthonormal columns.
Revised: Nguyen, 2013) maintains Π 1 A and post-processing, computes R − 1 so that Π 1 AR − 1 has orthonormal columns.

Original: Previous work of (Drineas et al., 2012) had shown that the squared row norms of
Revised: Previous work of (Drineas et al., 2012) had shown that the squared row norms of

Original: AR − 1 are (1 + (cid:15) )
Revised: AR − 1 are (1 + (cid:15) )

Original: -approximations to the leverage scores of A .
Revised: -approximations to the leverage scores of A .

Original: Hence for a JL matrix Π 2 that gives (1 + (cid:15) )
Revised: Hence for a JL matrix Π 2 that gives (1 + (cid:15) )

Original: -approximations to the row norms of AR − 1 , we can compute A ( R − 1 Π 2 ) and output the row norms of AR Π 2 as the approximate leverage scores for each row.
Revised: -approximations to the row norms of AR − 1 , we can compute A ( R − 1 Π 2 ) and output the row norms of AR Π 2 as the approximate leverage scores for each row.

Original: Due to the sparsity of Π 1 and Π 2 , the total runtime is ˜ O (cid:0) 1 (cid:15) 2 · nnz( A ) (cid:1) .
Revised: Due to the sparsity of Π 1 and Π 2 , the total runtime is ˜ O (cid:0) 1 (cid:15) 2 · nnz( A ) (cid:1) .

Original: Computing R − 1 takes additional ˜ O (cid:0) d ω (cid:15) 2 (cid:1) runtime.
Revised: Computing R − 1 takes additional ˜ O (cid:0) d ω (cid:15) 2 (cid:1) runtime.

Original: Now since the squared row norms of AR − 1 are (1 + (cid:15) )
Revised: Now since the squared row norms of AR − 1 are (1 + (cid:15) )

Original: -approximations to the leverage scores of A , it sufﬁces to take the rows of AR − 1 with large squared norms.
Revised: -approximations to the leverage scores of A , it sufﬁces to take the rows of AR − 1 with large squared norms.

Original: To that effect, we randomly sample a CountSketch matrix T and maintain TA .
Revised: To that effect, we randomly sample a CountSketch matrix T and maintain TA .

Original: Once R − 1 is computed, we can post-processing right multiply to obtain TAR − 1 , similar to Algorithm 5.
Revised: Once R − 1 is computed, we can post-processing right multiply to obtain TAR − 1 , similar to Algorithm 5.

Original: It follows that any row of TAR − 1 that is at least200 Td -heavy
Revised: It follows that any row of TAR − 1 that is at least200 Td -heavy

Original: (with respect to squared Frobenius norm) has leverage score at least 1 100 Td .
Revised: (with respect to squared Frobenius norm) has leverage score at least 1 100 Td .

Original: Thus we can obtain these rows by querying the CountSketch data structure while using space ˜ O
Revised: Thus we can obtain these rows by querying the CountSketch data structure while using space ˜ O

Original: ( Td ) .
Revised: ( Td ) .

Original: Due to the sparsity of the CountSketch matrix, the total runtime is ˜ O (cid:0) nnz( A ) +
Revised: Due to the sparsity of the CountSketch matrix, the total runtime is ˜ O (cid:0) nnz( A ) +

Original: [Equation]
Revised: [Equation]

Original: (cid:1) .
Revised: (cid:1) .

Original: Finally, (Mahabadi et al., 2020) show that the error guarantee on each reported heavy row required by Theorem 2.3.
Revised: Finally, (Mahabadi et al., 2020) show that the error guarantee on each reported heavy row required by Theorem 2.3.

Original: By reporting the outer products of each of the heavy rows rather than the heavy rows, we obtain
Revised: By reporting the outer products of each of the heavy rows rather than the heavy rows, we obtain

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: A. A PPROXIMATE SGD WITH I MPORTANCE S AMPLING
Revised: A. A PPROXIMATE SGD WITH I MPORTANCE S AMPLING

Original: Proof of Lemma 2.4:
Revised: Proof of Lemma 2.4:

Original: For any t ∈ [ T ] and i ∈ [ n ] , (cid:107)
Revised: For any t ∈ [ T ] and i ∈ [ n ] , (cid:107)

Original: A i x t (cid:107) 22 ≥ 1 100 Td (cid:107)
Revised: A i x t (cid:107) 22 ≥ 1 100 Td (cid:107)

Original: Ax t (cid:107) 2 F only if there exists a row in A i whose leverage score is at least 1 100 Td , since there are d rows in A i .
Revised: Ax t (cid:107) 2 F only if there exists a row in A i whose leverage score is at least 1 100 Td , since there are d rows in A i .

Original: Algorithmcalculates a 2 -approximation to each leverage score and maintains T separate instances of the L 1 , 2 ,d samplers for any matrix containing a row with approximate leverage score at least 1 100 Td .
Revised: Algorithmcalculates a 2 -approximation to each leverage score and maintains T separate instances of the L 1 , 2 ,d samplers for any matrix containing a row with approximate leverage score at least 1 100 Td .

Original: Thus for these indices i ∈ [ n ] , we maintain T separate instances of the L 1 , 2 ,d samplers for A i by explicitly maintaining the heavy row.
Revised: Thus for these indices i ∈ [ n ] , we maintain T separate instances of the L 1 , 2 ,d samplers for A i by explicitly maintaining the heavy row.

Original: Otherwise, for all j ∈ [ β ] so that h ( i ) (cid:54) =
Revised: Otherwise, for all j ∈ [ β ] so that h ( i ) (cid:54) =

Original: j for any index i ∈ [ n ] such that (cid:107)
Revised: j for any index i ∈ [ n ] such that (cid:107)

Original: A
Revised: A

Original: i x t (cid:107) 22 <
Revised: i x t (cid:107) 22 <

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 99100 by Bernstein’s inequality and a union bound over j ∈ [ β ] for β = Θ( Td ) with sufﬁciently high constant.
Revised: with probability at least 99100 by Bernstein’s inequality and a union bound over j ∈ [ β ] for β = Θ( Td ) with sufﬁciently high constant.

Original: Intuitively, by excluding the hash indices containing “heavy” matrices, the remaining hash indices contain only a small fraction of the mass with high probability.
Revised: Intuitively, by excluding the hash indices containing “heavy” matrices, the remaining hash indices contain only a small fraction of the mass with high probability.

Original: Then the probability that any j ∈ [ β ] with (cid:80) i : h ( i )=
Revised: Then the probability that any j ∈ [ β ] with (cid:80) i : h ( i )=

Original: j (cid:107)
Revised: j (cid:107)

Original: A i x t (cid:107) 2 ≤ 110 T (cid:107)
Revised: A i x t (cid:107) 2 ≤ 110 T (cid:107)

Original: Ax t (cid:107) 1 , 2 ,d is sampled more than once is at most 1 100 T for any t ∈
Revised: Ax t (cid:107) 1 , 2 ,d is sampled more than once is at most 1 100 T for any t ∈

Original: [ T ] provided there is no row in any A i with h ( i ) =
Revised: [ T ] provided there is no row in any A i with h ( i ) =

Original: j whose (cid:96) 2 leverage score is at least 1 100 Td .
Revised: j whose (cid:96) 2 leverage score is at least 1 100 Td .

Original: Thus, the probability that some bucket j ∈ [ β ] is sampled twice across T steps is at most
Revised: Thus, the probability that some bucket j ∈ [ β ] is sampled twice across T steps is at most

Original: [Equation]
Revised: [Equation]

Original: In summary, we maintain T separate instances of L 1 , 2 ,d samplers for the heavy matrices and one
Revised: In summary, we maintain T separate instances of L 1 , 2 ,d samplers for the heavy matrices and one

Original: L 1 , 2 ,d sampler for each hash index that does not contain a heavy matrix.
Revised: L 1 , 2 ,d sampler for each hash index that does not contain a heavy matrix.

Original: With probability at least 98100 , any hash index not containing a heavy matrix is sampled only once, so each time t ∈ [ T ] has access to a fresh L 1 , 2 ,d sampler.
Revised: With probability at least 98100 , any hash index not containing a heavy matrix is sampled only once, so each time t ∈ [ T ] has access to a fresh L 1 , 2 ,d sampler.

Original: (cid:50)
Revised: (cid:50)

Original: B E MPIRICAL E VALUATIONS
Revised: B E MPIRICAL E VALUATIONS

Original: We again emphasize that our primary contribution is the theoretical design of a nearly input sparsity time streaming algorithm that simulates the optimal importance sampling distribution for variance reduction in stochastic gradient descent without computing the full gradient.
Revised: We again emphasize that our primary contribution is the theoretical design of a nearly input sparsity time streaming algorithm that simulates the optimal importance sampling distribution for variance reduction in stochastic gradient descent without computing the full gradient.

Original: Thus our theory is optimized to minimize the number of SGD iterations without asymptotic wall-clock time penalties; we do not attempt to further optimize wall-clock runtimes.
Revised: Thus our theory is optimized to minimize the number of SGD iterations without asymptotic wall-clock time penalties; we do not attempt to further optimize wall-clock runtimes.

Original: Nevertheless, in this section we implement a scaled-down version of our algorithm and compare its performance across multiple iterations on large-scale real world data sets to SGD with uniform sampling on both linear regression and
Revised: Nevertheless, in this section we implement a scaled-down version of our algorithm and compare its performance across multiple iterations on large-scale real world data sets to SGD with uniform sampling on both linear regression and

Original: SGDIterations
Revised: SGDIterations

Original: O b j ec ti v e
Revised: O b j ec ti v e

Original: V a l u e (
Revised: V a l u e (

Original: [Figure]
Revised: [Figure]

Original: Step size η = 1 × 10 −    400 . 1 .2 · 

Revised: Step size η = 1 × 10 −    400 . 1 .2 ·

Original: SGDIterations
Revised: SGDIterations

Original: O b j ec ti v e
Revised: O b j ec ti v e

Original: V a l u e (
Revised: V a l u e (

Original: [Figure]
Revised: [Figure]

Original: Step size η = 5 × 10 −    400 . 1 .2 · 

Revised: Step size η = 5 × 10 −    400 . 1 .2 ·

Original: SGDIterations
Revised: SGDIterations

Original: O b j ec ti v e
Revised: O b j ec ti v e

Original: V a l u e
Revised: V a l u e

Original: [Figure]
Revised: [Figure]

Original: support-vector machines (SVMs).
Revised: support-vector machines (SVMs).

Original: Because most rows have roughly uniformly small leverage scores in real-world data, we assume that no bucket contains a row with a signiﬁcantly large leverage score and thus the implementation of our importance sampling algorithm does not create multiple samplers for any buckets.
Revised: Because most rows have roughly uniformly small leverage scores in real-world data, we assume that no bucket contains a row with a signiﬁcantly large leverage score and thus the implementation of our importance sampling algorithm does not create multiple samplers for any buckets.

Original: By similar reasoning, our implementation uniformly samples a number of indices i and estimates (cid:107) Ax (cid:107) 1 , 2 ,d = (cid:80) j (cid:107)
Revised: By similar reasoning, our implementation uniformly samples a number of indices i and estimates (cid:107) Ax (cid:107) 1 , 2 ,d = (cid:80) j (cid:107)

Original: A j x (cid:107) 1 , 2 ,d by scaling up (cid:107)
Revised: A j x (cid:107) 1 , 2 ,d by scaling up (cid:107)

Original: A
Revised: A

Original: i x (cid:107) 1 , 2 ,d . Observe that although these simpliﬁcations to our algorithm decreases the wall-clock running time and the total space used by our algorithm, they only decrease the quality of our solution for each SGD iteration.
Revised: i x (cid:107) 1 , 2 ,d . Observe that although these simpliﬁcations to our algorithm decreases the wall-clock running time and the total space used by our algorithm, they only decrease the quality of our solution for each SGD iteration.

Original: Nevertheless, our implementations signiﬁcantly improve upon SGD with uniform sampling.
Revised: Nevertheless, our implementations signiﬁcantly improve upon SGD with uniform sampling.

Original: The experiments in this section were performed on a Dell Inspiron 15-7579 device with an Intel Core i7-7500U dual core processor, clocked at 2.70 GHz and 2.90 GHz, in contrast to the logistic regression experiments that were performed on a GPU.
Revised: The experiments in this section were performed on a Dell Inspiron 15-7579 device with an Intel Core i7-7500U dual core processor, clocked at 2.70 GHz and 2.90 GHz, in contrast to the logistic regression experiments that were performed on a GPU.

Original: Linear Regression.
Revised: Linear Regression.

Original: We performed linear regression on the CIFAR-10 dataset to compare the performance of our importance sampling algorithm to the uniform sampling SGD algorithm.
Revised: We performed linear regression on the CIFAR-10 dataset to compare the performance of our importance sampling algorithm to the uniform sampling SGD algorithm.

Original: We trained using a data batch of 100000 points and 3072 features and tested the performance on a separate batch of data points.
Revised: We trained using a data batch of 100000 points and 3072 features and tested the performance on a separate batch of data points.

Original: We aggregated the objective values across 10 separate instances.
Revised: We aggregated the objective values across 10 separate instances.

Original: Each instance generated a random starting location as an initial position for both importance sampling and uniform sampling.
Revised: Each instance generated a random starting location as an initial position for both importance sampling and uniform sampling.

Original: We then ran 40 iterations of SGD for each algorithm and observed the objective value on the test data for each of these iterations.
Revised: We then ran 40 iterations of SGD for each algorithm and observed the objective value on the test data for each of these iterations.

Original: Finally, we computed the average performance on each iteration across these 10 separate instances.
Revised: Finally, we computed the average performance on each iteration across these 10 separate instances.

Original: As we ran our algorithm for 40 iterations, we created 1600 buckets that partitioned the data values for the importance sampling algorithm.
Revised: As we ran our algorithm for 40 iterations, we created 1600 buckets that partitioned the data values for the importance sampling algorithm.

Original: The sampled gradients were generally large in magnitude for both importance sampling and uniform sampling and thus we required small step-size.
Revised: The sampled gradients were generally large in magnitude for both importance sampling and uniform sampling and thus we required small step-size.

Original: For step-sizes η = 1 × 10 − 13 , η = 5 × 10 − 12 , and η = 1 × 10 − 12 , the objective value of the solution output by our importance sampling algorithm quickly and signiﬁcantly improved over the objective value of the solution output by uniform sampling.
Revised: For step-sizes η = 1 × 10 − 13 , η = 5 × 10 − 12 , and η = 1 × 10 − 12 , the objective value of the solution output by our importance sampling algorithm quickly and signiﬁcantly improved over the objective value of the solution output by uniform sampling.

Original: Our algorithm performance is much more sensitive to the choice of larger step-sizes, as choices of step-sizes larger than 5 × 10 − 11 generally caused the importance sampling algorithm to diverge, while the uniform sampling algorithm still slowly converged.
Revised: Our algorithm performance is much more sensitive to the choice of larger step-sizes, as choices of step-sizes larger than 5 × 10 − 11 generally caused the importance sampling algorithm to diverge, while the uniform sampling algorithm still slowly converged.

Original: We give our results in Figure 3.
Revised: We give our results in Figure 3.

Original: Support-Vector Machines.
Revised: Support-Vector Machines.

Original: We also compared the performance of our importance sampling algorithm to the uniform sampling SGD algorithm using support-vector machines (SVM) on the a9a
Revised: We also compared the performance of our importance sampling algorithm to the uniform sampling SGD algorithm using support-vector machines (SVM) on the a9a

Original: Adult data set collected by UCI and retrieved from LibSVM (Chang & Lin, 2011).
Revised: Adult data set collected by UCI and retrieved from LibSVM (Chang & Lin, 2011).

Original: The features correspond to responses from the 1994 Census database and the prediction task is to determine whether a person makes over 50K USD a year.
Revised: The features correspond to responses from the 1994 Census database and the prediction task is to determine whether a person makes over 50K USD a year.

Original: We trained using a data batch of 32581 points and 123 features and tested the performance on a separate batch of 16281 data points.
Revised: We trained using a data batch of 32581 points and 123 features and tested the performance on a separate batch of 16281 data points.

Original: We assume the data is not linearly separable and thus use the hinge loss function so that we aim to minimizen (cid:80) ni =1 max(0 , 1 − y i ( w · X i − b ))
Revised: We assume the data is not linearly separable and thus use the hinge loss function so that we aim to minimizen (cid:80) ni =1 max(0 , 1 − y i ( w · X i − b ))

Original: + λ (cid:107) w (cid:107) 22 , where X is the data matrix,
Revised: + λ (cid:107) w (cid:107) 22 , where X is the data matrix,

Original: y i is the corresponding label, and w is the desired maximum-margin hyperplane.
Revised: y i is the corresponding label, and w is the desired maximum-margin hyperplane.

Original: For each evaluation, we generatedrandom initial positions shared for both importance sampling and uniform sampling.
Revised: For each evaluation, we generatedrandom initial positions shared for both importance sampling and uniform sampling.

Original: We then raniterations of SGD for each algorithm, creating 1125 buckets for the importance sampling algorithm and computed the average performance on each iteration across these 5 separate instances.
Revised: We then raniterations of SGD for each algorithm, creating 1125 buckets for the importance sampling algorithm and computed the average performance on each iteration across these 5 separate instances.

Original: The sampled gradients were generally smaller than those from linear regression on CIFAR-10 and thus we were able to choose signiﬁcantly larger step-sizes.
Revised: The sampled gradients were generally smaller than those from linear regression on CIFAR-10 and thus we were able to choose signiﬁcantly larger step-sizes.

Original: Nevertheless, our algorithm performance was sensitive to both the step-size and the regularization parameter.
Revised: Nevertheless, our algorithm performance was sensitive to both the step-size and the regularization parameter.

Original: For step-sizes η = 0 .
Revised: For step-sizes η = 0 .

Original: 25 , η = 0 .and
Revised: 25 , η = 0 .and

Original: regularization parameters λ = 0 , λ = 0 .
Revised: regularization parameters λ = 0 , λ = 0 .

Original: 001 and λ = 0 . 0001 , the objective value of the solution output by our importance sampling algorithm quickly and signiﬁcantly improved over the objective value of the solution output by uniform sampling.
Revised: 001 and λ = 0 . 0001 , the objective value of the solution output by our importance sampling algorithm quickly and signiﬁcantly improved over the objective value of the solution output by uniform sampling.

Original: We give our results in Figure 5.
Revised: We give our results in Figure 5.

Original: Our algorithm performance degraded with larger values of λ , as well step-sizes larger than η = 1 .
Revised: Our algorithm performance degraded with larger values of λ , as well step-sizes larger than η = 1 .

Original: We also compared step-size η = 1 and regularization parameters λ = 0 , λ = 0 .
Revised: We also compared step-size η = 1 and regularization parameters λ = 0 , λ = 0 .

Original: 001 and λ = 

Revised: 001 and λ =

Original: .with a hybrid sampling scheme that selects the better gradient between importance sampling and uniform sampling at each step, as well as a hybrid sampling scheme that uses a few steps of importance sampling, followed by uniform sampling in the remaining steps.
Revised: .with a hybrid sampling scheme that selects the better gradient between importance sampling and uniform sampling at each step, as well as a hybrid sampling scheme that uses a few steps of importance sampling, followed by uniform sampling in the remaining steps.

Original: Our experiments show that the hybrid sampling algorithms perform better at the beginning and thus our importance sampling algorithm may be used in conjunction with existing techniques in ofﬂine settings to accelerate SGD.
Revised: Our experiments show that the hybrid sampling algorithms perform better at the beginning and thus our importance sampling algorithm may be used in conjunction with existing techniques in ofﬂine settings to accelerate SGD.

Original: Surprisingly, the hybrid sampling algorithms do not necessarily remain better than our importance sampling algorithm thus indicating that even if uniform sampling were run for a signiﬁcantly larger number of iterations, its performance may not exceed our importance sampling algorithm.
Revised: Surprisingly, the hybrid sampling algorithms do not necessarily remain better than our importance sampling algorithm thus indicating that even if uniform sampling were run for a signiﬁcantly larger number of iterations, its performance may not exceed our importance sampling algorithm.

Original: We give our results in Figure 6.
Revised: We give our results in Figure 6.

Original: Finally, we compare wall-clock times of each of the aforementioned sampling schemes with step-size η = 1 and regularization 0 across 100 iterations.
Revised: Finally, we compare wall-clock times of each of the aforementioned sampling schemes with step-size η = 1 and regularization 0 across 100 iterations.

Original: Our results in Figure 4 show that as expected, uniform sampling has the fastest running time.
Revised: Our results in Figure 4 show that as expected, uniform sampling has the fastest running time.

Original: However, each iteration of importance sampling takes about 15 iterations of uniform sampling, which empirically shows that even using wall-clock times for comparison, rather than total number of SGD iterations, the performance of our importance sampling algorithm still surpasses that of uniform sampling.
Revised: However, each iteration of importance sampling takes about 15 iterations of uniform sampling, which empirically shows that even using wall-clock times for comparison, rather than total number of SGD iterations, the performance of our importance sampling algorithm still surpasses that of uniform sampling.

Original: Moreover, the runtime experiments reveal the main bottleneck of our experiments: each of the 100 iterations took approximatelyseconds on average after including the evaluation of the objective on each gradient step.
Revised: Moreover, the runtime experiments reveal the main bottleneck of our experiments: each of the 100 iterations took approximatelyseconds on average after including the evaluation of the objective on each gradient step.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]