Original: We assume that the absolute values of the inner products of vectors in v 1 , · · · , v n are at most ϵ < 1 / ( n 2 (cid:80)
Revised: We assume that the absolute values of the inner products of vectors in v 1 , · · · , v n are at most ϵ < 1 / ( n 2 (cid:80)

Original: A i ∈ b ∥
Revised: A i ∈ b ∥

Original: A i ∥ 2 ) and the absolute values of the inner products of the normalized vectors of w 1 , · · · , w k are at most β
Revised: A i ∥ 2 ) and the absolute values of the inner products of the normalized vectors of w 1 , · · · , w k are at most β

Original: = O ( n − 3 k − 32 ) .
Revised: = O ( n − 3 k − 32 ) .

Original: Suppose that bin b contains the row A k + r .
Revised: Suppose that bin b contains the row A k + r .

Original: Then, the squared projection of A k + r onto the direction of w orthogonal to span ( { w 1 , · · · , w k } \ { w } ) is at most ∥
Revised: Then, the squared projection of A k + r onto the direction of w orthogonal to span ( { w 1 , · · · , w k } \ { w } ) is at most ∥

Original: A k + r ∥ 42 ∥ w ∥+ O ( n − 2 ) and is
Revised: A k + r ∥ 42 ∥ w ∥+ O ( n − 2 ) and is

Original: at least ∥ A k + r ∥ 42 ∥ w ∥− O ( n − 2 ) .
Revised: at least ∥ A k + r ∥ 42 ∥ w ∥− O ( n − 2 ) .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, we assume that A k + r is mapped to b k ; w = w k .
Revised: Without loss of generality, we assume that A k + r is mapped to b k ; w = w k .

Original: First, we provide an upper and a lower bound for |⟨ v k + r , w k ⟩| where for each i ≤ k , we let w i = w i ∥ w i ∥ 2 denote the normalized vector of w i .
Revised: First, we provide an upper and a lower bound for |⟨ v k + r , w k ⟩| where for each i ≤ k , we let w i = w i ∥ w i ∥ 2 denote the normalized vector of w i .

Original: Recall that by definition v k + r =
Revised: Recall that by definition v k + r =

Original: A k + r ∥
Revised: A k + r ∥

Original: A k + r ∥ 2 .
Revised: A k + r ∥ 2 .

Original: |⟨ w
Revised: |⟨ w

Original: [Equation]
Revised: [Equation]

Original: Now, let { e 1 , · · · , e k } be an orthonormal basis for the subspace spanned by { w 1 , · · · , w k } guaranteed by Lemma F.5.
Revised: Now, let { e 1 , · · · , e k } be an orthonormal basis for the subspace spanned by { w 1 , · · · , w k } guaranteed by Lemma F.5.

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we expand the orthonormal basis to include e k +1 so that we can write v k + r = (cid:80)
Revised: we expand the orthonormal basis to include e k +1 so that we can write v k + r = (cid:80)

Original: k +1 j =1 b
Revised: k +1 j =1 b

Original: j e j .
Revised: j e j .

Original: By a similar approach to the proof of Lemma F.5, we can show that for each j ≤ k − 1 , b 2 j ≤ β 2 j 2 .
Revised: By a similar approach to the proof of Lemma F.5, we can show that for each j ≤ k − 1 , b 2 j ≤ β 2 j 2 .

Original: Moreover,
Revised: Moreover,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ▷ similar to the proof of Lemma F.
Revised: ▷ similar to the proof of Lemma F.

Original: [Equation]
Revised: [Equation]

Original: and, | b k | ≥
Revised: and, | b k | ≥

Original: [Equation]
Revised: [Equation]

Original: ⟨ w k ,
Revised: ⟨ w k ,

Original: [Table]
Revised: [Table]

Original: ⟩| − k
Revised: ⟩| − k

Original: [Table]
Revised: [Table]

Original: j
Revised: j

Original: [Table]
Revised: [Table]

Original: |
Revised: |

Original: [Table]
Revised: [Table]

Original: w
Revised: w

Original: [Table]
Revised: [Table]

Original: , e
Revised: , e

Original: [Table]
Revised: [Table]

Original: | ) ≥
Revised: | ) ≥

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Table]
Revised: [Table]

Original: j = β 2 · j▷ since |⟨ w k , e k ⟩| ≤
Revised: j = β 2 · j▷ since |⟨ w k , e k ⟩| ≤

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: j =

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ▷ by (F.5)
Revised: ▷ by (F.5)

Original: [Equation]
Revised: [Equation]

Original: Hence, the squared projection of A k + r onto e k is at most ∥
Revised: Hence, the squared projection of A k + r onto e k is at most ∥

Original: A k + r ∥ 42 ∥ w k
Revised: A k + r ∥ 42 ∥ w k

Original: ∥
Revised: ∥

Original: + O ( n − 2 ) and is at least ∥
Revised: + O ( n − 2 ) and is at least ∥

Original: A k + r ∥ 42 ∥ w k ∥ 22 −
Revised: A k + r ∥ 42 ∥ w k ∥ 22 −

Original: O ( n − 2 ) .
Revised: O ( n − 2 ) .

Original: Now, we show that at the end of the algorithm no light row will be assigned to the bins that contain heavy rows.
Revised: Now, we show that at the end of the algorithm no light row will be assigned to the bins that contain heavy rows.

Original: Lemma F.12.
Revised: Lemma F.12.

Original: We assume that the absolute values of the inner products of vectors in v 1 , · · · , v n are at most ϵ < min { n − 2 k − 53 , ( n (cid:80)
Revised: We assume that the absolute values of the inner products of vectors in v 1 , · · · , v n are at most ϵ < min { n − 2 k − 53 , ( n (cid:80)

Original: A i ∈ w ∥
Revised: A i ∈ w ∥

Original: A i ∥ 2 ) − 1 } .
Revised: A i ∥ 2 ) − 1 } .

Original: At iteration k + r , the greedy algorithm will assign the light row
Revised: At iteration k + r , the greedy algorithm will assign the light row

Original: A k + r to a bin that does not contain a heavy row.
Revised: A k + r to a bin that does not contain a heavy row.

Original: Proof.
Revised: Proof.

Original: The proof is by induction.
Revised: The proof is by induction.

Original: Lemma F.8 implies that no light row has been mapped to a bin that contains a heavy row for the first k iterations.
Revised: Lemma F.8 implies that no light row has been mapped to a bin that contains a heavy row for the first k iterations.

Original: Next, we assume that this holds for the first k + r −iterations and show that is also must hold for the ( k + r ) -th iteration.
Revised: Next, we assume that this holds for the first k + r −iterations and show that is also must hold for the ( k + r ) -th iteration.

Original: To this end, we compare the sum of squared projection coefficients when A k + r avoids and collides with a heavy row.
Revised: To this end, we compare the sum of squared projection coefficients when A k + r avoids and collides with a heavy row.

Original: First, we upper bound β = max i ̸ =
Revised: First, we upper bound β = max i ̸ =

Original: j ≤ k |⟨ w i , w j ⟩| / ( ∥ w i ∥ 

Revised: j ≤ k |⟨ w i , w j ⟩| / ( ∥ w i ∥

Original: ∥ w j ∥ 2 ) .
Revised: ∥ w j ∥ 2 ) .

Original: Let c i and c j respectively denote the number of rows assigned to b i and b j .
Revised: Let c i and c j respectively denote the number of rows assigned to b i and b j .

Original: [Equation]
Revised: [Equation]

Original: If A k + r is assigned to a bin that contains c light rows and no heavy rows.
Revised: If A k + r is assigned to a bin that contains c light rows and no heavy rows.

Original: In this case, the projection loss of the heavy rows A 1 , · · · , A s onto row ( SA ) remains zero.
Revised: In this case, the projection loss of the heavy rows A 1 , · · · , A s onto row ( SA ) remains zero.

Original: Thus, we only need to bound the change in the sum of squared projection coefficients of the light rows before and after iteration k + r .
Revised: Thus, we only need to bound the change in the sum of squared projection coefficients of the light rows before and after iteration k + r .

Original: Without loss of generality, let w k denote the bin that contains A k + r .
Revised: Without loss of generality, let w k denote the bin that contains A k + r .

Original: Since
Revised: Since

Original: S k − 1 = span ( { w 1 , · · · , w k − 1 } ) has not changed, we only need to bound the difference in cost between projecting onto the component of w k − A k + r orthogonal to S k − 1 and the component of w k orthogonal to S k − 1 , respectively denoted as e k and e k .
Revised: S k − 1 = span ( { w 1 , · · · , w k − 1 } ) has not changed, we only need to bound the difference in cost between projecting onto the component of w k − A k + r orthogonal to S k − 1 and the component of w k orthogonal to S k − 1 , respectively denoted as e k and e k .

Original: By Claim
Revised: By Claim

Original: F
Revised: F

Original: , for the light rows that are not
Revised: , for the light rows that are not

Original: yet processed i.e., A j for j > k + r )
Revised: yet processed i.e., A j for j > k + r )

Original: , the squared projection of each onto e k is at most β 2 k 2 .
Revised: , the squared projection of each onto e k is at most β 2 k 2 .

Original: Hence, the total decrease in the squared
Revised: Hence, the total decrease in the squared

Original: projection
Revised: projection

Original: is at most ( n − k − r ) · β 2 k 2 .
Revised: is at most ( n − k − r ) · β 2 k 2 .

Original: By Claim F.10, for the processed light rows that are not mapped to the last bin, the squared projection of each onto e k is at most 4 β 2 k 2 .
Revised: By Claim F.10, for the processed light rows that are not mapped to the last bin, the squared projection of each onto e k is at most 4 β 2 k 2 .

Original: Hence, the total decrease in the squared projection cost is at most ( r − 1) · 4 β 2 k 2 .
Revised: Hence, the total decrease in the squared projection cost is at most ( r − 1) · 4 β 2 k 2 .

Original: For
Revised: For

Original: each row
Revised: each row

Original: A i ̸
Revised: A i ̸

Original: =
Revised: =

Original: A k + r that is mapped to the last bin, by Claim F.11 and the fact ∥
Revised: A k + r that is mapped to the last bin, by Claim F.11 and the fact ∥

Original: A i ∥ 42 = ∥
Revised: A i ∥ 42 = ∥

Original: A i ∥ 22 = 1 , the squared projection of A i onto e k is at most ∥
Revised: A i ∥ 22 = 1 , the squared projection of A i onto e k is at most ∥

Original: A i ∥ 

Revised: A i ∥

Original: ∥ w k − A k + r ∥ 22 + O ( n − 2 ) and the squared projection of A i onto e k is at least ∥
Revised: ∥ w k − A k + r ∥ 22 + O ( n − 2 ) and the squared projection of A i onto e k is at least ∥

Original: A i ∥ 

Revised: A i ∥

Original: ∥ w k ∥ 22 − O ( n − 2 ) .
Revised: ∥ w k ∥ 22 − O ( n − 2 ) .

Original: Moreover, the squared projection of A k + r onto e k compared to e k increases by at least ( ∥ A k + r ∥ 22 ∥ w k ∥ 22 −
Revised: Moreover, the squared projection of A k + r onto e k compared to e k increases by at least ( ∥ A k + r ∥ 22 ∥ w k ∥ 22 −

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: Hence, the total squared projection of the rows in the bin b k decreases by at least:
Revised: Hence, the total squared projection of the rows in the bin b k decreases by at least:

Original: [Equation]
Revised: [Equation]

Original: Observation
Revised: Observation

Original: [Equation]
Revised: [Equation]

Original: Hence, summing up the bounds in items 1 to 3 above, the total decrease in the sum of squared projection coefficients is at most O ( n − 1 ) .
Revised: Hence, summing up the bounds in items 1 to 3 above, the total decrease in the sum of squared projection coefficients is at most O ( n − 1 ) .

Original: If A k + r is assigned to a bin that contains a heavy row.
Revised: If A k + r is assigned to a bin that contains a heavy row.

Original: Without loss of generality, we can assume that A k
Revised: Without loss of generality, we can assume that A k

Original: + r is mapped to b k that contains the heavy row A s .
Revised: + r is mapped to b k that contains the heavy row A s .

Original: In this case, the distance of heavy rows A 1 , · · · , A s − 1 onto the space spanned by the rows of SA is zero.
Revised: In this case, the distance of heavy rows A 1 , · · · , A s − 1 onto the space spanned by the rows of SA is zero.

Original: Next, we bound the amount of change in the squared distance of A s and light rows onto the space spanned by the rows of SA .
Revised: Next, we bound the amount of change in the squared distance of A s and light rows onto the space spanned by the rows of SA .

Original: Note that the ( k − 1) -dimensional space corresponding to w 1 , · · · , w k − 1 has not changed.
Revised: Note that the ( k − 1) -dimensional space corresponding to w 1 , · · · , w k − 1 has not changed.

Original: Hence, we only need to bound the decrease in the projection distance of A k + r onto e k compared to e k (where e k , e k are defined similarly as in the last part).
Revised: Hence, we only need to bound the decrease in the projection distance of A k + r onto e k compared to e k (where e k , e k are defined similarly as in the last part).

Original: For the light rows other than A k + r , the squared projection of each onto e k is at most β 2 k 2 .
Revised: For the light rows other than A k + r , the squared projection of each onto e k is at most β 2 k 2 .

Original: Hence, the total increase in the squared projection of light rows onto e k is at most ( n − k ) · β 2 k 2 =
Revised: Hence, the total increase in the squared projection of light rows onto e k is at most ( n − k ) · β 2 k 2 =

Original: O ( n − 1 ) .
Revised: O ( n − 1 ) .

Original: [Equation]
Revised: [Equation]

Original: Hence, in this case
Revised: Hence, in this case

Original: [Equation]
Revised: [Equation]

Original: the total decrease in the squared projection is at least
Revised: the total decrease in the squared projection is at least

Original: [Equation]
Revised: [Equation]

Original: Thus, for a sufficiently large value of ℓ , the greedy algorithm will assign A k + r to a bin that only contains light rows.
Revised: Thus, for a sufficiently large value of ℓ , the greedy algorithm will assign A k + r to a bin that only contains light rows.

Original: This completes the inductive proof and in particular implies that at the end of the algorithm, heavy rows are assigned to isolated bins.
Revised: This completes the inductive proof and in particular implies that at the end of the algorithm, heavy rows are assigned to isolated bins.

Original: Corollary F.13.
Revised: Corollary F.13.

Original: The approximation loss of the best rank- k approximate solution in the rowspace
Revised: The approximation loss of the best rank- k approximate solution in the rowspace

Original: S g A for A ∼ A sp ( s, ℓ ) , where A ∈ R n × d for d = Ω( n 4 k 4 log n ) and S g is the CountSketch constructed by the greedy algorithm with non-increasing order, is at most n − s .
Revised: S g A for A ∼ A sp ( s, ℓ ) , where A ∈ R n × d for d = Ω( n 4 k 4 log n ) and S g is the CountSketch constructed by the greedy algorithm with non-increasing order, is at most n − s .

Original: Proof.
Revised: Proof.

Original: First, we need to show that the absolute values of the inner products of vectors in v 1 , · · · , v n are at most ϵ < min { n − 2 k − 2 , ( n (cid:80)
Revised: First, we need to show that the absolute values of the inner products of vectors in v 1 , · · · , v n are at most ϵ < min { n − 2 k − 2 , ( n (cid:80)

Original: A i ∈ w ∥
Revised: A i ∈ w ∥

Original: A i ∥ 2 ) − 1 } so that we can apply Lemma F.12.
Revised: A i ∥ 2 ) − 1 } so that we can apply Lemma F.12.

Original: To show this, note that by Observation F.3, ϵ ≤ 2 (cid:113) log nd ≤ n − 2 k − 2 since d = Ω( n 4 k 4 log n ) .
Revised: To show this, note that by Observation F.3, ϵ ≤ 2 (cid:113) log nd ≤ n − 2 k − 2 since d = Ω( n 4 k 4 log n ) .

Original: The proof follows from Lemma F.8 and Lemma F.12.
Revised: The proof follows from Lemma F.8 and Lemma F.12.

Original: Since all heavy rows are mapped to isolated bins, the projection loss of the light rows is at most n − s .
Revised: Since all heavy rows are mapped to isolated bins, the projection loss of the light rows is at most n − s .

Original: Next, we bound the Frobenius norm error of the best rank- k -approximation solution constructed by the standard CountSketch with a randomly chosen sparsity pattern.
Revised: Next, we bound the Frobenius norm error of the best rank- k -approximation solution constructed by the standard CountSketch with a randomly chosen sparsity pattern.

Original: Lemma F.14.
Revised: Lemma F.14.

Original: Let s = αk where 0 .
Revised: Let s = αk where 0 .

Original: 7 < α < 1 .
Revised: 7 < α < 1 .

Original: The expected squared loss of the best rank- k approximate solution in the rowspace S r A for A ∈ R n × d ∼ A sp ( s, ℓ ) , where d = Ω( n 6 ℓ 2 ) and S r is the sparsity pattern of CountSketch is chosen uniformly at random, is at least n + ℓk 4 e − (1 + α ) k − n −O (1) .
Revised: The expected squared loss of the best rank- k approximate solution in the rowspace S r A for A ∈ R n × d ∼ A sp ( s, ℓ ) , where d = Ω( n 6 ℓ 2 ) and S r is the sparsity pattern of CountSketch is chosen uniformly at random, is at least n + ℓk 4 e − (1 + α ) k − n −O (1) .

Original: Proof.
Revised: Proof.

Original: We can interpret the randomized construction of the CountSketch as a “balls and bins” experiment.
Revised: We can interpret the randomized construction of the CountSketch as a “balls and bins” experiment.

Original: In particular, considering the heavy rows, we compute the expected number of bins (i.e., rows in S r A ) that contain a heavy row.
Revised: In particular, considering the heavy rows, we compute the expected number of bins (i.e., rows in S r A ) that contain a heavy row.

Original: Note that the expected number of rows in S r A that do not contain any heavy row is k · (1 − 1 k ) s ≥ k · e − sk − 1 .
Revised: Note that the expected number of rows in S r A that do not contain any heavy row is k · (1 − 1 k ) s ≥ k · e − sk − 1 .

Original: Hence, the number of rows in S r A that contain a heavy row of A is at most k (1 − e − sk − 1 ) .
Revised: Hence, the number of rows in S r A that contain a heavy row of A is at most k (1 − e − sk − 1 ) .

Original: Thus, at least s − k (1 − e − sk − 1 ) heavy rows are not mapped to an isolated bin (i.e., they collide with some other heavy rows).
Revised: Thus, at least s − k (1 − e − sk − 1 ) heavy rows are not mapped to an isolated bin (i.e., they collide with some other heavy rows).

Original: Then, it is straightforward to show that the squared loss of each such row is at least ℓ − n −O (1) .
Revised: Then, it is straightforward to show that the squared loss of each such row is at least ℓ − n −O (1) .

Original: Claim F.15 .
Revised: Claim F.15 .

Original: Suppose that heavy rows A r 1 , · · · , A r c are mapped to the same bin via a CountSketch
Revised: Suppose that heavy rows A r 1 , · · · , A r c are mapped to the same bin via a CountSketch

Original: S .
Revised: S .

Original: Then, the total squared distances of these rows from the subspace spanned by SA is at least ( c − 1) ℓ − O ( n − 1 ) .
Revised: Then, the total squared distances of these rows from the subspace spanned by SA is at least ( c − 1) ℓ − O ( n − 1 ) .

Original: Proof.
Revised: Proof.

Original: Let b denote the bin that contains the rows A r 1 , · · · , A r c and suppose that it has c ′ light rows as well.
Revised: Let b denote the bin that contains the rows A r 1 , · · · , A r c and suppose that it has c ′ light rows as well.

Original: Note that by Claim F.10 and Claim F.11, the squared projection of each row
Revised: Note that by Claim F.10 and Claim F.11, the squared projection of each row

Original: A r i onto the subspace spanned by the k bins is at most
Revised: A r i onto the subspace spanned by the k bins is at most

Original: [Equation]
Revised: [Equation]

Original: Hence, the total squared loss of these c heavy rows is at least cℓ − c · ( ℓc + O ( n − 1 )) ≥ ( c − 1) ℓ −
Revised: Hence, the total squared loss of these c heavy rows is at least cℓ − c · ( ℓc + O ( n − 1 )) ≥ ( c − 1) ℓ −

Original: O ( n − 1 ) .
Revised: O ( n − 1 ) .

Original: Thus, the expected total squared loss of the heavy rows is at least:
Revised: Thus, the expected total squared loss of the heavy rows is at least:

Original: [Equation]
Revised: [Equation]

Original: ▷ s =
Revised: ▷ s =

Original: α · ( k − 1) where 0 .
Revised: α · ( k − 1) where 0 .

Original: 7 < α <
Revised: 7 < α <

Original: [Equation]
Revised: [Equation]

Original: Next, we compute a lower bound on the expected squared loss of the light rows.
Revised: Next, we compute a lower bound on the expected squared loss of the light rows.

Original: Note that Claim F. and Claim F.11 imply that when a light row collides with other rows, its contribution to the total squared loss (note that the loss accounts for the amount it decreases from the squared projection of the other rows in the bin as well) is at least 1 − O ( n − 1 ) .
Revised: Note that Claim F. and Claim F.11 imply that when a light row collides with other rows, its contribution to the total squared loss (note that the loss accounts for the amount it decreases from the squared projection of the other rows in the bin as well) is at least 1 − O ( n − 1 ) .

Original: Hence, the expected total squared loss of the light rows is at least:
Revised: Hence, the expected total squared loss of the light rows is at least:

Original: [Equation]
Revised: [Equation]

Original: Hence, the expected squared loss of a CountSketch whose sparsity is picked at random is at least
Revised: Hence, the expected squared loss of a CountSketch whose sparsity is picked at random is at least

Original: [Equation]
Revised: [Equation]

Original: Corollary F.16.
Revised: Corollary F.16.

Original: Let s
Revised: Let s

Original: = α ( k − 1)
Revised: = α ( k − 1)

Original: where 0 .
Revised: where 0 .

Original: 7 < α < 1 and let ℓ ≥ (4 e +1) n αk .
Revised: 7 < α < 1 and let ℓ ≥ (4 e +1) n αk .

Original: Let S g be the
Revised: Let S g be the

Original: CountSketch whose sparsity pattern is learned over a training set drawn from A sp via the greedy approach.
Revised: CountSketch whose sparsity pattern is learned over a training set drawn from A sp via the greedy approach.

Original: Let S r be a CountSketch whose sparsity pattern is picked uniformly at random.
Revised: Let S r be a CountSketch whose sparsity pattern is picked uniformly at random.

Original: Then, for an n × d matrix A ∼ A sp where d = Ω( n 6 ℓ 2 ) , the expected loss of the best rank- k approximation of A returned by S r is worse than the approximation loss of the best rank- k approximation of A returned by S g by at least a constant factor.
Revised: Then, for an n × d matrix A ∼ A sp where d = Ω( n 6 ℓ 2 ) , the expected loss of the best rank- k approximation of A returned by S r is worse than the approximation loss of the best rank- k approximation of A returned by S g by at least a constant factor.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: ▷ Lemma F.
Revised: ▷ Lemma F.

Original: [Equation]
Revised: [Equation]

Original: Corollary F.
Revised: Corollary F.

Original: F. Z IPFIAN ON SQUARED ROW NORMS .
Revised: F. Z IPFIAN ON SQUARED ROW NORMS .

Original: Each matrix A ∈ R n × d ∼ A zipf has rows which are uniformly random and orthogonal.
Revised: Each matrix A ∈ R n × d ∼ A zipf has rows which are uniformly random and orthogonal.

Original: Each A has 2 i +1 rows of squared norm n 2 / 2 2 i for i ∈ [1 , . . .
Revised: Each A has 2 i +1 rows of squared norm n 2 / 2 2 i for i ∈ [1 , . . .

Original: , O (log( n ))] .
Revised: , O (log( n ))] .

Original: We also assume that each row has the same squared norm for all members of A zipf .
Revised: We also assume that each row has the same squared norm for all members of A zipf .

Original: In this section, the s rows with largest norm are called the heavy rows and the remaining are the light rows.
Revised: In this section, the s rows with largest norm are called the heavy rows and the remaining are the light rows.

Original: For convenience, we number the heavy rows 1 , . . .
Revised: For convenience, we number the heavy rows 1 , . . .

Original: , s ; however, the heavy rows can appear at any indices, as long as any row of a given index has the same norm for all members of A zipf .
Revised: , s ; however, the heavy rows can appear at any indices, as long as any row of a given index has the same norm for all members of A zipf .

Original: Also, we assume that s ≤ k/ 2 and, for simplicity, s = (cid:80) h s i
Revised: Also, we assume that s ≤ k/ 2 and, for simplicity, s = (cid:80) h s i

Original: =
Revised: =

Original: 1 2 i +1 for some h s ∈ Z + .
Revised: 1 2 i +1 for some h s ∈ Z + .

Original: That means the minimum squared norm of a heavy row is n 2 / 2 2 h s and the maximum squared norm of a light row is n 2 / 2 2 h s +2 .
Revised: That means the minimum squared norm of a heavy row is n 2 / 2 2 h s and the maximum squared norm of a light row is n 2 / 2 2 h s +2 .

Original: The analysis of the greedy algorithm ordered by non-increasing row norms on this family of matrices is similar to our analysis for the spiked covariance model.
Revised: The analysis of the greedy algorithm ordered by non-increasing row norms on this family of matrices is similar to our analysis for the spiked covariance model.

Original: Here we analyze the case in which rows are orthogonal.
Revised: Here we analyze the case in which rows are orthogonal.

Original: By continuity, if the rows are close enough to being orthogonal, all decisions made by the greedy algorithm will be the same.
Revised: By continuity, if the rows are close enough to being orthogonal, all decisions made by the greedy algorithm will be the same.

Original: As a first step, by Lemma F.8, at the end of iteration k the first k rows are assigned to different bins.
Revised: As a first step, by Lemma F.8, at the end of iteration k the first k rows are assigned to different bins.

Original: Then, via a similar inductive proof, we show that none of the light rows are mapped to a bin that contains one of the top s heavy rows.
Revised: Then, via a similar inductive proof, we show that none of the light rows are mapped to a bin that contains one of the top s heavy rows.

Original: Lemma F.17.
Revised: Lemma F.17.

Original: At each iteration k + r , the greedy algorithm picks the position of the non-zero value in the ( k + r ) -th column of the CountSketch matrix S so that the light row A k + r is mapped to a bin that does not contain any of top s heavy rows.
Revised: At each iteration k + r , the greedy algorithm picks the position of the non-zero value in the ( k + r ) -th column of the CountSketch matrix S so that the light row A k + r is mapped to a bin that does not contain any of top s heavy rows.

Original: Proof.
Revised: Proof.

Original: We prove the statement by induction.
Revised: We prove the statement by induction.

Original: The base case r = 0 trivially holds as the first k rows are assigned to distinct bins.
Revised: The base case r = 0 trivially holds as the first k rows are assigned to distinct bins.

Original: Next we assume that in none of the first k + r − 1 iterations a light row is assigned to a bin that contains a heavy row.
Revised: Next we assume that in none of the first k + r − 1 iterations a light row is assigned to a bin that contains a heavy row.

Original: Now we consider the following cases:
Revised: Now we consider the following cases:

Original: If A k + r is assigned to a bin that only contains light rows.
Revised: If A k + r is assigned to a bin that only contains light rows.

Original: Without loss of generality we can assume that A k + r is assigned to b k .
Revised: Without loss of generality we can assume that A k + r is assigned to b k .

Original: Since the vectors are orthogonal, we only need to bound the difference in the projection of A k + r and the light rows that are assigned to b k onto the direction of w k before and after adding A k + r to b k .
Revised: Since the vectors are orthogonal, we only need to bound the difference in the projection of A k + r and the light rows that are assigned to b k onto the direction of w k before and after adding A k + r to b k .

Original: In this case, the total squared loss corresponding to rows in b k and A k + r before and after adding A k +1 are respectively before adding A k + r to b k :
Revised: In this case, the total squared loss corresponding to rows in b k and A k + r before and after adding A k +1 are respectively before adding A k + r to b k :

Original: ∥
Revised: ∥

Original: A k + r ∥ 22 + (cid:88)
Revised: A k + r ∥ 22 + (cid:88)

Original: [Equation]
Revised: [Equation]

Original: after adding A k + r to b k : ∥
Revised: after adding A k + r to b k : ∥

Original: A k + r ∥ 22 + (cid:88)
Revised: A k + r ∥ 22 + (cid:88)

Original: [Equation]
Revised: [Equation]

Original: b k
Revised: b k

Original: [Equation]
Revised: [Equation]

Original: Thus, the amount of increase in the squared loss is
Revised: Thus, the amount of increase in the squared loss is

Original: [Equation]
Revised: [Equation]

Original: If A k + r is assigned to a bin that contains a heavy row.
Revised: If A k + r is assigned to a bin that contains a heavy row.

Original: Without loss of generality and by the induction hypothesis, we assume that A k + r is assigned to a bin b that only contains a heavy row A j .
Revised: Without loss of generality and by the induction hypothesis, we assume that A k + r is assigned to a bin b that only contains a heavy row A j .

Original: Since the rows are orthogonal, we only need to bound the difference in the projection of A k + r and
Revised: Since the rows are orthogonal, we only need to bound the difference in the projection of A k + r and

Original: A j
Revised: A j

Original: In this case, the total squared loss corresponding to A j and A k + r before and after adding A k + to b are respectively before adding A k + r to b k :
Revised: In this case, the total squared loss corresponding to A j and A k + r before and after adding A k + to b are respectively before adding A k + r to b k :

Original: ∥
Revised: ∥

Original: A k + r ∥after adding A k + r to b k : ∥
Revised: A k + r ∥after adding A k + r to b k : ∥

Original: A k + r ∥ 22 + ∥
Revised: A k + r ∥ 22 + ∥

Original: A j ∥ 22 − ( ∥ A k + r ∥ 42 + ∥
Revised: A j ∥ 22 − ( ∥ A k + r ∥ 42 + ∥

Original: A j ∥∥ A k + r ∥ 22 + ∥
Revised: A j ∥∥ A k + r ∥ 22 + ∥

Original: [Equation]
Revised: [Equation]

Original: Thus, the amount of increase in the squared loss is
Revised: Thus, the amount of increase in the squared loss is

Original: [Equation]
Revised: [Equation]

Original: Then (F.7) is larger than (F.6) if ∥ A j ∥ 22 ≥ (cid:80)
Revised: Then (F.7) is larger than (F.6) if ∥ A j ∥ 22 ≥ (cid:80)

Original: A
Revised: A

Original: i ∈
Revised: i ∈

Original: b k ∥
Revised: b k ∥

Original: A i ∥ 22 .
Revised: A i ∥ 22 .

Original: Next, we show that at every inductive iteration, there exists a bin b which only contains light rows and whose squared norm is smaller than the squared norm of any heavy row.
Revised: Next, we show that at every inductive iteration, there exists a bin b which only contains light rows and whose squared norm is smaller than the squared norm of any heavy row.

Original: For each value m , define h m so that m =
Revised: For each value m , define h m so that m =

Original: (cid:80)
Revised: (cid:80)

Original: h m i =1 

Revised: h m i =1

Original: i +

Revised: i +

Original: = 2 h m +2 − 2 .
Revised: = 2 h m +2 − 2 .

Original: Recall that all heavy rows have squared norm at least n 2 2 2 hs .
Revised: Recall that all heavy rows have squared norm at least n 2 2 2 hs .

Original: There must be a bin b that only contains light rows and has squared norm at most ∥
Revised: There must be a bin b that only contains light rows and has squared norm at most ∥

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: ▷
Revised: ▷

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: 2 and k
Revised: 2 and k

Original: [Equation]
Revised: [Equation]

Original: 2 h k +
Revised: 2 h k +

Original: [Equation]
Revised: [Equation]

Original: Hence, the greedy algorithm will map A k + r to a bin that only contains light rows.
Revised: Hence, the greedy algorithm will map A k + r to a bin that only contains light rows.

Original: Corollary F.18.
Revised: Corollary F.18.

Original: The squared loss of the best rank- k approximate solution in the rowspace of S g A for A ∈ R n × d ∼ A zipf where A ∈ R n × d and S g is the CountSketch constructed by the greedy algorithm with non-increasing order, is <
Revised: The squared loss of the best rank- k approximate solution in the rowspace of S g A for A ∈ R n × d ∼ A zipf where A ∈ R n × d and S g is the CountSketch constructed by the greedy algorithm with non-increasing order, is <

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: At the end of iteration k , the total squared loss is (cid:80) h n i = h
Revised: At the end of iteration k , the total squared loss is (cid:80) h n i = h

Original: k
Revised: k

Original: +1 

Revised: +1

Original: i +1 · n 2 2 2 i .
Revised: i +1 · n 2 2 2 i .

Original: After that, in each iteration k + r , by (F.6), the squared loss increases by at most ∥
Revised: After that, in each iteration k + r , by (F.6), the squared loss increases by at most ∥

Original: A k + r ∥ 22 .
Revised: A k + r ∥ 22 .

Original: Hence, the total squared loss in the solution returned by S g is at most
Revised: Hence, the total squared loss in the solution returned by S g is at most

Original: [Equation]
Revised: [Equation]

Original: Next, we bound the squared loss of the best rank- k -approximate solution constructed by the standard
Revised: Next, we bound the squared loss of the best rank- k -approximate solution constructed by the standard

Original: CountSketch with a randomly chosen sparsity pattern.
Revised: CountSketch with a randomly chosen sparsity pattern.

Original: Observation F.19 .
Revised: Observation F.19 .

Original: Let us assume that the orthogonal rows A r 1 , · · · , A r c are mapped to the same bin and for each i ≤ c , ∥
Revised: Let us assume that the orthogonal rows A r 1 , · · · , A r c are mapped to the same bin and for each i ≤ c , ∥

Original: A r 1 ∥ 22 ≥ ∥
Revised: A r 1 ∥ 22 ≥ ∥

Original: A r i ∥ 22 .
Revised: A r i ∥ 22 .

Original: Then, the total squared loss of A r 1 , · · · , A r c after projecting onto A r 1 ± · · · ± A r c is
Revised: Then, the total squared loss of A r 1 , · · · , A r c after projecting onto A r 1 ± · · · ± A r c is

Original: at least ∥
Revised: at least ∥

Original: A r 2 ∥ 22 + · · · + ∥ A r c ∥ 22 .
Revised: A r 2 ∥ 22 + · · · + ∥ A r c ∥ 22 .

Original: Proof.
Revised: Proof.

Original: Note that since A r 1 , · · · , A r c are orthogonal, for each i ≤ c , the squared projection of A r i onto A r 1 ± · · · ±
Revised: Note that since A r 1 , · · · , A r c are orthogonal, for each i ≤ c , the squared projection of A r i onto A r 1 ± · · · ±

Original: A r c is ∥
Revised: A r c is ∥

Original: A r i ∥ 42 / (cid:80) cj
Revised: A r i ∥ 42 / (cid:80) cj

Original: =1 (cid:13)(cid:13)
Revised: =1 (cid:13)(cid:13)

Original: A r j (cid:13)(cid:13) 2 2 .
Revised: A r j (cid:13)(cid:13) 2 2 .

Original: Hence, the sum of squared projection coefficients of A r 1 , · · · , A r c onto A r 1 ± · · · ± A r c is
Revised: Hence, the sum of squared projection coefficients of A r 1 , · · · , A r c onto A r 1 ± · · · ± A r c is

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∥

Original: Hence, the total projection loss of A r 1 , · · · , A r c onto A r 1 ± · · · ± A r c is at least
Revised: Hence, the total projection loss of A r 1 , · · · , A r c onto A r 1 ± · · · ± A r c is at least

Original: [Equation]
Revised: [Equation]

Original: In particular, Observation F.19 implies that whenever two rows are mapped into the same bin, the squared norm of the row with smaller norm fully contributes to the total squared loss of the solution.
Revised: In particular, Observation F.19 implies that whenever two rows are mapped into the same bin, the squared norm of the row with smaller norm fully contributes to the total squared loss of the solution.

Original: Lemma F.20.
Revised: Lemma F.20.

Original: For k > 2 10 − 2 , the expected squared loss of the best rank- k approximate solution in the rowspace of S r A for A n × d ∼ A zipf , where S r is the sparsity pattern of a CountSketch chosen uniformly at random, is at least 1 .
Revised: For k > 2 10 − 2 , the expected squared loss of the best rank- k approximate solution in the rowspace of S r A for A n × d ∼ A zipf , where S r is the sparsity pattern of a CountSketch chosen uniformly at random, is at least 1 .

Original: 095 n 2 2 hk − 2 .
Revised: 095 n 2 2 hk − 2 .

Original: Proof.
Revised: Proof.

Original: In light of Observation F.19, we need to compute the expected number of collisions between rows with “large” norm.
Revised: In light of Observation F.19, we need to compute the expected number of collisions between rows with “large” norm.

Original: We can interpret the randomized construction of the CountSketch as a “balls and bins” experiment.
Revised: We can interpret the randomized construction of the CountSketch as a “balls and bins” experiment.

Original: For each 0 ≤ j ≤ h k , let A j denote the set of rows with squared norm n 2 

Revised: For each 0 ≤ j ≤ h k , let A j denote the set of rows with squared norm n 2

Original: [Equation]
Revised: [Equation]

Original: ) and let A >j = (cid:83) j<i ≤ h k A i .
Revised: ) and let A >j = (cid:83) j<i ≤ h k A i .

Original: Note that for each j , |A
Revised: Note that for each j , |A

Original: j | = 2 h k − j +1 and |A >j | = (cid:80)
Revised: j | = 2 h k − j +1 and |A >j | = (cid:80)

Original: h
Revised: h

Original: k
Revised: k

Original: i
Revised: i

Original: =
Revised: =

Original: j +1 

Revised: j +1

Original: h k − i +1 =
Revised: h k − i +1 =

Original: (cid:80)
Revised: (cid:80)

Original: h k − j i =1 2 i = 2(

Revised: h k − j i =1 2 i = 2(

Original: h k − j − 1) .
Revised: h k − j − 1) .

Original: Moreover, note that k = 2(

Revised: Moreover, note that k = 2(

Original: h k
Revised: h k

Original: +1 − 1) .
Revised: +1 − 1) .

Original: Next, for a row A r in A j ( 0 ≤ j < h k ), we compute the probability that at least one row in A >j collides with A r .
Revised: Next, for a row A r in A j ( 0 ≤ j < h k ), we compute the probability that at least one row in A >j collides with A r .

Original: Pr [ at least one row in A >j collides with A r ] =
Revised: Pr [ at least one row in A >j collides with A r ] =

Original: (1 − (1 − 1 k ) |A >j | )
Revised: (1 − (1 − 1 k ) |A >j | )

Original: [Equation]
Revised: [Equation]

Original: Hence, by Observation F.19, the contribution of rows in A j to the total squared loss is at least (
Revised: Hence, by Observation F.19, the contribution of rows in A j to the total squared loss is at least (

Original: [Equation]
Revised: [Equation]

Original: Thus, the contribution of rows with “large” squared norm, i.e., A > 0 , to the total squared loss is at least
Revised: Thus, the contribution of rows with “large” squared norm, i.e., A > 0 , to the total squared loss is at least

Original: [Equation]
Revised: [Equation]

Original: Corollary F.21.
Revised: Corollary F.21.

Original: Let S g be a CountSketch whose sparsity pattern is learned over a training set drawn from A sp via the greedy approach.
Revised: Let S g be a CountSketch whose sparsity pattern is learned over a training set drawn from A sp via the greedy approach.

Original: Let S r be a CountSketch whose sparsity pattern is picked uniformly at random.
Revised: Let S r be a CountSketch whose sparsity pattern is picked uniformly at random.

Original: Then, for an n × d matrix A ∼ A zipf , for a sufficiently large value of k , the expected loss of the best rank- k approximation of A returned by S r is worse than the approximation loss of the best rank- k approximation of A returned by S g by at least a constant factor.
Revised: Then, for an n × d matrix A ∼ A zipf , for a sufficiently large value of k , the expected loss of the best rank- k approximation of A returned by S r is worse than the approximation loss of the best rank- k approximation of A returned by S g by at least a constant factor.

Original: Proof.
Revised: Proof.

Original: The proof follows from Lemma F.20 and Corollary F.18.
Revised: The proof follows from Lemma F.20 and Corollary F.18.

Original: Remark F.22 .
Revised: Remark F.22 .

Original: We have provided evidence that the greedy algorithm that examines the rows of A according to a non-increasing order of their norms (i.e., greedy with non-increasing order ) results in a better rank- k solution compared to the CountSketch whose sparsity pattern is chosen at random.
Revised: We have provided evidence that the greedy algorithm that examines the rows of A according to a non-increasing order of their norms (i.e., greedy with non-increasing order ) results in a better rank- k solution compared to the CountSketch whose sparsity pattern is chosen at random.

Original: However, still other implementations of the greedy algorithm may result in a better solution compared to the greedy algorithm with non-increasing order.
Revised: However, still other implementations of the greedy algorithm may result in a better solution compared to the greedy algorithm with non-increasing order.

Original: To give an example, in the following simple instance the greedy algorithm that checks the rows of A in a random order (i.e., greedy with random order ) achieves a rank- k solution whose cost is a constant factor better than the solution returned by the greedy with non-increasing order.
Revised: To give an example, in the following simple instance the greedy algorithm that checks the rows of A in a random order (i.e., greedy with random order ) achieves a rank- k solution whose cost is a constant factor better than the solution returned by the greedy with non-increasing order.

Original: Let A be a matrix with four orthogonal rows u, u, v, w where
Revised: Let A be a matrix with four orthogonal rows u, u, v, w where

Original: ∥ u ∥ 2 = 1 and ∥ v ∥ 2 = ∥ w ∥ 2 = 1 + ϵ and suppose that the goal is to compute a rank - 2 approximation of A .
Revised: ∥ u ∥ 2 = 1 and ∥ v ∥ 2 = ∥ w ∥ 2 = 1 + ϵ and suppose that the goal is to compute a rank - 2 approximation of A .

Original: Note that in the greedy algorithm with non-decreasing order, v and w will be assigned to different bins and by a simple calculation we can show that the copies of u also will be assigned to different bins.
Revised: Note that in the greedy algorithm with non-decreasing order, v and w will be assigned to different bins and by a simple calculation we can show that the copies of u also will be assigned to different bins.

Original: Hence, the squared loss in the computed rank - 2 solution is 1 + (1+ ϵ ) 2 2+(

Revised: Hence, the squared loss in the computed rank - 2 solution is 1 + (1+ ϵ ) 2 2+(

Original: + ϵ ) 2 .
Revised: + ϵ ) 2 .

Original: However, the optimal solution will assign v and w to one bin and the two copies of u to the other bin which results in a squared loss of (1 + ϵ ) 2 which is a constant factor smaller than the solution returned by the greedy algorithm with non-increasing order for sufficiently small values of ϵ .
Revised: However, the optimal solution will assign v and w to one bin and the two copies of u to the other bin which results in a squared loss of (1 + ϵ ) 2 which is a constant factor smaller than the solution returned by the greedy algorithm with non-increasing order for sufficiently small values of ϵ .

Original: On the other hand, in the greedy algorithm with a random order, with a constant probability of ( 13 + 18 ), the computed solution is the same as the optimal solution.
Revised: On the other hand, in the greedy algorithm with a random order, with a constant probability of ( 13 + 18 ), the computed solution is the same as the optimal solution.

Original: Otherwise, the greedy algorithm with random order returns the same solution as the greedy algorithm with a non-increasing order.
Revised: Otherwise, the greedy algorithm with random order returns the same solution as the greedy algorithm with a non-increasing order.

Original: Hence, in expectation, the solution returned by the greedy with random order is better than the solution returned by the greedy algorithm with non-increasing order by a constant factor.
Revised: Hence, in expectation, the solution returned by the greedy with random order is better than the solution returned by the greedy algorithm with non-increasing order by a constant factor.

Original: G E XPERIMENT D ETAILS G. L OW -R ANK A PPROXIMATION
Revised: G E XPERIMENT D ETAILS G. L OW -R ANK A PPROXIMATION

Original: In this section, we describe the experimental parameters in our experiments.
Revised: In this section, we describe the experimental parameters in our experiments.

Original: We first introduce some parameters in Stage 2 of our approach proposed in Section 3.
Revised: We first introduce some parameters in Stage 2 of our approach proposed in Section 3.

Original: • bs : batch size, the number of training samples used in one iteration.
Revised: • bs : batch size, the number of training samples used in one iteration.

Original: • lr : learning rate of gradient descent.
Revised: • lr : learning rate of gradient descent.

Original: • iter : the number of iterations of gradient descent.
Revised: • iter : the number of iterations of gradient descent.

Original: For a given m , the dimensions of the four sketches were: S ∈ R m × n
Revised: For a given m , the dimensions of the four sketches were: S ∈ R m × n

Original: [Table]
Revised: [Table]

Original: R
Revised: R

Original: [Table]
Revised: [Table]

Original: Parameters of the algorithm: bs = 1 , lr = 1 . 0 , 10 . 0 for hyper and video respectively, num _ it = 1000 .
Revised: Parameters of the algorithm: bs = 1 , lr = 1 . 0 , 10 . 0 for hyper and video respectively, num _ it = 1000 .

Original: For our algorithm 4, we use the average of all training matrix as the input to the algorithm.
Revised: For our algorithm 4, we use the average of all training matrix as the input to the algorithm.

Original: [Table]
Revised: [Table]

Original: Parameters of the algorithm: bs = 1 , lr = 1 . 0 , 10 . 0 for hyper and video respectively, num _ it = 1000 .
Revised: Parameters of the algorithm: bs = 1 , lr = 1 . 0 , 10 . 0 for hyper and video respectively, num _ it = 1000 .

Original: For our algorithm 4, we use the sum of all training matrix as the input to the algorithm.
Revised: For our algorithm 4, we use the sum of all training matrix as the input to the algorithm.

Original: G. S ECOND -O RDER O PTIMIZATION
Revised: G. S ECOND -O RDER O PTIMIZATION

Original: As we state in Section 6, when we fix the positions of the non-zero entries (uniformly chosen in each column or sampled according to the heavy leverage score distribution), we aim to optimize the values by gradient descent, as mentioned in Section 3.
Revised: As we state in Section 6, when we fix the positions of the non-zero entries (uniformly chosen in each column or sampled according to the heavy leverage score distribution), we aim to optimize the values by gradient descent, as mentioned in Section 3.

Original: Here the loss function is given in Section 6.
Revised: Here the loss function is given in Section 6.

Original: In our implementation, we use PyTorch (Paszke et al.
Revised: In our implementation, we use PyTorch (Paszke et al.

Original: (2019)), which can compute the gradient automatically (here we can use torch.qr() and torch.svd() to define our loss function).
Revised: (2019)), which can compute the gradient automatically (here we can use torch.qr() and torch.svd() to define our loss function).

Original: For a more nuanced loss function, which may be beneficial, one can use the package released in Agrawal et al.
Revised: For a more nuanced loss function, which may be beneficial, one can use the package released in Agrawal et al.

Original: (2019), where the authors studied the problem of computing the gradient of functions which involve the solution to certain convex optimization problem.
Revised: (2019), where the authors studied the problem of computing the gradient of functions which involve the solution to certain convex optimization problem.

Original: As mentioned in Section 2, each column of the sketch matrix S has exactly one non-zero entry.
Revised: As mentioned in Section 2, each column of the sketch matrix S has exactly one non-zero entry.

Original: Hence, the i -th coordinate of p can be seen as the non-zero position of the i -th column of S .
Revised: Hence, the i -th coordinate of p can be seen as the non-zero position of the i -th column of S .

Original: In the implementation, to sample p randomly, we can sample a random integer in { 1 , . . .
Revised: In the implementation, to sample p randomly, we can sample a random integer in { 1 , . . .

Original: , m } for each coordinate of p .
Revised: , m } for each coordinate of p .

Original: For the heavy rows mentioned in Section 6, we can allocate positions 1 , . . .
Revised: For the heavy rows mentioned in Section 6, we can allocate positions 1 , . . .

Original: , k to the k heavy rows, and for the other rows, we randomly sample an integer in { k + 1 , . . .
Revised: , k to the k heavy rows, and for the other rows, we randomly sample an integer in { k + 1 , . . .

Original: , m } .
Revised: , m } .

Original: We note that once the vector p , which contains the information of the non-zero positions in each column of S , is chosen, it will not be changed during the optimization process in Section 3.
Revised: We note that once the vector p , which contains the information of the non-zero positions in each column of S , is chosen, it will not be changed during the optimization process in Section 3.

Original: Next, we introduce the parameters for our experiments:
Revised: Next, we introduce the parameters for our experiments:

Original: • bs : batch size, the number of training samples used in one iteration.
Revised: • bs : batch size, the number of training samples used in one iteration.

Original: • lr : learning rate of gradient descent.
Revised: • lr : learning rate of gradient descent.

Original: • iter : the number of iterations of gradient descent
Revised: • iter : the number of iterations of gradient descent

Original: In our experiments, we set bs = 20 , iter = 1000 for all datasets.
Revised: In our experiments, we set bs = 20 , iter = 1000 for all datasets.

Original: We set lr =
Revised: We set lr =

Original: Electric
Revised: Electric