Original: Approximate Euclidean lengths and distances beyond Johnson-Lindenstrauss
Revised: Approximate Euclidean lengths and distances beyond Johnson-Lindenstrauss

Original: Abstract
Revised: Abstract

Original: A classical result of Johnson and Lindenstrauss states that a set of n high dimensional data points can be projected down to O (log n/(cid:15) 2 ) dimensions such that the square of their pairwise distances is preserved up to a small distortion (cid:15) ∈ (0 , 1) .
Revised: A classical result of Johnson and Lindenstrauss states that a set of n high dimensional data points can be projected down to O (log n/(cid:15) 2 ) dimensions such that the square of their pairwise distances is preserved up to a small distortion (cid:15) ∈ (0 , 1) .

Original: It has been proved that the JL lemma is optimal for the general case, therefore, improvements can only be explored for special cases.
Revised: It has been proved that the JL lemma is optimal for the general case, therefore, improvements can only be explored for special cases.

Original: This work aims to improve the (cid:15) − 2 dependency based on techniques inspired by the Hutch++ Algorithm [34], which reduces (cid:15) − 2 to (cid:15) − 1 for the related problem of implicit matrix trace estimation.
Revised: This work aims to improve the (cid:15) − 2 dependency based on techniques inspired by the Hutch++ Algorithm [34], which reduces (cid:15) − 2 to (cid:15) − 1 for the related problem of implicit matrix trace estimation.

Original: We ﬁrst present an algorithm to estimate the Euclidean lengths of the rows of a matrix.
Revised: We ﬁrst present an algorithm to estimate the Euclidean lengths of the rows of a matrix.

Original: We prove for it element-wise probabilistic bounds that are at least as good as standard JL approximations in the worst-case, but are asymptotically better for matrices with decaying spectrum.
Revised: We prove for it element-wise probabilistic bounds that are at least as good as standard JL approximations in the worst-case, but are asymptotically better for matrices with decaying spectrum.

Original: Moreover, for any matrix, regardless of its spectrum, the algorithm achieves (cid:15) -accuracy for the total, Frobenius norm-wise relative error using only O ( (cid:15) − 1 ) queries.
Revised: Moreover, for any matrix, regardless of its spectrum, the algorithm achieves (cid:15) -accuracy for the total, Frobenius norm-wise relative error using only O ( (cid:15) − 1 ) queries.

Original: This is a quadratic improvement over the norm-wise error of standard JL approximations.
Revised: This is a quadratic improvement over the norm-wise error of standard JL approximations.

Original: We also show how these results can be extended to estimate (i) the Euclidean distances between data points and (ii) the statistical leverage scores of tall-and-skinny data matrices, which are ubiquitous for many applications, with analogous theoretical improvements.
Revised: We also show how these results can be extended to estimate (i) the Euclidean distances between data points and (ii) the statistical leverage scores of tall-and-skinny data matrices, which are ubiquitous for many applications, with analogous theoretical improvements.

Original: Proof-of-concept numerical experiments are presented to validate the theoretical analysis.
Revised: Proof-of-concept numerical experiments are presented to validate the theoretical analysis.

Original:  Introduction
Revised:  Introduction

Original: The Johnson-Lindenstrauss (JL) lemma [29] is a fundamental concept in dimensionality reduction and data science.
Revised: The Johnson-Lindenstrauss (JL) lemma [29] is a fundamental concept in dimensionality reduction and data science.

Original: Given a set of n high dimensional data points X
Revised: Given a set of n high dimensional data points X

Original: = { x 1 , ..., x n } , where each x i ∈ R d , the goal is to ﬁnd a projection f : R d → R k that maps the vectors to a much smaller dimension k (cid:28) d such that the geometry of the original set is approximately preserved.
Revised: = { x 1 , ..., x n } , where each x i ∈ R d , the goal is to ﬁnd a projection f : R d → R k that maps the vectors to a much smaller dimension k (cid:28) d such that the geometry of the original set is approximately preserved.

Original: Speciﬁcally, the projection should preserve the pairwise distances up to a small distortion (cid:15) ∈ (0 , 1) , that is
Revised: Speciﬁcally, the projection should preserve the pairwise distances up to a small distortion (cid:15) ∈ (0 , 1) , that is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for all i, j ∈ [ n ] .
Revised: for all i, j ∈ [ n ] .

Original: If f satisﬁes this property, then it is called an (cid:15) -isometry.
Revised: If f satisﬁes this property, then it is called an (cid:15) -isometry.

Original: Johnson and Lindenstrauss proved that, given (cid:15) , such an f can be found in randomized polynomial time and that the projected dimension is no larger than O (log n/(cid:15) 2 ) .
Revised: Johnson and Lindenstrauss proved that, given (cid:15) , such an f can be found in randomized polynomial time and that the projected dimension is no larger than O (log n/(cid:15) 2 ) .

Original: In the last decades the JL lemma has made an impact in many areas, including Graph Algorithms [6, 42], Machine Learning [5, 11, 16, 24], Numerical Linear
Revised: In the last decades the JL lemma has made an impact in many areas, including Graph Algorithms [6, 42], Machine Learning [5, 11, 16, 24], Numerical Linear

Original: Algebra [14, 33, 40, 44] and Optimization [20, 22, 38].
Revised: Algebra [14, 33, 40, 44] and Optimization [20, 22, 38].

Original: In the existing literature, a common approach to approximate the metric is to ﬁrst ﬁnd a map that preserves Euclidean lengths instead of distances.
Revised: In the existing literature, a common approach to approximate the metric is to ﬁrst ﬁnd a map that preserves Euclidean lengths instead of distances.

Original: The approximate isometry property is then achieved by applying this map to all the pairwise difference vectors, since the Euclidean distance between x and y is equal to the length of x − y ; cf.
Revised: The approximate isometry property is then achieved by applying this map to all the pairwise difference vectors, since the Euclidean distance between x and y is equal to the length of x − y ; cf.

Original: In this work we follow the same methodology.
Revised: In this work we follow the same methodology.

Original: We ﬁrst study the problem of approximating the Euclidean lengths of the rows of an arbitrary matrix A ∈ R n × d in the so-called matrix-vector query model.
Revised: We ﬁrst study the problem of approximating the Euclidean lengths of the rows of an arbitrary matrix A ∈ R n × d in the so-called matrix-vector query model.

Original: In this model, the matrix A might not be explicitly available, but we have access to a linear operator that computes the product
Revised: In this model, the matrix A might not be explicitly available, but we have access to a linear operator that computes the product

Original: Ax , for an arbitrary vector x .
Revised: Ax , for an arbitrary vector x .

Original: For non-symmetric and rectangular matrices we assume that we can compute both Ax and A (cid:62) x .
Revised: For non-symmetric and rectangular matrices we assume that we can compute both Ax and A (cid:62) x .

Original: This model is particularly useful when dealing with matrix functions, i.e. when A = f ( B ) is implicitly deﬁned as a function of another matrix B .
Revised: This model is particularly useful when dealing with matrix functions, i.e. when A = f ( B ) is implicitly deﬁned as a function of another matrix B .

Original: Two applications arise from network science.
Revised: Two applications arise from network science.

Original: For a graph G , let A be its adjacency matrix, B its edge-incidence matrix, W the diagonal matrix containing non-negative edge weights and L = B (cid:62) WB its Laplacian.
Revised: For a graph G , let A be its adjacency matrix, B its edge-incidence matrix, W the diagonal matrix containing non-negative edge weights and L = B (cid:62) WB its Laplacian.

Original: The exponential function of the adjacency matrix, e βA , provides information about node centrality measures [10, 23], while the diagonal entries of BL † B are the so-called “effective resistances” of the edges [33, 36, 42], which can be used to identify important edges.
Revised: The exponential function of the adjacency matrix, e βA , provides information about node centrality measures [10, 23], while the diagonal entries of BL † B are the so-called “effective resistances” of the edges [33, 36, 42], which can be used to identify important edges.

Original: Another example, which is detailed in Section 4, is for computing the leverage scores of a matrix, which can be found in the diagonal of the orthogonal projector H
Revised: Another example, which is detailed in Section 4, is for computing the leverage scores of a matrix, which can be found in the diagonal of the orthogonal projector H

Original: =
Revised: =

Original: A ( A (cid:62) A ) † A (cid:62) .
Revised: A ( A (cid:62) A ) † A (cid:62) .

Original: In all of these applications, explicitly evaluating the corresponding matrix functions has typically cubic complexity, which can be prohibitively expensive.
Revised: In all of these applications, explicitly evaluating the corresponding matrix functions has typically cubic complexity, which can be prohibitively expensive.

Original: However, after some algebra, the quantities of interest can be expressed as the Euclidean lengths of the rows of a matrix function in the matrix-vector query model.
Revised: However, after some algebra, the quantities of interest can be expressed as the Euclidean lengths of the rows of a matrix function in the matrix-vector query model.

Original: One can therefore use techniques related to the JL lemma to derive fast approximations.
Revised: One can therefore use techniques related to the JL lemma to derive fast approximations.

Original: To this end, we ﬁrst recall the concept of
Revised: To this end, we ﬁrst recall the concept of

Original: Johnson-Lindenstrauss transforms, as deﬁned in [40].
Revised: Johnson-Lindenstrauss transforms, as deﬁned in [40].

Original: Deﬁnition 1 (Johnson-Lindenstrauss transform [40]) .
Revised: Deﬁnition 1 (Johnson-Lindenstrauss transform [40]) .

Original: A random matrix S ∈ R r × d forms a JohnsonLindenstrauss transform with parameters (cid:15), δ ∈ (0 , 1 / 2) and positive integer n , or ( (cid:15), δ, n ) -JLT for short, if with probability at least 1 − δ , for any ﬁxed set V
Revised: A random matrix S ∈ R r × d forms a JohnsonLindenstrauss transform with parameters (cid:15), δ ∈ (0 , 1 / 2) and positive integer n , or ( (cid:15), δ, n ) -JLT for short, if with probability at least 1 − δ , for any ﬁxed set V

Original: ⊆ R d with n elements it holds that
Revised: ⊆ R d with n elements it holds that

Original: [Equation]
Revised: [Equation]

Original: ) (cid:107) v (cid:107) 2 ≤ (cid:107)
Revised: ) (cid:107) v (cid:107) 2 ≤ (cid:107)

Original: Sv (cid:107) 2 ≤ (1 + (cid:15) )
Revised: Sv (cid:107) 2 ≤ (1 + (cid:15) )

Original: (cid:107) v (cid:107) 2 for all v ∈ V .
Revised: (cid:107) v (cid:107) 2 for all v ∈ V .

Original: It is known that Gaussian matrices can provide JLTs; c.f.
Revised: It is known that Gaussian matrices can provide JLTs; c.f.

Original: Lemma 1 (Gaussian random projections [5, 29]) .
Revised: Lemma 1 (Gaussian random projections [5, 29]) .

Original: Let G ∈ R r × d with i.i.d.
Revised: Let G ∈ R r × d with i.i.d.

Original: elements from
Revised: elements from

Original: N (0 , 1 / √ r ) and (cid:15) ∈ (0 , 1 / 2) .
Revised: N (0 , 1 / √ r ) and (cid:15) ∈ (0 , 1 / 2) .

Original: For a ﬁxed x ∈ R d it holds that
Revised: For a ﬁxed x ∈ R d it holds that

Original: [Equation]
Revised: [Equation]

Original: For a set X ⊂ R d of n vectors and for δ ∈ (0 , 1 / 2) , as long as r ≥ 4log(2 n/δ ) (cid:15) 2 − (cid:15), then G forms an ( (cid:15), δ, n ) -JLT.
Revised: For a set X ⊂ R d of n vectors and for δ ∈ (0 , 1 / 2) , as long as r ≥ 4log(2 n/δ ) (cid:15) 2 − (cid:15), then G forms an ( (cid:15), δ, n ) -JLT.

Original: The dimension r of G depends on 1 /(cid:15) 2 , which can quickly become very large if a high accuracy is needed.
Revised: The dimension r of G depends on 1 /(cid:15) 2 , which can quickly become very large if a high accuracy is needed.

Original: Consequently, if r is very large, then it is also very expensive to compute the product GA .
Revised: Consequently, if r is very large, then it is also very expensive to compute the product GA .

Original: There is therefore no advantage in taking an approximate solution over computing the true solution.
Revised: There is therefore no advantage in taking an approximate solution over computing the true solution.

Original: Here, we would like to improve this dependency on (cid:15) .
Revised: Here, we would like to improve this dependency on (cid:15) .

Original: To achieve this, we will also use a powerful generalization of JLT, the so-called of Oblivious Subspace Embeddings [40], which extend the JLT deﬁnition for an entire subspace, instead of a ﬁnite set.
Revised: To achieve this, we will also use a powerful generalization of JLT, the so-called of Oblivious Subspace Embeddings [40], which extend the JLT deﬁnition for an entire subspace, instead of a ﬁnite set.

Original: We use the deﬁnitions from [44].
Revised: We use the deﬁnitions from [44].

Original: Deﬁnition 2 (Oblivious Subspace Embedding [44]) .
Revised: Deﬁnition 2 (Oblivious Subspace Embedding [44]) .

Original: Let D be a distribution on r × n matrices S , where r is a function of n, d and (cid:15), δ ∈ (0 , 1 / 2) .
Revised: Let D be a distribution on r × n matrices S , where r is a function of n, d and (cid:15), δ ∈ (0 , 1 / 2) .

Original: We call S an ( (cid:15), δ )
Revised: We call S an ( (cid:15), δ )

Original: Oblivious Subspace
Revised: Oblivious Subspace

Original: Embedding, or ( (cid:15), δ ) -OSE if for any ﬁxed n × d matrix A , S ∼ D is a (1 ± (cid:15) )
Revised: Embedding, or ( (cid:15), δ ) -OSE if for any ﬁxed n × d matrix A , S ∼ D is a (1 ± (cid:15) )

Original: l 2 -Subspace Embedding for A with probability at least 1 − δ , that is, for all x ∈ R d it holds that
Revised: l 2 -Subspace Embedding for A with probability at least 1 − δ , that is, for all x ∈ R d it holds that

Original: [Equation]
Revised: [Equation]

Original: Notation.
Revised: Notation.

Original: By default, the Householder notation is used, denoting matrices with capital letters, vectors with small letters, and scalars with Greek letters.
Revised: By default, the Householder notation is used, denoting matrices with capital letters, vectors with small letters, and scalars with Greek letters.

Original: [ n ] is the set { 1 , 2 , ..., n } , where n ∈ N .
Revised: [ n ] is the set { 1 , 2 , ..., n } , where n ∈ N .

Original: I n is the identity matrix of size n × n and e i its i -th column.
Revised: I n is the identity matrix of size n × n and e i its i -th column.

Original: A i,j is the element of A in row i and column j .
Revised: A i,j is the element of A in row i and column j .

Original: A k denotes the best rank- k approximation of A in the 2 -norm.
Revised: A k denotes the best rank- k approximation of A in the 2 -norm.

Original: (cid:107) A (cid:107) F is the Frobenius norm of A , while the 2-norm is assumed for both matrices and vectors when the norm subscript is omitted.
Revised: (cid:107) A (cid:107) F is the Frobenius norm of A , while the 2-norm is assumed for both matrices and vectors when the norm subscript is omitted.

Original: A (cid:62) is the transpose of A and A † is the pseudoinverse.
Revised: A (cid:62) is the transpose of A and A † is the pseudoinverse.

Original: P [ α ] ∈ [0 , 1] denotes the probability of an event α to occur.
Revised: P [ α ] ∈ [0 , 1] denotes the probability of an event α to occur.

Original: N ( µ, σ ) is the normal distribution with mean value µ and standard deviation σ .
Revised: N ( µ, σ ) is the normal distribution with mean value µ and standard deviation σ .

Original: σ i ( A ) denotes the i -th largest singular value of A .
Revised: σ i ( A ) denotes the i -th largest singular value of A .

Original: nnz ( A ) is the number of nonzeros A .
Revised: nnz ( A ) is the number of nonzeros A .

Original: ˜ O
Revised: ˜ O

Original: ( k ) :
Revised: ( k ) :

Original: = O ( k log c ( k )) for some constant c .
Revised: = O ( k log c ( k )) for some constant c .

Original: We refer to matrices with i.i.d.
Revised: We refer to matrices with i.i.d.

Original: elements from N (0 , 1) as Gaussian matrices.
Revised: elements from N (0 , 1) as Gaussian matrices.

Original: In the complexity analysis, ω denotes the fast matrix multiplication exponent, where 2 ≤ ω < 2 . 37286 [4].
Revised: In the complexity analysis, ω denotes the fast matrix multiplication exponent, where 2 ≤ ω < 2 . 37286 [4].

Original: Why Gaussians?
Revised: Why Gaussians?

Original: In this work we focus on Gaussian random projections.
Revised: In this work we focus on Gaussian random projections.

Original: Other constructions satisfying the ( (cid:15), δ, n ) -JLT deﬁnition exist in the literature, such as randomized Fourier/Hadamard
Revised: Other constructions satisfying the ( (cid:15), δ, n ) -JLT deﬁnition exist in the literature, such as randomized Fourier/Hadamard

Original: [2, 3, 43] or sparse [1, 15, 17, 30, 32, 35] transforms.
Revised: [2, 3, 43] or sparse [1, 15, 17, 30, 32, 35] transforms.

Original: When the input matrix is explicitly available, such constructions are faster to apply than Gaussian random projections.
Revised: When the input matrix is explicitly available, such constructions are faster to apply than Gaussian random projections.

Original: However, as already mentioned, this is not the case in the matrix-vector query model.
Revised: However, as already mentioned, this is not the case in the matrix-vector query model.

Original: Consider the following simple example.
Revised: Consider the following simple example.

Original: Assume that we are interested to compute the Euclidean row norms of the matrix
Revised: Assume that we are interested to compute the Euclidean row norms of the matrix

Original: A 2 , where
Revised: A 2 , where

Original: A ∈ R n × n is a dense input matrix.
Revised: A ∈ R n × n is a dense input matrix.

Original: As already detailed, these norms can be approximated by the
Revised: As already detailed, these norms can be approximated by the

Original: Euclidean row norms of the matrix ˜
Revised: Euclidean row norms of the matrix ˜

Original: A =
Revised: A =

Original: A 2 S , where S is an ( (cid:15), δ, n ) -JLT matrix with r columns.
Revised: A 2 S , where S is an ( (cid:15), δ, n ) -JLT matrix with r columns.

Original: If
Revised: If

Original: S is a Gaussian matrix, then ˜
Revised: S is a Gaussian matrix, then ˜

Original: A can be evaluated in two steps, i.e. by computing B = AS and then ˜
Revised: A can be evaluated in two steps, i.e. by computing B = AS and then ˜

Original: A = AB , with a total complexity of O ( n 2 r ω − 2 ) .
Revised: A = AB , with a total complexity of O ( n 2 r ω − 2 ) .

Original: On the other hand, if S is a CountSketch [15, 35], then the matrix B = AS can be evaluated in O ( n 2 ) , which is faster than O ( n 2 r ω − 2 ) .
Revised: On the other hand, if S is a CountSketch [15, 35], then the matrix B = AS can be evaluated in O ( n 2 ) , which is faster than O ( n 2 r ω − 2 ) .

Original: However, since
Revised: However, since

Original: B is dense and it has no special structure, then the second step ˜
Revised: B is dense and it has no special structure, then the second step ˜

Original: A = AB still requires O ( n 2 r ω − 2 ) operations.
Revised: A = AB still requires O ( n 2 r ω − 2 ) operations.

Original: The total complexity is therefore still dominated by the O ( n 2 r ω − 2 ) term.
Revised: The total complexity is therefore still dominated by the O ( n 2 r ω − 2 ) term.

Original: It is also known that Gaussian matrices require asymptotically less columns r than the aforementioned fast transforms to satisfy the ( (cid:15), δ, n ) -JLT deﬁnition.
Revised: It is also known that Gaussian matrices require asymptotically less columns r than the aforementioned fast transforms to satisfy the ( (cid:15), δ, n ) -JLT deﬁnition.

Original: This means that Gaussian matrices can in fact be the fastest option, since r is minimized.
Revised: This means that Gaussian matrices can in fact be the fastest option, since r is minimized.

Original: Nevertheless, all results of this work are derived as structural results, i.e. they do not necessarily require Gaussian matrices.
Revised: Nevertheless, all results of this work are derived as structural results, i.e. they do not necessarily require Gaussian matrices.

Original: Any matrix satisfying the properties that are detailed in the proofs can be used instead.
Revised: Any matrix satisfying the properties that are detailed in the proofs can be used instead.

Original: All aforementioned fast transforms are for example excellent candidates.
Revised: All aforementioned fast transforms are for example excellent candidates.

Original: Contributions.
Revised: Contributions.

Original: In Algorithm 1, we present the main algorithm of this work to approximate the
Revised: In Algorithm 1, we present the main algorithm of this work to approximate the

Original: Euclidean row norms of a matrix A ∈ R n × d , which is inspired by the Hutch++ algorithm [34].
Revised: Euclidean row norms of a matrix A ∈ R n × d , which is inspired by the Hutch++ algorithm [34].

Original: Following [34], this algorithm is called “Adaptive,” since it needs to make two passes over the input matrix A .
Revised: Following [34], this algorithm is called “Adaptive,” since it needs to make two passes over the input matrix A .

Original: The main contributions are the following:
Revised: The main contributions are the following:

Original: The proposed algorithms require asymptotically less matrix-vector queries to achieve the same accuracy as standard JL random projections for matrices with decaying spectrum, that is, spectral decay properties are reﬂected in the approximation bounds.
Revised: The proposed algorithms require asymptotically less matrix-vector queries to achieve the same accuracy as standard JL random projections for matrices with decaying spectrum, that is, spectral decay properties are reﬂected in the approximation bounds.

Original: To the best of our knowledge, this is the ﬁrst work to provably reduce the number of required matrix-vector queries for Euclidean length approximations.
Revised: To the best of our knowledge, this is the ﬁrst work to provably reduce the number of required matrix-vector queries for Euclidean length approximations.

Original: For any matrix, regardless of its spectrum, the proposed algorithms require a number of matrix-vector queries that depends on 1 /(cid:15) to achieve (cid:15) -accuracy for the total, Frobenius norm-wise error, as opposed to 1 /(cid:15) 2 for standard JL.
Revised: For any matrix, regardless of its spectrum, the proposed algorithms require a number of matrix-vector queries that depends on 1 /(cid:15) to achieve (cid:15) -accuracy for the total, Frobenius norm-wise error, as opposed to 1 /(cid:15) 2 for standard JL.

Original: For the worst-case inputs, that is, for matrices with ﬂat spectrum, the approximated values are at least as good as standard JL.
Revised: For the worst-case inputs, that is, for matrices with ﬂat spectrum, the approximated values are at least as good as standard JL.

Original: The techniques can be directly applied to and give similar improvements for the related problems of approximate pairwise Euclidean distances and approximate leverage scores
Revised: The techniques can be directly applied to and give similar improvements for the related problems of approximate pairwise Euclidean distances and approximate leverage scores

Original: Algorithm 1 Adaptive Euclidean Norm Estimation
Revised: Algorithm 1 Adaptive Euclidean Norm Estimation

Original: Input: Matrix A ∈ R n × d , n ≥ d , positive integer m < d .
Revised: Input: Matrix A ∈ R n × d , n ≥ d , positive integer m < d .

Original: Output: ˜ x i ≈ (cid:107) e (cid:62) i
Revised: Output: ˜ x i ≈ (cid:107) e (cid:62) i

Original: A (cid:107) 2 .
Revised: A (cid:107) 2 .

Original: # Step 1: Low-rank approximation
Revised: # Step 1: Low-rank approximation

Original: (cid:46) O ( dm )
Revised: (cid:46) O ( dm )

Original: (cid:46) O
Revised: (cid:46) O

Original: ( T MM ( A m ))
Revised: ( T MM ( A m ))

Original: (cid:46) and compute
Revised: (cid:46) and compute

Original: Compute
Revised: Compute

Original: A = AQ and C = AG .
Revised: A = AQ and C = AG .

Original: (cid:46) O
Revised: (cid:46) O

Original: ( T MM ( A, m )) 5:
Revised: ( T MM ( A, m )) 5:

Original: Compute
Revised: Compute

Original: ˜∆ =
Revised: ˜∆ =

Original: A ( I − QQ (cid:62) )
Revised: A ( I − QQ (cid:62) )

Original: G = C − ˜
Revised: G = C − ˜

Original: A ( Q (cid:62) G ) .
Revised: A ( Q (cid:62) G ) .

Original: (cid:46) O
Revised: (cid:46) O

Original: ( nm 2 ) 6: return ˜ x
Revised: ( nm 2 ) 6: return ˜ x

Original: i = (cid:107) e (cid:62) i ˜
Revised: i = (cid:107) e (cid:62) i ˜

Original: A (cid:107) 2 + (cid:107) e (cid:62)
Revised: A (cid:107) 2 + (cid:107) e (cid:62)

Original: i
Revised: i

Original: ˜∆ (cid:107) 2 , for all i ∈ [ n ] .
Revised: ˜∆ (cid:107) 2 , for all i ∈ [ n ] .

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: In Table 1 we summarize the approximation guarantees of the proposed Algorithms 1, 2, and 3 for the aforementioned problems.
Revised: In Table 1 we summarize the approximation guarantees of the proposed Algorithms 1, 2, and 3 for the aforementioned problems.

Original: We also compare it to the corresponding bounds of existing JL-based approximations to highlight the achieved improvements.
Revised: We also compare it to the corresponding bounds of existing JL-based approximations to highlight the achieved improvements.

Original: For the precise statements we refer to the corresponding sections.
Revised: For the precise statements we refer to the corresponding sections.

Original: Regarding the complexity of Algorithm 1, by T MM ( A, m ) we denote the complexity of computing the product AB , where B is a dense matrix with m columns.
Revised: Regarding the complexity of Algorithm 1, by T MM ( A, m ) we denote the complexity of computing the product AB , where B is a dense matrix with m columns.

Original: For example, if A is just a dense matrix, T MM ( A, m ) =
Revised: For example, if A is just a dense matrix, T MM ( A, m ) =

Original: O ( ndm ω − 2 ) , by leveraging fast matrix multiplication [4].
Revised: O ( ndm ω − 2 ) , by leveraging fast matrix multiplication [4].

Original: Another example is when A is implicitly available as a function of a given sparse matrix C , e.g. if A = C 3 then
Revised: Another example is when A is implicitly available as a function of a given sparse matrix C , e.g. if A = C 3 then

Original: [Table] [Table]
Revised: [Table] [Table]

Original: MM ( A, m ) = 

Revised: MM ( A, m ) =

Original: × T MM ( C, m ) =
Revised: × T MM ( C, m ) =

Original: O ( nnz ( C ) m ) .
Revised: O ( nnz ( C ) m ) .

Original: The results are stated for general T MM ( A, m ) , but they will be specialized, where applicable, for the targeted applications.
Revised: The results are stated for general T MM ( A, m ) , but they will be specialized, where applicable, for the targeted applications.

Original: Related work.
Revised: Related work.

Original: A related topic is stochastic matrix trace estimation [7, 27, 28, 34, 37, 39].
Revised: A related topic is stochastic matrix trace estimation [7, 27, 28, 34, 37, 39].

Original: Intuitively, a set of data points can be seen as the columns of a matrix.
Revised: Intuitively, a set of data points can be seen as the columns of a matrix.

Original: In various applications the trace of such a matrix contains useful information like triangle counts in graphs [6].
Revised: In various applications the trace of such a matrix contains useful information like triangle counts in graphs [6].

Original: Hutchinson [27] proposed a randomized algorithm to rapidly approximate the trace of such a matrix, which uses similar ideas to
Revised: Hutchinson [27] proposed a randomized algorithm to rapidly approximate the trace of such a matrix, which uses similar ideas to

Original: JL: it projects the rows of the matrix onto a low-dimensional subspace so that the trace can be quickly computed.
Revised: JL: it projects the rows of the matrix onto a low-dimensional subspace so that the trace can be quickly computed.

Original: Avron and Toledo showed that the dimension of that subspace needs to be proportional to (cid:15) − 2 in order to guarantee a worst-case (cid:15) -approximation for the trace [7].
Revised: Avron and Toledo showed that the dimension of that subspace needs to be proportional to (cid:15) − 2 in order to guarantee a worst-case (cid:15) -approximation for the trace [7].

Original: This dependence on (cid:15) matches the requirements for the (cid:15) -isometry of JL.
Revised: This dependence on (cid:15) matches the requirements for the (cid:15) -isometry of JL.

Original: The (cid:15) − 2 overhead can be prohibitive when (cid:15) is small, i.e. in applications where high accuracy is needed.
Revised: The (cid:15) − 2 overhead can be prohibitive when (cid:15) is small, i.e. in applications where high accuracy is needed.

Original: Recently, in their seminal work, Meyer,
Revised: Recently, in their seminal work, Meyer,

Original: Musco, Musco, and Woodruff [34] proved a remarkable result: their Hutch++ algorithm is the ﬁrst to obtain (cid:15) -accuracy for stochastic trace estimation while requiring only 1 /(cid:15) matrix-vector queries.
Revised: Musco, Musco, and Woodruff [34] proved a remarkable result: their Hutch++ algorithm is the ﬁrst to obtain (cid:15) -accuracy for stochastic trace estimation while requiring only 1 /(cid:15) matrix-vector queries.

Original: For the related problem of estimating the diagonal elements of a matrix, which was also recently studied in depth [9, 26], Baston and Nakatsukasa [9] achieved (cid:15) -accuracy for the total, norm-wise error of the entire diagonal using O (1 /(cid:15) )
Revised: For the related problem of estimating the diagonal elements of a matrix, which was also recently studied in depth [9, 26], Baston and Nakatsukasa [9] achieved (cid:15) -accuracy for the total, norm-wise error of the entire diagonal using O (1 /(cid:15) )

Original: matrix-vector queries, but not for each individual diagonal element, which should not be possible due to the optimality of the JL lemma [31].
Revised: matrix-vector queries, but not for each individual diagonal element, which should not be possible due to the optimality of the JL lemma [31].

Original: It is worth noting that the squared row norms of a matrix A can be found in the diagonal of AA (cid:62) , therefore, our work is closely related.
Revised: It is worth noting that the squared row norms of a matrix A can be found in the diagonal of AA (cid:62) , therefore, our work is closely related.

Original: Our results for the total norm-wise error, however (see e.g. Theorem 2), are tighter than simply using [9] on AA (cid:62) , since we are exploiting the special structure of AA (cid:62) .
Revised: Our results for the total norm-wise error, however (see e.g. Theorem 2), are tighter than simply using [9] on AA (cid:62) , since we are exploiting the special structure of AA (cid:62) .

Original: From a Fine-Grained complexity perspective, estimating row norms can be easily reduced to diagonal estimation, but the opposite reduction is not straightforward, therefore, one can argue that diagonal estimation is harder, which justiﬁes our tighter bounds.
Revised: From a Fine-Grained complexity perspective, estimating row norms can be easily reduced to diagonal estimation, but the opposite reduction is not straightforward, therefore, one can argue that diagonal estimation is harder, which justiﬁes our tighter bounds.

Original: Outline.
Revised: Outline.

Original: The analysis of Algorithm 1 is given in Section 2.
Revised: The analysis of Algorithm 1 is given in Section 2.

Original: In Sections 3 and 4 we show two important applications of the main results, namely for the estimation of the pairwise Euclidean distances between a set of data points and for the estimation of the statistical leverage scores of a tall-and-skinny data matrix.
Revised: In Sections 3 and 4 we show two important applications of the main results, namely for the estimation of the pairwise Euclidean distances between a set of data points and for the estimation of the statistical leverage scores of a tall-and-skinny data matrix.

Original: In Section 5 we present indicative experiments to validate the theoretical analysis, before ﬁnally giving concluding remarks and future directions in Section 6.
Revised: In Section 5 we present indicative experiments to validate the theoretical analysis, before ﬁnally giving concluding remarks and future directions in Section 6.

Original:  Analysis of Algorithm
Revised:  Analysis of Algorithm

Original: In this section we provide the analysis of Algorithm 1.
Revised: In this section we provide the analysis of Algorithm 1.

Original: Preliminary results and long proofs which were omitted from the main text and can be found in the Appendix.
Revised: Preliminary results and long proofs which were omitted from the main text and can be found in the Appendix.

Original: We state the following general result for the element-wise bounds of Algorithm 1.
Revised: We state the following general result for the element-wise bounds of Algorithm 1.

Original: Lemma 2.
Revised: Lemma 2.

Original: (Proof in the Appendix) Let A ∈ R n × d .
Revised: (Proof in the Appendix) Let A ∈ R n × d .

Original: If we use Algorithm 1 with m matrix-vector queries to estimate the Euclidean lengths of the rows e (cid:62) i
Revised: If we use Algorithm 1 with m matrix-vector queries to estimate the Euclidean lengths of the rows e (cid:62) i

Original: A , i ∈ [ n ] , then as long as m ≥ l ≥ 32 log(4 n/δ ) it holds that
Revised: A , i ∈ [ n ] , then as long as m ≥ l ≥ 32 log(4 n/δ ) it holds that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ] , with probability at least 1 − δ for all i ∈ [ n ] simultaneously.
Revised: ] , with probability at least 1 − δ for all i ∈ [ n ] simultaneously.

Original: Evidently, this result implies that if we can determine a suitable bound for (cid:107) e (cid:62) i
Revised: Evidently, this result implies that if we can determine a suitable bound for (cid:107) e (cid:62) i

Original: A ( I − QQ (cid:62) )
Revised: A ( I − QQ (cid:62) )

Original: (cid:107) 

Revised: (cid:107)

Original: then we automatically get a proper bound for the element-wise approximations of Algorithm 1.
Revised: then we automatically get a proper bound for the element-wise approximations of Algorithm 1.

Original: If A has a fast decaying spectrum and Q captures the dominant eigenspace of A we can expect that our approximations are very accurate, even for small l .
Revised: If A has a fast decaying spectrum and Q captures the dominant eigenspace of A we can expect that our approximations are very accurate, even for small l .

Original: For the general case, however, the following
Revised: For the general case, however, the following

Original: Lemma 3 as well as the optimality of the JL lemma [31] already hint that this is not possible (see also
Revised: Lemma 3 as well as the optimality of the JL lemma [31] already hint that this is not possible (see also

Original: Appendix II, Limitations of low-rank projections).
Revised: Appendix II, Limitations of low-rank projections).

Original: Lemma 3.
Revised: Lemma 3.

Original: Let A ∈ R n × d .
Revised: Let A ∈ R n × d .

Original: For 1 ≤ k < d , it holds that (cid:107) e (cid:62) i
Revised: For 1 ≤ k < d , it holds that (cid:107) e (cid:62) i

Original: A − A k ) (cid:107) 22 ≤ σ 2 k +

Revised: A − A k ) (cid:107) 22 ≤ σ 2 k +

Original: ( A ) ≤ (cid:107)
Revised: ( A ) ≤ (cid:107)

Original: A k (cid:107) 2 F k .
Revised: A k (cid:107) 2 F k .

Original: Proof.
Revised: Proof.

Original: Clearly, (cid:107) e (cid:62) i
Revised: Clearly, (cid:107) e (cid:62) i

Original: ( A − A k ) (cid:107) 22 ≤ max (cid:107) x (cid:107) =1 (cid:107) x (cid:62) ( A − A k ) (cid:107) 

Revised: ( A − A k ) (cid:107) 22 ≤ max (cid:107) x (cid:107) =1 (cid:107) x (cid:62) ( A − A k ) (cid:107)

Original: =
Revised: =

Original: σ 2 k +1 ( A ) .
Revised: σ 2 k +1 ( A ) .

Original: For the second part we have that σ 2 k +1 ( A ) ≤ 1 k (cid:80) ki
Revised: For the second part we have that σ 2 k +1 ( A ) ≤ 1 k (cid:80) ki

Original: =1 σ 2 i ( A ) = (cid:107) A k (cid:107) 2 F k .
Revised: =1 σ 2 i ( A ) = (cid:107) A k (cid:107) 2 F k .

Original: 2.1 Projecting rows on randomly chosen subspaces
Revised: 2.1 Projecting rows on randomly chosen subspaces

Original: To proceed further with the analysis, we show some length-preserving properties of the orthogonal projector QQ (cid:62) , which is an orthogonal projector on a random subspace as obtained in line 3 of
Revised: To proceed further with the analysis, we show some length-preserving properties of the orthogonal projector QQ (cid:62) , which is an orthogonal projector on a random subspace as obtained in line 3 of

Original: Algorithm 1.
Revised: Algorithm 1.

Original: Note that Corollary 1 is stated for constant factor approximations.
Revised: Note that Corollary 1 is stated for constant factor approximations.

Original: Here we provide a brief proof sketch.
Revised: Here we provide a brief proof sketch.

Original: For the main result we refer to Lemma 8 in Appendix III.
Revised: For the main result we refer to Lemma 8 in Appendix III.

Original: Corollary 1 (Projection on rowspace( SA (cid:62) A ) ) .
Revised: Corollary 1 (Projection on rowspace( SA (cid:62) A ) ) .

Original: (Proof in the Appendix) Let δ ∈ (0 , 12 ) , ¯ A k =
Revised: (Proof in the Appendix) Let δ ∈ (0 , 12 ) , ¯ A k =

Original: A − A k , and S be such that
Revised: A − A k , and S be such that

Original: (i) S ∼ D , where D is an (1 / 3 , δ ) -OSE for any ﬁxed k -dimensional subspace; (ii) S is a (1 / 3 , δ, 2 n ) -JLT.
Revised: (i) S ∼ D , where D is an (1 / 3 , δ ) -OSE for any ﬁxed k -dimensional subspace; (ii) S is a (1 / 3 , δ, 2 n ) -JLT.

Original: If Q is a matrix that forms an orthonormal basis for rowspace( SA (cid:62) A ) , then, with probability at least 1 − 2 δ , for all i ∈ [ n ] simultaneously, it holds that
Revised: If Q is a matrix that forms an orthonormal basis for rowspace( SA (cid:62) A ) , then, with probability at least 1 − 2 δ , for all i ∈ [ n ] simultaneously, it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: To prove the result it sufﬁces to ﬁnd a projector within rowspace( SA (cid:62) A ) with the desired properties.
Revised: To prove the result it sufﬁces to ﬁnd a projector within rowspace( SA (cid:62) A ) with the desired properties.

Original: To do this, we consider the matrix
Revised: To do this, we consider the matrix

Original: Π k = V k ( SV k Σ 2 k ) † SA (cid:62) A , where
Revised: Π k = V k ( SV k Σ 2 k ) † SA (cid:62) A , where

Original: V k , Σ k originate from the SVD of A k = U k Σ k V (cid:62) k .
Revised: V k , Σ k originate from the SVD of A k = U k Σ k V (cid:62) k .

Original: Clearly, this Π k is a rank- k matrix within rowspace( SA (cid:62) A ) .
Revised: Clearly, this Π k is a rank- k matrix within rowspace( SA (cid:62) A ) .

Original: After some algebra, the problem reduces to get a bound for the quantities | e (cid:62)
Revised: After some algebra, the problem reduces to get a bound for the quantities | e (cid:62)

Original: i AV k Σ
Revised: i AV k Σ

Original: 2 k C − 1 V (cid:62) k S (cid:62)
Revised: 2 k C − 1 V (cid:62) k S (cid:62)

Original: S ¯ V k ¯Σ 

Revised: S ¯ V k ¯Σ

Original: k ¯ V (cid:62)
Revised: k ¯ V (cid:62)

Original: k
Revised: k

Original: A (cid:62) e i | , for all i ∈ [ n ] , where the existence of C − 1 is guaranteed due to the (1 / 3 , δ ) -OSE property of S .
Revised: A (cid:62) e i | , for all i ∈ [ n ] , where the existence of C − 1 is guaranteed due to the (1 / 3 , δ ) -OSE property of S .

Original: For each i , this quantity is the absolute value of the inner product (cid:104) S ( V k C − 1 V (cid:62)
Revised: For each i , this quantity is the absolute value of the inner product (cid:104) S ( V k C − 1 V (cid:62)

Original: k A (cid:62)
Revised: k A (cid:62)

Original: [Equation]
Revised: [Equation]

Original: i , S
Revised: i , S

Original: [Equation]
Revised: [Equation]

Original: ) e i (cid:105) , which can be written in a simpliﬁed form as (cid:104)
Revised: ) e i (cid:105) , which can be written in a simpliﬁed form as (cid:104)

Original: Sx k , S ¯ x k (cid:105) .
Revised: Sx k , S ¯ x k (cid:105) .

Original: Therefore, we use an (1 / 3 , δ, 2 n ) -JLT to bound the inner products between the vectors of the set
Revised: Therefore, we use an (1 / 3 , δ, 2 n ) -JLT to bound the inner products between the vectors of the set

Original: [Equation]
Revised: [Equation]

Original: Having all pieces in-place, we can ﬁnally bound the element-wise approximations of Algorithm 1.
Revised: Having all pieces in-place, we can ﬁnally bound the element-wise approximations of Algorithm 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Proof in the Appendix) Let A ∈ R n × d and n ≥ d .
Revised: (Proof in the Appendix) Let A ∈ R n × d and n ≥ d .

Original: If we use Algorithm 1 with m matrix-vector queries to estimate the Euclidean lengths of the rows of A , then there exists a global constant C such that, as long as (i) m
Revised: If we use Algorithm 1 with m matrix-vector queries to estimate the Euclidean lengths of the rows of A , then there exists a global constant C such that, as long as (i) m

Original: ≥ l ≥ O (log( n/δ )) , such that G satisﬁes Lemma 1 and S forms an (1 / 3 , δ, 2 n ) -JLT, (ii) m ≥ O ( k + log(1 /δ )) , such that S forms an (1 / 3 , δ ) -OSE for a k -dimensional subspace, then it holds that
Revised: ≥ l ≥ O (log( n/δ )) , such that G satisﬁes Lemma 1 and S forms an (1 / 3 , δ, 2 n ) -JLT, (ii) m ≥ O ( k + log(1 /δ )) , such that S forms an (1 / 3 , δ ) -OSE for a k -dimensional subspace, then it holds that

Original: [Equation]
Revised: [Equation]

Original: for all i ∈ [ δ
Revised: for all i ∈ [ δ

Original: Discussion.
Revised: Discussion.

Original: We can investigate the bounds for special matrix cases.
Revised: We can investigate the bounds for special matrix cases.

Original: We highlight the approximation power of Algorithm 1 for matrices with decaying spectrum.
Revised: We highlight the approximation power of Algorithm 1 for matrices with decaying spectrum.

Original: For matrices with a linear decay it sufﬁces to take m (cid:38) O
Revised: For matrices with a linear decay it sufﬁces to take m (cid:38) O

Original: ( (cid:15) − 1 (cid:112) log( n/δ )) queries to achieve an almost (cid:15) -accuracy.
Revised: ( (cid:15) − 1 (cid:112) log( n/δ )) queries to achieve an almost (cid:15) -accuracy.

Original: For matrices with exponential decay we can use as few as m ≥ O (log(1 /(cid:15) ))
Revised: For matrices with exponential decay we can use as few as m ≥ O (log(1 /(cid:15) ))

Original: matrix-vector queries.
Revised: matrix-vector queries.

Original: For matrices with no decay, e.g., for orthogonal projector matrices, Lemma 2 already guarantees that Algorithmprovides at least as accurate element-wise approximations as standard JL projections.
Revised: For matrices with no decay, e.g., for orthogonal projector matrices, Lemma 2 already guarantees that Algorithmprovides at least as accurate element-wise approximations as standard JL projections.

Original: We recall once more that the JL lemma is optimal in the general case [31], therefore, improvements can only be derived for special cases, like the ones considered here.
Revised: We recall once more that the JL lemma is optimal in the general case [31], therefore, improvements can only be derived for special cases, like the ones considered here.

Original: 2.2 Frobenius norm bounds
Revised: 2.2 Frobenius norm bounds

Original: Due to the tightness of Lemma 3, which is crucial for the element-wise bounds, it is highly unlikely that low-rank projection-based methods can generally achieve better element-wise approximations.
Revised: Due to the tightness of Lemma 3, which is crucial for the element-wise bounds, it is highly unlikely that low-rank projection-based methods can generally achieve better element-wise approximations.

Original: However, if we carefully examine the total, Frobenius norm-wise error, we can in fact obtain a true (cid:15) -relative error approximation.
Revised: However, if we carefully examine the total, Frobenius norm-wise error, we can in fact obtain a true (cid:15) -relative error approximation.

Original: This cannot be done by “simply” adding together all element-wise bounds, i.e., we must use a different “collective” approach.
Revised: This cannot be done by “simply” adding together all element-wise bounds, i.e., we must use a different “collective” approach.

Original: This result also makes the elementwise bounds more appealing: even if there remain few outliers that violate the element-wise (cid:15) - approximation, the total error is still very small.
Revised: This result also makes the elementwise bounds more appealing: even if there remain few outliers that violate the element-wise (cid:15) - approximation, the total error is still very small.

Original: We note that this is a quadratic improvement over the norm-wise error of standard JL projections.
Revised: We note that this is a quadratic improvement over the norm-wise error of standard JL projections.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Proof in the Appendix)
Revised: (Proof in the Appendix)

Original: In Algorithm 1, for some absolute constants c, C , if l > c log(1 /δ ) , it holds that
Revised: In Algorithm 1, for some absolute constants c, C , if l > c log(1 /δ ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: where ˜ X is the sum of the returned approximations.
Revised: where ˜ X is the sum of the returned approximations.

Original: For l = k
Revised: For l = k

Original: = O (cid:18) √ log( 1 δ )
Revised: = O (cid:18) √ log( 1 δ )

Original: [Equation]
Revised: [Equation]

Original: (cid:19) , where (cid:15) ∈ (0 , 1 / 2) , setting m ≥ O ( k/δ + log( 1 δ )) , it follows that
Revised: (cid:19) , where (cid:15) ∈ (0 , 1 / 2) , setting m ≥ O ( k/δ + log( 1 δ )) , it follows that

Original: [Equation]
Revised: [Equation]

Original: 2.3 Complexity
Revised: 2.3 Complexity

Original: The complexity of Algorithm 1 is as follows.
Revised: The complexity of Algorithm 1 is as follows.

Original: In the ﬁrst step two matrices S and G must be generated with d × m random elements each.
Revised: In the ﬁrst step two matrices S and G must be generated with d × m random elements each.

Original: Hence, O ( dm ) calls to a random number generator are required.
Revised: Hence, O ( dm ) calls to a random number generator are required.

Original: In the second step, the products A (cid:62) ( AS ) and A (cid:62) ( AG ) can be both computed in
Revised: In the second step, the products A (cid:62) ( AS ) and A (cid:62) ( AG ) can be both computed in

Original: O ( T MM ( A, m ) + T MM ( A (cid:62) , m )) .
Revised: O ( T MM ( A, m ) + T MM ( A (cid:62) , m )) .

Original: Next we need to create an orthonormal basis for A (cid:62) AS which has size d × m .
Revised: Next we need to create an orthonormal basis for A (cid:62) AS which has size d × m .

Original: This can be done with a standard Householder QR or another orthogonal factorization in O ( dm 2 ) [25, Chapter 5].
Revised: This can be done with a standard Householder QR or another orthogonal factorization in O ( dm 2 ) [25, Chapter 5].

Original: The complexity of this operation can be improved using fast matrix multiplication primitives [19].
Revised: The complexity of this operation can be improved using fast matrix multiplication primitives [19].

Original: The product AQ costs O ( T MM ( A, m )) .
Revised: The product AQ costs O ( T MM ( A, m )) .

Original: We then have to compute ˜
Revised: We then have to compute ˜

Original: A ( Q (cid:62) G ) , which takes O ( dm 2 ) or O ( dm ω − 1 ) to ﬁrst obtain Q (cid:62) G
Revised: A ( Q (cid:62) G ) , which takes O ( dm 2 ) or O ( dm ω − 1 ) to ﬁrst obtain Q (cid:62) G

Original: and then the same cost to get ˜ A ( Q (cid:62) G ) .
Revised: and then the same cost to get ˜ A ( Q (cid:62) G ) .

Original: Finally, for the last step the squared row norms of 2 n vectors, the rows of ˜ A , and the rows of ˜ ∆ , are needed.
Revised: Finally, for the last step the squared row norms of 2 n vectors, the rows of ˜ A , and the rows of ˜ ∆ , are needed.

Original: For each row of ˜ A and ˜ ∆ the cost of computing the squared Euclidean norm is
Revised: For each row of ˜ A and ˜ ∆ the cost of computing the squared Euclidean norm is

Original: O ( m ) , therefore the cost for the last step is O ( nm ) .
Revised: O ( m ) , therefore the cost for the last step is O ( nm ) .

Original: Summing up, the total cost of Algorithm 1 is
Revised: Summing up, the total cost of Algorithm 1 is

Original: O ( d 2m + T MM ( A, m ) + nm ) .
Revised: O ( dm 2 + T MM ( A, m ) + nm ) .

Original:  Euclidean distances
Revised:  Euclidean distances

Original: In many applications it is desired to ﬁnd an approximate isometry for a set of data points.
Revised: In many applications it is desired to ﬁnd an approximate isometry for a set of data points.

Original: Let
Revised: Let

Original: A ∈ R n × d be a matrix whose rows deﬁne these d -dimensional data points.
Revised: A ∈ R n × d be a matrix whose rows deﬁne these d -dimensional data points.

Original: Assume we are interested to estimate all the (cid:0) n 2 (cid:1) distances between the rows of A .
Revised: Assume we are interested to estimate all the (cid:0) n 2 (cid:1) distances between the rows of A .

Original: Let B be a matrix with size (cid:0) n 2 (cid:1) × n and each row of B is equal to the vector ( e i − e j ) (cid:62) for some i, j ∈ [ n ] .
Revised: Let B be a matrix with size (cid:0) n 2 (cid:1) × n and each row of B is equal to the vector ( e i − e j ) (cid:62) for some i, j ∈ [ n ] .

Original: 1 Each row ( e i − e j ) (cid:62) of B , when multiplied with A , gives the difference vector e (cid:62) i
Revised: 1 Each row ( e i − e j ) (cid:62) of B , when multiplied with A , gives the difference vector e (cid:62) i

Original: A − e (cid:62) j A .
Revised: A − e (cid:62) j A .

Original: Therefore, to estimate the Euclidean distances between the rows of A , it is sufﬁcient to estimate the Euclidean lengths of the rows of the matrix BA .
Revised: Therefore, to estimate the Euclidean distances between the rows of A , it is sufﬁcient to estimate the Euclidean lengths of the rows of the matrix BA .

Original: In Algorithm 2 we describe this procedure for a general “incidence matrix” B , e.g., when one wants to estimate only a subset of the pairwise distances.
Revised: In Algorithm 2 we describe this procedure for a general “incidence matrix” B , e.g., when one wants to estimate only a subset of the pairwise distances.

Original: Since B has in general more rows than
Revised: Since B has in general more rows than

Original: A , the matrix multiplications must be computed in the correct order to minimize their complexity.
Revised: A , the matrix multiplications must be computed in the correct order to minimize their complexity.

Original: Bounds.
Revised: Bounds.

Original: Approximation bounds can be directly derived from Theorems 1 and 2, replacing A with
Revised: Approximation bounds can be directly derived from Theorems 1 and 2, replacing A with

Original: BA .
Revised: BA .

Original: For completeness, they can be found in Theorem 5 in the Appendix.
Revised: For completeness, they can be found in Theorem 5 in the Appendix.

Original: Complexity.
Revised: Complexity.

Original: The complexity of Algorithm 2 is as follows.
Revised: The complexity of Algorithm 2 is as follows.

Original: O ( dm ) operations are needed to generate G and S .
Revised: O ( dm ) operations are needed to generate G and S .

Original: The product ˜ S
Revised: The product ˜ S

Original: =
Revised: =

Original: A (cid:62) B (cid:62) BAS is evaluated in three steps.
Revised: A (cid:62) B (cid:62) BAS is evaluated in three steps.

Original: We ﬁrst compute AS in O ( T MM ( A, m )) , then B ( AS ) and B (cid:62) ( BAS ) in O ( nm ) , and ﬁnally A (cid:62) ( B (cid:62) BAS ) in
Revised: We ﬁrst compute AS in O ( T MM ( A, m )) , then B ( AS ) and B (cid:62) ( BAS ) in O ( nm ) , and ﬁnally A (cid:62) ( B (cid:62) BAS ) in

Original: O ( T MM ( A (cid:62) , m )) .
Revised: O ( T MM ( A (cid:62) , m )) .

Original: The intermediate products can be calculated in batches to save memory.
Revised: The intermediate products can be calculated in batches to save memory.

Original: The
Revised: The

Original: QR factorization of ˜ S requires O ( dm 2 ) .
Revised: QR factorization of ˜ S requires O ( dm 2 ) .

Original: The products ˜
Revised: The products ˜

Original: A = AQ and C = AG both require
Revised: A = AQ and C = AG both require

Original: O ( T MM ( A, m )) , whereas the product Q (cid:62) G can be performed in O ( dm 2 ) .
Revised: O ( T MM ( A, m )) , whereas the product Q (cid:62) G can be performed in O ( dm 2 ) .

Original: Accordingly, the product ˜
Revised: Accordingly, the product ˜

Original: A ( Q (cid:62) G ) needs O ( tm 2 ) and C − ˜
Revised: A ( Q (cid:62) G ) needs O ( tm 2 ) and C − ˜

Original: A ( Q (cid:62) G ) O ( tm ) .
Revised: A ( Q (cid:62) G ) O ( tm ) .

Original: In the last step each row norm costs O ( m ) operations, resulting in O ( nm ) .
Revised: In the last step each row norm costs O ( m ) operations, resulting in O ( nm ) .

Original: The total complexity of Algorithm 2 is therefore
Revised: The total complexity of Algorithm 2 is therefore

Original: [Equation]
Revised: [Equation]

Original:  Statistical leverage scores
Revised:  Statistical leverage scores

Original: We next consider the problem of approximating the leverage scores of a tall-and-skinny matrix.
Revised: We next consider the problem of approximating the leverage scores of a tall-and-skinny matrix.

Original: The leverage scores of the rows of A can be found in the diagonal of the orthogonal projector matrix
Revised: The leverage scores of the rows of A can be found in the diagonal of the orthogonal projector matrix

Original: P = AA † = UU
Revised: P = AA † = UU

Original: [Equation]
Revised: [Equation]

Original: , where U is any orthonormal basis for range( A ) .
Revised: , where U is any orthonormal basis for range( A ) .

Original: Speciﬁcally, the leverage score θ i of the i -th row of A is equal to all the following quantities
Revised: Speciﬁcally, the leverage score θ i of the i -th row of A is equal to all the following quantities

Original: [Equation]
Revised: [Equation]

Original: It is known that the leverage scores of a n × d matrix A with rank( A ) =
Revised: It is known that the leverage scores of a n × d matrix A with rank( A ) =

Original: r ≤ d sum to r : (cid:80) ni
Revised: r ≤ d sum to r : (cid:80) ni

Original: =

Revised: =

Original: θ i = r.
Revised: θ i = r.

Original: Leverage scores are important in outlier detection, graph sparsiﬁcation, and numerical linear algebra.
Revised: Leverage scores are important in outlier detection, graph sparsiﬁcation, and numerical linear algebra.

Original: We consider the general case where U is not explicitly available, and we only have access to A .
Revised: We consider the general case where U is not explicitly available, and we only have access to A .

Original: To simplify the analysis, we assume that the matrix A has full column rank.
Revised: To simplify the analysis, we assume that the matrix A has full column rank.

Original: The true rank r of A (or the numerical rank, if A is approximately low-rank) as well as a corresponding set of r linearly independent columns of A can be computed in O ( nnz ( A ) + d 4 / ( (cid:15) 2 δ )) , with provable approximation guarantees for the leverage scores of the selected column subset.
Revised: The true rank r of A (or the numerical rank, if A is approximately low-rank) as well as a corresponding set of r linearly independent columns of A can be computed in O ( nnz ( A ) + d 4 / ( (cid:15) 2 δ )) , with provable approximation guarantees for the leverage scores of the selected column subset.

Original: See sections 4 and 5 of [41] for details and [8, 12, 13] for related algorithms and lower bounds.
Revised: See sections 4 and 5 of [41] for details and [8, 12, 13] for related algorithms and lower bounds.

Original: To use Algorithm 1 to estimate leverage scores, we ﬁrst need a linear operator that computes AA † v , for an arbitrary vector v .
Revised: To use Algorithm 1 to estimate leverage scores, we ﬁrst need a linear operator that computes AA † v , for an arbitrary vector v .

Original: Since evaluating ( A (cid:62) A ) in order to compute its pseudoinverse and ultimately the orthogonal projector AA †
Revised: Since evaluating ( A (cid:62) A ) in order to compute its pseudoinverse and ultimately the orthogonal projector AA †

Original: =
Revised: =

Original: A ( A (cid:62) A ) † A (cid:62) is expensive, we opt for a fast approximate operator.
Revised: A ( A (cid:62) A ) † A (cid:62) is expensive, we opt for a fast approximate operator.

Original: For this we can use standard techniques from the literature.
Revised: For this we can use standard techniques from the literature.

Original: One of the ﬁrst approximation algorithms for tall-and-skinny leverage scores was proposed in [21].
Revised: One of the ﬁrst approximation algorithms for tall-and-skinny leverage scores was proposed in [21].

Original: In [41] it was shown that this algorithm is only efﬁcient for dense matrices, or more speciﬁcally for matrices with at least ω (log n ) nonzeros per row.
Revised: In [41] it was shown that this algorithm is only efﬁcient for dense matrices, or more speciﬁcally for matrices with at least ω (log n ) nonzeros per row.

Original: Given A ∈ R n × d with n (cid:29) d and ω (log n ) nonzeros per row, the idea consists of approximating the leverage scores of the rows of A with the squared Euclidean row norms of the matrix
Revised: Given A ∈ R n × d with n (cid:29) d and ω (log n ) nonzeros per row, the idea consists of approximating the leverage scores of the rows of A with the squared Euclidean row norms of the matrix

Original: A (Π 1 A )
Revised: A (Π 1 A )

Original: † Π 2 .
Revised: † Π 2 .

Original: Here, Π 1 is a subspace embedding for range( A ) and Π 2 is an (cid:15) -JLT.
Revised: Here, Π 1 is a subspace embedding for range( A ) and Π 2 is an (cid:15) -JLT.

Original: It can be proved that (Π 1 A ) † is in fact an approximate “orthogonalizer” for A , a property that we can leverage in our algorithm.
Revised: It can be proved that (Π 1 A ) † is in fact an approximate “orthogonalizer” for A , a property that we can leverage in our algorithm.

Original: Speciﬁcally, we apply Algorithm 1 to approximate the Euclidean row norms of A (Π 1 A ) † , instead of multiplying with Π 2 .
Revised: Speciﬁcally, we apply Algorithm 1 to approximate the Euclidean row norms of A (Π 1 A ) † , instead of multiplying with Π 2 .

Original: This procedure is described in Algorithm 3.
Revised: This procedure is described in Algorithm 3.

Original: 8: return Alg3 ( A, i ) =
Revised: 8: return Alg3 ( A, i ) =

Original: ˜ θ i
Revised: ˜ θ i

Original: = (cid:107) ( e (cid:62) i B )
Revised: = (cid:107) ( e (cid:62) i B )

Original: ˜
Revised: ˜

Original: A (cid:107) 2 + (cid:107) ( e (cid:62) i B ) ˜∆ (cid:107) 2 , for all i ∈ [ n ] .
Revised: A (cid:107) 2 + (cid:107) ( e (cid:62) i B ) ˜∆ (cid:107) 2 , for all i ∈ [ n ] .

Original: (cid:46) O ( nm )
Revised: (cid:46) O ( nm )

Original: The following theorem gives approximation bounds for the leverage scores returned by Algorithm 3.
Revised: The following theorem gives approximation bounds for the leverage scores returned by Algorithm 3.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let A ∈ R n × d , θ
Revised: Let A ∈ R n × d , θ

Original: i
Revised: i

Original: =
Revised: =

Original: (cid:107) e
Revised: (cid:107) e

Original: (cid:62) i AA † (cid:107) 2 and ˜ θ
Revised: (cid:62) i AA † (cid:107) 2 and ˜ θ

Original: i the values returned by Algorithm 3.
Revised: i the values returned by Algorithm 3.

Original: The following hold:
Revised: The following hold:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let ˆ θ i
Revised: Let ˆ θ i

Original: = (cid:107) e (cid:62) i
Revised: = (cid:107) e (cid:62) i

Original: A (Π 1 A ) † (cid:107) 2 , so that
Revised: A (Π 1 A ) † (cid:107) 2 , so that

Original: [Equation]
Revised: [Equation]

Original: From [21, Lemma 9] it follows that | ˆ θ i − θ i | ≤ (cid:15) 1 1 − (cid:15) 1 θ i .
Revised: From [21, Lemma 9] it follows that | ˆ θ i − θ i | ≤ (cid:15) 1 1 − (cid:15) 1 θ i .

Original: Subsequently, ˆ θ i ≤ (1 + (cid:15) 1 1 − (cid:15) 1 ) θ i = 1 1 − (cid:15) 1 θ i .
Revised: Subsequently, ˆ θ i ≤ (1 + (cid:15) 1 1 − (cid:15) 1 ) θ i = 1 1 − (cid:15) 1 θ i .

Original: From Theorem 1 we recall that for appropriate m, l, k,
Revised: From Theorem 1 we recall that for appropriate m, l, k,

Original: [Equation]
Revised: [Equation]

Original: Combining all these observations we ﬁnd that
Revised: Combining all these observations we ﬁnd that

Original: [Equation]
Revised: [Equation]

Original: Rescaling (cid:15) 1 and (cid:15) 2 gives the element-wise bounds.
Revised: Rescaling (cid:15) 1 and (cid:15) 2 gives the element-wise bounds.

Original: For the Frobenius norm bounds we can use similar arguments in combination with Theorem 2.
Revised: For the Frobenius norm bounds we can use similar arguments in combination with Theorem 2.

Original: Complexity and choice of (cid:15) 1 , (cid:15) 2 .
Revised: Complexity and choice of (cid:15) 1 , (cid:15) 2 .

Original: The complexity of Algorithm 3 can be split into two parts: ( i ) the complexity of obtaining an (cid:15) 1 -approximate orthonormal basis for A , and ( ii ) the complexity of estimating the row norms of this basis.
Revised: The complexity of Algorithm 3 can be split into two parts: ( i ) the complexity of obtaining an (cid:15) 1 -approximate orthonormal basis for A , and ( ii ) the complexity of estimating the row norms of this basis.

Original: The complexity of the former has been heavily studied in the literature and depends on the choice of the subspace embedding.
Revised: The complexity of the former has been heavily studied in the literature and depends on the choice of the subspace embedding.

Original: For very tall-and-skinny matrices, an efﬁcient construction is to use a combination of a CountSketch [15, 35], a Subsampled Randomized
Revised: For very tall-and-skinny matrices, an efﬁcient construction is to use a combination of a CountSketch [15, 35], a Subsampled Randomized

Original: Hadamard Transform (SRHT) [43] and a Gaussian subspace embedding; see e.g. [15].
Revised: Hadamard Transform (SRHT) [43] and a Gaussian subspace embedding; see e.g. [15].

Original: This provides a sketch Π 1 A with dimension O ( d/(cid:15) 2 ) × d in T (Π 1 A ) time.
Revised: This provides a sketch Π 1 A with dimension O ( d/(cid:15) 2 ) × d in T (Π 1 A ) time.

Original: Computing the QR factorization of the sketch requires O ( d 3 /(cid:15) 2 ) to obtain R .
Revised: Computing the QR factorization of the sketch requires O ( d 3 /(cid:15) 2 ) to obtain R .

Original: Since R is upper triangular, the computation of R − 1 G and
Revised: Since R is upper triangular, the computation of R − 1 G and

Original: R − 1 Q both take O ( d 2 m ) .
Revised: R − 1 Q both take O ( d 2 m ) .

Original: The products A ( R − 1 Q ) and A ( R − 1 G ) cost O ( T MM ( A, m )) each.
Revised: The products A ( R − 1 Q ) and A ( R − 1 G ) cost O ( T MM ( A, m )) each.

Original: The last step takes O ( nm ) .
Revised: The last step takes O ( nm ) .

Original: The total complexity is
Revised: The total complexity is

Original: O ( T (Π 1 A ) + d 3 /(cid:15) 21 + T MM
Revised: O ( T (Π 1 A ) + d 3 /(cid:15) 21 + T MM

Original: ( A, m ) + nm ) .
Revised: ( A, m ) + nm ) .

Original: To achieve (cid:15) -accuracy in the Frobenius norm, it sufﬁces to use m =
Revised: To achieve (cid:15) -accuracy in the Frobenius norm, it sufﬁces to use m =

Original: O ( (cid:112) log( n/δ ) /(cid:15) 2 ) .
Revised: O ( (cid:112) log( n/δ ) /(cid:15) 2 ) .

Original: With (cid:15) 1 = (cid:15) 2 = (cid:15) , the total complexity becomes
Revised: With (cid:15) 1 = (cid:15) 2 = (cid:15) , the total complexity becomes

Original: [Equation]
Revised: [Equation]

Original: If, instead, we use standard JL projections to estimate the row norms of AR − 1 , the total complexity is
Revised: If, instead, we use standard JL projections to estimate the row norms of AR − 1 , the total complexity is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) , to achieve the same Frobenius norm-wise accuracy.
Revised: ) , to achieve the same Frobenius norm-wise accuracy.

Original: For any matrix which is “tall-enough” such that the O ( T MM ( A, log( n/δ ) /(cid:15) 2 )) factor dominates the complexity, Algorithm 3 achieves a quadratic improvement over standard estimators.
Revised: For any matrix which is “tall-enough” such that the O ( T MM ( A, log( n/δ ) /(cid:15) 2 )) factor dominates the complexity, Algorithm 3 achieves a quadratic improvement over standard estimators.

Original:  Numerical experiments
Revised:  Numerical experiments

Original: Algorithm 1 was implemented in Python using NumPy.
Revised: Algorithm 1 was implemented in Python using NumPy.

Original: We conducted experiments to verify the approximation guarantees and the convergence improvements against standard Gaussian random projections.
Revised: We conducted experiments to verify the approximation guarantees and the convergence improvements against standard Gaussian random projections.

Original: Following [34], we generated synthetic matrices with decay in the spectrum.
Revised: Following [34], we generated synthetic matrices with decay in the spectrum.

Original: Speciﬁcally, d × d matrices A , with d = 5000 , were created as follows.
Revised: Speciﬁcally, d × d matrices A , with d = 5000 , were created as follows.

Original: We drew a random orthogonal d × d matrix Q .
Revised: We drew a random orthogonal d × d matrix Q .

Original: We then ﬁxed a diagonal d × d matrix Λ which deﬁnes the eigenvalues of the matrix.
Revised: We then ﬁxed a diagonal d × d matrix Λ which deﬁnes the eigenvalues of the matrix.

Original: Each element Λ i,i , i ∈
Revised: Each element Λ i,i , i ∈

Original: [ d ] is set to i − c for a given c ≥ 0 .
Revised: [ d ] is set to i − c for a given c ≥ 0 .

Original: The larger the c , the faster the spectral decay.
Revised: The larger the c , the faster the spectral decay.

Original: We ﬁnally constructed the symmetric A = Q Λ Q (cid:62) which were used in the numerical experiments.
Revised: We ﬁnally constructed the symmetric A = Q Λ Q (cid:62) which were used in the numerical experiments.

Original: Following [34], we applied four different decay factors, speciﬁcally c
Revised: Following [34], we applied four different decay factors, speciﬁcally c

Original: = { 0 . 5 , 1 , 1 . 5 , 2 } .
Revised: = { 0 . 5 , 1 , 1 . 5 , 2 } .

Original: The approximation errors of standard JL projections versus Algorithm 1 are compared in Figure 1.
Revised: The approximation errors of standard JL projections versus Algorithm 1 are compared in Figure 1.

Original: We plot the approximation errors of both methods as the number of samples increases.
Revised: We plot the approximation errors of both methods as the number of samples increases.

Original: We plot two types of errors, the maximum element-wise and the Frobenius norm-wise errors
Revised: We plot two types of errors, the maximum element-wise and the Frobenius norm-wise errors

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where ˜ x i are the approximated row norms and ˜ X is their sum returned by either Algorithm 1 or standard JL projections.
Revised: where ˜ x i are the approximated row norms and ˜ X is their sum returned by either Algorithm 1 or standard JL projections.

Original: The exact same number of matrix vector queries is used in both methods.
Revised: The exact same number of matrix vector queries is used in both methods.

Original: Standard JL projections involve only one random matrix G , which is multiplies A from the right.
Revised: Standard JL projections involve only one random matrix G , which is multiplies A from the right.

Original: G has size d × m , m being the number of samples.
Revised: G has size d × m , m being the number of samples.

Original: In Algorithm 1, on the other hand, A is multiplied four times with a matrix from the right.
Revised: In Algorithm 1, on the other hand, A is multiplied four times with a matrix from the right.

Original: Therefore, we set G, S, and Q in Algorithm 1 to have size d × m/ 4 , so that both algorithms are tested with the same number of matrix-vector products.
Revised: Therefore, we set G, S, and Q in Algorithm 1 to have size d × m/ 4 , so that both algorithms are tested with the same number of matrix-vector products.

Original: In each plot we illustrate the mean error over 10 independent runs and the standard deviation.
Revised: In each plot we illustrate the mean error over 10 independent runs and the standard deviation.

Original: Standard JL approximations perform marginally better than Algorithm 1 only for the element-wise errors and only for the matrix with very slow decay.
Revised: Standard JL approximations perform marginally better than Algorithm 1 only for the element-wise errors and only for the matrix with very slow decay.

Original: In all other cases, Algorithm 1 performs signiﬁcantly better.
Revised: In all other cases, Algorithm 1 performs signiﬁcantly better.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed an adaptive algorithm to estimate the Euclidean row norms of a matrix A .
Revised: We proposed an adaptive algorithm to estimate the Euclidean row norms of a matrix A .

Original: This algorithm improves standard Johnson-Lindenstrauss estimators in the following aspects: ( i ) Quadratically less matrix-vector queries are required to achieve the same Frobenius norm-wise accuracy for all matrices; ( ii ) Asymptotically less matrix-vector queries are needed to achieve the same element-wise accuracy for matrices with decaying spectrum; ( iii ) At least as accurate element-wise approximations as standard JL are achieved for worst-case input matrices, that is, for matrices with ﬂat spectrum.
Revised: This algorithm improves standard Johnson-Lindenstrauss estimators in the following aspects: ( i ) Quadratically less matrix-vector queries are required to achieve the same Frobenius norm-wise accuracy for all matrices; ( ii ) Asymptotically less matrix-vector queries are needed to achieve the same element-wise accuracy for matrices with decaying spectrum; ( iii ) At least as accurate element-wise approximations as standard JL are achieved for worst-case input matrices, that is, for matrices with ﬂat spectrum.

Original: We also showed how these results can be applied to other important problems, speciﬁcally to estimate
Revised: We also showed how these results can be applied to other important problems, speciﬁcally to estimate

Original: Euclidean distances between data points, which is related to the fundamental concept of approximate isometries that has many applications in data science, as well as for statistical leverage scores estimations, which are ubiquitous quantities not only in data science and statistics, but also in numerical linear algebra and spectral graph theory.
Revised: Euclidean distances between data points, which is related to the fundamental concept of approximate isometries that has many applications in data science, as well as for statistical leverage scores estimations, which are ubiquitous quantities not only in data science and statistics, but also in numerical linear algebra and spectral graph theory.

Original: As future work, several directions can be envisioned.
Revised: As future work, several directions can be envisioned.

Original: Most prominently, it would be interesting to determine whether the studied techniques can be used to improve Oblivious Subspace Embeddings [40, 44].
Revised: Most prominently, it would be interesting to determine whether the studied techniques can be used to improve Oblivious Subspace Embeddings [40, 44].

Original: Such improvements would have an immediate impact in many problems in NLA, e.g. least squares regression, low-rank approximations and column subset selection.
Revised: Such improvements would have an immediate impact in many problems in NLA, e.g. least squares regression, low-rank approximations and column subset selection.

Original: Two other relevant topics concern ( a ) the possibility to derive lower bounds similar to [34] for Euclidean row norms estimation and ( b ) to make the algorithms non-adaptive, like the non-adaptive versions of Hutch++ [28, 34] which are based on results from [14], or the Nyström++ of [37].
Revised: Two other relevant topics concern ( a ) the possibility to derive lower bounds similar to [34] for Euclidean row norms estimation and ( b ) to make the algorithms non-adaptive, like the non-adaptive versions of Hutch++ [28, 34] which are based on results from [14], or the Nyström++ of [37].

Original: [Figure]
Revised: [Figure]

Original: Acknowledgements
Revised: Acknowledgements

Original: The authors would like to thank Cameron Musco for helpful commentsI Background in Linear Algebra
Revised: The authors would like to thank Cameron Musco for helpful comments

Original: In this section we state some elementary results that we will use for our main proofs.

Original: I.1 Johnson-Lindenstrauss and subspace embeddings

Original: A useful deﬁnition for our proofs is the JL moment property, which bounds the moments of the length of Sx .

Original: Deﬁnition 3 (JL moment property, [30]) .

Original: A distribution D on matrices S ∈ R k × d has the ( (cid:15), δ, l ) -JL moment property if given S ∼ D , for all x ∈ R d with (cid:107) x (cid:107) 2 = 1 it holds that E (cid:104)(cid:12)(cid:12) (cid:107) Sx (cid:107) 2 − 1 (cid:12)(cid:12) l (cid:105) ≤ (cid:15) l δ.

Original: We mention a corollary from [40] which states that JLTs also preserve pairwise angles, which is an important by-product that we will use in our proofs.

Original: Corollary 2 ([40]) .

Original: If S is an ( (cid:15), δ, n ) -JLT, then for any ﬁxed n -element set V ⊂ R d , with probability at least 1 − δ it holds that (cid:104) u, v (cid:105) − (cid:15) (cid:107) u (cid:107)(cid:107) v (cid:107) ≤ (cid:104) Su, Sv (cid:105) ≤ (cid:104) u, v (cid:105) + (cid:15) (cid:107) u (cid:107)(cid:107) v (cid:107) , for all pairs u, v ∈ V .

Original: The next Lemma is part of the proof of [44, Lemma 4.2], which we state here as a separate result to save some space from the longer proofs that follow later.

Original: Lemma 4.

Original: Let S be a ( (cid:15), δ ) -OSE for a d × k matrix U k with orthonormal columns, k < d .

Original: Then there exists a k × k invertible matrix C such that ( SU k ) † =

Original: C − 1 ( SU k ) (cid:62) and (cid:107) C − 1 (cid:107) ≤ 11 − (cid:15) with probability at least 1 − δ .

Original: Proof.

Original: This is part of the proof of [44, Lemma 4.2].

Original: We also repeat here Lemma 4.2 from [44] regarding low-rank approximations as it will be used in upcoming proofs.

Original: Lemma 5 (Restatement Lemma 4.2 in [44]) .

Original: Let S be an (1 / 3 , δ ) -OSE for a ﬁxed k -dimensional subspace and let S also satisfy the ( (cid:112) (cid:15)/k, δ, l ) -JL moment property for some l ≥ 2 .

Original: Then the rowspace of SA , for some A , contains a (1 + (cid:15) )

Original: rank-

Original: k approximation to A , that is, there exists a k -dimensional subspace M within rowspace( SA ) with an orthogonal projector matrix Π k such that

Original: [Equation]

Original: I.2 Properties of Gaussian matrices

Original: In this section we specialize the deﬁnitions to the case of Gaussian matrices.

Original: In our analysis we will use the following restatement of JLTs, addressing the error as a function of the rows of G .

Original: Corollary 3.

Original: Let G ∈ R r × d with i.i.d.

Original: elements from N (0 , 1 / √ r ) and (cid:15) ∈ (0 , 1 / 2) .

Original: For a set

Original: X ⊂ R d of n vectors and for δ ∈ (0 , 1 / 2) , as long as r > 32 log(2 n/δ ) , then with probability at least 1 − δ for all x ∈ X it holds that

Original: [Equation]

Original: x (cid:107)

Original: [Equation]

Original: Proof.

Original: From Lemma 1 we have that

Original: [Equation]

Original: From (cid:15) ∈ (0 , 1 / 2) it follows that

Original: [Equation]

Original: As long as r > 32 log(2 n/δ ) , we can set (cid:15) =

Original: (cid:113) 8log(2 n/δ ) r

Original: and then replace (cid:15) in the exponent to obtain exp (cid:16) − r ( (cid:15) 2 ) 8 (cid:17) =

Original: exp ( − log(2 n/δ ))

Original: =

Original: δ/ (2 n ) .

Original: Applying a union bound we ﬁnd that, as long as r > 32 log(2 n/δ ) , for all x ∈ X simultaneously, it holds that

Original: [Equation]

Original: with probability at least 1 − δ .

Original: Next, we also state the required dimension for a scaled Gaussian matrix to satisfy the ( (cid:15), δ, 2) -JL moment property and the ( (cid:15), δ ) -OSE property.

Original: Lemma 6.

Original: Let G be a r × d matrix with i.i.d elements from N (0 , 1 / √ r ) , and (cid:15), δ ∈ (0 , 1 / 2) .

Original: If r ≥ 2 / ( (cid:15) 2 δ ) , then G satisﬁes the ( (cid:15), δ, 2) -JL moment property.

Original: Proof.

Original: From Deﬁnition 3, it must be shown that E ( (cid:107) Gx

Original: [Equation]

Original: 22 −

Original: [Equation]

Original: 2 = E

Original: [Equation]

Original: Gx

Original: [Equation]

Original: 42 −

Original: [Equation]

Original: E

Original: [Equation]

Original: Gx

Original: [Equation]

Original: 22 + 1 ≤ (cid:15) 2 δ .

Original: From the rotation invariance property of the Gaussian distribution, Gx can be replaced with a vector g ∈ R r with i.i.d elements from N (0 , 1 / √ r ) .

Original: We then calculate

Original: [Equation]

Original: as well as

Original: [Equation]

Original: | g i | 4 ) + E r (cid:88) i = (cid:88)

Original: j (cid:54) =

Original: i | g

Original: i | 2 | g j |= r (cid:88) i =

Original: E ( | g i | 4 ) + r (cid:88) i = (cid:88)

Original: j

Original: (cid:54) =

Original: i

Original: E | g i | 2 | g j |= r (cid:88) i = 3 /r 2 + r (cid:88) i = (cid:88)

Original: j (cid:54) =

Original: i 1 /r= 3 /r +

Original: ( r − 1)

Original: /r =

Original: ( r + 2) /r.

Original: Putting everything together we obtain

Original: [Equation]

Original: (cid:15)

Original: [Equation]

Original: The next theorem from [44] states the required dimension for a Gaussian matrix to provide an ( (cid:15), δ ) -OSE for a ﬁxed k -dimensional subspace.

Original: Theorem 4 (Thm. 2.3 in [44]) .

Original: Let G ∈ R r × d with i.i.d elements from N (0 , 1 √ r ) , and (cid:15), δ ∈ (0 , 1 / 2) .

Original: If r = Θ (cid:16) k +log(1 /δ )

Original: (cid:15)(cid:17) , then G is an ( (cid:15), δ ) -OSE for any ﬁxed k -dimensional subspace.

Original: I.3 Basic inequalities

Original: The next lemma gives a bound on powers of logarithms which we will use to simplify the terms in several proofs.

Original: Lemma 7.

Original: Let n ≥ 1 be an integer, and δ ∈ (0 , 1 / 2) .

Original: Then for any constant c ≥ 2 it holds that log( c ( n/δ )) ≤ (1 + log 2 c ) log( n/δ ) .

Original: Proof.

Original: log( c nδ ) =

Original: log(2 log 2 c ( nδ )) ≤ log(( 1 δ ) log 2 c ( nδ ))

Original: =

Original: log( n δ

Original: [Equation]

Original: c ) .

Original: II Limitations of low-rank projections

Original: We give a “problematic” example to demonstrate the limitations of low-rank projection based methods.

Original: Let A = I d and assume we want to use a rank- k approximation to estimate the norms of the rows of A for some k < d .

Original: That is, we want to ﬁnd a matrix Q k with k orthonormal columns such that the quantities (cid:107) e (cid:62) i ( A − AQ k Q (cid:62) k ) (cid:107) 22 are small.

Original: More formally, Q k should be a minimizer for the maximum norm of all

Original: i ∈

Original: [ d ] ,

Original: [Equation]

Original: Since the maximum is larger than or equal to the average, then for every Q we have that

Original: [Equation]

Original: where the last equality comes from the fact that A = I d by assumption and therefore (cid:107) I − QQ (cid:62) (cid:107) F = √ d − k since I − QQ (cid:62) is an orthogonal projector on a ( d − k ) -dimensional subspace.

Original: Subsequently,

Original: [Equation]

Original: If there exists a Q that satisﬁes this lower bound, then it is also a minimizer.

Original: For simplicity let us assume that k divides d exactly.

Original: We can construct Q k as d/k copies of a k × k orthogonal matrix

Original: H , that is, Q (cid:62) k

Original: = (cid:0) H (cid:62)

Original: H (cid:62)

Original: H (cid:62) ...

Original: H (cid:62) (cid:1) .

Original: This way Q k Q (cid:62) k is a matrix with d/k diagonals in equally spaced positions.

Original: We can also scale Q k Q (cid:62) k by k/d to ensure that it is an orthogonal projector.

Original: All rows of AQ k Q (cid:62) k have the same length, equal to (cid:107) e (cid:62) i Q k Q (cid:62)

Original: k (cid:107) 2 = k/d .

Original: Therefore, (cid:107) e (cid:62) i

Original: [-( I − Q k Q (cid:62) k ) (cid:107) 
-]
Revised: [-( I − Q k Q (cid:62) k ) (cid:107) 
-]

Original: = 1 − k/d = ( d − k )

Original: /d , for all i ∈ [ d ] , meaning that Q k is indeed a minimizer.

Original: It is evident that we need k to be almost equal to d in order for this quantity to be small.

Original: Speciﬁcally, if we want to achieve (cid:107) e (cid:62)

Original: i ( I − Q k Q (cid:62) k ) (cid:107) 2 ≤ (cid:15) , then we need ( d − k ) /d ≤ (cid:15) ⇔ d − k ≤ (cid:15)d ⇔ k ≥ d (1 − (cid:15) ) .

Original: Hence, to obtain a small (cid:15) -accuracy k must be set almost as large as d .

Original: We can ﬁnally conclude that there exist corner cases where O ( d ) samples are needed to achieve (cid:15) -accuracy for element-wise Euclidean norm estimation based on low rank projections.

Original: III Projecting rows on randomly chosen subspaces

Original: We state the following main lemma regarding projections on randomly chosen low-rank subspaces.

Original: Lemma 8 (Oblique projection from rowspace( A ) on rowspace( SA (cid:62) A ) ) .

Original: Let A ∈ R n × d , (cid:15), δ, (cid:15) (cid:48) , δ (cid:48) ∈ (0 , 1 / 2) be input parameters, and S have the following properties:

Original: • S ∼ D , where D is an ( (cid:15), δ ) -OSE for any ﬁxed k -dimensional subspace; • S is an ( (cid:15) (cid:48) , δ (cid:48) , 2 n ) -JLT.

Original: Then there exists a rank- k projection matrix Π k within rowspace( SA (cid:62) A ) such that for all i ∈ [ n ] simultaneously, it holds that

Original: [Equation]

Original: with probability at least 1 − δ − δ (cid:48) .

Original: Proof.

Original: Let A k

Original: = U k Σ k V (cid:62) k be the compact SVD of A k .

Original: The existence of Π k is proved by construction, that is, we consider the matrix

Original: Π k = V k ( SV k Σ 2 k ) † SA (cid:62) A .

Original: Clearly, this Π k is a rank- k matrix within rowspace( SA (cid:62) A ) .

Original: It is also straightforward to verify that it is a projector matrix, since

Original: [Equation]

Original: Clearly

Original: [Equation]

Original: I − Π k is also a projector since ( I −

Original: [Equation]

Original: k

Original: [Equation]

Original: We can then prove the lemma by starting from the left-hand-side of the target inequality.

Original: We write (cid:107) e

Original: [Equation]

Original: i

Original: [Equation]

Original: by linearity of the trace)

Original: [Equation]

Original: by the

Original: [Equation]

Original: [Equation]

Original: cyclic property)

Original: [Equation]

Original: where the last equality comes from Lemma 4.

Original: To conclude the proof, it sufﬁces to determine a bound for the rightmost term.

Original: To do this, notice that | e (cid:62)

Original: i AV k Σ 2 k C − 1 V (cid:62) k S

Original: [Equation]

Original: [-S ¯ V k ¯Σ 
-]
Revised: [-S ¯ V k ¯Σ 
-]

Original: k ¯ V (cid:62)

Original: k

Original: A (cid:62) e i | is the absolute value of the inner product (cid:104) S

Original: [Equation]

Original: ( ¯ V k ¯ V

Original: [Equation]

Original: i (cid:105) , which can be written in a simpliﬁed form as (cid:104)

Original: Sx k , S ¯ x k (cid:105) .

Original: Therefore, we must bound the inner products between vectors from a speciﬁc set.

Original: We consider the following set of vectors:

Original: [Equation]

Original: Clearly, | V | = 2 n .

Original: Recall that by assumption S is an ( (cid:15) (cid:48) , δ (cid:48) , 2 n ) -JLT.

Original: Therefore, by Corollary 2, S preserves all inner products between all pairs of vectors in V with probability at least 1 − δ (cid:48) .

Original: We have

Original: [Equation]

Original: Combining the two inequalities

Original: we ﬁnd that

Original: [Equation]

Original: [Table]

Original: A Proofs for Section 2 (Analysis of Algorithm 1)

Original: Lemma 2.

Original: Let A ∈ R n × d .

Original: If we use Algorithm 1 with m matrix-vector queries to estimate the

Original: Euclidean lengths of the rows e (cid:62) i

Original: A , i ∈ [ n ] , then as long as m ≥ l ≥ 32 log(4 n/δ ) it holds that

Original: [Equation]

Original: for all i ∈ [ n ] , with probability at least 1 − δ for all i ∈ [ n ] simultaneously.

Original: Proof.

Original: We start by noting that

Original: [Equation]

Original: Since G is a scaled Gaussian matrix with l columns, we can use Corollary 3, which implies that as long as l ≥ 32 log(4 n/δ ) it holds that

Original: [Equation]

Original: with probability at least 1 − δ for all i ∈ [ n ] .

Original: Corollary 1 (Projection on rowspace( SA (cid:62) A ) ) .

Original: Let δ ∈ (0 , 12 ) , ¯ A k

Original: =

Original: A − A k , and S be such that

Original: (i) S ∼ D , where D is an (1 / 3 , δ ) -OSE for any ﬁxed k -dimensional subspace; (ii) S is a (1 / 3 , δ, 2 n ) -JLT.

Original: If Q is a matrix that forms an orthonormal basis for rowspace( SA (cid:62) A ) , then, with probability at least 1 − 2 δ , for all i ∈ [ n ] simultaneously, it holds that

Original: [Equation]

Original: e (cid:62) i ¯ A k (cid:107) .

Original: Proof.

Original: From Lemma 8 we have that inside rowspace( SA (cid:62) A ) there exists a subspace with the desired properties.

Original: An orthogonal projection on that subspace via QQ (cid:62) is sufﬁcient to satisfy the results of

Original: Lemma 8 for (cid:15) =

Original: (cid:15) (cid:48)

Original: = 1 / 3 .

Original: The second part comes from the fact that (cid:107) e (cid:62) i

Original: ( A − A k ) (cid:107) ≤ (cid:107) e (cid:62) i

Original: A (cid:107) and (cid:107) e (cid:62) i

Original: A k (cid:107) ≤ (cid:107) e (cid:62) i

Original: A (cid:107) , as well as the fact that σ k +1 ( A ) ≤ σ k

Original: ( A ) .

Original: With a union bound we have that both Lemma 4 and Corollary 2 hold at the same time with probability at least 1 − 2 δ .

Original: Theorem 1.

Original: Let A ∈ R n × d and n ≥ d .

Original: If we use Algorithm 1 with m matrix-vector queries to estimate the Euclidean lengths of the rows of A , then there exists a global constant C such that, as long as

Original: (i) m ≥ l ≥

Original: O (log( n/δ ))

Original: , such that G satisﬁes Lemma 1 and S forms an (1 / 3 , δ, 2 n ) -JLT,

Original: (ii) m ≥ O

Original: ( k +

Original: log δ )) , such that S forms an (1 / 3 , δ ) -OSE for a k -dimensional subspace, then it holds that

Original: [Equation]

Original: for all i ∈ [ n ] with probability at least 1 − 3 δ .

Original: Proof.

Original: For the ﬁrst inequality, we start directly from Lemma 2, which gives a ﬁrst bound for the approximated values.

Original: We can then use Corollary 1 to bound the rightmost term.

Original: By assumption, S satisﬁes the conditions of Corollary 1, which implies that

Original: [Equation]

Original: with probability at least 1 − 2 δ .

Original: Combining this with Equation (2) we have that

Original: [Equation]

Original: We then recall Lemma 7 to move the constants outside the logarithms.

Original: There are three random events and each of them fails with probability at most δ .

Original: Therefore, by taking a union bound we ﬁnd that the algorithm succeeds with probability at least 1 − 3 δ .

Original: The last inequality of the theorem comes from

Original: Lemma 3, which gives bounds for (cid:107) e (cid:62) i ( A − A k ) (cid:107) .

Original: Theorem 2.

Original: In Algorithm 1, for some absolute constants c, C , if l > c log(1 /δ ) , it holds that

Original: [Equation]

Original: where ˜ X is the sum of the returned approximations.

Original: For l = k

Original: = O (cid:18)

Original: [Equation]

Original: (cid:19) , where (cid:15) ∈ (0 , 1 / 2) , setting m ≥ O ( k/δ + log( 1 δ )) , it follows that

Original: [Equation]

Original: Proof.

Original: The proof is a direct application of [34, Theorem 3.1].

Original: We state it for completeness.

Original: Let ˜

Original: A, ∆ , Q, G, S be as in Algorithm 1, that is, ˜

Original: A = AQQ (cid:62) , ∆ =

Original: A ( I − QQ (cid:62) ) , and Q is an orthonormal basis for range( A (cid:62) AS ) .

Original: Because

Original: [Equation]

Original: we can then write

Original: [Equation]

Original: This is identical to using the Hutch++ Algorithm [34] on A (cid:62) A instead of A .

Original: Note that A (cid:62) A is always symmetric and positive semi-deﬁnite.

Original: It can be shown that the following two conditions hold, which allows to apply [34, Theorem 3.1]:

Original: [Equation]

Original: and

Original: [Equation]

Original: The ﬁrst condition, Equation 3, is straightforward to prove

Original: [Equation]

Original: the trace cyclic property)

Original: [Equation]

Original: For the second condition

Original: [Equation]

Original: we have

Original: [Equation]

Original: From Lemma 5, we know that as long as Q has Ω( k/δ + log(1 /δ ))

Original: columns, then

Original: [Equation]

Original: holds with probability at least 1 − δ .

Original: Therefore, both conditions of [34, Theorem 3.1] are satisﬁed with probability at least 1 − δ , which implies that there exist constants c, C such that, if l > c log(1 /δ ) , the quantity

Original: [Equation]

Original: satisﬁes:

Original: [Equation]

Original: With a trivial union bound both events of the proof hold with probability at least 1 − 2 δ .

Original: Rescaling δ concludes the proof.

Original: B Proofs for Section 3 (Euclidean distances)

Original: Theorem 5.

Original: Let A ∈ R t × d , t ≥ d , for which we want to estimate the distances between pairs of rows.

Original: Let B ∈ R n × t be a matrix such that each row of B is equal to ( e i − e j ) (cid:62) , meaning that ( e i − e j ) (cid:62) A gives the distance vector between the i -th and the j -th row of A which we want to estimate.

Original: Let

Original: M = BA .

Original: If we apply Algorithm 2 with m matrix-vector queries to estimate the Euclidean lengths of rows of M by the values ˜ x

Original: i , i ∈ [ n ] , then there exists a global constant C such that, as long as

Original: (i) m

Original: ≥

Original: l

Original: ≥ O log n/

Original: δ )

Original: ) , such that G satisﬁes Lemma 1 and S forms an

Original: 1 / 3 , δ,

Original: 2 n

Original: ) -JLT,

Original: (ii)

Original: m ≥

Original: O k

Original: +

Original: log

Original: /δ )) , such that S forms an (1 / 3 , δ ) -OSE for a k -dimensional subspace, then it holds that

Original: [Equation]

Original: for all i ∈ [ n ] , with probability at least 1 − 3 δ .

Original: In addition, for some absolute constants c, C , if l > c log(1 /δ ) , it holds that

Original: [Equation]

Original: where ˜ X is the sum of the approximations.

Original: For l = k

Original: = O (cid:18) √ log

Original: [Equation]

Original: [Equation]

Original: (cid:19) , where (cid:15) ∈ (0 , 1 / 2) , setting m ≥ O ( k/δ + log( 1 δ )) , it follows that

Original: [Equation]

Original: [Table]

Original: Proof.

Original: The bounds are a direct application of Theorems 1 and 2 on BA instead of A .