Original: O N
Revised: O N

Original: T HE
Revised: T HE

Original: R ELATIVE
Revised: R ELATIVE

Original: E RROR
Revised: E RROR

Original: OF
Revised: OF

Original: R ANDOM
Revised: R ANDOM

Original: F OURIER
Revised: F OURIER

Original: F EATURES FOR P RESERVING K ERNEL D ISTANCE
Revised: F EATURES FOR P RESERVING K ERNEL D ISTANCE

Original: A BSTRACT
Revised: A BSTRACT

Original: The method of random Fourier features (RFF), proposed in a seminal paper by
Revised: The method of random Fourier features (RFF), proposed in a seminal paper by

Original: Rahimi and Recht (NIPS’07), is a powerful technique to find approximate lowdimensional representations of points in (high-dimensional) kernel space, for shift-invariant kernels.
Revised: Rahimi and Recht (NIPS’07), is a powerful technique to find approximate lowdimensional representations of points in (high-dimensional) kernel space, for shift-invariant kernels.

Original: While RFF has been analyzed under various notions of error guarantee, the ability to preserve the kernel distance with relative error is less understood.
Revised: While RFF has been analyzed under various notions of error guarantee, the ability to preserve the kernel distance with relative error is less understood.

Original: We show that for a significant range of kernels, including the well-known Laplacian kernels, RFF cannot approximate the kernel distance with small relative error using low dimensions.
Revised: We show that for a significant range of kernels, including the well-known Laplacian kernels, RFF cannot approximate the kernel distance with small relative error using low dimensions.

Original: We complement this by showing as long as the shift-invariant kernel is analytic, RFF with poly( ε − 1 log n ) dimensions achieves ε -relative error for pairwise kernel distance of n points, and the dimension bound is improved to poly( ε − 1 log k ) for the specific application of kernel k -means.
Revised: We complement this by showing as long as the shift-invariant kernel is analytic, RFF with poly( ε − 1 log n ) dimensions achieves ε -relative error for pairwise kernel distance of n points, and the dimension bound is improved to poly( ε − 1 log k ) for the specific application of kernel k -means.

Original: Finally, going beyond RFF, we make the first step towards dataoblivious dimension-reduction for general shift-invariant kernels, and we obtain a similar poly( ε − 1 log n ) dimension bound for Laplacian kernels.
Revised: Finally, going beyond RFF, we make the first step towards dataoblivious dimension-reduction for general shift-invariant kernels, and we obtain a similar poly( ε − 1 log n ) dimension bound for Laplacian kernels.

Original: We also validate the dimension-error tradeoff of our methods on simulated datasets, and they demonstrate superior performance compared with other popular methods including random-projection and Nyström methods.
Revised: We also validate the dimension-error tradeoff of our methods on simulated datasets, and they demonstrate superior performance compared with other popular methods including random-projection and Nyström methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We study the ability of the random Fourier features (RFF) method (Rahimi & Recht, 2007) for preserving the relative error for the kernel distance.
Revised: We study the ability of the random Fourier features (RFF) method (Rahimi & Recht, 2007) for preserving the relative error for the kernel distance.

Original: Kernel method (Schölkopf & Smola, 2002) is a systematic way to map the input data into a (indefinitely) high dimensional feature space to introduce richer structures, such as non-linearity.
Revised: Kernel method (Schölkopf & Smola, 2002) is a systematic way to map the input data into a (indefinitely) high dimensional feature space to introduce richer structures, such as non-linearity.

Original: In particular, for a set of n data points P , a kernel function
Revised: In particular, for a set of n data points P , a kernel function

Original: K : P × P → R implicitly defines a feature mapping φ : P → H to a feature space H which is a Hilbert space, such that ∀ x, y, K ( x, y )
Revised: K : P × P → R implicitly defines a feature mapping φ : P → H to a feature space H which is a Hilbert space, such that ∀ x, y, K ( x, y )

Original: = ⟨ φ ( x ) , φ ( y ) ⟩ .
Revised: = ⟨ φ ( x ) , φ ( y ) ⟩ .

Original: Kernel methods have been successfully applied to classical machine learning (Boser et al., 1992; Schölkopf et al., 1998; Girolami, 2002), and it has been recently established that in a certain sense the behavior of neural networks may be modeled as a kernel (Jacot et al., 2018).
Revised: Kernel methods have been successfully applied to classical machine learning (Boser et al., 1992; Schölkopf et al., 1998; Girolami, 2002), and it has been recently established that in a certain sense the behavior of neural networks may be modeled as a kernel (Jacot et al., 2018).

Original: Despite the superior power and wide applicability, the scalability has been an outstanding issue of applying kernel methods.
Revised: Despite the superior power and wide applicability, the scalability has been an outstanding issue of applying kernel methods.

Original: Specifically, the representation of data points in the feature space is only implicit, and solving for the explicit representation, which is crucially required in many algorithms, takes at least Ω( n 2 ) time in the worst case.
Revised: Specifically, the representation of data points in the feature space is only implicit, and solving for the explicit representation, which is crucially required in many algorithms, takes at least Ω( n 2 ) time in the worst case.

Original: While for many problems such as kernel SVM, it is possible to apply the so-called “kernel trick” to rewrite the objective in terms of K ( x, y ) , the explicit representation is still often preferred, since the representation is compatible with a larger range of solvers/algorithms which allows better efficiency.
Revised: While for many problems such as kernel SVM, it is possible to apply the so-called “kernel trick” to rewrite the objective in terms of K ( x, y ) , the explicit representation is still often preferred, since the representation is compatible with a larger range of solvers/algorithms which allows better efficiency.

Original: In a seminal work (Rahimi & Recht, 2007), Rahimi and Recht addressed this issue by introducing the method of random Fourier features (see Section 2 for a detailed description), to compute an explicit low-dimensional mapping φ ′ :
Revised: In a seminal work (Rahimi & Recht, 2007), Rahimi and Recht addressed this issue by introducing the method of random Fourier features (see Section 2 for a detailed description), to compute an explicit low-dimensional mapping φ ′ :

Original: P → R D (for D ≪ n ) such that ⟨ φ ′
Revised: P → R D (for D ≪ n ) such that ⟨ φ ′

Original: ( x ) , φ ′ ( y ) ⟩ ≈ ⟨ φ ( x ) , φ ( y ) ⟩ =
Revised: ( x ) , φ ′ ( y ) ⟩ ≈ ⟨ φ ( x ) , φ ( y ) ⟩ =

Original: K ( x, y ) , for shift-invariant kernels (i.e., there exists K : P → R , such that K ( x, y )
Revised: K ( x, y ) , for shift-invariant kernels (i.e., there exists K : P → R , such that K ( x, y )

Original: =
Revised: =

Original: K ( x − y ) ) which includes widely-used Gaussian kernels, Cauchy kernels and Laplacian kernels.
Revised: K ( x − y ) ) which includes widely-used Gaussian kernels, Cauchy kernels and Laplacian kernels.

Original: Towards understanding this fundamental method of RFF, a long line of research has focused on analyzing the tradeoff between the target dimension D and the accuracy of approximating K under certain error measures.
Revised: Towards understanding this fundamental method of RFF, a long line of research has focused on analyzing the tradeoff between the target dimension D and the accuracy of approximating K under certain error measures.

Original: This includes additive error max x,y |⟨ φ ( x ) , φ ( y ) ⟩−⟨ φ ′ ( x ) , φ ′ ( y ) ⟩|
Revised: This includes additive error max x,y |⟨ φ ( x ) , φ ( y ) ⟩−⟨ φ ′ ( x ) , φ ′ ( y ) ⟩|

Original: (Rahimi & Recht, 2007; Sriperumbudur & Szabó, 2015; Sutherland & Schneider, 2015), spectral error (Avron et al., 2017; Choromanski et al., 2018; Zhang et al., 2019;
Revised: (Rahimi & Recht, 2007; Sriperumbudur & Szabó, 2015; Sutherland & Schneider, 2015), spectral error (Avron et al., 2017; Choromanski et al., 2018; Zhang et al., 2019;

Original: Erdélyi et al., 2020;
Revised: Erdélyi et al., 2020;

Original: Ahle et al., 2020), and the generalization error of several learning tasks such as kernel SVM and kernel ridge regression (Avron et al., 2017; Sun et al., 2018; Li et al., 2021).
Revised: Ahle et al., 2020), and the generalization error of several learning tasks such as kernel SVM and kernel ridge regression (Avron et al., 2017; Sun et al., 2018; Li et al., 2021).

Original: A more comprehensive overview of the study of RFF can be found in a recent survey (Liu et al., 2021).
Revised: A more comprehensive overview of the study of RFF can be found in a recent survey (Liu et al., 2021).

Original: We focus on analyzing RFF with respect to the kernel distance .
Revised: We focus on analyzing RFF with respect to the kernel distance .

Original: Here, the kernel distance of two data points x, y is defined as their (Euclidean) distance in the feature space, i.e., dist φ ( x, y ) =
Revised: Here, the kernel distance of two data points x, y is defined as their (Euclidean) distance in the feature space, i.e., dist φ ( x, y ) =

Original: ∥ φ ( x ) − φ ( y ) ∥ 2 .
Revised: ∥ φ ( x ) − φ ( y ) ∥ 2 .

Original: While previous results on the additive error of K ( x, y ) (Rahimi & Recht, 2007; Sriperumbudur & Szabó, 2015; Sutherland & Schneider, 2015; Avron et al., 2017) readily implies additive error guarantee of dist φ ( x, y ) , the relative error guarantee is less understood.
Revised: While previous results on the additive error of K ( x, y ) (Rahimi & Recht, 2007; Sriperumbudur & Szabó, 2015; Sutherland & Schneider, 2015; Avron et al., 2017) readily implies additive error guarantee of dist φ ( x, y ) , the relative error guarantee is less understood.

Original: As far as we know, Chen & Phillips (2017) is the only previous work that gives a relative error bound for kernel distance, but unfortunately, only Gaussian kernel is studied in that work, and whether or not the kernel distance for other shift-invariant kernels is preserved by RFF, is still largely open.
Revised: As far as we know, Chen & Phillips (2017) is the only previous work that gives a relative error bound for kernel distance, but unfortunately, only Gaussian kernel is studied in that work, and whether or not the kernel distance for other shift-invariant kernels is preserved by RFF, is still largely open.

Original: In spirit, this multiplicative error guarantee of RFF, if indeed exists, makes it a kernelized version of
Revised: In spirit, this multiplicative error guarantee of RFF, if indeed exists, makes it a kernelized version of

Original: Johnson-Lindenstrauss Lemma (Johnson & Lindenstrauss, 1984) which is one of the central result in dimension reduction.
Revised: Johnson-Lindenstrauss Lemma (Johnson & Lindenstrauss, 1984) which is one of the central result in dimension reduction.

Original: This guarantee is also very useful for downstream applications, since one can combine it directly with classical geometric algorithms such as k-means++ (Arthur & Vassilvitskii, 2007), locality sensitive hashing (Indyk & Motwani, 1998) and fast geometric matching algorithms (Raghvendra & Agarwal, 2020) to obtain very efficient algorithms for kernelized k -means clustering, nearest neighbor search, matching and many more.
Revised: This guarantee is also very useful for downstream applications, since one can combine it directly with classical geometric algorithms such as k-means++ (Arthur & Vassilvitskii, 2007), locality sensitive hashing (Indyk & Motwani, 1998) and fast geometric matching algorithms (Raghvendra & Agarwal, 2020) to obtain very efficient algorithms for kernelized k -means clustering, nearest neighbor search, matching and many more.

Original: 1. O UR C ONTRIBUTIONS
Revised: 1. O UR C ONTRIBUTIONS

Original: Our main results are characterizations of the kernel functions on which RFF preserves the kernel distance with small relative error using poly log target dimensions.
Revised: Our main results are characterizations of the kernel functions on which RFF preserves the kernel distance with small relative error using poly log target dimensions.

Original: Furthermore, we also explore how to obtain data-oblivious dimension-reduction for kernels that cannot be handled by RFF.
Revised: Furthermore, we also explore how to obtain data-oblivious dimension-reduction for kernels that cannot be handled by RFF.

Original: As mentioned, it has been shown that RFF with small dimension preserves the additive error of kernel distance for all shift-invariant kernels (Rahimi & Recht, 2007; Sriperumbudur & Szabó, 2015;
Revised: As mentioned, it has been shown that RFF with small dimension preserves the additive error of kernel distance for all shift-invariant kernels (Rahimi & Recht, 2007; Sriperumbudur & Szabó, 2015;

Original: Sutherland & Schneider, 2015).
Revised: Sutherland & Schneider, 2015).

Original: In addition, it has been shown in Chen & Phillips (2017) that RFF indeed preserves the relative error of kernel distance for Gaussian kernels (which is shift-invariant).
Revised: In addition, it has been shown in Chen & Phillips (2017) that RFF indeed preserves the relative error of kernel distance for Gaussian kernels (which is shift-invariant).

Original: Hence, by analogue to the additive case and as informally claimed in Chen & Phillips (2017), one might be tempted to expect that RFF also preserves the relative error for general shift-invariant kernels as well.
Revised: Hence, by analogue to the additive case and as informally claimed in Chen & Phillips (2017), one might be tempted to expect that RFF also preserves the relative error for general shift-invariant kernels as well.

Original: Lower Bounds.
Revised: Lower Bounds.

Original: Surprisingly, we show that this is not the case.
Revised: Surprisingly, we show that this is not the case.

Original: In particular, we show that for a wide range of kernels, including the well-known Laplacian kernels, it requires unbounded target dimension for RFF to preserve the kernel distance with constant multiplicative error.
Revised: In particular, we show that for a wide range of kernels, including the well-known Laplacian kernels, it requires unbounded target dimension for RFF to preserve the kernel distance with constant multiplicative error.

Original: We state the result for a Laplacian kernel in the following, and the full statement of the general conditions of kernels can be found in Theorem 4.1.
Revised: We state the result for a Laplacian kernel in the following, and the full statement of the general conditions of kernels can be found in Theorem 4.1.

Original: In fact, what we show is a quantitatively stronger result, that if the input is (∆ , ρ ) -bounded , then preserving any constant multiplicative error requires Ω(poly(∆ /ρ ))
Revised: In fact, what we show is a quantitatively stronger result, that if the input is (∆ , ρ ) -bounded , then preserving any constant multiplicative error requires Ω(poly(∆ /ρ ))

Original: target dimension.
Revised: target dimension.

Original: Here, a point x ∈ R d is (∆ , ρ ) -bounded if ∥ x ∥ ∞ ≤ ∆ and min i : x i ̸ =

Revised: Here, a point x ∈ R d is (∆ , ρ ) -bounded if ∥ x ∥ ∞ ≤ ∆ and min i : x i ̸ =

Original: | x i | ≥ ρ , i.e., the magnitude is (upper) bounded by ∆ and the resolution is (lower) bounded by ρ .
Revised: | x i | ≥ ρ , i.e., the magnitude is (upper) bounded by ∆ and the resolution is (lower) bounded by ρ .

Original: Theorem 1.1 (Lower bound; see Remark 4.1) .
Revised: Theorem 1.1 (Lower bound; see Remark 4.1) .

Original: For every ∆ ≥ ρ > 0 and some feature mapping φ : R d → H of a Laplacian kernel K ( x, y ) = exp( −∥ x − y ∥ 1 ) , if for every x, y ∈ R d that are (∆ , ρ ) -bounded, the RFF mapping π for K with target dimension D satisfies dist π ( x, y ) ∈ (1 ± ε ) · dist φ ( x, y ) with constant probability, then D ≥ Ω( 1 ε 2 ∆ ρ ) .
Revised: For every ∆ ≥ ρ > 0 and some feature mapping φ : R d → H of a Laplacian kernel K ( x, y ) = exp( −∥ x − y ∥ 1 ) , if for every x, y ∈ R d that are (∆ , ρ ) -bounded, the RFF mapping π for K with target dimension D satisfies dist π ( x, y ) ∈ (1 ± ε ) · dist φ ( x, y ) with constant probability, then D ≥ Ω( 1 ε 2 ∆ ρ ) .

Original: This holds even when d = 1 .
Revised: This holds even when d = 1 .

Original: Upper Bounds.
Revised: Upper Bounds.

Original: Complementing the lower bound, we show that RFF can indeed preserve the kernel distance within 1 ± ε error using poly( ε − 1 log n ) target dimensions with high probability, as long as the kernel function is shift-invariant and analytic, which includes Gaussian kernels and
Revised: Complementing the lower bound, we show that RFF can indeed preserve the kernel distance within 1 ± ε error using poly( ε − 1 log n ) target dimensions with high probability, as long as the kernel function is shift-invariant and analytic, which includes Gaussian kernels and

Original: Cauchy kernels.
Revised: Cauchy kernels.

Original: Our target dimension nearly matches (up to the degree of polynomial of parameters) that is achievable by the Johnson-Lindenstrauss transform (Johnson & Lindenstrauss, 1984), which is shown to be tight (Larsen & Nelson, 2017).
Revised: Our target dimension nearly matches (up to the degree of polynomial of parameters) that is achievable by the Johnson-Lindenstrauss transform (Johnson & Lindenstrauss, 1984), which is shown to be tight (Larsen & Nelson, 2017).

Original: This upper bound also greatly generalizes the result of Chen & Phillips (2017) which only works for Gaussian kernels (see Section G for a detailed comparison).
Revised: This upper bound also greatly generalizes the result of Chen & Phillips (2017) which only works for Gaussian kernels (see Section G for a detailed comparison).

Original: Theorem 1.2 (Upper bound) .
Revised: Theorem 1.2 (Upper bound) .

Original: Let K : R d × R d → R be a kernel function which is shift-invariant and analytic at the origin, with feature mapping φ : R d → H for some feature space H .
Revised: Let K : R d × R d → R be a kernel function which is shift-invariant and analytic at the origin, with feature mapping φ : R d → H for some feature space H .

Original: For every 0 < δ ≤ ε ≤ 2 − 16 , every d, D ∈ N , D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , if π : R d → R D is an RFF mapping for K with target dimension D , then for every x, y ∈ R d ,
Revised: For every 0 < δ ≤ ε ≤ 2 − 16 , every d, D ∈ N , D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , if π : R d → R D is an RFF mapping for K with target dimension D , then for every x, y ∈ R d ,

Original: Pr[ | dist π ( x, y ) − dist φ ( x, y ) | ≤ ε · dist φ ( x, y )] ≥ 1 − δ.
Revised: Pr[ | dist π ( x, y ) − dist φ ( x, y ) | ≤ ε · dist φ ( x, y )] ≥ 1 − δ.

Original: The technical core of our analysis is a moment bound for RFF, which is derived by analysis techniques such as Taylor expansion and Cauchy’s integral formula for multi-variate functions.
Revised: The technical core of our analysis is a moment bound for RFF, which is derived by analysis techniques such as Taylor expansion and Cauchy’s integral formula for multi-variate functions.

Original: The moment bound is slightly weaker than the moment bound of Gaussian variables, and this is the primary reason that we obtain a bound weaker than that of the Johnson-Lindenstrauss transform.
Revised: The moment bound is slightly weaker than the moment bound of Gaussian variables, and this is the primary reason that we obtain a bound weaker than that of the Johnson-Lindenstrauss transform.

Original: Finally, several additional steps are required to fit this moment bound in Bernstein’s inequality, which implies the bound in Theorem 1.2.
Revised: Finally, several additional steps are required to fit this moment bound in Bernstein’s inequality, which implies the bound in Theorem 1.2.

Original: Improved Dimension Bound for Kernel k -Means.
Revised: Improved Dimension Bound for Kernel k -Means.

Original: We show that if we focus on a specific application of kernel k -means, then it suffices to set the target dimension D
Revised: We show that if we focus on a specific application of kernel k -means, then it suffices to set the target dimension D

Original: = poly( ε − 1 log k ) , instead of D = poly( ε − 1 log n ) , to preserve the kernel k -means clustering cost for every k -partition.
Revised: = poly( ε − 1 log k ) , instead of D = poly( ε − 1 log n ) , to preserve the kernel k -means clustering cost for every k -partition.

Original: This follows from the probabilistic guarantee of RFF in Theorem 1.2 plus a generalization of the dimension-reduction result proved in a recent paper (Makarychev et al., 2019).
Revised: This follows from the probabilistic guarantee of RFF in Theorem 1.2 plus a generalization of the dimension-reduction result proved in a recent paper (Makarychev et al., 2019).

Original: Here, given a data set P ⊂ R d and a kernel function K : R d × R d → R , denoting the feature mapping as φ : R d → H , the kernel k -means problem asks to find a k -partition C :
Revised: Here, given a data set P ⊂ R d and a kernel function K : R d × R d → R , denoting the feature mapping as φ : R d → H , the kernel k -means problem asks to find a k -partition C :

Original: = { C 1 , . .
Revised: = { C 1 , . .

Original: , C k } of P , such that cost φ ( P, C ) = (cid:80)
Revised: , C k } of P , such that cost φ ( P, C ) = (cid:80)

Original: ki
Revised: ki

Original: =1 min c i ∈H (cid:80) x ∈ C i ∥ φ ( x ) − c i ∥ 22 is minimized.
Revised: =1 min c i ∈H (cid:80) x ∈ C i ∥ φ ( x ) − c i ∥ 22 is minimized.

Original: Theorem 1.3 (Dimension reduction for clustering; see Theorem 3.1) .
Revised: Theorem 1.3 (Dimension reduction for clustering; see Theorem 3.1) .

Original: For kernel k -means problem whose kernel function K : R d × R d → R is shift-invariant and analytic at the origin, for every data set P ⊂ R d , the RFF mapping π : R d → R D with target dimension D ≥ O ( 1 ε 2 (log 3 kδ + log 3 1 ε )) , with probability at least 1 − δ , preserves the clustering cost within 1 ± ε error for every k -partition simultaneously.
Revised: For kernel k -means problem whose kernel function K : R d × R d → R is shift-invariant and analytic at the origin, for every data set P ⊂ R d , the RFF mapping π : R d → R D with target dimension D ≥ O ( 1 ε 2 (log 3 kδ + log 3 1 ε )) , with probability at least 1 − δ , preserves the clustering cost within 1 ± ε error for every k -partition simultaneously.

Original: Applying RFF to speed up kernel k -means has also been considered in Chitta et al.
Revised: Applying RFF to speed up kernel k -means has also been considered in Chitta et al.

Original: , but their error bound is much weaker than ours (and theirs is not a generic dimension-reduction bound).
Revised: , but their error bound is much weaker than ours (and theirs is not a generic dimension-reduction bound).

Original: Also, similar dimension-reduction bounds (i.e., independent of n ) for kernel k -means were obtained using Nyström methods (Musco & Musco, 2017; Wang et al., 2019), but their bound is poly( k ) which is worse than our poly log( k ) ; furthermore, our RFF-based approach is unique in that it is data-oblivious, which enables great applicability in other relevant computational settings such as streaming and distributed computing.
Revised: Also, similar dimension-reduction bounds (i.e., independent of n ) for kernel k -means were obtained using Nyström methods (Musco & Musco, 2017; Wang et al., 2019), but their bound is poly( k ) which is worse than our poly log( k ) ; furthermore, our RFF-based approach is unique in that it is data-oblivious, which enables great applicability in other relevant computational settings such as streaming and distributed computing.

Original: Going beyond RFF.
Revised: Going beyond RFF.

Original: Finally, even though we have proved RFF cannot preserve the kernel distance for every shift-invariant kernels, it does not rule out the existence of other efficient data-oblivious dimension reduction methods for those kernels, particularly for Laplacian kernel which is the primary example in our lower bound.
Revised: Finally, even though we have proved RFF cannot preserve the kernel distance for every shift-invariant kernels, it does not rule out the existence of other efficient data-oblivious dimension reduction methods for those kernels, particularly for Laplacian kernel which is the primary example in our lower bound.

Original: For instance, in the same paper where RFF was proposed, Rahimi and
Revised: For instance, in the same paper where RFF was proposed, Rahimi and

Original: Recht (Rahimi & Recht, 2007) also considered an alternative embedding called “binning features” that can work for Laplacian kernels.
Revised: Recht (Rahimi & Recht, 2007) also considered an alternative embedding called “binning features” that can work for Laplacian kernels.

Original: Unfortunately, to achieve a relative error of ε , it requires a dimension that depends linearly on the magnitude/aspect-ratio of the dataset, which may be exponential in the input size.
Revised: Unfortunately, to achieve a relative error of ε , it requires a dimension that depends linearly on the magnitude/aspect-ratio of the dataset, which may be exponential in the input size.

Original: Follow-up works, such as (Backurs et al., 2019), also suffer similar issues.
Revised: Follow-up works, such as (Backurs et al., 2019), also suffer similar issues.

Original: We make the first successful attempt towards this direction, and we show that Laplacian kernels do admit an efficient data-oblivious dimension reduction.
Revised: We make the first successful attempt towards this direction, and we show that Laplacian kernels do admit an efficient data-oblivious dimension reduction.

Original: Here, we focus on the (∆ , ρ ) -bounded case,
Revised: Here, we focus on the (∆ , ρ ) -bounded case,

Original: Here, we use a similar setting to our lower bound (Theorem 1.1) where we focus on the (∆ , ρ ) - bounded case.
Revised: Here, we use a similar setting to our lower bound (Theorem 1.1) where we focus on the (∆ , ρ ) - bounded case.

Original: Theorem 1.4 (Oblivious dimension-reduction for Laplacian kernels, see Theorem F.1) .
Revised: Theorem 1.4 (Oblivious dimension-reduction for Laplacian kernels, see Theorem F.1) .

Original: Let K be a Laplacian kernel, and denote its feature mapping as φ : R d → H .
Revised: Let K be a Laplacian kernel, and denote its feature mapping as φ : R d → H .

Original: For every  D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , every ∆ ≥ ρ > 0 , there is a mapping π : R d → R D , such that for every0 < δ ≤ ε ≤ 2 − 16 , ε g > 0 and x, y ∈ R d that are (∆ , ρ ) -bounded, it holds that
Revised: For every 0 < δ ≤ ε ≤ 2 − 16 , every D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , every ∆ ≥ ρ > 0 , there is a mapping π : R d → R D , such that for every x, y ∈ R d that are (∆ , ρ ) -bounded, it holds that

Original: [Equation]
Revised: [Equation]

Original: The time for evaluating π is dD · poly(log ∆ ρ  log ε − 1g ) .
Revised: The time for evaluating π is dD · poly(log ∆ ρ , log δ − 1 ) .

Original: Our target dimension only depends on log ∆ ρ which may be interpreted as the precision of the input.
Revised: Our target dimension only depends on log ∆ ρ which may be interpreted as the precision of the input.

Original: Hence, as an immediate corollary, for any n -points dataset with precision 1 / poly( n ) , we have an embedding with target dimension D = poly( ε − 1 log n ) , where the success probability is 1 − 1 / poly( n ) and the overall running time of embedding the n points is O ( n poly( dε − 1 log n )) .
Revised: Hence, as an immediate corollary, for any n -points dataset with precision 1 / poly( n ) , we have an embedding with target dimension D = poly( ε − 1 log n ) , where the success probability is 1 − 1 / poly( n ) and the overall running time of embedding the n points is O ( n poly( dε − 1 log n )) .

Original: Our proof relies on the fact that every ℓ 1 metric space can be embedded into a squared ℓ 2 metric space isometrically.
Revised: Our proof relies on the fact that every ℓ 1 metric space can be embedded into a squared ℓ 2 metric space isometrically.

Original: We explicitly implement an approximate version of this embedding (Kahane, 1981), and eventually reduce our problem of Laplacian kernels to Gaussian kernels.
Revised: We explicitly implement an approximate version of this embedding (Kahane, 1981), and eventually reduce our problem of Laplacian kernels to Gaussian kernels.

Original: After this reduction, we use the RFF for Gaussian kernels to obtain the final mapping.
Revised: After this reduction, we use the RFF for Gaussian kernels to obtain the final mapping.

Original: However, since the embedding to squared ℓ 2 is only of very high dimension, to implement this whole idea efficiently, we need to utilize the special structures of the embedding, combined with an application of space bounded pseudo-random generators (PRGs) (Nisan, 1992).
Revised: However, since the embedding to squared ℓ 2 is only of very high dimension, to implement this whole idea efficiently, we need to utilize the special structures of the embedding, combined with an application of space bounded pseudo-random generators (PRGs) (Nisan, 1992).

Original: Even though our algorithm utilizes the special property of Laplacian kernels and eventually still partially use the RFF for Gaussian kernels, it is still of conceptual importance.
Revised: Even though our algorithm utilizes the special property of Laplacian kernels and eventually still partially use the RFF for Gaussian kernels, it is still of conceptual importance.

Original: It opens up the direction of exploring general methods for Johnson-Lindenstrauss style dimension reduction for shift-invariant kernels.
Revised: It opens up the direction of exploring general methods for Johnson-Lindenstrauss style dimension reduction for shift-invariant kernels.

Original: Furthermore, the lower bound suggests that the Johnson-Lindenstrauss style dimension reduction for general shift-invariant kernels has to be not differentiable, which is a fundamental difference to RFF.
Revised: Furthermore, the lower bound suggests that the Johnson-Lindenstrauss style dimension reduction for general shift-invariant kernels has to be not differentiable, which is a fundamental difference to RFF.

Original: This requirement of “not analytical” seems very counter-intuitive, but our construction of the mapping for Laplacian kernels indeed provides valuable insights on how the non-analytical mapping behaves.
Revised: This requirement of “not analytical” seems very counter-intuitive, but our construction of the mapping for Laplacian kernels indeed provides valuable insights on how the non-analytical mapping behaves.

Original: Experiments and Comparison to Other Methods.
Revised: Experiments and Comparison to Other Methods.

Original: Apart from RFF, the Nyström and the random-projection methods are alternative popular methods for kernel dimension reduction.
Revised: Apart from RFF, the Nyström and the random-projection methods are alternative popular methods for kernel dimension reduction.

Original: In Section 6, we conduct experiments to compare their empirical dimension-error tradeoffs with that of our methods on a simulated dataset.
Revised: In Section 6, we conduct experiments to compare their empirical dimension-error tradeoffs with that of our methods on a simulated dataset.

Original: Since we focus on the error, we use the “ideal” implementation of both methods that achieve the best accuracy, so they are only in favor of the two baselines – for Nyström, we use SVD on the kernel matrix, since Nyström methods can be viewed as fast and approximate low-rank approximations to the kernel matrix; for random-projection, we apply the Johnson-Lindenstrauss transform on the explicit representations of points in the feature space.
Revised: Since we focus on the error, we use the “ideal” implementation of both methods that achieve the best accuracy, so they are only in favor of the two baselines – for Nyström, we use SVD on the kernel matrix, since Nyström methods can be viewed as fast and approximate low-rank approximations to the kernel matrix; for random-projection, we apply the Johnson-Lindenstrauss transform on the explicit representations of points in the feature space.

Original: We run two experiments to compare each of RFF (on a Gaussian kernel) and our new algorithm in Theorem 1.4 (on a Laplacian kernel) with the two baselines respectively.
Revised: We run two experiments to compare each of RFF (on a Gaussian kernel) and our new algorithm in Theorem 1.4 (on a Laplacian kernel) with the two baselines respectively.

Original: Our experiments indicate that the Nyström method is indeed incapable of preserving the kernel distance in relative error, and more interestingly, our methods perform the best among the three, even better than the
Revised: Our experiments indicate that the Nyström method is indeed incapable of preserving the kernel distance in relative error, and more interestingly, our methods perform the best among the three, even better than the

Original: Johnson-Lindenstrauss transform which is the optimal in the worst case.
Revised: Johnson-Lindenstrauss transform which is the optimal in the worst case.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Variants of the vanilla RFF, particularly those that use information in the input data set and/or sample random features non-uniformly, have also been considered, including leverage score sampling random Fourier features (LSS-RFF)
Revised: Variants of the vanilla RFF, particularly those that use information in the input data set and/or sample random features non-uniformly, have also been considered, including leverage score sampling random Fourier features (LSS-RFF)

Original: (Rudi et al., 2018; Liu et al., 2020;
Revised: (Rudi et al., 2018; Liu et al., 2020;

Original: Erdélyi et al., 2020; Li et al., 2021), weighted random features (Rahimi & Recht, 2008; Avron et al., 2016; Chang et al., 2017;
Revised: Erdélyi et al., 2020; Li et al., 2021), weighted random features (Rahimi & Recht, 2008; Avron et al., 2016; Chang et al., 2017;

Original: Dao et al., 2017), and kernel alignment (Shahrampour et al., 2018; Zhen et al., 2020).
Revised: Dao et al., 2017), and kernel alignment (Shahrampour et al., 2018; Zhen et al., 2020).

Original: The RFF-based methods usually work for shift-invariant kernels only.
Revised: The RFF-based methods usually work for shift-invariant kernels only.

Original: For general kernels, techniques that are based on low-rank approximation of the kernel matrix, notably Nyström method (Williams & Seeger, 2000; Gittens & Mahoney, 2016; Musco & Musco, 2017; Oglic &
Revised: For general kernels, techniques that are based on low-rank approximation of the kernel matrix, notably Nyström method (Williams & Seeger, 2000; Gittens & Mahoney, 2016; Musco & Musco, 2017; Oglic &

Original: Gärtner, 2017; Wang et al., 2019) and incomplete Cholesky factorization (Fine & Scheinberg, 2001;
Revised: Gärtner, 2017; Wang et al., 2019) and incomplete Cholesky factorization (Fine & Scheinberg, 2001;

Original: Bach & Jordan, 2002; Chen et al., 2021; Jia et al., 2021)) were developed.
Revised: Bach & Jordan, 2002; Chen et al., 2021; Jia et al., 2021)) were developed.

Original: Moreover, specific sketching techniques were known for polynomial kernels (Avron et al., 2014; Woodruff & Zandieh, 2020;
Revised: Moreover, specific sketching techniques were known for polynomial kernels (Avron et al., 2014; Woodruff & Zandieh, 2020;

Original: Ahle et al., 2020; Song et al., 2021), a basic type of kernel that is not shift-invariant.
Revised: Ahle et al., 2020; Song et al., 2021), a basic type of kernel that is not shift-invariant.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Random Fourier Features.
Revised: Random Fourier Features.

Original: RFF was first introduced by Rahimi and Recht (Rahimi & Recht, 2007).
Revised: RFF was first introduced by Rahimi and Recht (Rahimi & Recht, 2007).

Original: It is based on the fact that, for shift-invariant kernel K : R d → R such that K (0) =(this can be assumed w.l.o.g. by normalization), function p : R d → R
Revised: It is based on the fact that, for shift-invariant kernel K : R d → R such that K (0) =(this can be assumed w.l.o.g. by normalization), function p : R d → R

Original: such that p ( ω ) =
Revised: such that p ( ω ) =

Original: 12 π (cid:82) R d K ( x ) e − i ⟨ ω,x ⟩ d x , which is the Fourier transform of K ( · ) , is a probability distribution (guaranteed by Bochner’s theorem (Bochner, 1933; Rudin, 1991)).
Revised: 12 π (cid:82) R d K ( x ) e − i ⟨ ω,x ⟩ d x , which is the Fourier transform of K ( · ) , is a probability distribution (guaranteed by Bochner’s theorem (Bochner, 1933; Rudin, 1991)).

Original: Then, the RFF mapping is defined as
Revised: Then, the RFF mapping is defined as

Original: [Equation]
Revised: [Equation]

Original:  where ω 1 , ω
Revised:  where ω 1 , ω

Original: ω
Revised: ω

Original: D ∈ R
Revised: D ∈ R

Original: d are i.i.d.
Revised: d are i.i.d.

Original: samples from distribution with densitiy p .
Revised: samples from distribution with densitiy p .

Original: Theorem 2.1 (Rahimi & Recht 2007) .
Revised: Theorem 2.1 (Rahimi & Recht 2007) .

Original: E [ ⟨ π ( x ) , π ( y ) ⟩ ] = 1 D (cid:80) Di =1 E [cos ⟨ ω
Revised: E [ ⟨ π ( x ) , π ( y ) ⟩ ] = 1 D (cid:80) Di =1 E [cos ⟨ ω

Original: i , x − y ⟩ ] =
Revised: i , x − y ⟩ ] =

Original: K ( x − y ) .
Revised: K ( x − y ) .

Original: Fact 2.1.
Revised: Fact 2.1.

Original: Let ω be a random variable with distribution p over R d .
Revised: Let ω be a random variable with distribution p over R d .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: and Var(cos ⟨ ω, x − y ⟩ )
Revised: and Var(cos ⟨ ω, x − y ⟩ )

Original: =
Revised: =

Original: 1+ K (
Revised: 1+ K (

Original: [Equation]
Revised: [Equation]

Original: ( x
Revised: ( x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  U PPER B OUNDS
Revised:  U PPER B OUNDS

Original: We present two results in this section.
Revised: We present two results in this section.

Original: We start with Section 3.1 to show RFF preserves the relative error of kernel distance using poly( ε − 1 log n ) target dimensions with high probability, when the kernel function is shift-invariant and analytic at origin.
Revised: We start with Section 3.1 to show RFF preserves the relative error of kernel distance using poly( ε − 1 log n ) target dimensions with high probability, when the kernel function is shift-invariant and analytic at origin.

Original: Then in Section 3.2, combining this bound with a generalized analysis from a recent paper (Makarychev et al., 2019), we show that RFF also preserves the clustering cost for kernel k -clustering problems with ℓ p -objective, with target dimension only poly( ε − 1 log k ) which is independent of n .
Revised: Then in Section 3.2, combining this bound with a generalized analysis from a recent paper (Makarychev et al., 2019), we show that RFF also preserves the clustering cost for kernel k -clustering problems with ℓ p -objective, with target dimension only poly( ε − 1 log k ) which is independent of n .

Original: 3. P ROOF OF T HEOREM 1.2: T HE R ELATIVE E RROR FOR P RESERVING K ERNEL D ISTANCE
Revised: 3. P ROOF OF T HEOREM 1.2: T HE R ELATIVE E RROR FOR P RESERVING K ERNEL D ISTANCE

Original: Since K is shift-invariant, we interpret K as a function on R d instead of R d × R d , such that
Revised: Since K is shift-invariant, we interpret K as a function on R d instead of R d × R d , such that

Original: K ( x, y )
Revised: K ( x, y )

Original: =
Revised: =

Original: K ( x − y ) .
Revised: K ( x − y ) .

Original: As in Section 2, let p : R d → R be the Fourier transform of K , and suppose in the RFF mapping π , the random variables ω 1 , . .
Revised: As in Section 2, let p : R d → R be the Fourier transform of K , and suppose in the RFF mapping π , the random variables ω 1 , . .

Original: . , ω d ∈ R d are i.i.d.
Revised: . , ω d ∈ R d are i.i.d.

Original: sampled from the distribution with density p .
Revised: sampled from the distribution with density p .

Original: When we say K is analytic at the origin, we mean there exists some constant r s.t.
Revised: When we say K is analytic at the origin, we mean there exists some constant r s.t.

Original: K is analytic in { x ∈ R d : ∥ x ∥ 1 < r } .
Revised: K is analytic in { x ∈ R d : ∥ x ∥ 1 < r } .

Original: We pick r K to be the maximum of such constant r .
Revised: We pick r K to be the maximum of such constant r .

Original: Also notice that in D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , there are constants about K hidden inside the Θ , i.e. R K as in Lemma 3.2.
Revised: Also notice that in D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , there are constants about K hidden inside the Θ , i.e. R K as in Lemma 3.2.

Original: Fact 3.1.
Revised: Fact 3.1.

Original: The following holds.
Revised: The following holds.

Original: • dist π ( x, y ) = (cid:113) 2 − 2 /D (cid:80) Di =1 cos ⟨ ω i , x − y ⟩ , and dist φ ( x, y ) =
Revised: • dist π ( x, y ) = (cid:113) 2 − 2 /D (cid:80) Di =1 cos ⟨ ω i , x − y ⟩ , and dist φ ( x, y ) =

Original: (cid:112) 2 − 2 K ( x − y ) .
Revised: (cid:112) 2 − 2 K ( x − y ) .

Original: • Pr[ | dist π ( x, y ) − dist φ ( x, y ) | ≤ ε · dist φ ( x, y )] ≥ Pr[ | dist π ( x, y ) 2 − dist φ ( x, y ) 2 | ≤ ε · dist φ ( x, y ) 2 ] .
Revised: • Pr[ | dist π ( x, y ) − dist φ ( x, y ) | ≤ ε · dist φ ( x, y )] ≥ Pr[ | dist π ( x, y ) 2 − dist φ ( x, y ) 2 | ≤ ε · dist φ ( x, y ) 2 ] .

Original: Define X i 
Revised: Define X i (

Original: [Equation]
Revised: [Equation]

Original: As a crucial step, we next analyze the moment of random variables
Revised: As a crucial step, we next analyze the moment of random variables

Original: X i ( x − y ) .
Revised: X i ( x − y ) .

Original: This bound will be plugged into Bernstein’s inequality to conclude the proof.
Revised: This bound will be plugged into Bernstein’s inequality to conclude the proof.

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: If for some r > 0 , K is analytic in { x ∈ R d : ∥ x ∥ 1 < r } , then for every k ≥ 1 being even and every x s.t.
Revised: If for some r > 0 , K is analytic in { x ∈ R d : ∥ x ∥ 1 < r } , then for every k ≥ 1 being even and every x s.t.

Original: ∥ x ∥ 1 < r , we have E [ | X i ( x ) | k ] ≤
Revised: ∥ x ∥ 1 < r , we have E [ | X i ( x ) | k ] ≤

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof can be found in Section A.
Revised: The proof can be found in Section A.

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: For kernel K which is shift-invariant and analytic at the origin, there exist c K , R K >such that for all ∥ x
Revised: For kernel K which is shift-invariant and analytic at the origin, there exist c K , R K >such that for all ∥ x

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof can be found in Section B.
Revised: The proof can be found in Section B.

Original: Proof sketch of Theorem 1.2.
Revised: Proof sketch of Theorem 1.2.

Original: We present a proof sketch for Theorem 1.2, and the full proof can be found in Section C.
Revised: We present a proof sketch for Theorem 1.2, and the full proof can be found in Section C.

Original: We focus on the case when ∥ x − y ∥ 1 ≤ R K (the other case can be found in the full proof).
Revised: We focus on the case when ∥ x − y ∥ 1 ≤ R K (the other case can be found in the full proof).

Original: Then by Lemma 3.
Revised: Then by Lemma 3.

Original: [Equation]
Revised: [Equation]

Original: we have 2 − 2 K ( x − y ) ≥ c
Revised: we have 2 − 2 K ( x − y ) ≥ c

Original: ∥ x − y ∥ 21 .
Revised: ∥ x − y ∥ 21 .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: We take r = r K for simplicity of exhibition.
Revised: We take r = r K for simplicity of exhibition.

Original: Assume δ ≤ min { ε, 2 − 16 } , let k = log(

Revised: Assume δ ≤ min { ε, 2 − 16 } , let k = log(

Original: D 2 /δ ) , t = 64 k 2 /r 2 , is even.
Revised: D 2 /δ ) , t = 64 k 2 /r 2 , is even.

Original: By Markov’s inequality and Lemma 3.1:
Revised: By Markov’s inequality and Lemma 3.1:

Original: [Equation]
Revised: [Equation]

Original: For simplicity denote X i ( x − y ) by X i , ∥ x − y ∥ 21 by ℓ and define X ′ i = 

Revised: For simplicity denote X i ( x − y ) by X i , ∥ x − y ∥ 21 by ℓ and define X ′ i =

Original: [ | X i |≥ tℓ ] tℓ · sgn( X i )
Revised: [ | X i |≥ tℓ ] tℓ · sgn( X i )

Original: +[ | X i | <tℓ ] X i , note that E [ X i ] = 0 .
Revised: +[ | X i | <tℓ ] X i , note that E [ X i ] = 0 .

Original: By some further calculations and plugging in the parameters t, δ, D , we can eventually obtain E [ | X ′ i | ] ≤ δℓ .
Revised: By some further calculations and plugging in the parameters t, δ, D , we can eventually obtain E [ | X ′ i | ] ≤ δℓ .

Original: Denote σ ′ 2 as the variance of X ′ i , then again by Lemma 3.1 we immediately have σ ′ ≤ 64 ℓ/r 2 .
Revised: Denote σ ′ 2 as the variance of X ′ i , then again by Lemma 3.1 we immediately have σ ′ ≤ 64 ℓ/r 2 .

Original: The theorem follows by a straightforward application of Bernstein’s inequality.
Revised: The theorem follows by a straightforward application of Bernstein’s inequality.

Original: 3. D IMENSION R EDUCTION FOR K ERNEL C LUSTERING
Revised: 3. D IMENSION R EDUCTION FOR K ERNEL C LUSTERING

Original: We present the formal statement for Theorem 1.3 in Theorem 3.1.
Revised: We present the formal statement for Theorem 1.3 in Theorem 3.1.

Original: In fact, we consider the more general k -clustering problem with ℓ p 2 -objective defined in the following Definition 3.1, which generalizes kernel k -means (by setting p = 2 ).
Revised: In fact, we consider the more general k -clustering problem with ℓ p 2 -objective defined in the following Definition 3.1, which generalizes kernel k -means (by setting p = 2 ).

Original: Definition 3.1.
Revised: Definition 3.1.

Original: Given a data set P ⊂ R d and kernel function K : R d × R d → R , denoting the feature mapping as φ : R d → H , the kernel k -clustering problem with ℓ p -objective asks for a k -partition C
Revised: Given a data set P ⊂ R d and kernel function K : R d × R d → R , denoting the feature mapping as φ : R d → H , the kernel k -clustering problem with ℓ p -objective asks for a k -partition C

Original: = { C 1 , C 2 , ..., C k } of P that minimizes the cost function: cost φp ( P, C ) :
Revised: = { C 1 , C 2 , ..., C k } of P that minimizes the cost function: cost φp ( P, C ) :

Original: = (cid:80) ki
Revised: = (cid:80) ki

Original: =1 min c i ∈H (cid:80) x ∈ C i ∥ φ ( x ) − c i ∥ p 2 .
Revised: =1 min c i ∈H (cid:80) x ∈ C i ∥ φ ( x ) − c i ∥ p 2 .

Original: Theorem 3.1 (Generalization of Makarychev et al.
Revised: Theorem 3.1 (Generalization of Makarychev et al.

Original: 2019, Theorem 3.6) .
Revised: 2019, Theorem 3.6) .

Original: For kernel k -clustering problem with ℓ p 2 -objective whose kernel function K : R d × R d → R is shift-invariant and analytic at the origin, for every data set P ⊂ R d , the RFF mapping π : R d → R D with target dimension
Revised: For kernel k -clustering problem with ℓ p 2 -objective whose kernel function K : R d × R d → R is shift-invariant and analytic at the origin, for every data set P ⊂ R d , the RFF mapping π : R d → R D with target dimension

Original: D = Ω( p 2 log 3 kα + p 5 log 3 1 ε + p 8 )
Revised: D = Ω( p 2 log 3 kα + p 5 log 3 1 ε + p 8 )

Original: /ε 2 satisfies
Revised: /ε 2 satisfies

Original: Pr[ ∀ k -partition C of P : cost πp ( P, C ) ∈ (1 ± ε ) · cost φp ( P, C )] ≥ 1 − δ.
Revised: Pr[ ∀ k -partition C of P : cost πp ( P, C ) ∈ (1 ± ε ) · cost φp ( P, C )] ≥ 1 − δ.

Original: Proof.
Revised: Proof.

Original: The proof can be found in Section D.
Revised: The proof can be found in Section D.

Original:  L OWER B OUNDS
Revised:  L OWER B OUNDS

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Consider ∆ ≥ ρ > 0 , and a shift-invariant kernel function K : R d → R , denoting its feature mapping φ : R d → H .
Revised: Consider ∆ ≥ ρ > 0 , and a shift-invariant kernel function K : R d → R , denoting its feature mapping φ : R d → H .

Original: Then there exists x,
Revised: Then there exists x,

Original: y ∈ R d that are (∆ , ρ ) -bounded, such that for every 0 < ε < 1 , the RFF mapping π for K with target dimension D satisfies
Revised: y ∈ R d that are (∆ , ρ ) -bounded, such that for every 0 < ε < 1 , the RFF mapping π for K with target dimension D satisfies

Original: Pr[ | dist
Revised: Pr[ | dist

Original: [Equation]
Revised: [Equation]

Original: x, y
Revised: x, y

Original: [Equation]
Revised: [Equation]

Original: dist
Revised: dist

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: where s (∆ ,ρ ) K := sup (∆ ,ρ ) -bounded x ∈ R d s K ( x ) , and s K
Revised: where s (∆ ,ρ ) K := sup (∆ ,ρ ) -bounded x ∈ R d s K ( x ) , and s K

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof can be found in Section E.
Revised: The proof can be found in Section E.

Original: Note that the right hand side of (2) is always less than 1 , since the first term 2 √
Revised: Note that the right hand side of (2) is always less than 1 , since the first term 2 √

Original: [Equation]
Revised: [Equation]

Original: (cid:82) ∞ 
Revised: (cid:82) ∞ 6 ε

Original: [Equation]
Revised: [Equation]

Original: D/s (
Revised: D/s (

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: e − s 2 / 2 d s achieves its maximum at Dε 2 s (∆ ,ρ ) K = 0 , and this maximum is 1.
Revised: e − s 2 / 2 d s achieves its maximum at Dε 2 s (∆ ,ρ ) K = 0 , and this maximum is 1.

Original: On the other hand, we need the right hand side of (2) to be > 0 in order to obtain a useful lower bound, and a typical setup to achieve this is when D = Θ (cid:16) s (∆ ,ρ ) K (cid:17) .
Revised: On the other hand, we need the right hand side of (2) to be > 0 in order to obtain a useful lower bound, and a typical setup to achieve this is when D = Θ (cid:16) s (∆ ,ρ ) K (cid:17) .

Original: Intuition of s K .
Revised: Intuition of s K .

Original: Observe that s K ( x ) measures the ratio between the variance of RFF and the (squared) expectation evaluated at x .
Revised: Observe that s K ( x ) measures the ratio between the variance of RFF and the (squared) expectation evaluated at x .

Original: The intuition of considering this comes from the central limit theorem.
Revised: The intuition of considering this comes from the central limit theorem.

Original: Indeed, when the number of samples/target dimension is sufficiently large, the error/difference behaves like a Gaussian distribution where with constant probability the error ≈ Var .
Revised: Indeed, when the number of samples/target dimension is sufficiently large, the error/difference behaves like a Gaussian distribution where with constant probability the error ≈ Var .

Original: Hence, this s K measures the “typical” relative error when the target dimension is sufficiently large, and an upper bound of s (∆ ,ρ ) K is naturally a necessary condition for the bounded relative error.
Revised: Hence, this s K measures the “typical” relative error when the target dimension is sufficiently large, and an upper bound of s (∆ ,ρ ) K is naturally a necessary condition for the bounded relative error.

Original: The following gives a simple (sufficient) condition for kernels that do not have a bounded s K ( x ) .
Revised: The following gives a simple (sufficient) condition for kernels that do not have a bounded s K ( x ) .

Original: Remark 4.1 (Simple sufficient conditions for lower bounds) .
Revised: Remark 4.1 (Simple sufficient conditions for lower bounds) .

Original: Assume the input dimension is 1 , so
Revised: Assume the input dimension is 1 , so

Original: K : R → R , and assume ∆ = 1 , ρ < 1 .
Revised: K : R → R , and assume ∆ = 1 , ρ < 1 .

Original: Then the (∆ , ρ ) -bounded property simply requires ρ ≤ | x | ≤ 1 .
Revised: Then the (∆ , ρ ) -bounded property simply requires ρ ≤ | x | ≤ 1 .

Original: We claim that, if K ’s first derivative at 0 is non-zero, i.e., K ′ (0) ̸
Revised: We claim that, if K ’s first derivative at 0 is non-zero, i.e., K ′ (0) ̸

Original: = 0 , then RFF cannot preserve relative error for such K .
Revised: = 0 , then RFF cannot preserve relative error for such K .

Original: To see this, we use Taylor’s expansion for K at the origin, and simply use the approximation to degree one, i.e., K ( x ) ≈ 1+ ax (noting that x ≤ 1 so this is a good approximation), where a = K ′ (0) .
Revised: To see this, we use Taylor’s expansion for K at the origin, and simply use the approximation to degree one, i.e., K ( x ) ≈ 1+ ax (noting that x ≤ 1 so this is a good approximation), where a = K ′ (0) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: So if a = K ′ (0) ̸ = 0 , then for sufficiently small ρ and | x | ≥ ρ , s K ( ρ ) ≥ Ω(1 /ρ ) .
Revised: So if a = K ′ (0) ̸ = 0 , then for sufficiently small ρ and | x | ≥ ρ , s K ( ρ ) ≥ Ω(1 /ρ ) .

Original: This also implies the claim in Theorem 1.1 for Laplacian kernels (even though one needs to slightly modify this analysis since strictly speaking K ′ is not well defined at 0 for Laplacian kernels).
Revised: This also implies the claim in Theorem 1.1 for Laplacian kernels (even though one needs to slightly modify this analysis since strictly speaking K ′ is not well defined at 0 for Laplacian kernels).

Original: As a sanity check, for shift-invariant kernels that are analytic at the origin (which include Gaussian kernels), it is necessary that K ′ (0) = 0 .
Revised: As a sanity check, for shift-invariant kernels that are analytic at the origin (which include Gaussian kernels), it is necessary that K ′ (0) = 0 .

Original:  B EYOND RFF: O BLIVIOUS E MBEDDING FOR L APLACIAN K ERNEL
Revised:  B EYOND RFF: O BLIVIOUS E MBEDDING FOR L APLACIAN K ERNEL

Original: In this section we provide a proof sketch for theorem 1.4.
Revised: In this section we provide a proof sketch for theorem 1.4.

Original: A more detailed proof is in section F.
Revised: A more detailed proof is deferred to section F.

Original: Embedding To handle a Laplacian kernel function K ( x, y )
Revised: Embedding To handle a Laplacian kernel function K ( x, y )

Original: =
Revised: =

Original: e − ∥ x − y ∥ 1 c with some constant c , we cannot directly use the RFF mapping ϕ  since our lower bound shows that the output dimension has to be very large when K is not analytical around the origin.
Revised: e − ∥ x − y ∥ 1 c with some constant c , we cannot directly use the RFF mapping ϕ , since our lower bound shows that the output dimension has to be very large when K is not analytical around the origin.

Original: To overcome this issue, we come up with the following idea.
Revised: To overcome this issue, we come up with the following idea.

Original: Notice that L ( x, y ) relies on the ℓ 1 -distance between x, y .
Revised: Notice that L ( x, y ) relies on the ℓ 1 -distance between x, y .

Original: If one can embed (embedding function f ) the data points from the original ℓ 1 metric space to a new metric space and ensure that there is ananalytical kernel function K ′  for the new space
Revised: If one can embed (embedding function f ) the data points from the original ℓ 1 metric space to a new metric space and ensure that there is an kernel function K ′ , analytical around the origin, for the new space

Original: s.t. K ( x, y )
Revised: s.t. K ( x, y )

Original: =
Revised: =

Original: K ′ ( f ( x ) , f ( y )) for every pair of original data points x, y , then one can use the function composition ϕ ◦ f to get a desired mapping.
Revised: K ′ ( f ( x ) , f ( y )) for every pair of original data points x, y , then one can use the function composition ϕ ◦ f to get a desired mapping.

Original: Indeed, we find that ℓ 1 can be embedded to ℓ 22 isometrically (Kahane, 1981) in the following way.
Revised: Indeed, we find that ℓ 1 can be embedded to ℓ 22 isometrically (Kahane, 1981) in the following way.

Original: Here for simplicity of exhibition we only handle the case where input data are from N d  .
Revised: Here for simplicity of exhibition we only handle the case where input data are from N d , upper bounded by a natural number N .

Original: Notice that even though input data points are only consisted of integers, the mapping construction needs to handle fractions, as we will later consider some numbers generated from Gaussian distributions or numbers computed in the RFF mapping.
Revised: Notice that even though input data points are only consisted of integers, the mapping construction needs to handle fractions, as we will later consider some numbers generated from Gaussian distributions or numbers computed in the RFF mapping.

Original: So we first setup two numbers, ∆ ′
Revised: So we first setup two numbers, ∆ ′

Original: =
Revised: =

Original: poly( N ) large enough and ρ ′
Revised: poly( N, δ − 1 ) large enough and ρ ′

Original: = 1 / poly( N ) small enough.
Revised: = 1 / poly( N, δ − 1 ) small enough.

Original: All our following operations are working on numbers that are (∆ ′ , ρ ′ )
Revised: All our following operations are working on numbers that are (∆ ′ , ρ ′ )

Original: -bounded.
Revised: -bounded.

Original: For each dimension we do the following transformation.
Revised: For each dimension we do the following transformation.

Original: Let π 1 : N → R N be such that for every x ∈ N , x ≤ N ,  π 1 ( x )[ j ] = 1 , if j ∈ [1 , x ] and π 1 ( x )[ j ] = 0 otherwise.
Revised: Let π 1 : N → R N be such that for every x ∈ N , x ≤ N , the first x entries of π 1 ( x ) is the number 1 , while all the remaining entries are 0 .

Original: Then consider all  dimensions.
Revised: Then consider all d dimensions.

Original: The embedding function π ( d ): N d → R Nd be such that for every x ∈ 
Revised: The embedding function π ( d ): N d → R Nd be such that for every x ∈ N d , x i ≤

Original: N d , x i ≤ N, ∀ i ∈ [ d ] , we have π ( d ) 1 ( x ) being the concatenation of d vectors π 1 ( x i ) , i ∈ [ d ] .
Revised: N, ∀ i ∈ [ d ] , we have π ( d ) 1 ( x ) being the concatenation of d vectors π 1 ( x i ) , i ∈ [ d ] .

Original: After embedding, consider a new kernel function 
Revised: After embedding, consider a new kernel function K ′ =

Original: [Equation] , where x ′ =
Revised: e − ∥ x ′− y ′∥ 22 c , where x ′ =

Original: π ( d ) 1 ( x ) , y ′ =
Revised: π ( d ) 1 ( x ) , y ′ =

Original: π ( d ) 1 ( y ) .
Revised: π ( d ) 1 ( y ) .

Original: One can see immediately that K ′ ( x ′ , y ′ )
Revised: One can see immediately that K ′ ( x ′ , y ′ )

Original: =
Revised: =

Original: K ( x, y ) .
Revised: K ( x, y ) .

Original: Hence, we can apply RFF then, i.e. the mapping is ϕ ◦ π ( d ) 1 , which has a small output dimension.
Revised: Hence, we can apply RFF then, i.e. the mapping is ϕ ◦ π ( d ) 1 , which has a small output dimension.

Original: Detailed proofs can be seen in section F.1.
Revised: Detailed proofs can be seen in section F.1.

Original: However, there is another issue.
Revised: However, there is another issue.

Original: In our setting, if the data is (∆ , ρ ) bounded, then we have to pick
Revised: In our setting, if the data is (∆ , ρ ) bounded, then we have to pick

Original: N = O ( ∆ ρ ) .
Revised: N = O ( ∆ ρ ) .

Original: The computing time has a linear factor in N , which is too large.
Revised: The computing time has a linear factor in N , which is too large.

Original: Polynomial Time Construction To reduce computing time, we start from the following observation about the RFF mapping ϕ
Revised: Polynomial Time Construction To reduce computing time, we start from the following observation about the RFF mapping ϕ

Original: ( x ′ ) .
Revised: ( x ′ ) .

Original: Each output dimension is actually a function of ⟨ ω, x ′ ⟩ , where ω is a vector of i.i.d Gaussian random variables.
Revised: Each output dimension is actually a function of ⟨ ω, x ′ ⟩ , where ω is a vector of i.i.d Gaussian random variables.

Original: For simplicity of description we only consider that x has only one dimension and x ′ =
Revised: For simplicity of description we only consider that x has only one dimension and x ′ =

Original: π 1 ( x ) .
Revised: π 1 ( x ) .

Original: So x ′ is just a vector consists of x number of 1 ’sentries starting from the left and then all the remaining entries are 0 ’s. Notice that a summation of Gaussian random variables is still a Gaussian.
Revised: So x ′ is just a vector consists of x number of 1 ’s starting from the left and then all the remaining entries are 0 ’s. Notice that a summation of Gaussian random variables is still a Gaussian.

Original: So given x , one can generate ⟨ ω, x ′ ⟩ according to the summation of Gaussians.
Revised: So given x , one can generate ⟨ ω, x ′ ⟩ according to the summation of Gaussians.

Original: But here comes another problem.
Revised: But here comes another problem.

Original: For two data points x, y , we need to use the same ω .
Revised: For two data points x, y , we need to use the same ω .

Original: So if we generate ⟨ ω, x ′ ⟩ andthen ⟨ ω ′ , y ′ ⟩ , then ω, ω ′ are independent.
Revised: So if we generate ⟨ ω, x ′ ⟩ and ⟨ ω ′ , y ′ ⟩ separately, then ω, ω ′ are independent.

Original: To bypass this issue, first consider the following alternate way to generate ⟨ ω, x ′ ⟩ .
Revised: To bypass this issue, first consider the following alternate way to generate ⟨ ω, x ′ ⟩ .

Original: Let h be the smallest integer s.t. N ≤ 2 h .
Revised: Let h be the smallest integer s.t. N ≤ 2 h .

Original: Consider a binary tree where each node has exactly 2 children.
Revised: Consider a binary tree where each node has exactly 2 children.

Original: The depth is h .
Revised: The depth is h .

Original: So it has exactly 2 h≥ N leaf nodes in the last layer.
Revised: So it has exactly 2 h leaf nodes in the last layer.

Original: For each node v , we attach a random variable α v in the following way.
Revised: For each node v , we attach a random variable α v in the following way.

Original: For the root, we attach a Gaussian variable which is the summation of 2 h independent Gaussian variable with distribution ω 0 .
Revised: For the root, we attach a Gaussian variable which is the summation of 2 h independent Gaussian variable with distribution ω 0 .

Original: Then we proceed layer by layer from the root to leaves.
Revised: Then we proceed layer by layer from the root to leaves.

Original: For each u, v being children of a common parent w , assume that α w is the summation of 2 l independent ω 0 distributions.
Revised: For each u, v being children of a common parent w , assume that α w is the summation of 2 l independent ω 0 distributions.

Original: Then let α u be the summation of the first 2 l − 1 distributions among them and α v  the summation of the second 2 l − 1 distributions.
Revised: Then let α u be the summation of the first 2 l − 1 distributions among them and α v be the summation of the second 2 l − 1 distributions.

Original: That is α w =
Revised: That is α w =

Original: α u + α v with α u , α v being independent.
Revised: α u + α v with α u , α v being independent.

Original: Notice that conditioned on α w = a , then α u takes the value b with probability Pr α u ,α v i.i.d.
Revised: Notice that conditioned on α w = a , then α u takes the value b with probability Pr α u ,α v i.i.d.

Original: [ α u = b | α u + α v = a ] .
Revised: [ α u = b | α u + α v = a ] .

Original: α v takes the value a − b when α u takes value b .
Revised: α v takes the value a − b when α u takes value b .

Original: The randomness for generating every random variable corresponding to a node, are presented as a sequence, in the order from root to leaves, layer by layer, from left to right.
Revised: The randomness for generating every random variable corresponding to a node, is presented as a sequence, in the order from root to leaves, layer by layer, from left to right.

Original: We define α x to be the summation of the random variables corresponding to the first x leaves.
Revised: We define α x to be the summation of the random variables corresponding to the first x leaves.

Original: Notice that α x can be sampled efficiently in the following way.
Revised: Notice that α x can be sampled efficiently in the following way.

Original: Consider the path from the root to the x -th leaf.
Revised: Consider the path from the root to the x -th leaf.

Original: First we sample the root, which can be computed using the corresponding  randomness.
Revised: First we sample the root, which can be computed using the corresponding part of the randomness.

Original: We use a variable z to record this sample outcome, calling z a accumulator for convenience.
Revised: We use a variable z to record this sample outcome, calling z an accumulator for convenience.

Original: Then we visit each node along the path.
Revised: Then we visit each node along the path.

Original: When visiting v , assume its parent is w , where α w has already been sampled previously with outcome a .
Revised: When visiting v , assume its parent is w , where α w has already been sampled previously with outcome a .

Original: If v is a left child of w , then we sample α v conditioned on α w = a .
Revised: If v is a left child of w , then we sample α v conditioned on α w = a .

Original: Assume this sampling has outcome b .
Revised: Assume this sampling has outcome b .

Original: Then we add − a + b to the current accumulator z .
Revised: Then we add − a + b to the current accumulator z .

Original: If v is a right child of a node w , then we keep the current accumulator z unchanged.
Revised: If v is a right child of a node w , then we keep the current accumulator z unchanged.

Original: After visiting all nodes in the path, z is the sample outcome for α x .
Revised: After visiting all nodes in the path, z is the sample outcome for α x .

Original: We can show that the joint distribution α x , x = 0 , 1 , . . . , N has  the same distribution as ⟨ ω, π 1 ( x ) ⟩ , x = 0 , 1 , . . . , N .
Revised: We can show that the joint distribution α x , α y has basically the same distribution as ⟨ ω, π 1 ( x ) ⟩ , ⟨ ω, π 1 ( y ) ⟩ .

Original: See lemma F.2.
Revised: See lemma F.2.

Original: The advantage of this alternate construction is that given any x , to generate α x , one only needs to visit the path from the root to the x -th leaf, using the above generating procedure.
Revised: The advantage of this alternate construction is that given any x , to generate α x , one only needs to visit the path from the root to the x -th leaf, using the above generating procedure.

Original: To finally reduce the time complexity, the last issue is that the uniform random string for generating random variables here is very long.
Revised: To finally reduce the time complexity, the last issue is that the uniform random string for generating random variables here is very long.

Original: If we sweep the random tape to locate the randomness used to generate a variable corresponding to a node, then we still need a linear time of N .
Revised: If we sweep the random tape to locate the randomness used to generate a variable corresponding to a node, then we still need a linear time of N .

Original: Fortunately, PRGs for space bounded computation, e.g. Nisan’s PRG (Nisan, 1992), can be used here to replace the uniform randomness.
Revised: Fortunately, PRGs for space bounded computation, e.g. Nisan’s PRG (Nisan, 1992), can be used here to replace the uniform randomness.

Original: Because the whole procedure for deciding if ∥ ϕ ◦ π 1 ( x ) − ϕ ◦ π 1 ( y ) ∥ 2 approximates (cid:112) 2 − K ( x, y ) within (1 ± ε ) multiplicative error, is in poly-logarithmic space.
Revised: Because the whole procedure for deciding whether ∥ ϕ ◦ π 1 ( x ) − ϕ ◦ π 1 ( y ) ∥ 2 approximates (cid:112) 2 − K ( x, y ) within (1 ± ε ) multiplicative error, is in poly-logarithmic space.

Original: Also the computation of such PRGs can be highly efficient, i.e. given any index of its output, one can compute that bit in time polynomial of the seed length, which is poly-logarithmic of N .
Revised: Also the computation of such PRGs can be highly efficient, i.e. given any index of its output, one can compute that bit in time polynomial of the seed length, which is poly-logarithmic of N .

Original: Hence the computing time of the mapping only has a factor poly-logarithmic in N instead of a factor linear in N .
Revised: Hence the computing time of the mapping only has a factor poly-logarithmic in N instead of a factor linear in N .

Original: Now we have shown our construction for the case that all input data points are from Nd .
Revised: Now we have shown our construction for the case that all input data points are from N .

Original: One can generalize this to the case where all numbers are (∆ , ρ ) bounded, by doing some simple roundings and shiftings of numbers.
Revised: One can generalize this to the case where all numbers are (∆ , ρ ) bounded, by doing some simple roundings and shiftings of numbers.

Revised: Then this can be further generalized to the case where the input data has d dimension, by simply handling each dimension and then concatenating them together.

Original: More details of this part are deferred to section F.4.
Revised: More details of this part are deferred to section F.4.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the empirical relative error of our methods on a simulated dataset.
Revised: We evaluate the empirical relative error of our methods on a simulated dataset.

Original: Specifically, we do two experiments, one to evaluate RFF on a Gaussian kernel, and the other one to evaluate the newalgorithm in Theorem F.1, which we call “New-Lap”, on a Laplacian kernel. In each experiment, we compare against two other popular methods, particularly Nyström and random-projection methods. Baselines. Observe that there are many possible implementations of these two methods.
Revised: Specifically, we do two experiments, one to evaluate RFF on a Gaussian kernel, and the other one to evaluate the new

Original: However, since we focus on the accuracy evaluation, we choose computationally-heavy but more accurate implementations as the two baselines (hence the evaluation of the error is only in the baseline’s favor).

Original: In particular, we consider 1) SVD low-rank approximation which we call “SVD”, and 2) the

Original: [Figure]
Revised: [Figure]

Original: R e l a t i v e
Revised: e l a t i v e

Revised: [Figure]

Original: E rr o r
Revised: rr o r

Original: MethodRFFSVDJL
Revised: MethodRFFSVDJL

Original: [Figure]
Revised: [Figure]

Revised: algorithm in Theorem F.1, which we call “New-Lap”, on a Laplacian kernel.

Revised: In each experiment, we compare against two other popular methods, particularly Nyström and random-projection methods.

Original:  vanilla Johnson-Lindenstrauss algorithm performed on top of the high-dimensional representation of points in the feature space, which we call “JL”.
Revised: Baselines. Observe that there are many possible implementations of these two methods. However, since we focus on the accuracy evaluation, we choose computationally-heavy but more accurate implementations as the two baselines (hence the evaluation of the error is only in the baseline’s favor). In particular, we consider 1) SVD low-rank approximation which we call “SVD”, and 2) the vanilla Johnson-Lindenstrauss algorithm performed on top of the high-dimensional representation of points in the feature space, which we call “JL”.

Original: Note that SVD is the “ideal” goal/form of Nyström methods and that Johnson-Lindenstrauss applied on the feature space can obtain a theoretically-tight target-dimension bound (in the worst-case sense).
Revised: Note that SVD is the “ideal” goal/form of Nyström methods and that Johnson-Lindenstrauss applied on the feature space can obtain a theoretically-tight target-dimension bound (in the worst-case sense).

Original: Experiment Setup.
Revised: Experiment Setup.

Original: Both experiments are conducted on a synthesized dataset X which consists of N = 100 points with d = 60 dimensions generated i.i.d.
Revised: Both experiments are conducted on a synthesized dataset X which consists of N = 100 points with d = 60 dimensions generated i.i.d.

Original: from a Gaussian distribution.
Revised: from a Gaussian distribution.

Original: For the experiment that we evaluate RFF, we use a Gaussian kernel K ( x ) = exp( − 0 .
Revised: For the experiment that we evaluate RFF, we use a Gaussian kernel K ( x ) = exp( − 0 .

Original: 5 · ∥ x ∥ 2 ) , and for that we evaluate New-Lap, we use a Laplacian kernel K ( x ) = exp( − 0 .
Revised: 5 · ∥ x ∥ 2 ) , and for that we evaluate New-Lap, we use a Laplacian kernel K ( x ) = exp( − 0 .

Original: ∥ x ∥ 1 ) .
Revised: ∥ x ∥ 1 ) .

Original: In each experiment, for each method, we run it for varying target dimension D (for SVD, D is the target rank), and we report its empirical relative error, which is defined as
Revised: In each experiment, for each method, we run it for varying target dimension D (for SVD, D is the target rank), and we report its empirical relative error, which is defined as

Original: [Equation]
Revised: [Equation]

Original: where d K is the kernel distance and d ′ is the approximated distance.
Revised: where d K is the kernel distance and d ′ is the approximated distance.

Original: To make the result stabilized, we conduct this entire experiment for every D for T = 20 times and report the average and 95% confident interval.
Revised: To make the result stabilized, we conduct this entire experiment for every D for T = 20 times and report the average and 95% confident interval.

Original: We plot these dimension-error tradeoff curves, and we depict the results in
Revised: We plot these dimension-error tradeoff curves, and we depict the results in

Original: Figure 1.
Revised: Figure 1.

Original: Results.
Revised: Results.

Original: We conclude that in both experiments, our methods can indeed well preserve the relative error of the kernel distance, which verifies our theorem.
Revised: We conclude that in both experiments, our methods can indeed well preserve the relative error of the kernel distance, which verifies our theorem.

Original: In particular, the dimension-error curve is comparable (and even slightly better) to the computationally heavy Johnson-Lindenstrauss algorithm (which is theoretically optimal in the worst case).
Revised: In particular, the dimension-error curve is comparable (and even slightly better) to the computationally heavy Johnson-Lindenstrauss algorithm (which is theoretically optimal in the worst case).

Original: On the contrary, the popular Nyström (low-rank approximation) method is largely incapable of preserving the relative error of the kernel distance.
Revised: On the contrary, the popular Nyström (low-rank approximation) method is largely incapable of preserving the relative error of the kernel distance.

Original: In fact, we observe that d ′ SV D ( x, y )
Revised: In fact, we observe that d ′ SV D ( x, y )

Original: =
Revised: =

Original: or ≈ 0 often happens for some pairs of ( x, y ) such that d ( x, y ) ̸ = 0 , which explains the high relative error.
Revised: or ≈ 0 often happens for some pairs of ( x, y ) such that d ( x, y ) ̸ = 0 , which explains the high relative error.

Original: This indicates that our methods can indeed well preserve the kernel distance in relative error, but existing methods struggle to achieve this.
Revised: This indicates that our methods can indeed well preserve the kernel distance in relative error, but existing methods struggle to achieve this.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Research is partially supported by a national key R&D program of China No. 2021YFA1000900, startup funds from Peking University, and the Advanced Institute of Information Technology, Peking
Revised: Research is partially supported by a national key R&D program of China No. 2021YFA1000900, startup funds from Peking University, and the Advanced Institute of Information Technology, Peking

Original: University.
Revised: University.

Original: Appendices A P ROOF OF L EMMA 3. 
Revised: Appendices A P ROOF OF L EMMA 3.

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: If for some r > 0 , K is analytic in { x ∈ R d : ∥ x ∥ 1 < r } , then for every k ≥ 1 being even and every x s.t.
Revised: If for some r > 0 , K is analytic in { x ∈ R d : ∥ x ∥ 1 < r } , then for every k ≥ 1 being even and every x s.t.

Original: ∥ x ∥ 1 < r , we have E [ | X i ( x ) | k ] ≤ (cid:18) 4 k ∥ x ∥r (cid:19) 2 k .
Revised: ∥ x ∥ 1 < r , we have E [ | X i ( x ) | k ] ≤ [Equation]

Original: We first introduce following two lemmas to show the properties of E [ X i ( x ) k ] .
Revised: We first introduce following two lemmas to show the properties of E [ X i ( x ) k ] .

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For any ω i sampled in RFF and k ≥ 0 , we have E [cos k ⟨ ω i , x ⟩ ] =
Revised: For any ω i sampled in RFF and k ≥ 0 , we have E [cos k ⟨ ω i , x ⟩ ] =

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By eq.
Revised: By eq.

Original: (1) it is sufficient to prove that
Revised: (1) it is sufficient to prove that

Original: [Equation]
Revised: [Equation]

Original: We prove this by induction.
Revised: We prove this by induction.

Original: In the case of k = 

Revised: In the case of k =

Original: the lemma holds obviously.
Revised: the lemma holds obviously.

Original: If for k the lemma holds, we have
Revised: If for k the lemma holds, we have

Original: [Equation]
Revised: [Equation]

Original: x ⟩
Revised: where in the third equality we use the fact that 2 cos α cos

Original: [Equation]
Revised: [Equation]

Original: where in the third equality we use the fact that 2 cos α cos β = cos( α + β ) + cos( α − β ) .
Revised: = [Equation] ( α − β ) .

Original: Lemma A.2.
Revised: Lemma A.2.

Original: If there exists r > 0 such that K is analytic in { x ∈ R d : ∥ x ∥ 1 < r } , then ∀ k ≥ 0 , lim 
Revised: If there exists r > 0 such that K is analytic in { x ∈ R d : ∥ x ∥ 1 < r } , then ∀ k ≥ 0 , lim x → 0 E [ X i ( x [Equation] k ] [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: c for some constant c .
Revised: c for some constant c .

Original: Proof.
Revised: Proof.

Original: We denote analytic function K ( x ) as Taylor series around origin as
Revised: We denote analytic function K ( x ) as Taylor series around origin as

Original: [Equation]
Revised: [Equation]

Original: where x β := (cid:81) di
Revised: where x β := (cid:81) di

Original: =1 x β i
Revised: =1 x β i

Original: i is a monomial and its coefficient is c β .
Revised: i is a monomial and its coefficient is c β .

Original: By definition, c 0 = 1 since K (0) = 1 .
Revised: By definition, c 0 = 1 since K (0) = 1 .

Original: We let s : N d → N , s ( β ) :
Revised: We let s : N d → N , s ( β ) :

Original: = (cid:80) di
Revised: = (cid:80) di

Original: =1 β i denote the degree of x β .
Revised: =1 β i denote the degree of x β .

Original: Since K ( x − y )
Revised: Since K ( x − y )

Original: =
Revised: =

Original: K ( x, y ) = K ( y, x ) =
Revised: K ( x, y ) = K ( y, x ) =

Original: K ( y − x ) by definition, hence K ( x ) is an even function, so
Revised: K ( y − x ) by definition, hence K ( x ) is an even function, so

Original: c β = 0 for s ( β ) odd.
Revised: c β = 0 for s ( β ) odd.

Original: Recall that X i ( x ) := cos ⟨ ω i , x ⟩ − K ( x ) .
Revised: Recall that X i ( x ) := cos ⟨ ω i , x ⟩ − K ( x ) .

Original: In the following, we drop the subscripts in X i , ω i and write X, ω for simplicity.
Revised: In the following, we drop the subscripts in X i , ω i and write X, ω for simplicity.

Original: By the definition of X we have
Revised: By the definition of X we have

Original: [Equation]
Revised: [Equation]

Original: Note that by Lemma A.1, E [cos k − i ⟨ ω, x ⟩ ] =
Revised: Note that by Lemma A.1, E [cos k − i ⟨ ω, x ⟩ ] =

Original: [Equation]
Revised: [Equation]

Original: i (cid:80) k − i j =0 (cid:0) k − i j (cid:1) K ((2 j − ( k − i )) x ) .
Revised: (cid:80) k − i j =0 (cid:0) k − i j (cid:1) K ((2 j − ( k − i )) x ) .

Original: Plug this in eq.
Revised: Plug this in eq.

Original: E [

Original: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: j

Original: [Equation] (cid:18) [Equation] where the second equality comes from the Tyler expansion of K ((2 j − k + i ) x ) .
Revised: where the second equality comes from the Tyler expansion of K ((2 j − k + i ) x ) .

Original: Next we will show that E [ X ( x ) k ] is of degree at least 2 k .
Revised: Next we will show that E [ X ( x ) k ] is of degree at least 2 k .

Original: For β = 0 note that (cid:80) ki =0 (cid:0) ki (cid:1) K
Revised: For β = 0 note that (cid:80) ki =0 (cid:0) ki (cid:1) K

Original: ( x ) i ( − 1) k − i =
Revised: ( x ) i ( − 1) k − i =

Original: ( K ( x ) − 1) k , since K ( x ) is even and K (0)
Revised: ( K ( x ) − 1) k , since K ( x ) is even and K (0)

Original: = 1 , we have lim x → 0 ( K ( x ) − 1) k x t = 0 , ∀ t < 2 k .
Revised: = 1 , we have lim x → 0 ( K ( x ) − 1) k x t = 0 , ∀ t < 2 k .

Original: For β ̸ = 0 , we next show that every term of degree less than 2 k has coefficient zero.
Revised: For β ̸ = 0 , we next show that every term of degree less than 2 k has coefficient zero.

Original: Fix β ̸
Revised: Fix β ̸

Original: = 0 and take Tyler expansion for K ( x ) i
Revised: = 0 and take Tyler expansion for K ( x ) i

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, we assume β l +1 , ..., β i are all β s that equals 0 , so we have c β l +

Revised: Without loss of generality, we assume β l +1 , ..., β i are all β s that equals 0 , so we have c β l +

Original: = ...
Revised: = ...

Original: =
Revised: =

Original: c β i = 1 .
Revised: c β i = 1 .

Original: Now we consider the coefficient of term
Revised: Now we consider the coefficient of term

Original: c β c β 1 c β 2 . . .
Revised: c β c β 1 c β 2 . . .

Original: , c
Revised: , c

Original: β l x β + (cid:80) lj
Revised: β l x β + (cid:80) lj

Original: =1 β j , which would be:
Revised: =1 β j , which would be:

Original: [Equation]
Revised: [Equation]

Original: where ˜ C is the number of ordered sequence ( β 1 , β 2 , . .
Revised: where ˜ C is the number of ordered sequence ( β 1 , β 2 , . .

Original: . , β l ) , here, for β 1 = β 2 , ( β 1 , β 2 , . .
Revised: . , β l ) , here, for β 1 = β 2 , ( β 1 , β 2 , . .

Original: . , β l ) and ( β 2 , β 1 , . .
Revised: . , β l ) and ( β 2 , β 1 , . .

Original: . , β l ) are equivalent.
Revised: . , β l ) are equivalent.

Original: Next, we show if the degree of a monomial s ( β ) + (cid:80) lj
Revised: Next, we show if the degree of a monomial s ( β ) + (cid:80) lj

Original: =1 s ( β j ) < 2 k , its coefficient is zero.
Revised: =1 s ( β j ) < 2 k , its coefficient is zero.

Original: Since all β j ̸ = 0 , we may assume s ( β j ) ≥ 2 , therefore s ( β ) <
Revised: Since all β j ̸ = 0 , we may assume s ( β j ) ≥ 2 , therefore s ( β ) <

Original: 2 k − 2 l .
Revised: 2 k − 2 l .

Original: Suppose operator J is a mapping from a function space to itself, such that ∀ f : R → R , J ( f ) :
Revised: Suppose operator J is a mapping from a function space to itself, such that ∀ f : R → R , J ( f ) :

Original: R → R is defined by J ( f )( x ) :
Revised: R → R is defined by J ( f )( x ) :

Original: = f ( x + 1) .
Revised: = f ( x + 1) .

Original: Denote J 1 =
Revised: Denote J 1 =

Original: J, J k :
Revised: J, J k :

Original: =
Revised: =

Original: J ◦ J k − 1 as its k -time composition, define J 0 to be the identity mapping such that J 0 ( f ) = f .
Revised: J ◦ J k − 1 as its k -time composition, define J 0 to be the identity mapping such that J 0 ( f ) = f .

Original: Similarly we can define addition that ( J 1 + J 2 )( f ) =
Revised: Similarly we can define addition that ( J 1 + J 2 )( f ) =

Original: J 1 ( f ) + J 2 ( f ) and scalar multiplication that ( αJ )( f )
Revised: J 1 ( f ) + J 2 ( f ) and scalar multiplication that ( αJ )( f )

Original: =
Revised: =

Original: α ( J ( f )) .
Revised: α ( J ( f )) .

Original: By definition,
Revised: By definition,

Original: [Equation]
Revised: [Equation]

Original: Let L ( x ) =
Revised: Let L ( x ) =

Original: x s ( β ) , the coefficient can be rewritten as:
Revised: x s ( β ) , the coefficient can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: Let P =
Revised: Let P =

Original: ˜ C
Revised: ˜ C

Original: (cid:80) ki =0 (cid:0) ki (cid:1)(cid:0) il (cid:1) (cid:0) − 12 (cid:1) k − i (cid:80) k − i j =0 (cid:0) k − i j (cid:1)
Revised: (cid:80) ki =0 (cid:0) ki (cid:1)(cid:0) il (cid:1) (cid:0) − 12 (cid:1) k − i (cid:80) k − i j =0 (cid:0) k − i j (cid:1)

Original: J 2 j
Revised: J 2 j

Original: + i ( L ) , the above is P ( − k ) .
Revised: + i ( L ) , the above is P ( − k ) .

Original: Now we show P ≡
Revised: Now we show P ≡

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: = (cid:18) k
Revised: = (cid:18) k

Original: i (cid:19)(cid:18) i l (cid:19) (cid:18) − (cid:19) k
Revised: i (cid:19)(cid:18) i l (cid:19) (cid:18) − (cid:19) k

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ◦  
Revised: ◦  

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: j = (cid:18)
Revised: j = (cid:18)

Original: [Equation]
Revised: [Equation]

Original: Note that (cid:16) J − J 0 + J 2(cid:17)
Revised: Note that (cid:16) J − J 0 + J 2(cid:17)

Original: [Equation]
Revised: [Equation]

Original: f )
Revised: f )

Original: [Equation]
Revised: [Equation]

Original: x ) =
Revised: x ) =

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: x + 1) − f
Revised: x + 1) − f

Original: [Equation]
Revised: [Equation]

Original: x )) / 2 −
Revised: x )) / 2 −

Original: [Equation]
Revised: [Equation]

Original: f ( x + 2) − f ( x + 1)) / 2 calculates second order difference, namely, ∀ f
Revised: f ( x + 2) − f ( x + 1)) / 2 calculates second order difference, namely, ∀ f

Original: that is a polynomial of degree k ≥ 2 , (cid:16) J − J 0 + J 2(cid:17)
Revised: that is a polynomial of degree k ≥ 2 , (cid:16) J − J 0 + J 2(cid:17)

Original: ( f ) is a polynomial of degree k − 2 , and ∀ f that is a polynomial of degree k < 2 , (cid:16) J − J 0 + J 2(cid:17)
Revised: ( f ) is a polynomial of degree k − 2 , and ∀ f that is a polynomial of degree k < 2 , (cid:16) J − J 0 + J 2(cid:17)

Original: ( f ) is 0 .
Revised: ( f ) is 0 .

Original: Since L is a polynomial of degree less than 2( k − l ) , we have
Revised: Since L is a polynomial of degree less than 2( k − l ) , we have

Original: [Equation]
Revised: [Equation]

Original: Combining the above two cases, we have proved eq.
Revised: Combining the above two cases, we have proved eq.

Original: (4) is of degree at least 2 k , which completes our proof.
Revised: (4) is of degree at least 2 k , which completes our proof.

Original: Proof of Lemma 3.1.
Revised: Proof of Lemma 3.1.

Original: If 2 k ∥ x
Revised: If 2 k ∥ x

Original: ∥ 1 ≥ r , since | X i ( x ) | = | cos ⟨ ω i , x ⟩ − K ( x ) | ≤ 2 , we have
Revised: ∥ 1 ≥ r , since | X i ( x ) | = | cos ⟨ ω i , x ⟩ − K ( x ) | ≤ 2 , we have

Original: E
Revised: E

Original: [ X i ( x ) k ] ≤ 2 k ≤ (cid:18) 4 k ∥ x
Revised: [ X i ( x ) k ] ≤ 2 k ≤ (cid:18) 4 k ∥ x

Original: ∥ 1 r (cid:19) 2 k .
Revised: ∥ 1 r (cid:19) 2 k .

Original: Otherwise ∥ x ∥ 1 < r/ 2 k .
Revised: Otherwise ∥ x ∥ 1 < r/ 2 k .

Original: Define g k ( x ) :
Revised: Define g k ( x ) :

Original: = E [ X i ( x ) k ] , we have:
Revised: = E [ X i ( x ) k ] , we have:

Original: [Equation]
Revised: [Equation]

Original: where the second equation comes from Lemma A.2 and Taylor expansion with Lagrange remainder.
Revised: where the second equation comes from Lemma A.2 and Taylor expansion with Lagrange remainder.

Original: Lemma A.3 (Cauchy’s integral formula for multivariate functions Hormander 1966) .
Revised: Lemma A.3 (Cauchy’s integral formula for multivariate functions Hormander 1966) .

Original: For f ( z 1 , ..., z d ) analytic in ∆( z, r ) = (cid:8) ζ
Revised: For f ( z 1 , ..., z d ) analytic in ∆( z, r ) = (cid:8) ζ

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: If in addition | f | < M , we have the following evaluation:
Revised: If in addition | f | < M , we have the following evaluation:

Original: [Equation]
Revised: [Equation]

Original: Recall that g k ( x ) =
Revised: Recall that g k ( x ) =

Original: E [ X i
Revised: E [ X i

Original: [Equation]
Revised: [Equation]

Original: ] =
Revised: ] =

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: j −
Revised: j −

Original: [Equation]
Revised: [Equation]

Original: k − i ))
Revised: k − i ))

Original: x ) , so g k ( x ) = poly( K ( x ) , K ( − x ) , . .
Revised: x ) , so g k ( x ) = poly( K ( x ) , K ( − x ) , . .

Original: , K ( kx ) , K ( − kx )) is analytic when ∥ x ∥ 1 ≤ r/k .
Revised: , K ( kx ) , K ( − kx )) is analytic when ∥ x ∥ 1 ≤ r/k .

Original: Applying
Revised: Applying

Original: Cauchy’s integral formula Lemma A.3 (here ∥ z + θx ∥ 1 ≤ 2 · r/ 2 k is in the analytic area), g
Revised: Cauchy’s integral formula Lemma A.3 (here ∥ z + θx ∥ 1 ≤ 2 · r/ 2 k is in the analytic area), g

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: 1 d d z we have | g k ( x ) | ≤ sup
Revised: 1 d d z we have | g k ( x ) | ≤ sup

Original: | z i | = r/ 2 k | g k ( z + θx ) | (cid:18) 2 k r (cid:19) 2 k (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) d (cid:88) i
Revised: | z i | = r/ 2 k | g k ( z + θx ) | (cid:18) 2 k r (cid:19) 2 k (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) d (cid:88) i

Original: = x i (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) 2 k ≤ (cid:18) 4 k ∥ x ∥r (cid:19) 2 k .
Revised: = x i (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) 2 k ≤ (cid:18) 4 k ∥ x ∥r (cid:19) 2 k .

Original: B P ROOF OF L EMMA 3. 
Revised: B P ROOF OF L EMMA 3.

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: For kernel K which is shift-invariant and analytic at the origin, there exist c K , R K >such that for all ∥ x ∥ 1 ≤ R K , 1 − K ( x ) ∥ x ∥≥ c K 2 .
Revised: For kernel K which is shift-invariant and analytic at the origin, there exist c K , R K >such that for all ∥ x ∥ 1 ≤ R K , 1 − K ( x ) ∥ x ∥≥ c K 2 .

Original: Proof.
Revised: Proof.

Original: It suffices to prove that lim inf x → 0 1 − K ( x ) ∥ x ∥≥ c > 0 , for some c .
Revised: It suffices to prove that lim inf x → 0 1 − K ( x ) ∥ x ∥≥ c > 0 , for some c .

Original: Towards proving this, we show that K is strongly convex at origin.
Revised: Towards proving this, we show that K is strongly convex at origin.

Original: In fact, by definition, ℜ (cid:82) R d p ( ω ) e i ⟨ ω,tx ⟩ d ω = K ( tx ) for every fixed x , therefore K ′′ ( tx ) =
Revised: In fact, by definition, ℜ (cid:82) R d p ( ω ) e i ⟨ ω,tx ⟩ d ω = K ( tx ) for every fixed x , therefore K ′′ ( tx ) =

Original: ℜ (cid:82) R d ∥ ω ∥ 2 p ( ω ) e i ⟨ ω,tx ⟩ d ω > 0 , hence K ( tx ) is strongly convex with respect to t at origin, so is K ( x ) .
Revised: ℜ (cid:82) R d ∥ ω ∥ 2 p ( ω ) e i ⟨ ω,tx ⟩ d ω > 0 , hence K ( tx ) is strongly convex with respect to t at origin, so is K ( x ) .

Original: C P ROOF OF T HEOREM 1. 
Revised: C P ROOF OF T HEOREM 1.

Original: Theorem 1.2 (Upper bound) .
Revised: Theorem 1.2 (Upper bound) .

Original: Let K : R d × R d → R be a kernel function which is shift-invariant and analytic at the origin, with feature mapping φ : R d → H for some feature space H .
Revised: Let K : R d × R d → R be a kernel function which is shift-invariant and analytic at the origin, with feature mapping φ : R d → H for some feature space H .

Original: For every 0 < δ ≤ ε ≤ 2 − 16 , every d, D ∈ N , D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , if π : R d → R D is an RFF mapping for K with target dimension D , then for every x, y ∈ R d ,
Revised: For every 0 < δ ≤ ε ≤ 2 − 16 , every d, D ∈ N , D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , if π : R d → R D is an RFF mapping for K with target dimension D , then for every x, y ∈ R d ,

Original: Pr[ | dist π ( x, y ) − dist φ ( x, y ) | ≤ ε · dist φ ( x, y )] ≥ 1 − δ.
Revised: Pr[ | dist π ( x, y ) − dist φ ( x, y ) | ≤ ε · dist φ ( x, y )] ≥ 1 − δ.

Original: Proof.
Revised: Proof.

Original: When ∥ x − y ∥ 1 ≥ R K , consider the function g ( t )
Revised: When ∥ x − y ∥ 1 ≥ R K , consider the function g ( t )

Original: =
Revised: =

Original: K ( t ( x − y )) .
Revised: K ( t ( x − y )) .

Original: It follows from definition that g ′ (0) = 0 , g ′′ ( t ) =
Revised: It follows from definition that g ′ (0) = 0 , g ′′ ( t ) =

Original: −ℜ (cid:82) R d ∥ ω ∥ 2 ∥ x − y ∥
Revised: −ℜ (cid:82) R d ∥ ω ∥ 2 ∥ x − y ∥

Original: 2 p ( ω ) e i ⟨ ω,t ( x − y ) ⟩ d ω < 0 , so g ( t ) strictly decreases for all t > 0 .
Revised: 2 p ( ω ) e i ⟨ ω,t ( x − y ) ⟩ d ω < 0 , so g ( t ) strictly decreases for all t > 0 .

Original: So 2 − 2 K ( x − y ) ≥ 2 − 2 max ∥ x − y ∥ 1 = R K K ( x − y ) >
Revised: So 2 − 2 K ( x − y ) ≥ 2 − 2 max ∥ x − y ∥ 1 = R K K ( x − y ) >

Original: We denote t = 2 − 2 max ∥ x − y ∥ 1 = R K K ( x − y ) , so by Chernorff bound, when D ≥ 12 t (ln 1 δ + ln 2) , we have:
Revised: We denote t = 2 − 2 max ∥ x − y ∥ 1 = R K K ( x − y ) , so by Chernorff bound, when D ≥ 12 t (ln 1 δ + ln 2) , we have:

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: 2 D
Revised: 2 D

Original: D (cid:88) i =
Revised: D (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: When ∥ x − y ∥ 1 ≤ R K , by Lemma 3.2, we have 2 − 2 K ( x − y ) ≥ c
Revised: When ∥ x − y ∥ 1 ≤ R K , by Lemma 3.2, we have 2 − 2 K ( x − y ) ≥ c

Original: ∥ x − y ∥ 2 1 .
Revised: ∥ x − y ∥ 2 1 .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: We take r = r K for simplicity of exhibition.
Revised: We take r = r K for simplicity of exhibition.

Original: Assume δ ≤ min { ε, 2 − 16 } , let k = log(

Revised: Assume δ ≤ min { ε, 2 − 16 } , let k = log(

Original: D 2 /δ ) , t = 64 k 2 /r 2 , is even.
Revised: D 2 /δ ) , t = 64 k 2 /r 2 , is even.

Original: By Markov’s inequality and Lemma 3.1:
Revised: By Markov’s inequality and Lemma 3.1:

Original: [Equation]
Revised: [Equation]

Original: For simplicity denote X i ( x − y ) by X i , ∥ x − y ∥ 21 by ℓ and define X ′ i = 

Revised: For simplicity denote X i ( x − y ) by X i , ∥ x − y ∥ 21 by ℓ and define X ′ i =

Original: [ | X i |≥ tℓ ] tℓ · sgn( X i )
Revised: [ | X i |≥ tℓ ] tℓ · sgn( X i )

Original: +[ | X i | <tℓ ] X i , note that E [ X i ] = 0 .
Revised: +[ | X i | <tℓ ] X i , note that E [ X i ] = 0 .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Pr[ X i ≥ tℓ ] − Pr
Revised: Pr[ X i ≥ tℓ ] − Pr

Original: [ X i ≤ − tℓ ] | where tℓ · | Pr[ X i > tℓ ] − Pr[ X i < − tℓ ] | ≤ tℓ · Pr[ | X i | > tℓ ] ≤ tℓδ/ (2 D 2 ) .
Revised: [ X i ≤ − tℓ ] | where tℓ · | Pr[ X i > tℓ ] − Pr[ X i < − tℓ ] | ≤ tℓ · Pr[ | X i | > tℓ ] ≤ tℓδ/ (2 D 2 ) .

Original: The first inequality is by considering the two conditions | X i | <
Revised: The first inequality is by considering the two conditions | X i | <

Original: tℓ and | X i | ≥ tℓ , then taking a triangle inequality.
Revised: tℓ and | X i | ≥ tℓ , then taking a triangle inequality.

Original: The first and second equations are by definition of X i , X
Revised: The first and second equations are by definition of X i , X

Original: ′ i .
Revised: ′ i .

Original: The third equation is a straightforward computation.
Revised: The third equation is a straightforward computation.

Original: The last equation is due to E [ X i ] = 0 .
Revised: The last equation is due to E [ X i ] = 0 .

Original: By Lemma 3.1 for every integer α ,
Revised: By Lemma 3.1 for every integer α ,

Original: [Equation]
Revised: [Equation]

Original: The first equality is straightforward.
Revised: The first equality is straightforward.

Original: The first inequality is by Markov.
Revised: The first inequality is by Markov.

Original: The second equality is by
Revised: The second equality is by

Original: E [ | X i | √ α/ 8 ] ≤ (cid:0) αℓ 4 r 2 (cid:1) √ α/ 8 which follows from Lemma 3.1, and a rearrangement of parameters, where r is the parameter r in Lemma 3.1.
Revised: E [ | X i | √ α/ 8 ] ≤ (cid:0) αℓ 4 r 2 (cid:1) √ α/ 8 which follows from Lemma 3.1, and a rearrangement of parameters, where r is the parameter r in Lemma 3.1.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: The first inequality is by the property of absolute value.
Revised: The first inequality is by the property of absolute value.

Original: The second inequality is because we can divide the event |
Revised: The second inequality is because we can divide the event |

Original: X i | ≥ tℓ into | X i | ∈ [ αℓ/r 2 , ( α + 1) ℓ/r 2 ) , α = tr 2 , tr 2 + 1 , . . .
Revised: X i | ≥ tℓ into | X i | ∈ [ αℓ/r 2 , ( α + 1) ℓ/r 2 ) , α = tr 2 , tr 2 + 1 , . . .

Original: and when | X i | ∈ [ αℓ/r 2 , ( α + 1) ℓ/r 2 ) , | X i | < ( α + 1) ℓ/r 2 .
Revised: and when | X i | ∈ [ αℓ/r 2 , ( α + 1) ℓ/r 2 ) , | X i | < ( α + 1) ℓ/r 2 .

Original: The third inequality is by pluging in the previous bound for Pr[ | X i | ≥ αℓ/r 2 ] .
Revised: The third inequality is by pluging in the previous bound for Pr[ | X i | ≥ αℓ/r 2 ] .

Original: The last inequality is by a calculation of the integral.
Revised: The last inequality is by a calculation of the integral.

Original: By plugging in parameters t , δ , D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , we have E [ | X ′ i | ] ≤ δℓ .
Revised: By plugging in parameters t , δ , D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , we have E [ | X ′ i | ] ≤ δℓ .

Original: Note that the Θ( D ) hides a constant r .
Revised: Note that the Θ( D ) hides a constant r .

Original: Denote σ ′ 2 as the variance of X ′ i .
Revised: Denote σ ′ 2 as the variance of X ′ i .

Original: So σ ′ ≤ 64 ℓ/r 2 by Lemma 3.1.
Revised: So σ ′ ≤ 64 ℓ/r 2 by Lemma 3.1.

Original: Lemma C.1 (Bernstein’s Inequality) .
Revised: Lemma C.1 (Bernstein’s Inequality) .

Original: Let X 1 , .., X D be independent zero-mean random variables.
Revised: Let X 1 , .., X D be independent zero-mean random variables.

Original: Suppose that | X i | ≤ M, ∀ i , then for all positive t ,
Revised: Suppose that | X i | ≤ M, ∀ i , then for all positive t ,

Original: [Equation]
Revised: [Equation]

Original: Applying Bernstein’s Inequality to X ′ i ,
Revised: Applying Bernstein’s Inequality to X ′ i ,

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: 4 σ ′ 2 /ℓ(cid:19)(cid:27) .
Revised: 4 σ ′ 2 /ℓ(cid:19)(cid:27) .

Original: Since D ≥ max { Θ (cid:0) tε − 1 log(1 /δ ) (cid:1) , Θ (cid:0) ε − 2 log(1 /δ ) (cid:1) } , we have Pr (cid:104)(cid:80) Di =1 X
Revised: Since D ≥ max { Θ (cid:0) tε − 1 log(1 /δ ) (cid:1) , Θ (cid:0) ε − 2 log(1 /δ ) (cid:1) } , we have Pr (cid:104)(cid:80) Di =1 X

Original: i ≥ ε ( ℓ/σ ′ )
Revised: i ≥ ε ( ℓ/σ ′ )

Original: Dσ ′ (cid:105) ≤ δ/ 2 .
Revised: Dσ ′ (cid:105) ≤ δ/ 2 .

Original: With 1 − δ 2 probability, every X i ≤ tℓ, X ′ i
Revised: With 1 − δ 2 probability, every X i ≤ tℓ, X ′ i

Original: =
Revised: =

Original: X i .
Revised: X i .

Original: Therefore, Pr (cid:104)(cid:80) Di =1 X
Revised: Therefore, Pr (cid:104)(cid:80) Di =1 X

Original: i ≥ D ( δ + ε ) ℓ (cid:105) ≤ δ/ 2 .
Revised: i ≥ D ( δ + ε ) ℓ (cid:105) ≤ δ/ 2 .

Original: Combine it together, Pr[ | dist π ( x, y ) − dist φ ( x, y ) | ≤ ε · dist φ ( x, y )] ≥ 1 − δ .
Revised: Combine it together, Pr[ | dist π ( x, y ) − dist φ ( x, y ) | ≤ ε · dist φ ( x, y )] ≥ 1 − δ .

Original: D P ROOF OF T HEOREM 3. 
Revised: D P ROOF OF T HEOREM 3.

Original: Theorem 3.1 (Generalization of Makarychev et al.
Revised: Theorem 3.1 (Generalization of Makarychev et al.

Original: 2019, Theorem 3.6) .
Revised: 2019, Theorem 3.6) .

Original: For kernel k -clustering problem with ℓ p 2 -objective whose kernel function K : R d × R d → R is shift-invariant and analytic at the origin, for every data set P ⊂ R d , the RFF mapping π : R d → R D with target dimension
Revised: For kernel k -clustering problem with ℓ p 2 -objective whose kernel function K : R d × R d → R is shift-invariant and analytic at the origin, for every data set P ⊂ R d , the RFF mapping π : R d → R D with target dimension

Original: D = Ω( p 2 log 3 kα + p 5 log 3 1 ε + p 8 )
Revised: D = Ω( p 2 log 3 kα + p 5 log 3 1 ε + p 8 )

Original: /ε 2 satisfies
Revised: /ε 2 satisfies

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: C of
Revised: C of

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: The proof relies on a key notion of ( ε, δ, ρ ) -dimension reduction from (Makarychev et al., 2019), and we adopt it with respect to our setting/language of kernel distance as follows.
Revised: The proof relies on a key notion of ( ε, δ, ρ ) -dimension reduction from (Makarychev et al., 2019), and we adopt it with respect to our setting/language of kernel distance as follows.

Original: Definition D.1 (Makarychev et al.
Revised: Definition D.1 (Makarychev et al.

Original: 2019, Definition 2.1) .
Revised: 2019, Definition 2.1) .

Original: For ε, δ, ρ > 0 , a feature mapping φ : R d → H for some Hilbert space H , a random mapping π d,D : R d → R D is an ( ε, δ, ρ ) -dimension reduction, if
Revised: For ε, δ, ρ > 0 , a feature mapping φ : R d → H for some Hilbert space H , a random mapping π d,D : R d → R D is an ( ε, δ, ρ ) -dimension reduction, if

Original: • for every x, y ∈ R d , 11+ ε dist φ ( x, y ) ≤ dist π ( x, y ) ≤ (1 + ε ) dist φ ( x, y ) with probability at least 1 − δ , and • for every fixed p ∈ [1 , ∞ ) , E (cid:104) 1 { dist π ( x,y ) >
Revised: • for every x, y ∈ R d , 11+ ε dist φ ( x, y ) ≤ dist π ( x, y ) ≤ (1 + ε ) dist φ ( x, y ) with probability at least 1 − δ , and • for every fixed p ∈ [1 , ∞ ) , E (cid:104) 1 { dist π ( x,y ) >

Original: (1+ ε )dist φ ( x,y ) } (cid:16) dist π ( x,y ) p dist φ ( x,y ) p − (1 + ε ) p (cid:17)(cid:105) ≤ ρ .
Revised: (1+ ε )dist φ ( x,y ) } (cid:16) dist π ( x,y ) p dist φ ( x,y ) p − (1 + ε ) p (cid:17)(cid:105) ≤ ρ .

Original: In Makarychev et al.
Revised: In Makarychev et al.

Original: , most results are stated for a particular parameter setup of Definition D. resulted from Johnson-Lindenstrauss transform (Johnson & Lindenstrauss, 1984), but their analysis actually works for other similar parameter setups.
Revised: , most results are stated for a particular parameter setup of Definition D. resulted from Johnson-Lindenstrauss transform (Johnson & Lindenstrauss, 1984), but their analysis actually works for other similar parameter setups.

Original: The following is a generalized statement of (Makarychev et al., 2019, Theorem 3.5) which also reveals how alternative parameter setups affect the distortion.
Revised: The following is a generalized statement of (Makarychev et al., 2019, Theorem 3.5) which also reveals how alternative parameter setups affect the distortion.

Original: We note that this is simply a more precise and detailed statement of (Makarychev et al., 2019, Theorem 3.5), and it follows from exactly the same proof in Makarychev et al.
Revised: We note that this is simply a more precise and detailed statement of (Makarychev et al., 2019, Theorem 3.5), and it follows from exactly the same proof in Makarychev et al.

Original: Lemma D.1 (Makarychev et al.
Revised: Lemma D.1 (Makarychev et al.

Original: 2019, Theorem 3.5) .
Revised: 2019, Theorem 3.5) .

Original: Let 0 < ε, α < 1 and θ := min { ε p +1 3 − ( p +1)
Revised: Let 0 < ε, α < 1 and θ := min { ε p +1 3 − ( p +1)

Original: ( p +2) , αε p / (10 k (1 + ε ) 4 p − 1 ) , 1 / 10 p +1 } .
Revised: ( p +2) , αε p / (10 k (1 + ε ) 4 p − 1 ) , 1 / 10 p +1 } .

Original: If some ( ε, δ, ρ ) -dimension reduction π for feature mapping φ : R d → H of some kernel function satisfies δ ≤ min( θ 7 / 600 , θ/k ) , (cid:0) k 2 (cid:1) δ ≤ α 2 , ρ ≤ θ , then with probability at least 1 − α , for every partition C of P ,
Revised: If some ( ε, δ, ρ ) -dimension reduction π for feature mapping φ : R d → H of some kernel function satisfies δ ≤ min( θ 7 / 600 , θ/k ) , (cid:0) k 2 (cid:1) δ ≤ α 2 , ρ ≤ θ , then with probability at least 1 − α , for every partition C of P ,

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 3.1.
Revised: Proof of Theorem 3.1.

Original: We verify that setting D = Θ(log
Revised: We verify that setting D = Θ(log

Original: 3 kα + p 3 log 3 1 ε + p 6 ) /ε 2 , the RFF mapping π with target dimension D satisfies the conditions in Lemma D.1, namely, it is a ( ε, δ, ρ ) - dimension reduction .
Revised: 3 kα + p 3 log 3 1 ε + p 6 ) /ε 2 , the RFF mapping π with target dimension D satisfies the conditions in Lemma D.1, namely, it is a ( ε, δ, ρ ) - dimension reduction .

Original: In fact, Theorem 1.2 already implies such π satisfies that for every x, y ∈ R d ,
Revised: In fact, Theorem 1.2 already implies such π satisfies that for every x, y ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: dist
Revised: dist

Original: [Equation]
Revised: [Equation]

Original: x, y ) with probability at least 1 − δ , where δ = e − cf ( ε,D ) for some constant c , and f ( ε, D ) := max { ε 2 D, ε 1 / 3 D 1 / 3 } .
Revised: x, y ) with probability at least 1 − δ , where δ = e − cf ( ε,D ) for some constant c , and f ( ε, D ) := max { ε 2 D, ε 1 / 3 D 1 / 3 } .

Original: For the other part,
Revised: For the other part,

Original: [Equation]
Revised: [Equation]

Original: Where the third equality follows by Pr (cid:16) dist
Revised: Where the third equality follows by Pr (cid:16) dist

Original: π ( x,y ) dist φ ( x,y ) > m (cid:17) decays exponentially fast with respect to m .
Revised: π ( x,y ) dist φ ( x,y ) > m (cid:17) decays exponentially fast with respect to m .

Original: Observe that for p ≥ 1 , D ≥ ( p − 1) 3 8 c 3 , p (1 + t ) p − 1 e − cD 13 t 13 / 2 decrease when t ≥ ε , and for
Revised: Observe that for p ≥ 1 , D ≥ ( p − 1) 3 8 c 3 , p (1 + t ) p − 1 e − cD 13 t 13 / 2 decrease when t ≥ ε , and for

Original: D ≥ c ( p − 1) ε 2 , p (1 + t ) p − 1 e − ct 2 D/ 2 decrease when t ≥ ε .
Revised: D ≥ c ( p − 1) ε 2 , p (1 + t ) p − 1 e − ct 2 D/ 2 decrease when t ≥ ε .

Original: Hence for D ≥ max { ( p − 1) 3 8 c 3 , c ( p − 1) ε 2 } , we have
Revised: Hence for D ≥ max { ( p − 1) 3 8 c 3 , c ( p − 1) ε 2 } , we have

Original: [Equation]
Revised: [Equation]

Original: In conclusion, by setting D = Θ(log 3 kα + p 3 log 3 1 ε + p 6 ) /ε 2 , for δ = e − cf ( ε,D ) , ρ = c ′′ e − cf ( ε,D ) and f ( ε, D )
Revised: In conclusion, by setting D = Θ(log 3 kα + p 3 log 3 1 ε + p 6 ) /ε 2 , for δ = e − cf ( ε,D ) , ρ = c ′′ e − cf ( ε,D ) and f ( ε, D )

Original: = max { ε 2 D, ε 1 / 3 D 1 / 3 } , it satisfies δ ≤ min( θ 7 / 600 , θ/k ) , (cid:0) k 2 (cid:1) δ ≤ α 2 , ρ ≤ θ .
Revised: = max { ε 2 D, ε 1 / 3 D 1 / 3 } , it satisfies δ ≤ min( θ 7 / 600 , θ/k ) , (cid:0) k 2 (cid:1) δ ≤ α 2 , ρ ≤ θ .

Original: This verifies the condition of Lemma D.1.
Revised: This verifies the condition of Lemma D.1.

Original: Finally, we conclude the proof of Theorem 3.1 by plugging ε ′ =
Revised: Finally, we conclude the proof of Theorem 3.1 by plugging ε ′ =

Original: ε/ 3 p and the above mentioned RFF mapping π with target dimension D into Lemma D.1.
Revised: ε/ 3 p and the above mentioned RFF mapping π with target dimension D into Lemma D.1.

Original: E P ROOF OF T HEOREM 4. 
Revised: E P ROOF OF T HEOREM 4.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Consider ∆ ≥ ρ > 0 , and a shift-invariant kernel function K : R d → R , denoting its feature mapping φ : R d → H .
Revised: Consider ∆ ≥ ρ > 0 , and a shift-invariant kernel function K : R d → R , denoting its feature mapping φ : R d → H .

Original: Then there exists x,
Revised: Then there exists x,

Original: y ∈ R d that are (∆ , ρ ) -bounded, such that for every 0 < ε < 1 , the RFF mapping π for K with target dimension D satisfies
Revised: y ∈ R d that are (∆ , ρ ) -bounded, such that for every 0 < ε < 1 , the RFF mapping π for K with target dimension D satisfies

Original: [Equation]
Revised: [Equation]

Original: where s (∆ ,ρ ) K := sup (∆ ,ρ ) -bounded x ∈ R d s K ( x ) , and s K ( x ) :
Revised: where s (∆ ,ρ ) K := sup (∆ ,ρ ) -bounded x ∈ R d s K ( x ) , and s K ( x ) :

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Our proof requires the following anti-concentration inequality.
Revised: Our proof requires the following anti-concentration inequality.

Original: Lemma E.1 (Berry 1941; Esseen 1942) .
Revised: Lemma E.1 (Berry 1941; Esseen 1942) .

Original: For i.i.d.
Revised: For i.i.d.

Original: random variables ξ i ∈ R with mean 0 and variance 1, let X :=
Revised: random variables ξ i ∈ R with mean 0 and variance 1, let X :=

Original: 1 √ D (cid:80) Di =1 ξ i , then for any t,
Revised: 1 √ D (cid:80) Di =1 ξ i , then for any t,

Original: [Equation]
Revised: [Equation]

Original: Let X i ( x ) := cos ⟨ ω i , x ⟩ − K ( x ) , σ ( x ) :=
Revised: Let X i ( x ) := cos ⟨ ω i , x ⟩ − K ( x ) , σ ( x ) :=

Original: (cid:112) Var( X i ( x ))
Revised: (cid:112) Var( X i ( x ))

Original: = (cid:113) 

Revised: = (cid:113)

Original: + K (2( x )) − 2 K ( x ) 2, choose x, y such that s K ( x − y )
Revised: + K (2( x )) − 2 K ( x ) 2, choose x, y such that s K ( x − y )

Original: =
Revised: =

Original: s (∆ ,ρ ) K .
Revised: s (∆ ,ρ ) K .

Original: Clearly, such pair of x, y satisfies that ( x − y ) is (∆ , ρ ) -bounded.
Revised: Clearly, such pair of x, y satisfies that ( x − y ) is (∆ , ρ ) -bounded.

Original: In fact, it is without loss of generality to assume that both x and y are (∆ , ρ ) -bounded, since one may pick y ′ = 0 , x ′ =
Revised: In fact, it is without loss of generality to assume that both x and y are (∆ , ρ ) -bounded, since one may pick y ′ = 0 , x ′ =

Original: x − y and still have x ′ − y ′
Revised: x − y and still have x ′ − y ′

Original: = x − y .
Revised: = x − y .

Original: We next verify that such x, y satisfy our claimed properties.
Revised: We next verify that such x, y satisfy our claimed properties.

Original: Indeed,
Revised: Indeed,

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is by Lemma E.1, and the the second-last equality follows from the definition of s K ( · ) , and that of x, y such that s K ( x − y )
Revised: where the second inequality is by Lemma E.1, and the the second-last equality follows from the definition of s K ( · ) , and that of x, y such that s K ( x − y )

Original: = s (∆ ,ρ ) K .
Revised: = s (∆ ,ρ ) K .

Original: F B EYOND RFF: O BLIVIOUS E MBEDDING FOR L APLACIAN K ERNEL WITH S MALL C OMPUTING T IME
Revised: F B EYOND RFF: O BLIVIOUS E MBEDDING FOR L APLACIAN K ERNEL WITH S MALL C OMPUTING T IME

Original: In this section we show an oblivious feature mapping for Laplacian kernel dimension reduction with small computing time.
Revised: In this section we show an oblivious feature mapping for Laplacian kernel dimension reduction with small computing time.

Original: The following is the main theorem.
Revised: The following is the main theorem.

Original: Theorem F.1.
Revised: Theorem F.1.

Original: Let K be a Laplacian kernel with feature mapping φ : R d → H .
Revised: Let K be a Laplacian kernel with feature mapping φ : R d → H .

Original: For every 0 < δ ≤ ε ≤ 2 − 16 , everyε g > 0 , every d, D ∈ N , D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , every  ρ  0∈ (0 , 1) , there is a mapping π : R d → R D , such that for every x, y ∈ R d  (∆ , ρ ) -bounded,
Revised: For every 0 < δ ≤ ε ≤ 2 − 16 , every d, D ∈ N , D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , every ∆ ≥ ρ > 0 , there is a mapping π : R d → R D , such that for every x, y ∈ R d that are (∆ , ρ ) -bounded,

Original: Pr[ | dist π ( x, y ) − dist φ ( x, y ) | ≤ ε · dist φ ( x, y )] ≥ 1 − δ − ε g .
Revised: Pr[ | dist π ( x, y ) − dist φ ( x, y ) | ≤ ε · dist φ ( x, y )] ≥ 1 − δ.

Original: The time for evaluating π is dD poly(log ∆ ρ  log ε − 1g ) .
Revised: The time for evaluating π is dD poly(log ∆ ρ , log δ − 1 ) .

Original: For simplicity of exhibition, we first handle the case when the input data are from N d .
Revised: For simplicity of exhibition, we first handle the case when the input data are from N d .

Original: At the end we will describe how to handle the case when the input data are from R d by a simple transformation.
Revised: At the end we will describe how to handle the case when the input data are from R d by a simple transformation.

Original: Let N ∈ N be s.t.
Revised: Let N ∈ N be s.t.

Original: every entry of an input data point is at most N .
Revised: every entry of an input data point is at most N .

Original: Even though input data are only consisted of integers, the mapping construction needs to handle fractions, as we will later consider some numbers generated from Gaussian distributions or numbers computed in the RFF mapping.
Revised: Even though input data are only consisted of integers, the mapping construction needs to handle fractions, as we will later consider some numbers generated from Gaussian distributions or numbers computed in the RFF mapping.

Original: So we first setup two numbers, ∆ ′
Revised: So we first setup two numbers, ∆ ′

Original: =
Revised: =

Original: poly( N ) large enough and ρ ′
Revised: poly( N, δ − 1 ) large enough and ρ ′

Original: = 1 / poly( N ) small enough.
Revised: = 1 / poly( N, δ − 1 ) small enough.

Original: All our following operations are working on numbers that are (∆ ′ , ρ ′ )
Revised: All our following operations are working on numbers that are (∆ ′ , ρ ′ )

Original: -bounded.
Revised: -bounded.

Original: Denote ρ ′ / ∆ ′ as ρ 0 for convenience.
Revised: Denote ρ ′ / ∆ ′ as ρ 0 for convenience.

Original: F. E MBEDDING FROM ℓ 1 TO ℓ
Revised: F. E MBEDDING FROM ℓ 1 TO ℓ

Original: Now we describe an isometric embedding from ℓ 1 norm to ℓ 22 .
Revised: Now we describe an isometric embedding from ℓ 1 norm to ℓ 22 .

Original: This construction is based on Kahane (1981), in which the first such construction of finite dimension was given, to the best of our knowledge.
Revised: This construction is based on Kahane (1981), in which the first such construction of finite dimension was given, to the best of our knowledge.

Original: Let π 1 : N → R N be such that for every x ∈ N , x ≤ N , π 1 ( x )[ j ] = 1 , if j ∈ [1 , x ] and π 1 ( x )[ j ] = 0 otherwise.
Revised: Let π 1 : N → R N be such that for every x ∈ N , x ≤ N , π 1 ( x )[ j ] = 1 , if j ∈ [1 , x ] and π 1 ( x )[ j ] = 0 otherwise.

Original: Let π ( d ): N d → R Nd be such that for every x ∈ N d , x i ≤ N, ∀ i ∈ [ d ] , we have π ( d ) 1 ( x ) being the concatenation of d vectors π 1 ( x i ) , i ∈
Revised: Let π ( d ): N d → R Nd be such that for every x ∈ N d , x i ≤ N, ∀ i ∈ [ d ] , we have π ( d ) 1 ( x ) being the concatenation of d vectors π 1 ( x i ) , i ∈

Original: [ d ] .
Revised: [ d ] .

Original: Lemma F.1.
Revised: Lemma F.1.

Original: For every x, y ∈ N d with x i , y i ≤ N, i ∈ [ d ] , it holds that
Revised: For every x, y ∈ N d with x i , y i ≤ N, i ∈ [ d ] , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Notice that for every i ∈ [ d ] , π 1 ( x i ) has its first x i entries being 1 while π 1 ( y i ) has its first y i entries being 1 .
Revised: Notice that for every i ∈ [ d ] , π 1 ( x i ) has its first x i entries being 1 while π 1 ( y i ) has its first y i entries being 1 .

Original: Thus ∥
Revised: Thus ∥

Original: π 1 ( x i ) − π 1 ( y i ) ∥ 22 is exactly ∥ x
Revised: π 1 ( x i ) − π 1 ( y i ) ∥ 22 is exactly ∥ x

Original: i − y i ∥ 1 .
Revised: i − y i ∥ 1 .

Original: If we consider all the d dimensions, then by the construction of π ( d ) 1 , the lemma holds.
Revised: If we consider all the d dimensions, then by the construction of π ( d ) 1 , the lemma holds.

Original: F. F EATURE M APPING FOR L APLACIAN K ERNEL
Revised: F. F EATURE M APPING FOR L APLACIAN K ERNEL

Original: Notice that we can apply the mapping π ( d ) 1 and the RFF mapping ϕ from Theorem 1.2 for the kernel function exp( −∥ x − y ∥ 22 ) which is actually a Gaussian kernel.
Revised: Notice that we can apply the mapping π ( d ) 1 and the RFF mapping ϕ from Theorem 1.2 for the kernel function exp( −∥ x − y ∥ 22 ) which is actually a Gaussian kernel.

Original: This gives a mapping which preserves kernel distance for Laplacian kernel.
Revised: This gives a mapping which preserves kernel distance for Laplacian kernel.

Original: To be more precise, we setup the mapping to be π = ϕ ◦ π ( d ) 1 .
Revised: To be more precise, we setup the mapping to be π = ϕ ◦ π ( d ) 1 .

Original: The only drawback is that the running time is high, as in the above mapping we map d dimension to dN dimension.
Revised: The only drawback is that the running time is high, as in the above mapping we map d dimension to dN dimension.

Original: We formalize this as the following theorem.
Revised: We formalize this as the following theorem.

Original: Theorem F.2.
Revised: Theorem F.2.

Original: Let K be a Laplacian kernel with feature map φ : R d → H .
Revised: Let K be a Laplacian kernel with feature map φ : R d → H .

Original: For every 0 < δ ≤ ε ≤ 2 − 16 , every d, D, N ∈ N , D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , there exists a mapping π : R d → R D s.t.
Revised: For every 0 < δ ≤ ε ≤ 2 − 16 , every d, D, N ∈ N , D ≥ max { Θ( ε − 1 log 3 (1 /δ )) , Θ( ε − 2 log(1 /δ )) } , there exists a mapping π : R d → R D s.t.

Original: for every x, y ∈ N d , x, y ≤ N ,
Revised: for every x, y ∈ N d , x, y ≤ N ,

Original: [Equation]
Revised: [Equation]

Original: ≤ ε · dist φ ( x, y )] ≥ 1 − δ.
Revised: ≤ ε · dist φ ( x, y )] ≥ 1 − δ.

Original: The time of evaluating π is ˜ O ( dDN ) .
Revised: The time of evaluating π is ˜ O ( dDN ) .

Original: Proof.
Revised: Proof.

Original: Consider the map π defined above.
Revised: Consider the map π defined above.

Original: It follows by Lemma F.1 and Theorem 1.2.
Revised: It follows by Lemma F.1 and Theorem 1.2.

Original: The running time is as stated, since we need to compute a vector of length
Revised: The running time is as stated, since we need to compute a vector of length

Original: O ( N ) and then apply the RFF on this vector.
Revised: O ( N ) and then apply the RFF on this vector.

Original: The time complexity is a bit large, since we need to compute π ( d ) 1 which has a rather large dimension dN .
Revised: The time complexity is a bit large, since we need to compute π ( d ) 1 which has a rather large dimension dN .

Original: Next we show how to reduce the time complexity.
Revised: Next we show how to reduce the time complexity.

Original: F. A N A LTERNATE C ONSTRUCTION
Revised: F. A N A LTERNATE C ONSTRUCTION

Original: We give the following map π ′ which has the same output distribution as that of π = ϕ ◦ π ( d ) 1 .
Revised: We give the following map π ′ which has the same output distribution as that of π = ϕ ◦ π ( d ) 1 .

Original: Then in the next subsection we will use pseudorandom generators to replace the randomness in π ′ while using its highly efficiency in computation to reduce the time complexity.
Revised: Then in the next subsection we will use pseudorandom generators to replace the randomness in π ′ while using its highly efficiency in computation to reduce the time complexity.

Original: Notice that in computing ϕ ◦ π ( d ) 1 , for each output dimension, the crucial step is computing ⟨ ω, π ( d ) 1 ( x ) ⟩ for some Gaussian distribution ω ∈ R dN which has each dimension being an independent gaussian distribution ω 0 .
Revised: Notice that in computing ϕ ◦ π ( d ) 1 , for each output dimension, the crucial step is computing ⟨ ω, π ( d ) 1 ( x ) ⟩ for some Gaussian distribution ω ∈ R dN which has each dimension being an independent gaussian distribution ω 0 .

Original: The final output is a function of ⟨ ω, π ( d ) 1 ( x ) ⟩ .
Revised: The final output is a function of ⟨ ω, π ( d ) 1 ( x ) ⟩ .

Original: So we only need to present the construction for the first part, i.e. the inner product of an N dimension Gaussian distribution and π 1 ( x 1 ) .
Revised: So we only need to present the construction for the first part, i.e. the inner product of an N dimension Gaussian distribution and π 1 ( x 1 ) .

Original: For the other parts the computations are the same and finally we only need to sum them up.
Revised: For the other parts the computations are the same and finally we only need to sum them up.

Original: Hence to make the description simpler, we denote this inner product as ⟨ ω, π 1 ( x ) ⟩ , where now we let x ∈ N , x ≤ N and ω has N dimensions each being an independent ω 0 .
Revised: Hence to make the description simpler, we denote this inner product as ⟨ ω, π 1 ( x ) ⟩ , where now we let x ∈ N , x ≤ N and ω has N dimensions each being an independent ω 0 .

Original: Let h be the smallest integer s.t. N ≤ 2 h .
Revised: Let h be the smallest integer s.t. N ≤ 2 h .

Original: Consider a binary tree where each node has exactlychildren.
Revised: Consider a binary tree where each node has exactlychildren.

Original: The depth is h .
Revised: The depth is h .

Original: So it has exactly 2 h ≥ N leaf nodes in the last layer.
Revised: So it has exactly 2 h ≥ N leaf nodes in the last layer.

Original: For each node v , we attach a random variable α v in the following way.
Revised: For each node v , we attach a random variable α v in the following way.

Original: For the root, we attach a Gaussian variable which is the summation of 2 h independent Gaussian variable with distribution ω 0 .
Revised: For the root, we attach a Gaussian variable which is the summation of 2 h independent Gaussian variable with distribution ω 0 .

Original: Then we proceed layer by layer from the root to leaves.
Revised: Then we proceed layer by layer from the root to leaves.

Original: For each u, v being children of a common parent w , assume that α w is the summation of 2 l independent ω 0 distributions.
Revised: For each u, v being children of a common parent w , assume that α w is the summation of 2 l independent ω 0 distributions.

Original: Then let α u be the summation of the first 2 l − 1 distributions among them and α v  the summation of the second 2 l − 1 distributions.
Revised: Then let α u be the summation of the first 2 l − 1 distributions among them and α v be the summation of the second 2 l − 1 distributions.

Original: That is α w =
Revised: That is α w =

Original: α u + α v with α u , α v being independent.
Revised: α u + α v with α u , α v being independent.

Original: Notice that conditioned on α w = a , then α u takes the value b with probability Pr α u ,α v i.i.d.
Revised: Notice that conditioned on α w = a , then α u takes the value b with probability Pr α u ,α v i.i.d.

Original: [ α u = b | α u + α v = a ] .
Revised: [ α u = b | α u + α v = a ] .

Original: α v takes the value a − b when α u takes value b .
Revised: α v takes the value a − b when α u takes value b .

Original: The randomness for generating every random variable corresponding to a node, are presented as a sequence, in the order from root to leaves, layer by layer, from left to right.
Revised: The randomness for generating every random variable corresponding to a node, are presented as a sequence, in the order from root to leaves, layer by layer, from left to right.

Original: We define α x to be the summation of the random variables corresponding to the first x leaves.
Revised: We define α x to be the summation of the random variables corresponding to the first x leaves.

Original: Notice that α x can be sampled efficiently in the following way.
Revised: Notice that α x can be sampled efficiently in the following way.

Original: Consider the path from the root to the x -th leaf.
Revised: Consider the path from the root to the x -th leaf.

Original: First we sample the root, which can be computed using the corresponding randomness.
Revised: First we sample the root, which can be computed using the corresponding randomness.

Original: We use a variable z to record this sample outcome, calling z a accumulator for convenience.
Revised: We use a variable z to record this sample outcome, calling z an accumulator for convenience.

Original: Then we visit each node along the path.
Revised: Then we visit each node along the path.

Original: When visiting v , assume its parent is w , where α w has already been sampled previously with outcome a .
Revised: When visiting v , assume its parent is w , where α w has already been sampled previously with outcome a .

Original: If v is a left child of w , then we sample α v conditioned on α w = a .
Revised: If v is a left child of w , then we sample α v conditioned on α w = a .

Original: Assume this sampling has outcome b .
Revised: Assume this sampling has outcome b .

Original: Then we add − a + b to the current accumulator z .
Revised: Then we add − a + b to the current accumulator z .

Original: If v is a right child of a node w , then we keep the current accumulator z unchanged.
Revised: If v is a right child of a node w , then we keep the current accumulator z unchanged.

Original: After visiting all nodes in the path, z is the sample outcome for α x .
Revised: After visiting all nodes in the path, z is the sample outcome for α x .

Original: Lemma F.2.
Revised: Lemma F.2.

Original: The joint distribution α x , x = 0 , 1 , . .
Revised: The joint distribution α x , x = 0 , 1 , . .

Original: , N has the same distribution as ⟨ ω, π 1 ( x ) ⟩ , x = 0 , 1 , . . .
Revised: , N has the same distribution as ⟨ ω, π 1 ( x ) ⟩ , x = 0 , 1 , . . .

Original: , N .
Revised: , N .

Original: Proof.
Revised: Proof.

Original: According to our construction, each leaf is an independent distribution ω 0 .
Revised: According to our construction, each leaf is an independent distribution ω 0 .

Original: Hence if we take all the leaves and form a vector, then it has the same distribution as w .
Revised: Hence if we take all the leaves and form a vector, then it has the same distribution as w .

Original: Notice that for each parent w with two children u, v , by the construction, α w = α u + α v .
Revised: Notice that for each parent w with two children u, v , by the construction, α w = α u + α v .

Original: Here α u , α v are independent, each being a summation of l independent ω 0 , with l being the number of leaves derived from u .
Revised: Here α u , α v are independent, each being a summation of l independent ω 0 , with l being the number of leaves derived from u .

Original: Thus for each layer, for every node u in the layer, α u ’s are independent and the summation of them is their parent.
Revised: Thus for each layer, for every node u in the layer, α u ’s are independent and the summation of them is their parent.

Original: So for the last layer all the variables are independent and follow the distribution ω 0 .
Revised: So for the last layer all the variables are independent and follow the distribution ω 0 .

Original: And for each node w in the tree, α w is the summation of the random variables attached to the leaves of the subtree whose root is w .
Revised: And for each node w in the tree, α w is the summation of the random variables attached to the leaves of the subtree whose root is w .

Original: So α x is the summation of the first x leaf variables.
Revised: So α x is the summation of the first x leaf variables.

Original: We do the same operation for other dimensions of the output of π ( d ) 

Revised: We do the same operation for other dimensions of the output of π ( d )

Original: and then sum them up to get an alternate construction π ′ for π = ϕ ◦ π ( d ) 1 .
Revised: and then sum them up to get an alternate construction π ′ for π = ϕ ◦ π ( d ) 1 .

Original: We note that to generate an α v , we only need to simulate the conditional distributions.
Revised: We note that to generate an α v , we only need to simulate the conditional distributions.

Original: The distribution function F of the random variable is easy to derive, since its density function is a product of three Gaussian density functions, i.e.
Revised: The distribution function F of the random variable is easy to derive, since its density function is a product of three Gaussian density functions, i.e.

Original: [Equation]
Revised: [Equation]

Original: where α u , α v are Gaussians.
Revised: where α u , α v are Gaussians.

Original: To compute F we can use the taylor expansion of its density function to get an analytical form of F , and the evaluation then can be computed in time t τ = poly( ρ − 1 0 ) .
Revised: To compute F we can use the taylor expansion of its density function to get an analytical form of F , and the evaluation then can be computed in time t τ = poly( ρ − 1 0 ) .

Original: Recall that ρ 0 is defined to be ρ ′ / ∆ ′ .
Revised: Recall that ρ 0 is defined to be ρ ′ / ∆ ′ .

Original: To sample α u , we use τ = O (log ρ − 1 0 ) uniform random bits to generate a number p uniformly with precision poly( ρ − 1 0 ) small enough.
Revised: To sample α u , we use τ = O (log ρ − 1 0 ) uniform random bits to generate a number p uniformly with precision poly( ρ − 1 0 ) small enough.

Original: Then we use binary search to figure out an b such that F ( b ) ∈ [ p − ε 0 , p + ε 0 ] , for some small enough ε 

Revised: Then we use binary search to figure out an b such that F ( b ) ∈ [ p − ε 0 , p + ε 0 ] , for some small enough ε

Original: = poly( ρ 0 ) .
Revised: = poly( ρ 0 ) .

Original: and the space used is s τ = poly log( ρ − 1 0 ) .
Revised: and the space used is s τ = poly log( ρ − 1 0 ) .

Original: We remark that simulating a distribution in this way always has some simulating bias.
Revised: We remark that simulating a distribution using uniform random bits always has some simulating bias.

Revised: The above lemma is proved under the assumption that the simulation has no bias.

Original: But we can see that the statistical distance between the simulated distribution and the Gaussian is at most poly( ρ 0 )  , which is small enough when ρ is small enough, since our input are all natural numbers at most N .
Revised: But we can see that the statistical distance between the simulated distribution and the original distribution is at most poly( ρ 0 ) = 1 / poly( N ) , which is small enough by our picking of ∆ ′ = poly( N, δ − 1 ) , ρ ′

Original: F. R EDUCING THE T IME C OMPLEXITY U SING PRG S 
Revised: = 1 / poly( N, δ − 1 ) . So if we consider simulation bias, then we can show that for every subset S ⊆ { 0 , 1 , . . . , N } , the joint distribution α x , x ∈ S has a statistical distance

Revised: O ( | S | ε 0 ) to the joint distribution ⟨ ω, π 1 ( x ) ⟩ , x ∈ S .

Revised: Later we will only use the case that | S | = 2 , i.e. two points.

Revised: So the overall statistical distance is δ − Θ(1) which does not affect our analysis and parameters.

Revised: F. R EDUCING THE T IME C OMPLEXITY U SING PRG S

Original: Next we use a pseudorandom generator to replace the randomness used in the above construction.
Revised: Next we use a pseudorandom generator to replace the randomness used in the above construction.

Original: A function G : { 0 , 1 } r → { 0 , 1 } n a pseudorandom generator for space s computations with error parameter ε  , if for every probabilistic TM M with space s using n bits randomness in the read-once manner
Revised: A function G : { 0 , 1 } r → { 0 , 1 } n a pseudorandom generator for space s computations with error parameter ε g , if for every probabilistic TM M with space s using n bits randomness in the read-once manner

Original: [Equation]
Revised: [Equation]

Original: ] − Pr [ M ( U n ) = 1] | ≤ ε.

Original: Here r is called the seed length of G .
Revised: Here r is called the seed length of G .

Original: Theorem F.3 (Nisan 1992) .
Revised: Theorem F.3 (Nisan 1992) .

Original: For every n ∈ N and s ∈ N , there exists an pseudorandom generator
Revised: For every n ∈ N and s ∈ N , there exists an pseudorandom generator

Original: G : { 0 , 1 } r → { 0 , 1 } n for space s computations with parameter ε  , where r = O (log n (log n + s + log 1 ε  )) .
Revised: G : { 0 , 1 } r → { 0 , 1 } n for space s computations with parameter ε g , where r = O (log n (log n + s + log 1 ε g )) .

Original: G can be computed in polynomial time (in n and s ) and O ( s log n ) space.
Revised: G can be computed in polynomial time (in n, r ) and O ( r ) space.

Original: Moreover, given an index i ∈ [ n ] , the i -th bit of the output of G can be computed in time poly( r ) .
Revised: Moreover, given an index i ∈ [ n ] , the i -th bit of the output of G can be computed in time poly( r ) .

Original: Let G : { 0 , 1 } r → { 0 , 1 } ℓ , ℓ = 2 dDNτ be a pseudorandom generator for space s =
Revised: Let G : { 0 , 1 } r → { 0 , 1 } ℓ , ℓ = 2 dDNτ be a pseudorandom generator for space s =

Original: c 1 (log N + s τ ) , witherror ε g  , τ = c 2 log N for some large enough constants c 1 , c 2 .
Revised: c 1 (log N + s τ ) , with ε g = δ/ 2 , τ = c 2 log N for some large enough constants c 1 , c 2 .

Original: Again we only need to consider the construction corresponding to the first output dimension of ϕ ◦ π 1 .
Revised: Again we only need to consider the construction corresponding to the first output dimension of ϕ ◦ π 1 .

Original: We replace the randomness U ℓ used in the construction by output of G .
Revised: We replace the randomness U ℓ used in the construction by output of G .

Original: That is, when we need τ uniform random bits to construct a distribution α v in the tree, we first compute positions of these bits in U ℓ and then compute the corresponding bits in the output of G .
Revised: That is, when we need τ uniform random bits to construct a distribution α v in the tree, we first compute positions of these bits in U ℓ and then compute the corresponding bits in the output of G .

Original: Then use them to do the construction in the same way.
Revised: Then use them to do the construction in the same way.

Original: We denote this mapping using pseudorandomness as our final mapping π ∗ .
Revised: We denote this mapping using pseudorandomness as our final mapping π ∗ .

Original: Now we provide a test algorithm to show that the feature mapping provided by the pseudorandom distribution has roughly the same quality as that of the mapping provided by the true randomness.
Revised: Now we provide a test algorithm to show that the feature mapping provided by the pseudorandom distribution has roughly the same quality as that of the mapping provided by the true randomness.

Original: We denote the test algorithm as T =
Revised: We denote the test algorithm as T =

Original: T K,x,y,ε where x, y ∈ R d
Revised: T K,x,y,ε where x, y ∈ R d

Original: and K is a Laplacian kernel with feature mapping φ .
Revised: and K is a Laplacian kernel with feature mapping φ .

Original: T works as the following.
Revised: T works as the following.

Original: Its input is the randomness either being U ℓ or G ( U r ) .
Revised: Its input is the randomness either being U ℓ or G ( U r ) .

Original: T first computes dist φ ( x, y ) .
Revised: T first computes dist φ ( x, y ) .

Original: Notice that T actually does not have to compute φ since the distance can be directly computed as (cid:112) 2 − 2 K ( x, y ) .
Revised: Notice that T actually does not have to compute φ since the distance can be directly computed as (cid:112) 2 − 2 K ( x, y ) .

Original: Then T ( G ( U r )) computes dist π ∗ ( x, y ) and test [Equation]
Revised: Then T ( G ( U r )) computes dist π ∗ ( x, y ) and test | dist π ∗ ( x, y ) − dist φ ( x, y ) | ≤ ε dist φ ( x, y ) .

Original: Notice that when the input is U ℓ , then this algorithm T is instead testing
Revised: Notice that when the input is U ℓ , then this algorithm T is instead testing

Original: [Equation]
Revised: [Equation]

Original: Recall that π ′ is defined in the previous section as our mapping using true randomness.
Revised: Recall that π ′ is defined in the previous section as our mapping using true randomness.

Original: Next we consider using T on true randomness.
Revised: Next we consider using T on true randomness.

Original: Lemma F.3.
Revised: Lemma F.3.

Original: Pr[ T ( U ℓ ) = 1] ≥ 1 − δ .
Revised: Pr[ T ( U ℓ ) = 1] ≥ 1 − δ/ 2 .

Original: Proof.
Revised: Proof.

Original: By Lemma F.2, dist π ′ ( x, y )
Revised: By Lemma F.2, dist π ′ ( x, y )

Original: =
Revised: =

Original: dist π ( x, y ) .
Revised: dist π ( x, y ) .

Original: By Theorem 1.2and our setting of parameters,
Revised: By Theorem 1.2 setting the error probability to be δ/ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Notice that the event T ( U ℓ ) = 1 is indeed | dist π ′ ( x, y ) − dist φ ( x, y ) | ≤ ε dist φ ( x, y ) .
Revised: Notice that the event T ( U ℓ ) = 1 is indeed | dist π ′ ( x, y ) − dist φ ( x, y ) | ≤ ε dist φ ( x, y ) .

Original: Hence the lemma holds.
Revised: Hence the lemma holds.

Original: Now we show that T is actually a small space computation.
Revised: Now we show that T is actually a small space computation.

Original: Lemma F.4.
Revised: Lemma F.4.

Original: T runs in space c (log N + s τ ) for some constant c and the input is read-once.
Revised: T runs in space c (log N + s τ ) for some constant c and the input is read-once.

Original: Proof.
Revised: Proof.

Original: The computing of dist φ ( x, y ) is in space O (log N ) , since x, y ∈ N , x, y ≤ N and the kernel function K can be computed in that space.
Revised: The computing of dist φ ( x, y ) is in space O (log N ) , since x, y ∈ N , x, y ≤ N and the kernel function K can be computed in that space.

Original: Now we focus on the computation of π ′ .
Revised: Now we focus on the computation of π ′ .

Original: We claim that by the construction of α x in section F.3, π ′ can be computed using space O ( s τ +log N ) .
Revised: We claim that by the construction of α x in section F.3, π ′ can be computed using space O ( s τ +log N ) .

Original: The procedure proceeds as the following.
Revised: The procedure proceeds as the following.

Original: First it finds the path to the x -th leaf.
Revised: First it finds the path to the x -th leaf.

Original: This takes space O (log N ) .
Revised: This takes space O (log N ) .

Original: Then along this path, for each node we need to compute a distribution α v .
Revised: Then along this path, for each node we need to compute a distribution α v .

Original: This takes space O
Revised: This takes space O

Original: ( s τ ) .
Revised: ( s τ ) .

Original: Also notice that since the randomness is presented layer by layer, the procedure only needs to do a readonce sweep of the randomness.
Revised: Also notice that since the randomness is presented layer by layer, the procedure only needs to do a readonce sweep of the randomness.

Original: T needs to compute π ′ for both x and y , but this only blow up the space by 2 .
Revised: T needs to compute π ′ for both x and y , but this only blow up the space by 2 .

Original: So the overall space needed is as stated.
Revised: So the overall space needed is as stated.

Original: Finally we prove our theorem by using the property of the PRG.
Revised: Finally we prove our theorem by using the property of the PRG.

Original: Proof of Theorem F.1.
Revised: Proof of Theorem F.1.

Original: We first show our result assuming x, y ∈ N , x, y ≤ N for an integer N .
Revised: We first show our result assuming x, y ∈ N , x, y ≤ N for an integer N .

Original: We claim that π ∗ is the mapping we want.
Revised: We claim that π ∗ is the mapping we want.

Original: Notice that by Theorem F.2 and Lemma F.2, we have

Original: [Equation]

Original: Notice that the event is equivalent to T ( U ℓ )

Original: = 1 . Hence Pr[ T ( U ℓ ) = 1] ≥ 1 − δ.
Revised: By lemma F.3, Pr[ T ( U ℓ ) = 1] ≥ 1 − δ/ 2 .

Original: By Lemma F.4,
Revised: By Lemma F.4,

Original: T runs in space O
Revised: T runs in space O

Original: (log N + s τ ) and is read-once.
Revised: (log N + s τ ) and is read-once.

Original: As G is for space c (log N + s τ ) for some large enough constant c ,
Revised: As G is for space c (log N + s τ ) for some large enough constant c ,

Original: [Equation]
Revised: [Equation]

Original: where  r=
Revised: where seed length r

Original:  O (log( dDNs τ /ε g ) log( dDNτ )) .
Revised: = O (log( dDNs τ /ε g ) log( dDNτ )) .

Original: Notice that T ( G ( U r ))
Revised: Notice that T ( G ( U r ))

Original: = 1 is equivalent to | dist π ∗ ( x, y ) − dist
Revised: = 1 is equivalent to | dist π ∗ ( x, y ) − dist

Original: φ
Revised: φ

Original: ( x, y ) | ≤ ε · dist φ ( x, y ) .
Revised: ( x, y ) | ≤ ε · dist φ ( x, y ) .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: The running time is computed as the following.
Revised: The running time is computed as the following.

Original: We only need to consider one dimension of the input data and one output dimension of the mapping, since others can be computed using the same time.
Revised: We only need to consider one dimension of the input data and one output dimension of the mapping, since others can be computed using the same time.

Original: So actually we consider the time for sampling α x .
Revised: So actually we consider the time for sampling α x .

Original: For α x , recall that we visit the path from the root to the x -th leaf.
Revised: For α x , recall that we visit the path from the root to the x -th leaf.

Original: We don’t have to compute the whole output of G , but instead only need to use some parts of the output.
Revised: We don’t have to compute the whole output of G , but instead only need to use some parts of the output.

Original: For sampling each variable α v along the path, we use τ bits in the output of G .
Revised: For sampling each variable α v along the path, we use τ bits in the output of G .

Original: By Theorem F.3, the computing of each random bit in G ’s output, given the index of this bit, needs time poly( r ) .
Revised: By Theorem F.3, the computing of each random bit in G ’s output, given the index of this bit, needs time poly( r ) .

Original: Locating the τ bits of randomness for generating α v needs time
Revised: Locating the τ bits of randomness for generating α v needs time

Original: O (log N ) .
Revised: O (log N ) .

Original: Generating each of the Gaussian random variable using these random bits needs time t τ .
Revised: Generating each of the Gaussian random variable using these random bits needs time t τ .

Original: Summing up these variables takes less time than sampling all of them.
Revised: Summing up these variables takes less time than sampling all of them.

Original: After sampling, the cosine and sine function of the RFF can be computed in time poly(1 /ρ 0 )
Revised: After sampling, the cosine and sine function of the RFF can be computed in time poly(1 /ρ 0 )

Original: = poly(log N ) .
Revised: = poly(log N, δ − 1 ) .

Original: There are d input dimensions and D output dimensions.
Revised: There are d input dimensions and D output dimensions.

Original: So the total time complexity is dD poly(log N, log ε − 1g ) .
Revised: So the total time complexity is dD poly(log N, δ − 1 ) .

Original: For the case that x, y ∈ R d , we only need to modify the embedding π ( d ) 1 in the following way.
Revised: For the case that x, y ∈ R d , we only need to modify the embedding π ( d ) 1 in the following way.

Original: We first round every entry so that their decimal part is now finite.
Revised: We first round every entry so that their decimal part is now finite.

Original: The rounded parts are small enough (e.g. dropping all digits after the 10 log ρ − 1) -th position to the right of the decimal point) such that this only introduce some small additive errors.
Revised: The rounded parts are small enough (e.g. dropping all digits after the 10 log ρ − 1 -th position to the right of the decimal point.) such that this only introduce some small additive errors.

Original: Then we shift all the entries to be non-negative numbers by adding a common shift s .
Revised: Then we shift all the entries to be non-negative numbers by adding a common shift s .

Original: Then we multiply every entry of x by a common factor t s.t.
Revised: Then we multiply every entry of x by a common factor t s.t.

Original: every entry now only has an integer part.
Revised: every entry now only has an integer part.

Original: Notice that t and s can both be chosen according to ∆ ρ , for example t =
Revised: Notice that t and s can both be chosen according to ∆ ρ , for example t =

Original: s =
Revised: s =

Original: O ( ∆ ρ ) .
Revised: O ( ∆ ρ ) .

Original: And we can take N to be poly( ∆ ρ ) .
Revised: And we can take N to be poly( ∆ ρ ) .

Original: Then we apply π 1 , and multiply a factor (cid:112) 1 /t .
Revised: Then we apply π 1 , and multiply a factor (cid:112) 1 /t .

Original: Denote this map as ˜ π 1 .
Revised: Denote this map as ˜ π 1 .

Original: Notice that this ensures that ∥ x − y ∥ 

Revised: Notice that this ensures that ∥ x − y ∥

Original: =
Revised: =

Original: ∥
Revised: ∥

Original: ˜ π ( d ) 1 ( x ) − ˜ π
Revised: ˜ π ( d ) 1 ( x ) − ˜ π

Original: ( d ) 1 ( y ) ∥ 22 .
Revised: ( d ) 1 ( y ) ∥ 22 .

Original: Then we  apply the same construction  as we did for the above natural number case.
Revised: Then we can apply the same construction and analysis as we did for the above natural number case.

Original: The theorem follows by a a similar argument.
Revised: This shows the theorem.

Original: G R EMARKS AND C OMPARISONS TO C HEN & P HILLIPS (2017)
Revised: G R EMARKS AND C OMPARISONS TO C HEN & P HILLIPS (2017)

Original: Our upper bound in Theorem 1.2 is not directly comparable to that of Chen & Phillips (2017) which gave dimension reduction results for Gaussian kernels.
Revised: Our upper bound in Theorem 1.2 is not directly comparable to that of Chen & Phillips (2017) which gave dimension reduction results for Gaussian kernels.

Original: Chen & Phillips (2017) showed in their
Revised: Chen & Phillips (2017) showed in their

Original: Theorem 7 a slightly improved target dimension bound than ours, but it only works for the case of ∥ x − y ∥ ≥ σ , where σ is the parameter in the Gaussian kernel 1 .
Revised: Theorem 7 a slightly improved target dimension bound than ours, but it only works for the case of ∥ x − y ∥ ≥ σ , where σ is the parameter in the Gaussian kernel 1 .

Original: For the other case of ∥ x − y ∥ < σ , their Theorem 14 gave a related bound, but their guarantee is quite different from ours.
Revised: For the other case of ∥ x − y ∥ < σ , their Theorem 14 gave a related bound, but their guarantee is quite different from ours.

Original: Specifically, their target dimension depends linearly on the input dimension d .
Revised: Specifically, their target dimension depends linearly on the input dimension d .

Original: Hence, when d is large (e.g., d = log 2 n ), this Theorem 14 is worse than ours (for the case of ∥ x − y ∥ < σ .
Revised: Hence, when d is large (e.g., d = log 2 n ), this Theorem 14 is worse than ours (for the case of ∥ x − y ∥ < σ .

Original: Finally, we remark that there might be subtle technical issues in the proof of [CP17].
Revised: Finally, we remark that there might be subtle technical issues in the proof of [CP17].

Original: Their Theorem 7 crucially uses a bound for moment generating functions that is established in their Lemma 5.
Revised: Their Theorem 7 crucially uses a bound for moment generating functions that is established in their Lemma 5.

Original: However, we find various technical issues in the proof of Lemma 5 (found in their appendix).
Revised: However, we find various technical issues in the proof of Lemma 5 (found in their appendix).

Original: Specifically, the term E [ e − s 12 ω 2 ∥ ∆ ∥ 2 ] in the last line above “But” (in page 17), should actually be
Revised: Specifically, the term E [ e − s 12 ω 2 ∥ ∆ ∥ 2 ] in the last line above “But” (in page 17), should actually be

Original: E [ e s 12 ω 2 ∥ ∆ ∥ 2 ] .
Revised: E [ e s 12 ω 2 ∥ ∆ ∥ 2 ] .

Original: Even if one fixes this mistake (by negating the exponent), then eventually we can only obtain a weaker bound of ln M ( s ) ≤ s 2 4 ∥ ∆ ∥ 4 + s ∥ ∆ ∥ 2 in the very last step, since the term − s ∥ ∆ ∥ 2 is negated accordingly.
Revised: Even if one fixes this mistake (by negating the exponent), then eventually we can only obtain a weaker bound of ln M ( s ) ≤ s 2 4 ∥ ∆ ∥ 4 + s ∥ ∆ ∥ 2 in the very last step, since the term − s ∥ ∆ ∥ 2 is negated accordingly.

Original: Hence, it is not clear if the claimed bound can still be obtained in
Revised: Hence, it is not clear if the claimed bound can still be obtained in

Original: Theorem 7.
Revised: Theorem 7.