Q UADRATURE - BASED FEATURES FOR KERNEL APPROXIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the problem of improving kernel approximation via randomized feature maps.
These maps arise as Monte Carlo approximation to integral representations of kernel functions and scale up kernel methods for larger datasets.
We propose to use more efﬁcient numerical integration technique to obtain better estimates of the integrals compared to the state-of-the-art methods.
Our approach allows the use of information about the integrand to enhance approximation and facilitates fast computations.
We derive the convergence behavior and conduct an extensive empirical study that supports our hypothesis.
I NTRODUCTION
Kernel methods proved to be an efﬁcient technique in numerous real-world problems.
The core idea of kernel methods is the kernel trick – compute an inner product in a high-dimensional (or even inﬁnite-dimensional) feature space by means of a kernel function k :
[Equation]
where ψ : X → F is a non-linear feature map transporting elements of input space X into a feature space F .
It is a common knowledge that kernel methods incur space and time complexity infeasible to be used with large-scale datasets directly.
For example, kernel regression has O ( N 3 + Nd 2 ) training time, O ( N 2 ) memory, O ( Nd ) prediction time complexity for N data points in original d -dimensional space X .
One of the most successful techniques to handle this problem (Rahimi &
Recht (2008)) introduces a low-dimensional randomized approximation to feature maps:
[Equation]
This is essentially carried out by using Monte-Carlo sampling to approximate scalar product in (1).
A randomized D -dimensional mapping ˆΨ ( · ) applied to the original data input allows employing standard linear methods, i.e. reverting the kernel trick.
In doing so one reduces the complexity to that of linear methods, e.g. D -dimensional approximation admits O ( ND 2 ) training time, O ( ND ) memory and O ( N ) prediction time.
It is well known that as D → ∞ , the inner product in (2) converges to exact kernel k ( x , y ) .
Recent research (Yang et al.
Felix et al.
; Choromanski & Sindhwani (2016)) aims to improve the convergence of approximation so that a smaller D can be used to obtain the same quality of approximation.
This paper considers kernels that allow the following integral representation
[Equation]
where q ( w ) is a density associated with a kernel, e.g. the popular Gaussian kernel has q ( w ) =
p ( w ) , so the exact equality holds with g xy ( w ) =
f xy ( w ) =
φ ( w (cid:62) x ) (cid:62) φ ( w (cid:62) y ) , where φ ( · ) =
[cos( · ) , sin( · )] (cid:62) .
The class of kernels admitting the form in (3) covers shift-invariant kernels (e.g. radial basis function (RBF) kernels) and Pointwise Nonlinear Gaussian (PNG) kernels.
They are widely used in practice and have interesting connections with neural networks (Cho & Saul (2009), Williams (1997)).
For kernels with q ( w ) other than Gaussian (e.g. Laplacian kernel is associated with Gamma distribution) asymptotics imply that for large sample sizes one can obtain very accurate results with little effort by using Gaussian approximation of q ( w ) , which is essentially a reparameterization of q ( w ) and g ( w ) (Monahan & Genz (1997)).
The main challenge for the construction of low-dimensional feature maps is the approximation of the expectation in (3) which is d -dimensional integral with Gaussian weight.
While standard MonteCarlo rule is easy to implement, there are better quadrature rules for such kind of integrals.
For example, Yang et al.
(2014) apply quasi-Monte Carlo (QMC) rules and obtain better quality kernel matrix approximations compared to random Fourier features of Rahimi & Recht (2008).
Unlike other research studies we refrain from using simple Monte Carlo estimate of the integral, instead, we propose to use speciﬁc quadrature rules.
We now list our contributions:
We propose to use advanced quadrature rules to improve kernel approximation accuracy.
We also provide an analytical estimate of the error for the used quadrature rules.
We note that for kernels with speciﬁc integrand f xy ( w ) in (3) one can improve on its properties.
For example, for kernels with even function f xy ( w ) we derive the reduced quadrature rule which gives twice smaller embedded dimension D with the same accuracy.
This applies, for example, to any RBF kernel.
We use structured orthogonal matrices (so-called butterﬂy matrices ) when designing quadrature rule that allow fast matrix by vector multiplications.
As a result, we speed up the approximation of the kernel function and reduce memory requirements.
We demonstrate our approach on a set of regression and classiﬁcation problems.
Empirical results show that the proposed approach has a better quality of approximation of kernel function as well as better quality of classiﬁcation and regression when using different kernels.
Q UADRATURE - BASED RANDOM FEATURES
We start with rewriting the expectation in equation (3) as integral of f xy with respect to p ( w ) :
[Equation]
Integration can be performed by means of quadrature rules.
The rules usually take a form of interpolating function that is easy to integrate.
Given such a rule, one may sample points from the domain of integration and calculate the value of the rule at these points.
Then, the sample average of the rule values would yield the approximation of the integral.
We use the average of sampled quadrature rules developed by Genz & Monahan (1998) to yield unbiased estimates of I ( f xy ) .
A change of coordinates is the ﬁrst step to facilitate stochastic sphericalradial rules.
Now, let w = r z , with z (cid:62) z
= 1 , so that w (cid:62) w = r 2 for r ∈ [0 , ∞ ] , leaving us with
I ( f xy
[Equation]
= (
[Equation]
(cid:90)
[Equation]
where U d = { z : z (cid:62) z
= 1 , z ∈ R d } .
As mentioned earlier we are going to use a combination of radial R and spherical S rules.
We now describe the logic behind the used quadratures.
Stochastic radial rules.
Stochastic radial rule R ( h ) of degree 2 l + 1 has the form of weighted symmetric sums:
[Equation]
where h is an integrand in inﬁnite range integral T ( h )
= ∞ (cid:82)
[Equation]
| d − 1 h ( r ) dr .
To get an unbiased estimate for T ( h ) , points ρ
i are sampled from speciﬁc distributions which depend on the degree of the rule.
Weights w i are derived so that R has a polynomial degree 2 l +1 , i.e. is exact for integrands h ( r ) =
r p with p = 0 , 1 , . . .
, 2 l + 1 .
For radial rules of degree three R 3 the point ρ 0 = 0 , while ρ 1 ∼ χ ( d + 2) follows Chi-distribution with d + 2 degrees of freedom.
Higher degrees require samples from more complex distributions which are hard to sample from.
Stochastic spherical rules.
Spherical rule S ( s ) approximates an integral of a function s ( z ) over the surface of unit d -sphere U d and takes the following form:
[Equation]
where z j are points on U d , i.e. z (cid:62)
z
= 1 .
If we set weight (cid:101) w j = | U d | 2( d +1) and sum function s values at original and reﬂected vertices v j of randomly rotated d -simplex V , we will end up with a degree three rule:
[Equation]
where v j is the j ’th vertex of d -simplex V with vertices on U d and Q is a random d × d orthogonal matrix.
We justify the choice of the degree in Appendix A.
Since the value of the integral is approximated as the sample average, the key to unbiased estimate is proper randomization.
In this case, randomization is attained through the matrix Q .
It is crucial to generate uniformly random orthogonal matrices to achieve an unbiased estimate for spherical surface integrals.
We consider various designs of such matrices further in Section 3.
Stochastic spherical-radial rules.
Meanwhile, combining foregoing rules results in stochastic spherical-radial rule of degree three:
[Equation]
which we ﬁnally apply to the approximation of (4) by averaging the samples of SR 3 , 3 Q ,ρ :
[Equation]
where n is the number of sampled SR rules.
Speaking in terms of approximate feature maps, the new feature dimension D in case of quadrature based approximation equals 2 n ( d +1) as we sample n rules and evaluate each of them at 2( d + 1) points.
Surprisingly, empirical results (see Section 5) show that even a small number of rule samples n provides accurate approximations.
Properties of the integrand.
We also note here that for speciﬁc functions f xy ( w ) we can derive better versions of SR rule by taking on advantage of the knowledge about the integrand.
For example, the Gaussian kernel has f xy ( w ) = cos( w (cid:62) ( x − y )) .
Note that f is even, so we can discard an excessive term in the summation in (5), since f ( w ) =
f ( − w ) , i.e SR 3 , 3 rule reduces to
[Equation]
Variance of the error.
We contribute the variance estimation for the stochastic spherical-radial rules when applied to kernel function.
To the best of our knowledge, it has not been done before.
In case of kernel functions the integrand f xy can be represented as
[Equation]
where z 1 , z 2 are scalar values.
Using this representation of kernel function and its Taylor expansion we can obtain the following proposition (see Appendix B for detailed derivation of the result):
Proposition 2.1.
The quadrature rule (6) is an unbiased estimate of integral of any integrable function f .
If function f can be represented in the form (8), i.e. f ( w ) =
g ( z 1 , z 2 ) , z 1 = w (cid:62) x , z 2 = w (cid:62) y for some x , y ∈ R d , all 4 -th order partial derivatives of g are bounded and D =
2 n ( d +1) is the number of generated features, then
[Equation]
where M 1 = max
[Equation]
and L = max {(cid:107) x (cid:107) , (cid:107) y (cid:107)} .
Constants M 1 , M 2 in the proposition are upper bounds on the derivatives of function g and don’t depend on the data set, while L plays the role of the scale of inputs.
The proposition implies that the error of approximation is proportional to L – the less the scale, the better the accuracy (see Figure 1).
However, scaling input vectors is equivalent to changing the parameters of the kernel function.
For example, decreasing the norm of input variables for RBF kernel is equivalent to increasing the kernel width σ .
Therefore, the wide RBF kernels are approximated better than the narrow ones.
This result also gives us the rate of convergence O ( 1 / nd ) for the quadrature rule.
[Figure]
The quadrature rule (5) grants us some freedom in the choice of random orthogonal matrix Q .
The next section discusses such matrices and suggests butterﬂy matrices for fast matrix by vector multiplication as the SR 3 , 3 rule implementation involves multiplication of the matrix QV by the data vector x .
G ENERATING UNIFORMLY RANDOM ORTHOGONAL MATRICES
Previously described stochastic spherical-radial rules require a random orthogonal matrix Q (see equation (5)).
If Q follows Haar distribution on the set of all matrices in the orthogonal group O ( d ) in dimension d , then the averages of spherical rules S 3 Q i ( s ) provide unbiased degree three estimates for integrals over unit sphere.
Essentially, Haar distribution means that all orthogonal matrices in the group are equiprobable, i.e. uniformly random.
Methods for sampling such matrices vary in their complexity of generation and multiplication.
Techniques based on QR decomposition (Mezzadri (2006)) have complexity cubic in d , and the resulting matrix does not allow fast matrix by vector multiplications.
Another set of methods is based on a sequence of reﬂectors (Stewart (1980)) or rotators (Anderson et al.
The complexity is better (quadratic in d ), however the resulting matrix is unstructured and, thus, implicates no fast matrix by vector multiplication.
In Choromanski et al.
(2017) random orthogonal matrices are considered.
They are constructed as a product of random diagonal matrices and Hadamard matrices and therefore enable fast matrix by vector products.
Unfortunately, they are not guaranteed to follow the
Haar distribution.
To satisfy both our requirements, i.e low computational/space complexity and generation of Haar distributed orthogonal matrices, we propose to use so-called butterﬂy matrices .
Butterﬂy matrices.
The method from Genz (1998) generates Haar distributed random orthogonal matrix B .
As it happens to be a product of butterﬂy structured factors, a matrix of this type conveniently possesses the property of fast multiplication.
For d = 4 an example of butterﬂy orthogonal matrix is
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]

[Table]
Deﬁnition 3.1.
Let c i
= cos θ i , s i = sin
θ i for i = 1 , . . .
, d − 1 be given.
Assume d = 2 k with k > 0 .
Then an orthogonal matrix B ( d ) ∈ R d × d is deﬁned recursively as follows
[Equation]
where ˆB ( d ) is the same as B ( d ) with indexes i shifted by d , e.g.
[Equation]
) = (cid:20) c− ssc(cid:21) .
Matrix B ( d ) by vector product has computational complexity O
( d log d ) since B ( d ) has (cid:100) log d (cid:101) factors and each factor requires O ( d ) operations.
Another advantage is space complexity: B ( d ) is fully determined by d − 1 angles θ i , yielding O ( d ) memory complexity.
One can easily deﬁne butterﬂy matrix B ( d ) for the cases when d is not a power of two (see Appendix
C.1 for details).
The randomization is based on the sampling of angles θ
and we discuss it in
Appendix C.2.
The method that uses butterﬂy orthogonal matrices is denoted B in the experiments section.
K ERNELS
This section gives examples on how quadrature rules can be applied to a number of kernels.
4. G AUSSIAN KERNEL
Radial basis function (RBF) kernels are popular kernels widely used in kernel methods.
Gaussian kernel is a widely exploited RBF kernel and has the following form:
[Equation]
In this case the integral representation has φ ( w (cid:62) x ) =
[cos( w (cid:62) x ) , sin( w (cid:62) x )] (cid:62) .
Since f xy (0) = 1 ,
SR 3 , 3 rule for Gaussian kernel has the form ( σ appears due to scaling):
[Equation]
4. A RC - COSINE KERNELS
Arc-cosine kernels were originally introduced by Cho & Saul (2009) upon studying the connections between deep learning and kernel methods.
The integral representation of the b th -order arc-cosine kernel is
[Equation]
[Table]
where φ b ( w (cid:62) x ) =
Θ( w (cid:62) x )( w (cid:62) x ) b , Θ( · ) is the Heaviside function and p is the density of the standard Gaussian distribution.
Such kernels can be seen as an inner product between the representation produced by inﬁnitely wide single layer neural network with random Gaussian weights.
They have closed form expression in terms of the angle θ = cos −
[Table]
x
[Table]
between x and y .
0 th -order arc-cosine kernel is given by k 0 ( x , y )
= 1 − θπ , 1 st -order kernel is given by k 1 ( x , y )
= (cid:107) x (cid:107)(cid:107) y (cid:107) π
[Equation]
Let φ 0 ( w (cid:62) x ) =
Θ( w (cid:62) x ) and φ 1 ( w (cid:62) x ) = max(0 , w (cid:62) x ) , then we can rewrite the integral representation as follows: k b ( x , y ) =
2 (cid:82) R d φ b ( w (cid:62) x ) φ b ( w (cid:62) y ) p ( w ) d w ≈ 2 n n (cid:80) i
=1 SR 3 , 3 Q i , ρ i .
For arccosine kernel of order
the value of the function φ 0 (0) = Θ(0)
= 0 .
5 results in
SR 3 , 3 Q ,ρ ( f xy )
= 0 .(cid:18)
1 − d ρ(cid:19)
+ d d +d +1 (cid:88) j = f xy ( ρ Qv j ) ρ.
In the case of arc-cosine kernel of order 1 , the value of φ 1 (0) is 0 and the SR 3 , 3 rule reduces to
SR 3 , 3 Q ,ρ ( f xy )
=
d d +d +1 (cid:88) j = f xy ( ρ Qv j ) + f xy ( − ρ Qv j ) 2 ρ.
E XPERIMENTS
We extensively study the proposed method on several established benchmarking datasets: Powerplant, LETTER, USPS, MNIST, CIFAR100 (Krizhevsky & Hinton (2009)), LEUKEMIA (Golub et al.
In Section 5.2 we show kernel approximation error across different kernels and number of features.
We also report the quality of SVM models with approximate kernels on the same data sets in Section 5.3.
The compared methods are described below.
5. M ETHODS
We present a comparison of our method with estimators based on simple Monte Carlo 1 .
The Monte
Carlo approach has a variety of ways to generate samples: unstructured Gaussian (Rahimi & Recht (2008)), structured Gaussian (Felix et al.
(2016)), random orthogonal matrices (ROM) (Choromanski et al.
Monte Carlo integration.
The kernel is estimated as ˆ k ( x , y )
=
1 D φ ( Mx ) φ ( My ) , where M ∈
R D × d is a random weight matrix.
For unstructured Gaussian based approximation M = G , where
G is a random matrix with iid N (0 , 1) elements.
Structured Gaussian has M = G ort , where G ort =
DQ , Q is obtained from RQ decomposition of G (note that this is not the same Q used in the quadrature rules), D is a diagonal matrix with diagonal elements sampled from the χ ( d ) distribution.
In compliance with the previous work on ROM we use S -Rademacher with three blocks: M =
[Table]
i
[Table]
since three blocks have been shown to yield the best results.
[Figure]
Quadrature rules.
Our main method that uses stochastic spherical-radial rules with Q = (cid:101) B(butterﬂy matrix) is denoted by B .
As mentioned earlier we also include a variant of our algorithm that uses an orthogonal matrix Q based on a sequence of random reﬂectors (we denote it as H ).
5. K ERNEL APPROXIMATION
To measure kernel approximation quality we use relative error in Frobenius norm (cid:107) K − ˆK (cid:107) F (cid:107) K (cid:107) F , where
K and ˆK denote exact kernel matrix and its approximation.
We run experiments for the kernel approximation on a random subset of a dataset (see Appendix D for details).
Approximation was constructed for different number of SR samples n =
D 2( d +1) , where d is an original feature space dimensionality and D is the new one.
For the Gaussian kernel we set hyperparameter γ = 1 2 σ 2 to the same value for all the approximants, while arc-cosine kernels have no hyperparameters.
We run experiments for each [kernel, dataset, n ] tuple and plot 95% conﬁdence interval around the mean value line.
Figure 2 show results for kernel approximation error on LETTER, MNIST,
CIFAR100 and LEUKEMIA datasets.
We observe that for the most of the datasets and kernels the methods we propose in the paper ( B ,
H ) show better results than the baselines.
They do coincide almost everywhere, which is expected, as the B method is only different from H in the choice of the matrix Q to facilitate speed up.
5. C LASSIFICATION / REGRESSION WITH NEW FEATURES
We report accuracy and R 2 scores for the classiﬁcation and regression tasks on the same data sets (see Figure 3).
We examine the performance with the same setting (the number of runs for each [kernel, dataset, n ] tuple) as in experiments for kernel approximation error, except now we map the
[Figure]
one of our methods H here since its kernel approximation almost coincides with B .
whole dataset.
We use Support Vector Machines to obtain predictions.
We also drop one of our methods H here since its kernel approximation almost coincides with B .
Kernel approximation error does not fully deﬁne the ﬁnal prediction accuracy – the best performing kernel matrix approximant not necessarily yields the best accuracy or R 2 score.
However, the empirical results illustrate that our method delivers comparable and often the best quality on the ﬁnal tasks.
We also note that in many cases our method provides greater performance using less number of features n , e.g. LETTER and Powerplant datasets with arc-cosine kernel of the ﬁrst order.
R ELATED WORK
The most popular methods for scaling up kernel methods are based on a low-rank approximation of the kernel using either data-dependent or independent basis functions.
The ﬁrst one includes Nystr¨om method (Drineas & Mahoney (2005)), greedy basis selection techniques (Smola &
Sch¨olkopf (2000)), incomplete Cholesky decomposition (Fine & Scheinberg (2001)).
The construction of basis functions in these techniques utilizes the given training set making them more attractive for some problems compared to Random Fourier Features approach.
In general, data-dependent approaches perform better than data-independent approaches when there is a gap in the eigen-spectrum of the kernel matrix.
The rigorous study of generalization performance of both approaches can be found in (Yang et al.
In data-independent techniques, the kernel function is approximated directly.
Most of the methods (including the proposed approach) that follow this idea are based on Random Fourier Features (Rahimi & Recht (2008)).
They require so-called weight matrix that can be generated in a number of ways.
Le et al.
(2013) form the weight matrix as a product of structured matrices.
It enables fast computation of matrix-vector products and speeds up generation of random features.
Another work (Felix et al.
(2016)) orthogonalizes the features by means of orthogonal weight matrix.
This leads to less correlated and more informative features increasing the quality of approximation.
They support this result both analytically and empirically.
The authors also introduce matrices with some special structure for fast computations.
Choromanski et al.
(2017) propose a generalization of the ideas from (Le et al.
and (Felix et al.
, delivering an analytical estimate for the mean squared error (MSE) of approximation.
All these works use simple Monte Carlo sampling.
However, the convergence can be improved by changing Monte Carlo sampling to Quasi-Monte Carlo sampling.
Following this idea Yang et al.
(2014) apply quasi-Monte Carlo to Random Fourier Features.
In (Yu et al.
the authors make attempt to improve quality of the approximation of Random Fourier Features by optimizing sequences conditioning on a given dataset.
C ONCLUSION
In this work we proposed to apply advanced integration rule that allowed us to achieve higher quality of kernel approximation.
Our derivation of the variance of the error implies the dependence of the error on the scale of data, which in case of Gaussian kernel can be interpreted as width of the kernel.
However, as we have seen earlier, accuracy on the ﬁnal task has no direct dependence on the approximation quality, so we can only speculate whether better approximated wide kernels deliver better accuracy compared to the poorer approximated narrow ones.
It is interesting to explore this connection in the future work.
To speed up the computations we employed butterﬂy orthogonal matrices yielding the computational complexity O ( d log d ) .
Although the procedure we used to generate butterﬂy matrices claims to produce uniformly random orthogonal matrices, we found that it is not always so.
However, the comparison of the method H (uses properly distributed orthogonal matrices) with method B (sometimes fails to do so) did not reveal any differences.
We also leave it for the future investigation.
Our experimental study conﬁrms that for many kernels on the most datasets the proposed approach delivers better kernel approximation.
Additionally, the empirical results showed that the quality of the ﬁnal task (classiﬁcation/regression) is also higher than the state-of-the-art baselines.
The connection between the ﬁnal score and the kernel approximation error is to be explored as well.
A Q UADRATURE RULES DETAILS
The degree of the rules.
We now discuss the choice of the degree for the SR rule.
Genz &
Monahan (1999) show that higher degree rules, being more computationally expensive, often bring in only marginal improvement in performance.
For these reasons in our experiments we use the rule of degree three SR 3 , 3 , i.e. a combination of radial rule R 3 and spherical rule S 3 .
B V ARIANCE OF KERNEL FUNCTION APPROXIMATION USING QUADRATURE RULE
The function f xy in quadrature rule (7) can be considered as a function of two variables, i.e. f xy = φ ( w (cid:62) x ) φ ( w (cid:62) y )
=
g ( z 1 , z 2 ) , where z 1 = w (cid:62) x , z 2 = w (cid:62) y .
In the quadrature rule ρ ∼ χ ( d + 2) and Q is a random orthogonal matrix.
Therefore, random variables w i =
Qv i are uniformly distributed on a unit-sphere.
Now, let’s write down 4-th order Taylor expansion with Lagrange remainder of function g ( ρ w (cid:62) i x , ρ w (cid:62) i y )
+ g ( − ρ w (cid:62) i x , − ρ w (cid:62) i y ) around 0 (odd terms cancel out) g (
[Equation]
where c j = ∂ 2 g ∂z j 1 ∂z 2 − j(0 , 0) ,
d ij = ∂ 4 g ∂z j 1 ∂z 4 − j( (cid:15) i 1 , (cid:15) i 2 )+ ∂ 4 g ∂z j 1 ∂z 4 − j( (cid:15) i 3 , (cid:15) i 4 ) , (cid:15) i 1 is between 0 and ( ρ w (cid:62) i x ) , (cid:15) i 2 is between 0 and ( ρ w (cid:62) i y ) , (cid:15) i 3 is between 0 and ( − ρ w (cid:62) i x ) and (cid:15) i 4 is between 0 and ( − ρ w (cid:62) i y ) .
Plugging this expression into (5) we obtain
[Equation]
S
[Equation]
, where
S i
[Equation]
B
[Equation]
= (cid:80)
[Equation]
d ij
[Equation] [Equation]
j )!
( w
[Equation] [Equation]
( w
[Equation]
y ) 4 −
[Equation]
w j =
Qv j , matrix Q is a random orthogonal matrix uniformly distributed on a set of orthogonal matrices O ( n ) .
From uniformity of orthogonal matrix Q it follows that vector w j is uniform on a unit n -sphere.
Also note that A i and B j are independent if i (cid:54) =
j , B i and B j are independent if i (cid:54) =
j , however, A i and A j are dependent as they have common random variable ρ .
Therefore, Cov ( S i , S j ) =
Cov ( A i , A j ) =
E ( A i A j ) − ( E ( A i ))if i (cid:54) =
j .
Let us calculate the variance of the estimate.
[Equation]
Distribution of random variable ρ is χ ( d + 2) , therefore
[Equation]
Now, let us calculate the variance V S i from the ﬁrst term of equation (9)
[Equation]
Let (cid:107) x (cid:107) ≥ (cid:107) y (cid:107) .
Then
V ( B i ) =
V  
[Equation]
=
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
(10) where M 1 = max (cid:26)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:27) .
To calculate expectations of the form E ( w (cid:62) x ) k we will use expression for integral of monomial over unit sphere Baker (1997)
[Equation]
where k i =
2 s i , s i ∈
Z + , | k | = (cid:80) i
k
i , σ d is a volume of an d -dimensional unit sphere.
For example, let us show how to calculate E ( w (cid:62) x ) 4 :
[Equation]
thanks to symmetry all terms, for which at least one index doesn’t coincide with other indices, are equal to 0.
[Equation]
x (cid:107) 4 .
Using the same technique for (10) we obtain
[Equation]
For the variance of A i we have
[Equation]
we assume that (cid:107) x (cid:107) ≥ (cid:107)
[Equation]
[Equation]
where M 2 = max j =0 , 1 , 2 (cid:40)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
Let’s estimate covariance Cov
( A i , B i ) :
[Equation]
The ﬁrst term of the right hand side:
E
[Equation]

 ≤ (cid:34) we assume that (cid:107) x (cid:107) ≥ (cid:107) y (cid:107) (cid:35) ≤ 210 M 1 M 2 (cid:107) x
(cid:107)d 2 ( d +
[Equation]
( d + 4) .
The second term E A i E B i (again for (cid:107) x (cid:107) ≥ (cid:107) y (cid:107) )
[Equation]
Combining the derived inequalities we obtain
[Equation]
[Equation]
Now, let’s examine the expectation of the second term in (9):
[Equation]
we assume that
(cid:107)
x
(cid:107) ≥
[Equation]
y
[Equation]
[Figure]
Substituting (14) and (15) into (9) we obtain
[Equation]
And ﬁnally
V
[Equation]
C B UTTERFLY MATRICES GENERATION DETAILS C. N OT A POWER OF TWO
We discuss here the procedure to generate butterﬂy matrices of size d × d when d is not a power of 2 .
Let the number of butterﬂy factors k
= (cid:100) log d (cid:101) .
Then B ( d ) is constructed as a product of k factor matrices of size d × d obtained from k matrices used for generating B (2 k ) .
For each matrix in the product for B (2 k ) , we delete the last 2 k − d rows and columns.
We then replace with 1 every c i in the remaining d × d matrix that is in the same column as deleted s i .
For the cases when d is not a power of two, the resulting B has deﬁcient columns with zeros (Figure 4b, right), which introduces a bias to the integral estimate.
To correct for this bias one may apply additional randomization by using a product BP , where P ∈ { 0 , 1 } d × d is a permutation matrix.
Even better, use a product of several BP ’s: (cid:101)
B = ( BP ) 1 ( BP ) 2 . . .
( BP ) t .
We set t = 3 in the experiments.
C. B UTTERFLY RANDOMIZATION
The key to uniformly random orthogonal butterﬂy matrix B is the sequence of d − 1 angles θ i .
To get
B ( d ) Haar distributed, we follow Fang & Li (1997) algorithm that ﬁrst computes a uniform random point u from U d .
It then calculates the angles by taking the ratios of the appropriate u coordinates θ i =
u i
u
i +1 , followed by computing cosines and sines of the θ ’s.
D K ERNEL APPROXIMATION ERRORS ON DIFFERENT DATA SETS
Table 2 displays the settings for the experiments across datasets.
[Table]
[Figure]
Here we discuss the datasets that did not appear in the main body of the paper.
Figure 5 shows the results for the kernel approximation error on Powerplant, LETTER, USPS and MNIST datasets.
For these datasets we include QMC with Halton sequences into comparison.
Quasi-Monte Carlo integration boasts improved rate of convergence 1 / D compared to 1 / √ D of Monte
Carlo, however, empirical results illustrate its performance is poorer than that of orthogonal random features (Felix et al.
For QMC the weight matrix M is generated as a transformation of quasi-random sequences.
We run our experiments with Halton sequences in compliance with the previous work (see Figure 5 with QMC method included).
Although, for the arc-cosine kernels, our methods are the best performing estimators, for the Gaussian kernel the error is not always the lowest one and depends on the dataset, e.g. on the USPS dataset the lowest is Monte Carlo with ROM.
However, for the most of the datasets we demonstrate superiority of our approach with this kernel.
We also notice that the dataset with a small amount of features, Powerplant, enjoys Halton and
Orthogonal Random Features best, while ROM’s convergence stagnates at some point.
This could be due the small input feature space with d =
and we leave it for the future investigation.