A GENERALIZATION OF THE RANDOMIZED SINGULAR VALUE DECOMPOSITION
Anonymous authors
Paper under double-blind review
A BSTRACT
The randomized singular value decomposition (SVD) is a popular and effective algorithm for computing a near-best rank k approximation of a matrix
A using matrix-vector products with standard Gaussian vectors.
Here, we generalize the randomized SVD to multivariate Gaussian vectors, allowing one to incorporate prior knowledge of A into the algorithm.
This enables us to explore the continuous analogue of the randomized SVD for Hilbert–Schmidt (HS) operators using operator-function products with functions drawn from a Gaussian process (GP).
We then construct a new covariance kernel for GPs, based on weighted Jacobi polynomials, which allows us to rapidly sample the GP and control the smoothness of the randomly generated functions.
Numerical examples on matrices and HS operators demonstrate the applicability of the algorithm.
I NTRODUCTION
Computing the singular value decomposition (SVD) is a fundamental linear algebra task in machine learning (Paterek, 2007), statistics (Wold et al., 1987), and signal processing (Alter et al., 2000; Van
Der Veen et al., 1993).
The SVD of an m × n real matrix
A with m ≥ n is a factorization of the form A = UΣV ∗ , where U is an m × m orthogonal matrix of left singular vectors, Σ is an m × n diagonal matrix with entries σ 1 ( A ) ≥ · · · ≥ σ n ( A ) ≥ 0 , and V is an n × n orthogonal matrix of right singular vectors (Golub & Van Loan, 2013).
The SVD plays a central role because truncating it after k terms provides a best rank k approximation to A in the spectral and Frobenius norm (Eckart & Young, 1936; Mirsky, 1960).
Since computing the SVD of a large matrix can be computationally infeasible, there are various alternative algorithms that compute near-best rank k matrix approximations from matrix-vector products (Halko et al., 2011; Martinsson & Tropp, 2020;
Nakatsukasa, 2020; Nystr¨om, 1930; Williams & Seeger, 2001).
The randomized SVD uses matrixvector products with standard Gaussian random vectors and is one of the most popular algorithms for constructing a low-rank approximation to A (Halko et al., 2011; Martinsson & Tropp, 2020).
Currently, the randomized SVD is used and theoretically justiﬁed when it uses matrix-vector products with standard Gaussian random vectors.
In this paper, we consider the following generalizations.
Generalization 1.
We generalize the randomized SVD when the matrix-vector products are with multivariate Gaussian random vectors.
Our theory allows for multivariate Gaussian random input vectors that have a general symmetric positive semi-deﬁnite covariance matrix.
A key novelty of our work is that prior knowledge of A can be exploited to design covariance matrices that achieve lower approximation errors than the randomized SVD with standard Gaussian vectors.
Generalization 2.
We generalize the randomized SVD to Hilbert–Schmidt (HS) operators (Hsing & Eubank, 2015).
We design a practical algorithm for learning HS operators using random input functions, sampled from a Gaussian process (GP).
Examples of applications include learning integral kernels such as Green’s functions associated with linear partial differential equations (Boull´e &
Townsend, 2021; Boull ´ e et al., 2021).
The choice of the covariance kernel in the GP is crucial and impacts both the theoretical bounds and numerical results of the randomized SVD.
This leads us to introduce a new covariance kernel based on weighted Jacobi polynomials for learning HS operators.
One of the main advantages of this kernel is that it is directly expressed as a Karhunen–Lo`eve expansion (Karhunen, 1946; Lo`eve, 1946) so that it is faster to sample functions from the associated GP than using a standard squared- exponential kernel.
In addition, we show that the smoothness of the functions sampled from a GP with the Jacobi kernel can be controlled as it is related to the decay rate of the kernel’s eigenvalues.
Contributions.
We summarize our novel contributions as follows:
We provide new theoretical bounds for the randomized SVD for matrices or HS operators when using random input vectors generated from any multivariate Gaussian distribution.
This shows when it is beneﬁcial to use nonstandard Gaussian random vectors in the randomized SVD for constructing low-rank approximations.
We generalize the randomized SVD to HS operators and provide numerical examples to learn integral kernels.
We propose a covariance kernel based on weighted Jacobi polynomials and show that one can select the smoothness of the sampled random functions by choosing the decay rate of the kernel eigenvalues.
B ACKGROUND : T HE RANDOMIZED SVD FOR MATRICES
The randomized SVD computes a near-best rank k approximation to a matrix A .
First, one performs the matrix-vector products y
=
A x 1 , . .
, y k + p =
A x k + p , where x 1 , . . .
, x k + p are standard Gaussian random vectors with identically and independently distributed entries and p ≥ 1 is an oversampling parameter.
Then, one computes the economized QR factorization [ y· · · y k + p ] =
QR , before forming the rank ≤ k + p approximant QQ ∗ A .
Note that if A is symmetric, one can form
QQ ∗ A by computing Q ( AQ ) ∗ via matrix-vector products involving A .
The quality of the rank ≤ k + p approximant QQ ∗ A is characterized by the following theorem.
Theorem 1 (Halko et al.
Let A be an m × n matrix, k ≥ 1 an integer, and choose an oversampling parameter p ≥ 4 .
If Ω ∈ R n × ( k + p ) is a standard Gaussian random matrix and
QR =
AΩ is the economized QR decomposition of AΩ , then for all u
[Equation]
t ≥
[Equation]
with failure probability at most 2 t − p + e − u 2 .
The squared tail of the singular values of A , i.e., (cid:113)(cid:80) nj = k +1 σ 2 j ( A ) , gives the best rank k approximation error to A in the Frobenius norm.
This result shows that the randomized SVD can compute a near-best low-rank approximation to A with high probability.
G ENERALIZED RANDOMIZED SVD FOR MATRICES AND OPERATORS
The theory behind the randomized SVD has been recently extended to nonstandard covariance matrices and HS operators (Boull´e & Townsend, 2021).
However, the probability bounds, generalizing
Theorem 1, are not sharp enough to emphasize the improved performance of covariance matrices with prior information over the standard randomized SVD.
We provide new bounds for GPs with nonstandard covariance matrices in Theorem 2.
An upper bound on the expectation is also available in the Appendix (see Lemma 7).
While Theorem 2 is formulated with matrices, the same result holds for HS operators in inﬁnite dimensions.
For a ﬁxed target rank 1 ≤ k ≤ n , we deﬁne V 1 ∈ R n × k and V 2 ∈ R n × ( n − k ) to be the matrices containing the ﬁrst k and last n − k right singular vectors of A , respectively, and denote by Σ 2 ∈ R ( n − k ) × ( n − k ) , the diagonal matrix with entries σ k +1 ( A ) , . . .
, σ n ( A ) .
We consider a symmetric positive semi-deﬁnite covariance matrix K ∈ R n × n , with k th largest eigenvalue λ k > 0 .
Theorem 2 Let A be an m × n matrix, k ≥ 1 an integer, and choose an oversampling parameter p ≥ 4 .
If Ω ∈ R n × ( k + p ) is a Gaussian random matrix, where each column is sampled from a multivariate Gaussian distribution with covariance matrix K ∈ R n × n , and QR =
AΩ is the economized QR decomposition of AΩ , then for all u, t ≥ 1 ,
[Equation]
with failure probability at most t − p + [ ue − ( u 2 − 1) / 2 ] k + p .
Here, γ k = k/ ( λ 1 Tr(( V ∗ 1 KV 1 ) − 1 ))) denotes the covariance quality factor, and β k =
Tr( Σ 22 V ∗ 2 KV 2 ) /
( λ 1 (cid:107) Σ 2 (cid:107) 2 F ) , where λ 1 is the largest eigenvalue of K .
The factors γ k and β k , measuring the quality of the covariance matrix to learn A in Theorem 2, can be respectively bounded (Boull´e & Townsend 2021, Lem. 3.4; Lemma 9) using the eigenvalues λ 1 ≥ · · · ≥ λ n of the covariance matrix K and the singular values of A as:
[Equation]
This shows that the performance of the generalized randomized SVD depends on the decay rate of the sequence { λ j } .
The quantities α k and β
k depend on how much prior information of the k + 1 , . . .
, n right singular vectors of A is encoded in K .
In the ideal situation where these singular vectors are known, then one can deﬁne K such that β k = 0 for λ k +1 = · · · = λ n = 0 .
In particular, this highlights that a suitably chosen covariance matrix can outperform the randomized SVD with standard Gaussian vectors (see Section 5.1 for a numerical example).
3. R ANDOMIZED SVD FOR H ILBERT –S CHMIDT OPERATORS
We now describe the randomized SVD for learning HS operators (see Algorithm 1).
The algorithm is implemented in the Chebfun software system (Driscoll et al., 2014), which is a MATLAB package for computing with functions.
The Chebfun implementation of the randomized SVD for HS operators uses Chebfun’s capabilities, which offer continuous analogues of several matrix operations like the QR decomposition and numerical integration.
Indeed, the continuous analogue of a matrixvector multiplication AΩ for an HS integral operator F (see Hsing & Eubank, 2015 for deﬁnitions and properties of HS operators), with kernel G : D × D → R , is
[Equation]
where D ⊂ R d with d ≥ 1 , and L 2 ( D ) is the space of square-integrable functions on D .
i ( z ) d z 9: end for
The algorithm takes as input an integral operator that we aim to approximate.
Note that we focus here on learning an integral operator, but other HS operators would work similarly.
The ﬁrst step of the randomized SVD for HS operators consists of generating an ∞ × k quasimatrix Ω by sampling a GP k times, where k is the target rank (see Section 4).
An ∞ × k quasimatrix is an ordered set of k functions (Townsend & Trefethen, 2015), and generalizes the notion of matrix to inﬁnite dimensions.
Therefore, each column of Ω is an object, consisting of a polynomial approximation of a smooth random function sampled from the GP in the Chebyshev basis.
After evaluating the HS operator at Ω to obtain a quasimatrix Y , we use the QR algorithm (Townsend & Trefethen, 2015) to obtain an orthonormal basis Q for the range of the columns of Y .
Then, the randomized SVD for HS operators requires the left-evaluation of the operator F or, equivalently, the evaluation of its adjoint F t satisfying:
[Equation]
We evaluate the adjoint of F at each column vector of Q to construct an approximation G k of G .
Finally, the approximation error between the operator kernel G and the learned kernel G k can be computed in the L 2 -norm, corresponding to the HS norm of the integral operator.
C OVARIANCE KERNELS
To generate the random input functions f 1 , . .
, f k for the randomized SVD for HS operators, we draw them from a GP, denoted by GP (0 , K ) , for a certain covariance kernel K .
A widely employed covariance kernel is the squared-exponential function K SE (Rasmussen & Williams, 2006) given by
[Equation]
where (cid:96) > 0 is a parameter controlling the length-scale of the GP.
This kernel is isotropic as it only depends on | x − y | , is inﬁnitely differentiable, and its eigenvalues decay supergeometrically to 0 .
Since the bound in Theorem 2 degrades as the ratio λ 1 /λ j increases for j ≥ k + 1 , the randomized SVD for learning HS operators prefers covariance kernels with slowly decaying eigenvalues.
Our randomized SVD cannot hope to learn HS operators where the range of the operator has a rank greater than ˜ k , where ˜ k is such that the ˜ k th eigenvalue of K SE reaches machine precision.
Other popular kernels for GPs include the Mat ´ ern kernel (Rasmussen & Williams, 2006, Chapt. 4) and Brownian bridge (Nelsen & Stuart, 2020).
Prior information on the HS operator can also be enforced through the choice of the covariance kernel.
For instance, one can impose the periodicity of the samples by using the following squared-exponential periodic kernel:
[Equation]
where (cid:96) > 0 is the length-scale parameter.
4. S AMPLE RANDOM FUNCTIONS FROM A G AUSSIAN PROCESS
In ﬁnite dimensions, a random vector u ∼ N (0 , K ) , where K ∈ R n × n is a covariance matrix with
Cholesky factorization K = LL ∗ , can be generated from the matrix-vector product u
= L c .
Here, c ∈ R n is a vector whose entries follow the standard Gaussian distribution.
We now detail how this process extends to inﬁnite dimensions with a continuous covariance kernel.
Let K be a continuous symmetric positive-deﬁnite covariance function deﬁned on the domain [ a, b ] × [ a, b ] ⊂ R 2 with −∞ < a < b < ∞ .
We consider the continuous analogue of the Cholesky factorization to write K as (Townsend & Trefethen, 2015)
[Equation]
where r j is the j th row of L c , which —in Chebfun’s terminology— is a lower-triangular quasimatrix.
In practice, we truncate the series after n terms, either arbitrarily or when the n th largest kernel eigenvalue, λ n , falls below machine precision.
Then, if c ∈ R n follows the standard Gaussian distribution, a function u can be sampled from GP (0 , K ) as u
= L c c .
That is,
[Equation]
n (cid:88) j = [Equation] j [Equation]
x )
[Equation] [ a [Equation] b ] .
The continuous Cholesky factorization is implemented in Chebfun
(Townsend & Trefethen, 2013), which is the extension of Chebfun for computing with two-dimensional functions.
As an example, the polynomial approximation, which is accurate up to essentially machine precision, of the squaredexponential covariance kernel K SE with parameter (cid:96) = 0 .
01 on [ − 1 , 1] 2 yields a numerical rank of n = 503 .
The functions sampled from GP (0 , K SE ) become more oscillatory as the length-scale parameter (cid:96) decreases and hence the numerical rank of the kernel increases or, equivalently, the associated eigenvalues sequence {
λ j } decays slower to zero.
4. I NFLUENCE OF THE KERNEL ’ S EIGENVALUES AND M ERCER ’ S REPRESENTATION
The covariance kernel can also be deﬁned from its Mercer’s representation as
[Equation]
where { ψ j } is an orthonormal basis of L 2 ( D ) and λ 1 ≥ λ 2 ≥ · · · > 0 (Hsing & Eubank, 2015,
Thm. 4.6.5).
We prefer to construct K directly from Mercer’s representations for several reasons: 1. One can impose prior knowledge of the kernel of the HS operator on the eigenfunctions of K (such as periodicity or smoothness), 2. One can often generate samples from GP (0 , K ) efﬁciently using Equation (4), and 3. One can control the decay rate of the eigenvalues of K ,
Hence, the quantity γ k in the probability bound of Theorem 2 measures the quality of the covariance kernel K in GP (0 , K ) to generate random functions that can learn the HS operator F .
To minimize 1 /γ k
we would like to select the eigenvalues λ 1 ≥ λ 2 ≥ · · · > 0 of K so that they have the slowest possible decay rate while maintaining (cid:80) ∞
j =1 λ j < ∞ .
One needs { λ j } ∈ (cid:96) 1 to guarantee that ω ∼ GP (0 , K ) has ﬁnite expected squared L 2 -norm, i.e., E [ (cid:107) ω (cid:107) 2 L 2 ( D ) ]
= (cid:80) ∞
j =1 λ j < ∞ .
The best sequence of eigenvalues we know that satisﬁes this property is called the Rissanen sequence (Rissanen, 1983) and is given by λ j = R j := 2 − L ( j ) , where
[Equation]
and log ( i ) 2 ( j ) = log 2 ◦ · · · ◦ log 2 ( j ) is the composition of log 2 ( · ) i times.
Other options for the choice of eigenvalues include any sequence of the form
λ j = j − ν for ν > 1 .
4. J ACOBI COVARIANCE KERNEL
If D = [ − 1 , 1] , then a natural choice of orthonormal basis of L 2 ( D ) to deﬁne the Mercer’s representation of the kernel are weighted Jacobi polynomials (Deheuvels & Martynov, 2008;
Olver et al., 2010).
That is, for a weight function w α,β ( x ) =
(1 − x ) α (1+ x ) β with α, β > − 1 , and any positive eigenvalue sequence { λ j } , we consider the Jacobi kernel
[Equation]
5) where ˜ P ( α,β ) j is the scaled Jacobi polynomial of degree j .
The polynomials are normalized such that (cid:107) w 1 / 2 α,β ˜ P ( α,β ) j (cid:107) L 2 ([ − 1 , 1])
= 1 and K ( α,β ) Jac ∈ L 2 ([ − 1 , 1] 2 ) .
In this case, a random function can be sampled as
[Equation]
where c j ∼ N (0 , 1) for 0 ≤ j ≤ ∞ .
A desirable property of a covariance kernel is to be unbiased towards one spatial direction, i.e.,
K ( x, y )
=
K ( − y, − x ) for x, y ∈ [ − 1 , 1] , which motivates us to always select α = β .
Moreover, it is desirable to have the eigenfunctions of K ( α,β ) Jac to be polynomial so that one can generate samples from GP (0 , K ) efﬁciently.
This leads us to choose α and β to be even integers.
The choice of α = β = 0 gives the Legendre kernel (Foster et al., 2020; Habermann, 2021).
In Section 5, we use Equation (5) with α = β = 2 (see Figure 1) to ensure that functions sampled from the associated
GP satisfy homogeneous Dirichlet boundary conditions.
In this case, one must have (cid:80) ∞
j =1 jλ j < ∞ so that the series of functions in Equation (5) converges uniformly and K (2 , 2)
Jac is a continuous kernel (see Appendix B).
Under this additional constraint, the best choice of eigenvalues is given by a scaled Rissanen sequence: λ j = R j /j , for j ≥ 1 (cf.
Section 4.2).
Covariance kernels on higher dimensional domains of the form D
=
[ − 1 , 1] d , for d ≥ 2 , can be deﬁned using tensor products of weighted Jacobi polynomials.
4. S MOOTHNESS OF FUNCTIONS SAMPLED FROM A GP WITH J ACOBI KERNEL
We now connect the decay rate of the eigenvalues of the Jacobi covariance kernel K (2 , 2)
Jac to the smoothness of the samples from GP (0 , K (2 , 2) Jac ) .
Hence, the Jacobi covariance function allows the control of the decay rate of the eigenvalues { λ j } as well as the smoothness of the resulting randomly generated functions.
First, Lemma 3 asserts that if the coefﬁcients of an inﬁnite polynomial series have sufﬁcient decay, then the resulting series is smooth with regularity depending on the decay rate.
This result can be seen as a reciprocal to (Trefethen, 2019, Thm. 7.1).
Lemma 3 Let { p j } be a family of polynomials such that deg ( p j ) ≤ j and max x ∈ [ − 1 , 1] | p
j
( x ) | = 1 .
If f n ( x ) = (cid:80) nj =0 a j p j ( x ) with | a j | ≤ j − ν for ν > 1 , then f n converges uniformly to f ( x )
= (cid:80) ∞
j =0 a j p j ( x ) and f is µ times continuously differentiable for any integer µ such that µ < ( ν − 1) / 2 .
Note that the main application of this lemma occurs when deg ( p j ) =
j for all j ≥ 0 .
The proof of Lemma 3 is deferred to the supplementary material.
We now state the following theorem about the regularity of functions sampled from GP (0 , K (2 , 2) Jac ) , which guarantees that if the eigenvalues are chosen such that λ j = O (1 /j ν ) with ν > 3 , then f ∼ GP (0 , K (2 , 2) Jac ) is almost surely continuous.
Moreover, a faster decay of the eigenvalues of K (2 , 2) Jac implies higher regularity of the sampled functions, in an almost sure sense.
Theorem 4 Let { λ j } ∈ (cid:96) 1 ( R + ) be a positive sequence
such that λ j = O
( j − ν ) for ν > 3 .
If f is sampled from GP (0 , K (2 , 2) Jac ) , then f ∈ C µ ([ − 1 , 1]) almost surely for any integer µ < ( ν − 3) / 2 .
This theorem can be seen as a particular case of Driscoll’s zero-one law (Driscoll, 1973), which characterizes the regularity of functions samples from GPs (see also Kanagawa et al., 2018).
[Figure]
trate functions sampled from GP (0 , K (2 , 2) Jac ) with the different eigenvalue sequences.
The series for generating the random functions are truncated to n = 500 .
In Figure 1, we display the Jacobi kernel of type (2 , 2) with functions sampled from the corresponding GP.
We selected eigenvalue sequences of different decay rates: from the faster 1 /j 4 to the slower
Rissanen sequence R j /j
(Section 4.2).
For λ j = 1 /j 3 and λ j = R j
/j
we observe a large variation of the randomly generated functions near x = ± 1 , indicating a potential discontinuity of the samples at these two points as n → ∞ .
This is in agreement with Theorem 4, which only guarantees continuity (with probability one) of the randomly generated functions if λ j ∼ 1 /j ν with ν > 3 .
N UMERICAL EXPERIMENTS 5. A PPROXIMATION OF MATRICES USING NON - STANDARD COVARIANCE FUNCTIONS
The approximation error bound in Theorem 2 depends on the eigenvalues of the covariance matrix, which dictates the distribution of the column vectors of the input matrix Ω .
Roughly speaking, the more prior knowledge of the matrix A that can be incorporated into the covariance matrix, the better.
In this numerical example
[Figure]
we investigate whether the standard randomized SVD, which uses the identity as its covariance matrix
[Figure]
can be improved by using a different covariance matrix.
We then attempt to learn the discretized 2000 × 2000 matrix,
i
[Figure]
e., the discrete Green’s function, of the inverse of the following differential operator:
[Equation]
We vary the number of columns (i
[Figure]
e. samples from the GP) in the input matrix Ω from 1 to 2000 .
[Figure]
In Figure 2(left), we compare the ratios between the relative error in the Frobenius norm given by the randomized SVD and the best approximation error, obtained by truncating the SVD of A .
The prior covariance matrix K consists of the discretized 2000 × 2000 matrix of the Green’s function of the negative Laplace operator L u = − d 2 u/dx 2 on [0 , 1] to incorporate knowledge of the diffusion term in the matrix A .
We see that a nonstandard covariance matrix leads to a higher approximation accuracy, with a reduction of the error by a factor of 1 . 3 - 1 . 6 compared to the standard randomized SVD.
At the same time, the procedure is only 20% slower 1 on average (Figure 2 (right))
as one can precompute the eigenvalue decomposition of the covariance matrix (see Appendix C).
It is of interest to maximize the accuracy of the approximation matrix from a limited number of samples in applications where the sampling time is much higher than the numerical linear algebra costs.
5. A PPLICATIONS OF THE RANDOMIZED SVD FOR H ILBERT –S CHMIDT OPERATORS
We now apply the randomized SVD for HS operators to learn kernels of integral operators.
In this ﬁrst example, the kernel is deﬁned as (Townsend, 2013)
[Equation]
and is displayed in Figure 3(a).
We employ the squared-exponential covariance kernel K SE with parameter (cid:96) = 0 .
01 and k = 100 samples (see Equation (3)) to sample random functions from the associated GP.
The learned kernel G k is represented on the bottom panel of Figure 3(a) and has an approximation error around machine precision.
− 1 x −   y − 1 .− 0 .0 .0 .1 .− 1 x −   y − 1 .− 0 .0 .0 .1 .−
1 x −   y − 0 .− 0 .0 .0 .0 .− 1 x −   y − 0 .− 0 .0 .0 .0 .−
1 x −   y − 1 .− 0 .0 .0 .1 .− 1 x −   y − 1 .− 0 .0 .0 .1 .
[Figure]
) (b) (c)
K e r n e l
L ea r n e d
K e r n e l
As a second application of the randomized SVD for HS operators, we learn the kernel G ( x, y ) =
Ai ( − 13( x 2 y + y 2 )) for x, y ∈ [ − 1 , 1] , where Ai is the Airy function (NIS, Chapt. 9) deﬁned by
[Equation]
We plot the kernel and its low-rank approximant given by the randomized SVD for HS operators in
Figure 3(b) and obtain an approximation error (measured in the L 2 -norm) of 5 .
× 10 − 14 .
The two kernels have a numerical rank equal to 42 .
The last example consists of learning the HS operator associated with the kernel G ( x, y ) =
J 0 (100( xy + y 2 )) for x, y ∈ [ − 1 , 1] , where J 0 is the Bessel function of the ﬁrst kind (NIS, Chapt. 10) deﬁned as
[Equation]
and plotted in Figure 3(c).
The rank of this kernel is equal to 91 while its approximation is of rank 89 and the approximation error is equal to 4 .
88 × 10 − 13 .
We observe that in the three numerical examples displayed in Figure 3, the difference between the learned and the original kernel is visually not perceptible.
Finally, we evaluate the inﬂuence of the choice of covariance kernel and number of samples in
Figure 4.
Here, we vary the number of samples from k = 1 to k = 100 and use the randomized
SVD for HS operators with four different covariance kernels: the squared-exponential K SE with parameters (cid:96) = 0 .
01 , 0 . 1 , 1 , and the Jacobi kernel K (2 , 2) Jac with eigenvalues
λ j = 1 /j 3 , for j ≥ 1 .
In the left panel of Figure 4, we represent the eigenvalue ratio λ j /λ 1 of the four kernels and observe that this quantity falls below machine precision for the squared-exponential kernel with (cid:96)
= 1 and (cid:96) = 0 .
1 at j = 13 and j = 59 , respectively.
In Figure 4 (right), we observe that these two kernels fail to approximate kernels of high numerical rank.
The other two kernels have a much slower decay of eigenvalues and can capture (or learn) more complicated kernels.
We then see in the right panel of Figure 4 that the relative approximation errors obtained using K (2 , 2) Jac and K SE are close to the best approximation error given by the squared tail of the singular values of the integral kernel G ( x, y ) , i.e., ( (cid:80) j ≥ k +1 σ 2 j ) 1 / 2 .
The overshoot in the error at k = 100 compared to the machine precision is due to the decay of the eigenvalues of the covariance kernels.
Hence, spatial directions associated with small eigenvalues are harder to learn accurately.
This issue does not arise in ﬁnite dimensions with the standard randomized SVD because the covariance kernel used there
[Figure]
is isotropic, i.e., all its eigenvalues are equal to one.
However, this choice is no longer possible for learning HS integral operators as the covariance kernel K must be squared-integrable.
The relative approximation errors at k = 100 (averaged over 10 runs) using K (2 , 2) Jac and K SE with (cid:96) = 0 .
01 are
Error ( K (2 , 2) Jac )
≈ 2 .
6 × 10 − 11 , and Error ( K SE )
≈ 5 .
7 × 10 − 13 , which gives a ratio of
Error ( K (2 , 2) Jac ) / Error ( K SE )
≈ 45 . 6 .
However, the square-root of the ratio of the quality of the two kernels for k = 91 is equal to (cid:113)
[Equation]
[Equation]
(7) which is of the same order of magnitude of Equation (6) as predicted by Theorem 2.
In Equation (7), γ 91 ( K SE )
≈ 5 . 88 × 10 − 2 and γ 91 ( K (2 , 2) Jac )
≈ 4 .
24 × 10 − 6 are both computed using Chebfun.
In conclusion, this section provides numerical insights to motivate the choice of the covariance kernel to learn HS operators.
Following Figure 4, a kernel with slowly decaying eigenvalues is preferable and yields better approximation errors or higher learning rate with respect to the number of samples, especially when learning a kernel with a large numerical rank.
The optimal choice from a theoretical viewpoint is to select a covariance kernel whose eigenvalues have a decay rate similar to the Rissanen sequence (Rissanen, 1983), but other choices may be preferable in practice to ensure smoothness of the sample functions (cf.
Section 4.4).
C ONCLUSIONS
We have explored practical extensions of the randomized SVD to nonstandard covariance matrices and Hilbert–Schmidt operators.
This paper motivates new computational and algorithmic approaches for preconditioning the covariance kernel based on prior information to compute a lowrank approximation of matrices and impose properties on the learned matrix and random functions from the GP.
Numerical experiments demonstrate that covariance matrices with prior knowledge outperform the standard identity matrix used in the literature and lead to near-optimal approximation errors.
In addition, we proposed a covariance kernel based on weighted Jacobi polynomials, which allows the control of the smoothness of the random functions generated and may ﬁnd practical applications in PDE learning (Boull´e et al., 2020; 2021) as it imposes prior knowledge of
Dirichlet boundary conditions.
The algorithm presented in this paper is limited to matrices and HS operators and does not extend to unbounded operators such as differential operators.
Additionally, the theoretical bounds only offer probabilistic guarantees for Gaussian inputs, while sub-Gaussian distributions (Kahane, 1960) of the inputs would be closer to realistic application settings.
A R ANDOMIZED SVD WITH MULTIVARIATE G AUSSIAN INPUTS
Let m ≥ n ≥ 1 and A be an m × n real matrix with singular value decomposition A = UΣV ∗ , where U and V are orthonormal matrices, and Σ be an m × n diagonal matrix with entries σ 1 ( A ) ≥ · · · ≥ σ n ( A ) > 0 .
For a ﬁxed target rank k ≥ 1 , we deﬁne Σ 1 and Σ 2 to be the diagonal matrices, which respectively contain the ﬁrst k singular values of A : σ 1 ( A ) ≥ · · · ≥ σ k ( A ) , and the remaining singular values.
Let V 1 be the n × k matrix obtained by truncating V 1 after k columns and V 2 the remainder.
In this section, K denotes a symmetric positive semi-deﬁnite n × n matrix and Ω ∈ R n × (cid:96) a Gaussian random matrix with (cid:96) ≥ k independent columns sampled from a multivariate normal distribution with covariance matrix K .
Finally, we deﬁne Ω 1 :=
V ∗ 1 Ω and Ω 2 :=
V ∗ 2 Ω .
We ﬁrst reﬁne (Boull´e & Townsend, 2021, Lem. 3.7).
Lemma 5 Let (cid:96) ≥ 1 , Ω ∈ R n × (cid:96) be a Gaussian random matrix, where each column is sampled from a multivariate Gaussian distribution with covariance matrix K , and T be an (cid:96) × k matrix.
Then,
[Equation]
Proof.
Let K
= Q K ΛQ ∗ K be the eigenvalue decomposition of K , where Q K is orthonormal and Λ is a diagonal matrix containing the eigenvalues of K in decreasing order: λ 1 ≥ · · · ≥ λ n ≥ 0 .
We note that Ω can be expressed as Ω = Q K Λ 1 / 2 G , where G is a standard Gaussian matrix. Let S =
Σ 2 V ∗ 2 Q K Λ 1 / 2 , the proof follows from (Halko et al., 2011, Prop. A.1), which shows that E [Equation]
Note that one can bound the term
Tr( Σ 22 V ∗ 2 KV 2 ) by λ 1 (cid:107) Σ 2 (cid:107) 2 F , where λ 1 is the largest eigenvalue of
K (Boull ´ e & Townsend, 2021).
While this provides a simple upper bound, it does not demonstrate that the use of a covariance matrix containing prior information on the singular vectors of A can outperform the randomized SVD with standard Gaussian inputs.
Then, combining the proof of (Boull ´ e & Townsend, 2021, Thm. 1.1) and Lemma 5, we prove the following proposition, which bounds the expected approximation error of the randomized SVD with multivariate Gaussian inputs.
Proposition 6 Let A be an m × n matrix, k ≥ 1 an integer, and choose an oversampling parameter p ≥ 2 .
If Ω ∈ R n × ( k + p ) is a Gaussian random matrix, where each column is sampled from a multivariate Gaussian distribution with covariance matrix K ∈ R n × n , and QR =
AΩ is the economized QR decomposition of AΩ , then,
[Equation]
where γ k = k/ ( λ 1 Tr(( V ∗ 1 KV 1 ) − 1 ))) and β k =
Tr( Σ 22 V ∗ 2 KV 2 ) /
( λ 1 (cid:107) Σ 2 (cid:107) 2 F ) .
We remark that for standard Gaussian inputs, we have γ k = β k = 1 in Proposition 6, and we recover the average Frobenius error of the randomized SVD (Halko et al., 2011, Thm. 10.5) up to a factor of ( k + p ) due to the non-independence of Ω 1 and Ω 2 in general.
Lemma 7 With the notations introduced at the beginning of the section, for all s ≥ 0 , we have
[Equation]
Proof.
Let ω
j be the j th column of Ω for 1 ≤ j ≤ (cid:96) and v 1 , . .
. , v n be the n columns of the orthonormal matrix V .
We ﬁrst remark that
[Equation]
where the Z j are i.i.d because ω j ∼ N (0 , K ) are i.i.d.
Let λ 1 ≥ · · · ≥ λ n ≥ 0 be the eigenvalues of K with eigenvectors ψ 1 , . . .
, ψ n ∈ R n .
For 1 ≤ j ≤ (cid:96) , we have,
[Equation]
i ψ [Equation] , where c ( j ) i ∼ N (0 , 1) are i.i.d for 1 ≤ i ≤ n and 1 ≤ j ≤ (cid:96) .
Then,
Z j = n (cid:88) i =
{+( c ( j ) i ) 
+}
λ i n − k (cid:88) n 1 = σ
[Equation]
k
[Equation]
( A )( v
[Equation]
where the X i are independent.
Let γ i
= λ
i (cid:80) n − k n 1 =1 σ 2 k + n 1 ( A )( v ∗ k + n 1 ψ i ) 2 , then X i ∼ γ i χ 2 for 1 ≤ i ≤ n .
Let 0 < θ < 1 / (2 (cid:80) ni
=1 γ i ) , we can bound the moment generating function of (cid:80) ni
=
X i as
[Equation]
because the X i /γ i are independent and follow a chi-squared distribution.
The right inequality is obtained by showing by recurrence that, if a 1 , . .
. , a n ≥ 0 are such that (cid:80) ni
=1 a i ≤ 1 , then (cid:81) ni
=1 (1 − a i ) ≥ 1 − (cid:80) ni
=1 a i .
For convenience, we deﬁne C 1 := (cid:80) ni
=1 γ n , we have shown that
[Equation]
Moreover, we ﬁnd that
[Equation]
Let s ≥ 0 and 0 < θ < 1 / (2 C 1 ) , by Chernoff bound (Chernoff, 1952, Thm. 1), we obtain P
[Equation]
We minimize the bound over 0 < θ < 1 / (2 Tr( K )) by choosing θ = s/ (2(1 + s ) C 1 ) , which gives
[Equation]
The proof of Theorem 2 will require to bound the term (cid:107) Ω † 1 (cid:107) 2 F , which we achieve using the following result (Boull ´ e & Townsend, 2021, Lem. 3.5).
Lemma 8 (Boull´e & Townsend 2021) Let k, (cid:96) ≥ 1 such that (cid:96) − k ≥ 4 , with the notations introduced at the beginning of the section, for all t ≥ 1 , we have
[Equation] We now prove Theorem 2, which provides a reﬁned probability bound for the performance of the generalized randomized SVD on matrices.
Proof of Theorem 2.
Using (Boull ´ e & Townsend, 2021, Thm. 3.2) and the submultiplicativity of the Frobenius norm, we have
[Equation]
Let (cid:96) =
k
+ p with p ≥ 4 , combining Lemmas 7 and 8 to bound the terms (cid:107) Σ 2 Ω 2 (cid:107) 2 F and (cid:107) Ω † 1 (cid:107) 2 F in
Equation (10) yields the following probability estimate:
[Equation]
with failure probability at most t − p + (1 + s ) ( k + p ) / 2 e − s ( k + p ) / 2 .
Note that we introduced γ k := k/ ( λ 1 Tr
( V ∗ 1 KV 1 ) − 1 ))) and β k :=
Tr( Σ 22 V ∗ 2 KV 2 ) /
( λ 1 (cid:107) Σ 2 (cid:107) 2 F ) .
We conclude the proof by deﬁning u = √ 1 + s ≥ 1 .
The following Lemma provides an estimate of the quantity β k introduced in the statement of Theorem 2.
Lemma 9 Let β k
=
Tr( Σ 22 V [Equation]
2 KV 2 ) / ( λ 1 (cid:107) Σ 2 (cid:107) 2 F ) , then the following inequality holds
[Equation]
Proof.
Let µ 1 ≥ · · · ≥ µ n − k be the eigenvalues of the matrix V ∗ 2 KV 2 .
Using Von Neumann’s trace inequality (Mirsky, 1975; Von Neumann, 1937), we have
[Equation]
Then, the matrix V ∗ 2 KV 2 is a principal submatrix of V ∗ KV , which has the same eigenvalues of
K .
Therefore, by (Kato, 2013, Thm. 6.46), the eigenvalues of V ∗ 2 KV 2 are individually bounded by the eigenvalues of K , i.e., µ j ≤ λ j for 1 ≤ j ≤ n − k , which concludes the proof.
Finally, we highlight that the statement of Theorem 2 can be simpliﬁed by choosing p = 5 , t = 4 , and u = 3 .
Corollary 1 (Generalized randomized SVD) Let A be an n × n matrix and k ≥ 1 an integer.
If Ω ∈ R n × ( k +5) is a Gaussian random matrix, where each column is sampled from a multivariate
Gaussian distribution with symmetric positive semi-deﬁnite covariance matrix K ∈ R n × n , and
QR =
AΩ is the economized QR decomposition of AΩ , then
[Equation]
In contrast, a simpliﬁcation of Theorem 1 by choosing t = 6 and u = 4 gives the following result.
Corollary 2 (Randomized SVD) Let A be an n × n matrix and k ≥ 1 an integer.
If Ω ∈ R n × ( k +5) is a standard Gaussian random matrix and QR =
AΩ is the economized QR decomposition of AΩ , then
[Equation]
B C ONTINUITY OF THE J ACOBI KERNEL
Here, we show that if the kernel’s eigenvalue sequence, { λ j } , is such that (cid:80) ∞
j =1 jλ j < ∞ , then the Jacobi kernel K (2 , 2) Jac deﬁned by Equation (5) is continuous.
First, note that ˜ P (2 , 2) j is a scaled ultraspherical polynomial ˜ C (5 / 2) j with parameter 5 / 2 and degree j ≥
so it can be bounded by the following proposition.
Proposition 10 Let ˜ C
j be the ultraspherical polynomial of degree j with parameter 5 / 2 , normalized such that (cid:82) [Equation] x [Equation]
d x
[Equation]
Then,
[Equation]
Proof.
Let j ≥ 0 and x ∈ [ − 1 , 1] , according to (NIS, Table 18.3.1),
[Equation]
where C (5 / 2) j ( x ) is the standard ultraspherical polynomial.
Using (NIS, (18.9.8)), we have
[Equation]
By using (NIS, (18.9.7)), we have ( C (3 / 2) j +2 ( x ) − C (3 / 2)
j [Equation]
+ [Equation]
+ [Equation] x ) and hence,
[Equation]
We bound the two terms with (NIS, (18.14.4)) to obtain the following inequalities: [Equation] [Equation] +
[Equation]
[Equation] ) | [Equation] Hence, | (1 − x 2 ) C (5 / 2)
j [Equation] x ) | ≤ 2( j + 1)( j + 2) / 3 and following Equation (12) we obtain
[Equation]
which concludes the proof.
The bound given in Proposition 10 differs by a factor of 4 / 3 from the numerically observed upper bound ( 1 . 5 (cid:112)
j
+ 5 / 12 ).
This result shows that if (cid:80) ∞
j =1 jλ j < ∞ , then the series of functions in
Equation (5) converges uniformly and K (2 , 2)
Jac is continuous.
Proof of Lemma 3.
By Markov brothers’ inequality (Markov, 1889), for all j ≥ 0 and 0 ≤ µ ≤ j , we have max x ∈ [ − 1 , 1] | p ( µ ) j ( x ) | ≤ j 2 µ .
Therefore, | f ( µ ) n ( x ) | ≤ (cid:80) nj =0 | a j |(cid:107) p [Equation] µ ) [-j (cid:107) ∞ ≤ (cid:80) nj =
-][Equation]
{+∞ ≤ (cid:80) nj =
+}
j 2 µ − ν so | f ( µ ) n ( x ) | < ∞ if µ < ( ν − 1) / 2 .
The result follows from (Rudin, 1976, Thm. 7.17).
Proof of Theorem 4.
Since f ∼ GP (0 , K (2 , 2) Jac ) , f ∼ (cid:80) ∞
j =
c j (cid:112) λ j +
(1 − x 2 )
˜ P (2 , 2) j ( x ) , where c j ∼ N (0 , 1) for j ≥ 0 .
Let f n denote the truncation of f after n terms.
By letting M > 0 be the constant such that λ j +1 ≤ M ( j + 1) − ν , we ﬁnd that
[Equation]
where we used max x ∈ [ − 1 , 1] | (1 − x 2 ) ˜ P (2 , 2) j ( x
[Equation]
| ≤ 2 √ [Equation] 1 (cf.
Proposition 10).
Thus, we have
[Equation]
Here, S n ∼ X n = (cid:80) ∞ j = n +
Y j j (1 − ν ) / 2 , where Y j follows a half-normal distribution (Leone et al., 1961) with parameter σ = 1 and the ( Y j ) j are independent.
We want to show that X n a.s. −−→ 0 .
For (cid:15) > 0 , using Chebyshev’s inequality, we have:
[Equation]
which is ﬁnite if ν > 3 .
Therefore, using Borel–Cantelli Lemma (Durrett, 2019, Chapt. 2.3), X n converges to 0 almost surely and P (lim n →∞ X n = 0) = 1 .
Finally,
[Equation]
which proves that { f n } converges uniformly and hence f is continuous with probability one.
The statement for higher order derivatives follows the proof of Lemma 3.
C E XPERIMENTS WITH NON - STANDARD COVARIANCE MATRICES
In this section, we provide more detailed explanations regarding the numerical experiments conducted in Section 5.1.
Sampling a random vector from a multivariate normal distribution with an arbitrary covariance matrix K can be computationally expensive when the dimension, n , of the matrix is large as it requires the computation of a Cholesky factorization, which can be done in O ( n 3 ) operations.
We highlight that this step can be precomputed once, such that the overhead of the generalized SVD can be essentially expressed as the cost of an extra matrix-vector multiplication.
Then, the difference in timings between standard and prior covariance matrices is marginal as shown by the right panel of
Figure 2.
Additionally, we would like to highlight that prior covariance matrices can be designed and derived using physical knowledge of the problem, such as a diffusion behaviour of the system, which can also signiﬁcantly decrease the precomputation cost.
As an example, in Section 5.1, we employ the discretized Green’s function of the negative Laplacian operator with homogeneous
Dirichlet boundary conditions, given by L u = − d 2 u/dx 2 on [0 , 1] , for which we know the eigenvalue decomposition.
Hence, the eigenvalues and normalized eigenfunctions are respectively given by
[Equation]
Therefore, one can employ Mercer’s representation (see Equation (4)) to sample the random vectors and precompute the covariance matrix in O ( n 2 ) operations.
For a problem of size n = 2000 , it takes 0 .
16 s to precompute the matrix.