Minimax Optimal Kernel Operator Learning via Multilevel Training
Abstract
Learning mappings between infinite dimensional function spaces has achieved empirical success in many disciplines of machine learning, including generative modeling, functional data analysis, causal inference, and multi-agent reinforcement learning.
In this paper, we study the statistical limit of learning a Hilbert-Schmidt operator between two infinite-dimensional Sobolev reproducing kernel Hilbert spaces.
We establish the information-theoretic lower bound in terms of the Sobolev
Hilbert-Schmidt norm and show that a regularization that learns the spectral components below the bias contour and ignores the ones that above the variance contour can achieve optimal learning rate.
At the same time, the spectral components between the bias and variance contours give us the flexibility in designing computationally feasible machine learning algorithms.
Based on this observation, we develop a multilevel kernel operator learning algorithm that is optimal when learning linear operators between infinite-dimensional function spaces.
Introduction
The supervised learning of operators between two infinite-dimensional spaces has attracted attention in many machine learning applications, such as scientific computation [1, 2, 3, 4, 5], functional data analysis [6, 7, 8], learning mean-field games [9, 10], conditional probability regression [11, 12, 13] and econometrics [14, 15, 16].
Despite the empirical success of operator learning, the statistical limit of learning an infinite-dimensional operator is poorly studied.
In this paper, we study the problem of learning Hilbert Schmidt operators between infinite-dimensional Sobolev reproducing kernel Hilbert spaces H βK and H γL with given kernels k and l respectively and β, γ ∈ [0 , 1) [17, 18, 19].
Our goal is to derive the optimal sample complexity to learn the linear operator, i.e. how much data is required to achieve a certain performance level.
We first establish an information-theoretic lower bound for learning a Hilbert-Schmidt operator between Sobolev spaces respect to a general Sobolev norm.
Our information-theoretic lower bound indicates that the optimal learning rate is determined by the minimum of two polynomial rates: one is purely decided by the input Sobolev reproducing kernel Hilbert space and its evaluating norm, while the other one is purely determined by the output space along with its evaluating norm.
The rate is novel in the sense that all existing results [19, 20, 3] only establish rates that depend on the parameter of input space.
The reason is all previous works [21, 20, 3] only consider the case of the output space as a subspace of a trace bounded reproducing kernel Hilbert space but not a general Sobolev space.
We refer to Remark 2.1 for detail comparisons.
To design a learning algorithm for approximating an infinite-dimensional operator, we need to learn a finite dimensional restriction instead of the whole operator, as the latter would result in infinitevariance.
The finite dimensional selection leads to bias error but decreases the variance.
A natural task is then to study the shape of regularization that can lead to the optimal bias-variance trade-off and achieve the optimal learning rate.
In this paper, we consider the bias contour and the variance contour at the scale of optimal learning.
Once the regularization enables one to learn all the spectral part above the bias contour and below the variance contour, the learning is optimal.
Finally, utilizing the region in between the bias contour and variance contour , we developed a multilevel training algorithm [22, 23] which first learns the mapping on low frequency and then successively fine-tunes the machine learning models to fit the high-frequency output.
The intuition of our algorithm aligns with the original motivation of multilevel Monte Carlo [24, 25]: we use the next level to reduce bias while keeping the variance at the same scale.
We demonstrate that such multilevel algorithm can achieve optimal non-parametric rate for linear operator learning.
1.1 Related Work
Machine Learning Based PDE
Solver Solving PDEs plays a prominent role in many scientific and engineering discipline, such as physics, chemistry, operation management, macro-economy, etc.
The recent deep learning breakthrough has drawn attention to solving PDEs via machine learning methods [26, 27, 28, 29, 30, 31].
The statistical power and computational cost of these problem is well-studied by recent papers [32, 33, 34, 35].
This paper focuses on operator learning [36, 37, 38, 39, 40, 1, 2, 41, 42], i.e. learning a map between two infinite dimensional function spaces.
For example, one can learn a PDE solver that maps from the boundary condition to the solution or an inverse problem that maps from the boundary measurement to the coefficient field.
In terms of the mathematical foundation, [43] considers the learning rate of non-parametric operator learning.
However, non-parametric functional data analysis often suffers from slower-than-polynomial convergence rates [44], due to the small ball probability problem for the probability distributions in infinite dimensional spaces [45].
The most relevant works are [46, 47, 3], which consider the rates for learning a linear operator.
For the comparison between our work and [3], see Remark 2.1. multilevel Monte Carlo By combining biased estimators with multiple stepsizes, multilevel Monte
Carlo (MLMC) [24, 25] dramatically improves the rate of convergence and achieves in many settings the canonical square root convergence rate associated with unbiased Monte Carlo [48, 49].
Multilevel
Monte Carlo can also be used for random variable with infinite variance [50, 51].
To the best of our knowledge, this is the first paper that provides optimal sample complexity for multilevel Monte Carlo type algorithm for infinite variance problems in the non-parametric regime.
Very recently, [22, 23] developed a multilevel machine learning Monte Carlo algorithm (ML2MC) / multilevel fine-tuning algorithm for learning solution maps, by first learning the map on coarsest grid and then successively fine-tuning the network on samples generated at finer grids.
The authors also showed that, following the telescoping in MLMC, the multilevel training procedure can reduce the generalization error without spending more time on generating training samples.
[52, 53] consider such multi-scale algorithm for learning Green’s function.
However, the statistical power of such algorithm is still under investigation.
Another difference with [53] is that we consider the Green function in H −norm rather than the ℓ 1 norm used in [53].
In this paper, we qualify a specific setting where this multilevel procedure can and is necessary to achieve the minimax optimal learning rate.
1.2 Contribution
• We derive a novel information-theoretic lower bound of learning a linear operator between two infinite-dimensional Sobolev reproducing kernel Hilbert spaces.
The optimal learning rate is a minimum of two polynomial rates, one only dependent on the parameters of the input space while the other only on the parameters of the output space.
The first rate aligns with the previous works [20], while the second lower bound is novel to the literature.
• We study the shape of regularization that can lead to the optimal learning rate.
One should learn all the spectral parts under the bias contour at the level of the optimal learning rate but not the spectral parts above the variance contour at the level of learning rate.
This enables the estimator to enjoy an optimal balance of bias-variance.
• We qualify a specific setting where a multilevel training procedure [22, 23] is necessary and capable of achieving a minimax optimal learning rate for learning a linear operator.
We achieve the optimal learning rate via O (ln ln n ) ensemble of ridge regression models.
This is different from finite-dimensional operator learning where a single level estimator can be optimal.
Problem Formulation 2.1 Preliminary
Let P K be a distribution over the input space H K and define covariance operator C KK
= E u ∼ P K u ⊗ u .
Consider its spectral decomposition C KK = (cid:80) + ∞ i =1 µ 2 i e i ⊗ e i , where { µ 12 i e i } + ∞ i =1 is an orthogonal eigenbasis and { µ i } is the corresponding eigenvalues of C KK (here the g ⊗ h is an operator defined as g ⊗ h = gh ∗ :
f → ⟨ f, h ⟩ g ).
In the typical machine learning applications, the test distribution is the same as the training distribution, so we can assume that H K
=
(cid:110)(cid:80)
i a i µ 12 i e i : { a i } ∞ i =1 ∈ ℓ 2 (cid:111) without loss of generality.
Note that this automatically holds in the context of learning the conditional mean embedding (CME) [19, 21, 20].
Following [18, 19], we define the interpolation
Sobolev space
H βK = (cid:26) f
= (cid:80) i a i ( µ β
i e i ) : { a i } ∞
i =1 ∈ l(cid:27) for any β > 0 , equipped with Sobolev norm defined by the inner product (cid:68)(cid:80) i {+a i ( µ β/ 
+}
i e i ) , (cid:80) i b i ( µ β/ {+
+} i e i ) (cid:69) H βK
= (cid:80) i a
i b i .
For the output space, we fix a user-specified distribution Q L and a reproducing Kernel Hilbert Space.
We can similarly define the covariance operator C Q L and the Sobolev space H γL .
Natural choices of Q L include some distribution on kernel functions { ℓ ( y, · ) : y ∈ Y } of H L induced by some distribution Q L on Y , so that C Q L is a kernel integral operator with respect to Q L and H γL is an interpolation space between H L and
L 2 ( Q L ) ; see Example 2.1 for a specific choice of Q L and its practical implications.
Following [20], in this paper we consider the Hilbert-Schmidt norm between two Sobolev Spaces for all the operators, which is defined as following.
Definition 2.1 ( ( β, γ ) -norm) Let T : H K (cid:55)→ H L be a possibly unbounded linear operator.
I 1 ,β,P K :
H K (cid:55)→ H βK , β ∈ (0 , 1) is the canonical embedding mapping that takes u ∈ H K to the same element u in the larger space H βK , and I 1 ,γ,Q L : H L (cid:55)→ H γL , γ ∈ (0 , 1) is similarly defined.
Then the ( β, γ ) -norm of T is defined as
[Equation]
where we omit the dependence of ∥·∥ β,γ on P K and Q L since it will always be clear from context.
2.2 Problem Formulation
We consider the problem of learning an unknown linear operator A 0 : H K (cid:55)→ H L between two reproducing kernel Hilbert spaces corresponding to kernl k and l respectively.
We are given N noisy data pairs ( u i , v i ) , 1 ⩽
i ⩽ N related by v i =
A 0 u i + ε i , where u i i .
i .
d . ∼ P K for some unknown distribution P K and ε i is the noise drawn from some distribution with zero mean that may depend on u i .
We use P KL for the joint distribution of ( u i , v i ) .
Denote C KK = E u ∼ P K u ⊗ u ,
C KL =
E ( u,v ) ∼ P KL u ⊗ v and its adjoint C LK
= C ∗ KL be uncentered cross-covariance operators associated with P KL .
Then we can reformulate the ground turth operator as A 0 =
C LK C † KK , where † is the pseudo-inverse [21, 20].
With the goal of understanding the relative difficulty of learning different types of linear operators, we investigate the sample efficiency of learning A 0 under certain source assumptions imposed on the data model.
Source condition [54, 55, 56, 57, 19] assumes that the learning target lies in a parameterized function class and study the learning rate for different problems with different hardness.
Specifically, the source condition assume that the learning target is bounded in certain Sobolev norm.
In this paper, we consider learning an operator with bounded ( β, γ ) -norm, which is the Hilbert-Schmidt norm that maps from H βK to H γL .
We consider the generalization error/convergence rate under another ( β ′ , γ ′ )
-norm as in [19, 33, 21, 20].
Remark 2.
Although recent works have considered similar problems in the context of conditional mean embedding [21, 20] and functional data analysis [3].
In all these papers, the output space is a trace bounded reproducing kernel Hilbert space [3, Assumption 2.14 (vi)] rather than the general parameterized Sobolev space in our paper.
We then list all the assumptions imposed on the underlying kernel for our theoretical results.
We follow the standard capacity assumptions and embedding properties used in kernel regression [19, 21, 20].
Assumption 2.1 (Capacity Condition of the Covariance)
The eigenvalues { µ i } i ⩾ 1 of the covariance operator C KK = E u ∼ P K u ⊗ u satisfies µ i ∝ i − 1 p for some p ∈ (0 , 1) .
Similarly, the eigenvalues { ρ i } i ⩾ 1 of the covariance operator C Q L = E v ∼ Q L v ⊗ v satisfies ρ i ∝ i − 1 q for some q ∈ (0 , 1) .
Assumption 2.2 ( ℓ ∞ Embedding Property of the Input RKHS)
There exists a smallest α ∈ (0 , 1) such that (cid:13)(cid:13)(cid:13)(cid:0)
I ∗ 1 ,α,P K (cid:1) † f (cid:13)(cid:13)(cid:13)
H αK ⩽
A 1 a.s. under P K for some A 1 < + ∞ .
Assumption 2.3 ( ℓ ∞ Embedding Property of the Output RKHS)
There exists A 2 < + ∞ such that ∥ g ∥ H L ⩽
A 2 holds for all g in the range of A 0 , except from a Q L -null set.
Assumption 2.4 (Moment Condition)
There exists an operator V : H L (cid:55)→ H L with tr ( V ) ⩽ σsuch that for every u ∈ H K , We have E v ∼ P KL ( ·| u ) (cid:104) (( v − A 0 u ) ⊗ ( v − A 0 u ))
k (cid:105) ⪯ 12 (2 k )!
R 2 k − 2 V. holds for all k ≥ 2 .
Assumption 2.5 (Source Condition)
A 0 is bounded under ( β, γ ) -norm i.e. ∥A 0 ∥ β,γ ⩽ B for some
B ∈ (0 , + ∞ ) .
2.3 Examples
In this section, we will introduce two examples of our theory.
The first one is about learning a differential operator, for example inferring an advection-diffusion model [58] from observations or predicting the future [37, 1, 2, 39, 59].
The second example is about learning conditional mean embedding [11, 12, 13], which represents a conditional distribution as an RKHS element.
Thus conditional distribution regression can be reduced to a kernel operator learning.
Our theory can also be used for linear inverse problem such as radial electrical impedance tomography (EIT) [60] and the severely ill-posed inverse boundary problem for the Helmholtz equation with unknown wave-number parameter [61].
For detailed discussion, we refer to [3, Section 1.3]
Example 2.1 (Learning differential operators) Suppose that the ground-truth operator A 0 = ∆ t where ∆ is the Laplacian and t ∈ Z .
Let H K
= H m +2 t ([0 , 1]) be the Sobolev space with smoothness m + 2 t on [0 , 1] and H L
= H m ([0 , 1]) , then A 0 is a bounded operator from H K to H L which corresponds to the β = γ = 1 case.
However, we will see below that we can obtain a better characterization of the learning error using our theory.
Consider for example that the input has mean zero and the Matérn-type covariance operator C KK = σ 2 (cid:0) − ∆ + τ 2 I (cid:1) − s .
Its eigenvalues satisfy µ n ∝ n − 2 s .
On the other hand, we choose Q Y to be a distribution supported on { ℓ ( y, · ) : y ∈ [0 , 1] } induced by a uniform distribution on [0 , 1] , where ℓ is the kernel function of H L .
Then C Q Y is essentially the kernel integral operator on H L w.r.t. the uniform distribution, and its eigenvalues are ρ n ∝ n − 2 m .
The assumption ∥A 0 ∥ β,γ < + ∞ is satisfied if and only if (1 − γ ) m < (1 − β ) s − 12 ⇒ γ > 1 − 2(1 − β ) s − 1 2 m .
Example 2.2 (Conditional mean embedding) Suppose that we would like to learn the conditional distribution P ( y | x ) from a data set { ( x i , y i ) : 1 ⩽ i ⩽ N } ⊂ X × Y where x i i .
i .
d . ∼ P K .
Let H K and H L be two RKHSs on X and Y respectively, with measurable kernel k ( · , · ) and ℓ ( · , · ) .
Then we can define a conditional mean embedding (CME) operator C Y |
X that satisfies
C Y |
X k ( x, · ) =
E Y | x ℓ ( Y, · ) =: µ Y | x , and E Y
| x g ( Y ) =
(cid:10) g, µ Y | x (cid:11) ∀ x ∈ X.
We choose A 0 = C Y | x .
In this case, C KK = E P K k ( X, · ) ⊗ k ( X, · ) .
Assumption 2.2 states that sup x ∈ X k α ( x, x ) =
A 1 , while Assumption 2.3 is equivalent to sup x ∈ X (cid:13)(cid:13) µ Y | x (cid:13)(cid:13) ⩽
A 2 (for simplicity we only focus on the case ζ = 1 ).
According to Assumption 2.5, we assume that (cid:13)(cid:13) C Y | X (cid:13)(cid:13) β,γ ⩽ B .
The mis-specified setting where β < 1 has been studied in previous work [19, 21, 20].
However, they only consider the case γ = 1 .
Our results also cover the case γ < 1 , which allows us to obtain theoretical guarantee for computing conditional expectation of the larger function class H γL .
Information Theoretic Lower Bound
In this section, we provide an information-theoretic lower bound for the convergence rate of the operator learning problem formulated in Section 2.
Theorem 3.1 Suppose that H K and H L are two Hilbert spaces, P K and Q L are probability distributions on H K and H L respectively such that Assumptions 2.1 and 2.2 hold.
Then for any estimator
L : ( H K × H L ) ⊗ N (cid:55)→ HS (cid:16) H βK , H γL (cid:17) , there exists a linear operator A 0 and a joint data distribution P XY with marginal distribution P K on H K satisfying Assumptions 2.3 to 2.5, such that with probability ⩾ 0 .
99 over ( u i , v i ) i .
i .
d . ∼ P XY we have
[Equation]
Remark 3.1 Our lower bound is composed of a minimum of two parts.
The first rate N −
[Equation]
is the minimax optimal Sobolev learning rate for kernel regression [19, 21, 20, 33] and is fully determined by the parameter of the input Sobolev reproducing kernel Hilbert space.
Our second rate N − γ ′− γ 1 − γ is novel to the literature.
This bound shows that how the infinite dimensional problem is different from finite dimensional regression problem and is fully determined by the parameter of the output Sobolev reproducing kernel Hilbert space.
Our lower bound shows that the hardness of learning a linear operator is determined by the harder part between the input and output spaces.
We will explain why the lower bound has such structure in Remark 4.1 and Figure 2.
On the Shape of Regularization
In this section, we aim to understand the shape of regularization so that the constructed estimator ˆ A based on N i.i.d.
data { ( u i , v i ) } ni
=1 ∼ P ⊗ nKL for 1 ⩽
i ⩽ N enjoys an optimal learning rate.
Compared with existing approaches where a regularized least-squares estimator can achieve statistical optimality [19, 21, 20, 3] under ( β, 1) -norm, we study the learning rate under the ( β, γ ) -norm ( β ′ ∈ (0 , β ) , γ ′ ∈ ( γ, 1) ) which is defined in Definition 2.1 as
[Equation]
HS .
The norm of the additional C −
[Equation]
term is unbounded which make our setting harder than the convergence in ( β, 1) -norm in existing works.
Since C − 1 − γ [Equation] 2 Q L is bounded when restricted to the finite-dimensional space span (cid:16) ρ 12 i f i : 1 ⩽
i ⩽ n (cid:17) , we should also include another bias-variance trade-off via regularizing in the output shape.
As a result, we are interested in answering the following question
What is the optimal way to combine the regularization in the input space and regularization in the output space?
i.e. What is the optimal shape of regularization?
To answer this question, we investigate the problem in the spectral space, i.e. considering the spectral representation of operator
A
= (cid:80)
+
∞ i,j =1 a ij µ β 2 i e
[Equation]
f j .
The problem of estimating A then reduces to learning the coefficients “matrix" ( a ij ) ∞
i,j =1 .
The source condition Assumption 2.5 enforces (cid:80) ∞ i,j =1 a 2 ij ≤ B .
We show in Appendix B.1.1 that regularizing the basis e i ⊗ f j will introduce a bias of order (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
under the ( β ′ , γ [Equation] ) -norm.
On the other hand, when α ⩽ β + p , we show in Appendix B.1.2 that the variance of learning ( i, j ) from noisy data scales as 1 N µ − β
i
[Equation]
q .
Since the variance would accumulate for a fixed j , learning ( i, j ) for i ⩽ i max results in a variance of ∝ 1 N i β ′
[Equation]
max j
[Equation]
(Similar analysis can be carried out for the α > β + p case as well, but the variance now scales as 1 N i β ′ + α
[Equation]
max j 1 − γ ′ q ; see
Appendix B for detailed derivations.)
In summary, we need to make bias-variance trade off in the ( i, j ) − plane, i.e. decide whether we should learn or regularize over the basis e i ⊗ f j .
4.1 Regularization via variance contour
The underlying idea of regularization is that some components are intrinsically hard to learn due to large variance; these components are then neglected by adding regularization and are counted as bias.
The remaining components are easy to learn due to controllable variance.
This intuition works well when the estimation error results from the noise of the data and is well-studied in a line of works [19, 3, 21, 20].
This idea still works in our setting, but we need to re-evaluate the bias and variance of each component.
Since we work with the Hilbert-Schmidt norm, this can be done in a coordinate-wise manner, meaning that we can look at each a ij separately and decide whether to neglect it (contribute to bias) or to learn it from data (contribute to variance).
Since the variance term measures the hardness of learning, we naturally introduce the notion of variance contour , which is a curve on the R 2+ plane on which all points induce the same order of variance (here we work with real coordinates for convenience, although we only care about integer points).
Formally, we fix an arbitrary constant C > 0 and define
[Equation]
[Equation]
A reasonable regularization scheme is then to learn all coordinates ( i,
j ) ∈
Z 2 + below the curve ℓ C, var and ‘regularize out’ the remaining coordinates that are difficult to learn due to large variances.
This can gives us the estimator with smallest estimator at give variance level.
This observation motivates us to construct our estimator as
[Equation]
where λ j (1 ⩽ j ⩽ y N = C q 1 − γ ′ ) are the regularization coefficients imposed on different dimensions of the output space.
According to (1) and noting that µ i ∝ i − 1 p , we define
[Equation]
log N [Equation] (3) with C = N max (cid:110) 1 − β − β ′ max { α,β + p } , 1 − γ
[Equation]
1 − γ (cid:111) in (1).
The additional N − 1 α term in (3) is needed for controlling the error of approximating C KK via ˆ C KK (cf.
Theorem D.3) which is standard in the Sobolev learning [Figure] literature [19, 21, 33].
The following theorem describes the convergence rate of our estimator defined by (2) and (3).
Theorem 4.1 Consider the estimator ˆ A defined by (2) and (3) .
Suppose that Assumptions 2.1 to 2. hold, then there exists a universal constant C such that with probability ⩾ 1 − e − τ , we have
[Equation]
[Figure]
4.2 Regularization via Bias Contour
We have showed that if we learn all the spectral components under certain variance contour and regularize all other component can achieve optimal rate.
In this section, we introduce another scheme to design the optimal estimator via learning all the spectral component under a certain bias contour.
Specifically, we consider deciding the regularization strength according to the spectral elements induce a certain level of bias i.e. the bias contour ℓ C ′ , bias = (cid:110) ( x, y ) ∈ R 2+ : x [Equation]
y
[Equation]
q = C
[Equation]
(cid:111) .
does not coincide with ℓ C, var for any C ′ up to constant scaling.
Thus, there exists a point ( x ∗ , y ∗ ) on the variance contour with maximal contribution to bias.
Naturally, we can also construct our estimator using a bias contour that passes through ( x ∗ , y ∗ ) .
In this case, we may define λ j = max (cid:40)(cid:18)
[Equation]
for similar reasons as Section 4.1, which also yields optimal rate as stated in Theorem 4.2 below.
Remark 4.1 (On the optimal shape of regularization)
The discussion in Sections 4.1 and 4.2 reveals another understanding of our information theoretic lower bound.
Firstly, we should learn all the spectral components under the bias contour otherwise the bias will exceed the lower bound.
Secondly, we should not learn any spectral component over the variance contour since otherwise the variance will exceed the lower bound.
Thus the bias contour should always be under the variance contour , otherwise no estimator can be designed.
The bias and variance contours at the level of optimal learning rate are plotted in Figure 2.
They only meet at ( x ∗ , y ∗ ) with x ∗ = 1 or y ∗ = 1 , which has the largest contribution to the bias (resp. variance) among all points on the variance (resp. bias) contour, thus dominating the estimation error.
When the two curves meet at y ∗ = 1 , it reduces to the original kernel regression case.
When the two curves meet at x ∗ = 1 , it leads to our new rate that depends on the output space. [Figure]
[Figure]
, where the bias and variance contours overlap and we set x n +1 = 12 x n .
Each rectangular represents a certain level of regularization.
Theorem 4.2 Consider the estimator ˆ A defined by (2) with λ j defined above.
Suppose that Assumptions 2.1 to 2.5 hold, then there exists a universal constant C , such that (cid:13) (cid:13)(cid:13) ˆ A − A 0 (cid:13) (cid:13)(cid:13)
[Figure]
⩽
[Figure]
MultiLevel Kernel Operator Learning
In this section, we study a multilevel machine learning algorithm [22, 23, 53] but at each level we consider a cost-accuracy trade-off [62] to control the variance at a proper scale.
We show that the multilevel level algorithm can cover all the spectral component below the bias contour and achieve the optimal learning rate.
Our idea is similar to the multilevel Monte Carlo [24, 25], which reduces bias from multilevel algorithm.
Our multilevel estimator differs from the DeepONet [1] and the
PCA-Net [63] since we add different regularizations for each level.
Our theory indicates that the multilevel approach outperforms previous ones and achieves the optimal learning rate.
The basic idea is to design a minimum number of machine learning estimators that cover all the spectral elements under the bias contour but do not exceed the variance contour at the same time.
To achieve this, we choose sequences { x i } and { y i } for 1 ⩽
i ⩽ L N where y i denotes the i -th level and x i controls the corresponding regularization via the regularization coefficient
λ ( K ) i
[Equation]
The sequences are chosen in a staircase manner as plotted in Figure 3 (for formal definitions see
Appendix C).
The eigenbasis (cid:110) ρ 12 j f j (cid:111) of the output space is divided into defferent levels by { y i } .
The main idea behind our multilevel method is that different levels of the output need to be learned with different regularization.
Formally, we define our multilevel estimator as
[Equation]
The following theorem shows that the estimator (4) can achieve the optimal convergence rate with
L N =
O
(ln ln N )
when
[Equation]
We also show that O (ln N ) estimator is needed for the case when
[Equation]
3 Right) in Appendix C.
Theorem 5.1 Suppose that Assumptions 2.1 to 2.5 hold, then there exists a sequence { y i } 1 ⩽
i ⩽ L N with L N =
O (ln N ) when
[Equation]
and O (ln ln N ) otherwise, such that the estimator ˆ
[Equation]
satisfies
[Equation]
probability [Equation]
1 − e − τ , where C is a universal constant.
A Proof of the lower bound
In this section, we follow the lower bound proof in (author?)
[19] to give a lower bound of the convergence rate in our operator learning setting.
A.1 Preliminaries on Tools for Lower Bounds
In this section, we repeat the standard tools we use to establish the lower bound.
The main tool we use is the Fano’s inequality and the Varshamov-Gilber Lemma.
Lemma A.1 (Fano’s methods) Assume that V is a uniform random variable over set V , then for any Markov chain V
→ X → ˆ V , we always have
[Equation]
In our proof we will use a version from (author?)
Lemma A.2 [19, Theorem 20] Let M ⩾ 2 , (Ω , A ) be a measurable space, P 0 , P 1 , . .
, P M be probability measures on (Ω , A ) with P j ≪ P 0 for all j = 1 , . . .
, M , and 0 < α ∗ < ∞ with
[Equation]
Then, for all measurable functions Ψ : Ω → { 0 , 1 , . . .
, M } , the following bound is satisfied
[Equation]
Lemma A.3 (Varshamov-Gillbert Lemma,[64] Theorem 2.9) Let D ≥ 8 .
There exists a subset V = { τ (0) , · · · , τ (2 D/ 8 ) } of D − dimensional hypercube H D
= { 0 , 1 } D such that τ (0) =
(0 , 0 , · · · , 0) and the ℓ 1 distance between every two elements is larger than D D (cid:88) l [Equation] ∥
[Equation]
A.2 Proof of the Lower Bound
To prove our lower bound, we construct a sequence of linear operators as follows:
[Equation]
where m 1 and m 2 are hyper-parameters (scale as poly( N ) and will be selected later) and K is a constant that will be specified afterwards.
It’s easy to check that
[Equation]
By Gilbert-Varshamov Lemma it is possible to select M ε ⩾ 2 m 1 K/ 8 binary strings
[Equation] [Equation]
1 K such that (cid:13) (cid:13) ω ( i ) − ω
( j ) (cid:13) (cid:13) 2 2 ⩾ 4 ε .
Let Ω be the collection of this strings.
We now select the hyper-parameters to satisfies the assumptions made in Section 2.
First we have
[Equation]
where the last step follows from Assumption 2.1.
Similarly, we have ∥ C ω
∥ 2 α,≲ ε (2 m 1 ) α − β ′ p (2 m 2 ) γ ′− 1 q .
To the assumptions made in Section 2, we should make
[Equation]
be satisfied.
To be specific, with the previous selection of hyper-parameters, we can have ∥ C ω ∥ β,γ =
O (1) and
[Equation]
range
[Equation]
C
[Equation]
where the last step follows from our assumption on the input distribution Assumption 2.2.
This verifies that Assumptions 2.3 and 2.5 hold for C ω , ∀ ω ∈ Ω .
We now construct the hypothesis (probability distributions) as follows: for ∀ ω ∈ { 0 , 1 } m 1 , define
P ω (d f, d g
[Equation]
=
d N
[Equation]
C
[Equation] f, Σ [Equation] g
[Equation]
d P
[Equation]
f ) where the covariance operator Σ
=
σ 2 K (cid:80) Kj =1 ρ j + m 2 f j + m 2 ⊗ f j + m 2 for some constant σ > 0 .
It’s then easy to see that tr (Σ) =
σ 2 , which satisfies Assumption 2.4 .Note that the range of C ω is span( f m 2 ) and Σ is non-degenerate on this subspace.
As a result, we can view P ω , ω ∈ Ω as distributions on H K × span ( f j + m 2 : 1 ⩽ j ⩽ K ) , and we have for ∀ ω, ω
′ ∈
Ω
that
[Equation]
where the last step follows from E P K f ⊗ f = C KK = (cid:80) ∞ i =1 µ i e i ⊗ e i and recall that K is a constant.
Hence we deduce that
[Equation]
Applying Lemma A.2, we find that when
[Equation]
there exists a hypothesis P ω 0 such that for any estimator ˆ C ω 0 ,
[Equation]
holds with high probability.
Finally, we need to choose optimal m 1 and m 2 under the constraint (5).
It turns out that either m 1 = 1 or m 2 = 1 , and the resulting lower bound is
[Equation]
B Proof of the upper bound
In this section, we upper-bound the learning error of estimator (2) which defined as
[Equation]
where λ j , 1 ⩽ j ⩽ y N = N q 1 −
[Equation]
[Equation]
−
[Equation] [Equation]
p } , 1 − γ
[Equation]
1 − γ } are regularization coefficients that we impose on different dimensions of the output space.
In this section, we consider the following two ways to select regularization coefficients in Section 4:
• We regularize all spectral component below certain variance contour,
i
e. we set regularization strength λ j
[Equation]
max
[Equation]
• We regularize all spectral component below certain bias contour, i.e. we set regularization strength
λ j = max
[Equation]
To obtain the upper bound for our estimator, we decompose the learning error E ( ˆ A ) =
(cid:13)(cid:13)(cid:13) ˆ A − A 0 (cid:13)(cid:13)(cid:13) β
[Equation]
,γ ′ in to bias and variance via
[Equation]
where
[Equation]
B.1 Regularization via Variance Counter
In the following, we separately bound the bias term and the variance term.
We first assume α ⩽ β + p in Appendix B.1.1 and Appendix B.1.2, then the case α > β + p is treated in Appendix B.1.3.
Finally in Appendix B.2, we establish the same convergence rate for regularization via bias contour.
B.1.1 Bias
Lemma B.
[Equation]
Proof sketch : Since ∥A 0 ∥ β,γ ⩽ B , we can write A 0 :=
(cid:80) + ∞ i =1 (cid:80) + ∞
j =1 a ij µ β
i ρ 1 − γ
j f j ⊗ e i where the coefficient matrix A
= ( a ij ) 1 ⩽
i,j ⩽ + ∞ satisfies ∥
A 0 ∥ 2 F ⩽ B 2 .
The definition (7) implies that for 1 ⩽ j ⩽ y N
and i ⩾
we have
[Equation]
{++ β 2 i µ i + λ j ρ 1 − γ 
+}
j a ij .
The bias term can be bounded as follows:
[Equation]
We now prove that
[Equation]
Case 1.
If λ j = c 0 (cid:16) N log N (cid:17) 1 α , then
[Equation]
where we use α ⩽ β + p in the final step.
Case 2.
If λ j = [Equation]
N max
[Equation]
+ p , we need to consider two sub-cases: • If β ′ + p β + p > 1 − γ ′ 1 − γ , then we have λ j = (cid:16) N β ′ + p β
+ p j − 1 − γ ′ q (cid:17) 1 β ′ + p and thus
[Equation]
• If β ′ + p β + p < 1 − γ ′ 1 − γ , then similarly we have λ j = (cid:16) N 1 − γ
[Equation]
γ
[Equation]
γ
[Equation]
q (cid:17) 1 β ′ + p and
[Equation]
Hence
[Equation]
in all cases (9) holds and we have that
[Equation]
□
B.1.2 Variance
The variance term can be rewritten in the following way:
[Equation]
In (11), (11a) uses the definition of the Hilbert-Schmidt norm; (11b) follows from the definition of ˆ A (cf.(2)) and the fact that for any j ⩾ y N , we have (cid:68) ρ 12 j f j , (cid:16) ˆ A − A λ (cid:17) µ 12 i e i (cid:69) = 0 ; (11c) is obtained from re-arranging and (11d) follows from (cid:13)(cid:13)(cid:13)(cid:13)(cid:80) + ∞ i =
1 µ 2 − β ′ i µ i
+ λ j e i ⊗ e i (cid:13)(cid:13)(cid:13)(cid:13) =
max i ⩾ 1 µ 1 − β ′ i µ i + λ j ≲ λ − β ′
j and ρ j ≲ j − 1 q .
Note that
[Equation]
The U 1 j term is the variance of observational noise and U 2 j term is the variance of regularized bias.
Thus the U 1 j term is the dominating term.
Plugging the above decomposition into (11), we deduce that V ⩽ 2 ( V 1 + V 2 ) where
[Equation]
where ˆ E [ X ] = 1 N (cid:80) Nk
=1 X k denotes the empirical mean.
Define the event
[Equation]
Recall that m N ⩽ c 0 (cid:16) N log N (cid:17) pα , by Theorem D.3, we know that E 1 ,j holds with probability ⩾ 1 − 2 e − a 1 .
As a result E 1 = ∩ n N j =1 E 1 ,j holds with probability ⩾ 1 − 2 n N e − a 1 .
We assume event Eholds in all the following proof.
Bounding V 1 .
Let
[Equation]
and X k = ( X j,k : 1 ⩽ j ⩽ n N ) ∈
H y N K .
Then we have V 1 ≲ (cid:13)(cid:13)(cid:13) 1 N (cid:80)
Nk
=1 X k (cid:13)(cid:13)(cid:13) 2 where the norm here defined for H ⊗ y N K is induced by ⟨ a, b ⟩ = (cid:80) n N i =1 ⟨ a i , b i ⟩ H K .
Note that X k , k = 1 , 2 , · · · , N are i.i.d. random variables with mean zero, and
E ∥
X 1 ∥ 2 t = E P KL     n N (cid:88) j = ∥
X
[Equation]
k ∥
[Equation]  
[Equation]

 =
E P KL 
[Equation]

[Equation]
 ⩽ max 1 ⩽ j ⩽ [Equation] N [Equation]
)   j 1 − γ ′ [Equation] ′ p j (cid:13)(cid:13)(cid:13) ( C
[Equation]
j
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:124) (cid:123)(cid:122) (cid:125)
=: G  t −·
E ( u,v ) ∼ P KL 
[Equation]
  [Equation] (cid:69)
[Equation]
(cid:13)(cid:13)(cid:13) ( C KK [Equation] λ j I ) [Equation]
12 u (cid:13)(cid:13)(cid:13)    (cid:124) (cid:123)(cid:122) (cid:125) =: G
By Lemma D.2 we have
G [Equation]
For G 2 , note that for fixed u , Assumption 2.4 implies that
E v | u 
[Equation]
 
[Equation]
j 1 − γ [Equation] q i β ′ p j
[Equation] v − [Equation] ρ 12 j f j (cid:69) 2 (cid:13)(cid:13)(cid:13) ( C KK + λ j I ) − 12 u (cid:13)(cid:13)(cid:13)  
 ⩽2(2 t )!
R 2 t −n N (cid:88)
j = σ 2 j j 1 − γ ′
q λ − β ′ j
(cid:13)(cid:13)(cid:13) ( C KK + λ j I ) − 12 u (cid:13)(cid:13)(cid:13) 2 .
where σ 2 j = (cid:68) ρ 12 j f j , V ρ 12 f j (cid:69) .
As a result, we have
G
[Equation]
where in the second step we use (cid:80) + ∞ j =1 σ 2 j = tr ( V ) =
σ 2 and
[Equation]
We have shown that for some constant c 1 > 0 ,
[Equation]
By Bernstein’s inequality, the event
E
[Equation]
holds with probability ⩾ 1 − 2 e − a 2 .
By our definition of λ j , we have
[Equation]
and λ j ≳ N − 1 α (which implies that the 1 N 2 term is dominated by the 1 N term).
Hence, under E 1 ∩ Ewe have
[Equation]
with probability ⩾ 1 − 2 n N e − a 2 .
Bounding V 2 .
For any j ∈ Z + we have
[Equation]
14a
[Equation]
) where (14a) follows from E P K u ⊗ u = C KK , (14b) uses the fact that C KK and C KK + λ j I commute, and lastly (14c) follows from (cid:13)(cid:13)(cid:13)
[Equation]
KK
[Equation]
Let
[Equation]
and
Y
[Equation]
Then we have
[Equation]
Note that Y k , k = 1 , 2 , · · · , N are i.i.d.
random variables, and
E ∥
[Equation]
= E    
[Equation]
(cid:88)
[Equation]


[Equation]
 =
E
[Equation]
K    
[Equation] (cid:88) [Equation] u
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]


[Equation]

 ⩽ [Equation]
 
[Equation]
( C
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]


[Equation]
n
[Equation]
(cid:88)
j = j
[Equation]
λ
[Equation]
For any j ∈ Z + and u ∈ supp( P K ) we have n N (cid:88)
[Equation]
(16c) where (16a) uses the AM-GM inequality, (16b) follows from the assumption that ∥A 0 u ∥ ⩽ A 2 is uniformly bounded
[Equation]
and that
[Equation]
by Assumption 2.
[Equation]
and lastly (16c) follows from ∥A
∥ β,γ ⩽ B .
Plugging into (15), we deduce that
E ∥
Y 1 ∥ 2 t
[Equation]
≲
[Equation]
where the last step follows from (cid:80) + ∞
j =1 D j, 2 = ∥A
∥ 2 β,γ .
By Bernstein’s inequality, there exists a constant C 3 such that the event
E 3 =
[Equation]
holds with probability ⩾ 1 − 2 e − a 3 .
The definition of λ N implies that the 1 N 2 term is dominated by the 1 N term, so
[Equation]
holds under E 1 ∩ E 3 .
To summarize, under E 1 ∩ E 2 ∩ E 3 which holds with probability ⩾ 1 − 2 n N e − a 1 − 2 e − a 2 − 2 e − a 3 , we have
[Equation]
Recall that the bias term is upper bounded in (10).
This gives the final upper bound
[Equation]
B.1.3 The hard-learning regime
In the previous sections, we focus on the case where α ⩽ β + p and establish an upper bound for the convergence rate via an optimal bias-variance trade-off.
The opposite case, α > β + p is referred to as the hard-learning regime, for which the optimal rate is not known for several decades even in the case of γ = 1 (cf.
the discussion following [19, Theorem 2]).
In the hard learning regime the Vterm becomes the leading terms.
In this section, we use the technique developed in previous sections to obtain an upper bound in the hard-learning regime.
To do this, we need to re-define the truncation set S N as follows:
[Equation]
The definition implies that the variance can be controlled by N
[Equation]
and it remains to focus on the bias term.
Similar to the derivations in Appendix B.1.1, we have
[Equation]
The maximum value of the right hand side can be achieved in either of the following two cases:
• i
= O (1) .
Then we have j ≳ N q 1 − γ ′ (cid:16) 1 − min (cid:110)
β − β ′ α , γ ′− γ 1 − γ (cid:111)(cid:17)
so that
[Equation]
• j = O (1) .
In this case we must have i ≲ N min (cid:110) pα , p β
[Equation]
β ′ γ ′− γ 1 − γ (cid:111) , otherwise it falls into S N by definition.
Hence we have
[Equation]
On the other hand, for the variance term we still have V 1 ≲ 1 N max 1 ⩽ j ⩽ [Equation]
q i
[Equation]
+
[Equation]
j and V 2 ⩽
[Equation]
, so that
[Equation]
As a result
[Equation]
we can obtain the following convergence rate:
[Equation]
B.2 Regularization via bias contour
In this subsection [Equation] we analyze the convergence rate of regularization via bias contour (cf.
Figure 2).
Specifically, we consider the estimator (2) with the choice
[Equation]
It now remains to plug the above λ j into our bounds for bias and variance derived in the previous subsections.
Bounding the bias term.
It follows from (8) that
[Equation]
Bounding the variance term It follows from (13) and (17) that the variance is bounded by
[Equation]
j . As before [Equation] we consider the cases α ⩽ β + p
and α > β + p separately.
If α ⩽ β + p , then it follows that
[Equation]
where we use y γ ′− γ q N
[Equation]
by definition.
• If α > β + p , then similarly we have
[Equation]
Hence we deduce that
[Equation]
as desired.
B.3 Implication of the upper bound
In this section, we discuss the implications of our upper bounds under the ( β ′ , γ ′ ) -norm.
Note that (cid:13)(cid:13)(cid:13)(cid:13) [Equation]
for all v ∈ L [Equation]
( Q K ) (if one side of the equation is + ∞ then so is the other), we have that
[Equation]
where the last step follows from E u ∼ P K u ⊗ u = C P K .
Note that the above derivations hold for any 0 ⩽ β ′ <
β , so choosing β ′ =
yields the best upper bound.
We can see from (19) that our analysis implies an upper bound of the expected error of the learned solution evaluated under the H {+
+} − γ ′ L norm.
On the other hand, it is also possible to obtain a uniform convergence rate when β ′ ⩾ α :
[Equation]
C Proofs for the multi-level operator learning algorithm
In this section, we analyze the convergence rate of our multi-level algorithm described in Section 5.
We define η 1 = min (cid:110) β − β ′ max { α,β + p } , γ ′ − γ 1 − γ (cid:111) and η 2 = max (cid:110) 1 − β − β ′ max { α,β + p } , 1 − γ ′ 1 − γ (cid:111)
=
1 − η 1 .
We first restrict ourselves to the case when β − β ′ max { α,β + p } ̸ = γ [Equation] − γ 1 − γ ; the special case when the two terms are equal will be separately treated in Appendix C.1.
For the optimal bias and variance contours ℓ C 1 , bias and ℓ C 2 , var with C 1 = N η 1 and C 2 = N η 2 , we define a sequence { x n } as follows:
[Equation]
of
[Equation]
the
[Equation]
We first derive an explicit recursive formula for { x n } . Lemma C.1 Let u = β ′ +max { α − β,p }
[Equation]
− β ′
[Equation]
, then (1).
if u > 1 , then
[Equation]
if u < 1 , then x
[Equation]
Proof : (1).
Suppose that u > 1 , then we have η 1 = β − β ′ max { α,β + p } and η 2 = 1 − η 1 .
It follows from (20b) and (20c) that
[Equation]
Suppose that u < 1 , then we have η 1 = γ ′ − γ 1 − γ and η 2 = 1 − γ ′ 1 − γ , so that η 1 η 2 = γ ′ − γ 1 − γ ′ , and it follows from (20b) and (20c) that x β
[Equation]
x
[Equation]
+
[Equation]
= x un .
□
Lemma C.1 implies that when u ̸ = 1 , the sequence { x n } decreases super-exponentially.
Thus, there exists L N =
O (log log N ) such that x n ⩽ 2 for all n ⩾ L N .
Let λ ( K ) i
= x − 1 p i and λ ( L ) i =
y − 1 q i , then we construct the following estimator:
[Equation] [Equation]
where y − 1 :=
Note that each summand in the above equation is essentially a regularized leastsquares estimator and learns a rectangular region.
The following theorem states that the estimator ˆ A ml can achieve minimax optimal convergence rate.
Theorem C.1 Consider the estimator ˆ A ml defined by (4) .
Suppose that Assumptions 2.1 to 2.5 hold, then there exists a universal constant C , such that
[Equation]
holds with probability ⩾ 1 − e − τ .
Proof : The proof of Theorem 5.1 is similar to that of Theorems 4.1 and 4.2.
We consider the bias-variance decomposition of the estimation error
[Equation]
where
[Equation]
Bounding the bias term.
Since ∥A 0 ∥ β,γ ⩽ B , we can write
[Equation]
where the coefficient matrix A
= ( a ij [Equation]
i,j ⩽ + ∞ satisfies ∥
A 0 ∥ 2 F ⩽ B 2 .
We fix ( i, j
) ∈
Z 2+ and assume WLOG that y m j − 1 ⩽ j < y m j for some m ⩾ 0 , where y L N +1 = + ∞ .
It follows from (22) that
[Equation]
Thus
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] where we recall that η 1 = min (cid:110)
[Equation] and the last step follows from (20c).
Bounding the variance term.
The variance term can be rewritten in the following way:
[Equation]
for reasons similar to (11).
It now remains to bound (cid:13) (cid:13) G m j (cid:13) (cid:13) and (cid:13)(cid:13) (cid:13) U m j ρ 12 j f j (cid:13)(cid:13) (cid:13) for 1 ⩽ j ⩽ L N .
Note that these quantities have already been bounded in Appendix B.1.2 with λ m j replaced with λ j (there we use a different regularization for each j ).
Hence, those bounds can be directly applied here, so there exists a constant C > 0 such that
[Equation]
with probability ⩾ 1 − Ne − a .
Since j ⩽ y m j , by (20b) we have
[Equation]
Hence
[Equation]
Combining the bias and variance bounds, the conclusion directly follows.
□
C.
Special case:
[Equation]
Note that Lemma C.1 does not cover the case u = 1 , or equivalently
[Equation]
This case is special since the bias contour coincides with the variance contour, and we need to modify our construction of the multilevel estimator.
We define two sequences { x n } , { y n } as follows:
[Equation]
the solution of
[Equation]
where we recall that η 1 = β − β ′ max { α,β + p } = γ ′ − γ 1 − γ .
In this case, there exists L N =
O (ln N ) such that x n < 1 for all n ⩾ L N .
Let λ ( K ) i
= x − 1 p i , then we construct the following estimator:
[Equation]
Similar to Theorem C.1, we can establish the following result:
Theorem C.2 Consider the estimator ˆ A ml defined by (25) .
Suppose that Assumptions 2.1 to 2.5 hold, then there exists a universal constant C , such that
[Equation]
holds with probability ⩾ 1 − e − τ .
Proof : The proof of Theorem C.2 is similar to that of Theorems 4.1 and 4.2.
We consider the bias-variance decomposition
[Equation]
where
[Equation]
[Equation]
as defined in (25).
Bounding the bias term.
Let A 0 :
= (cid:80) + ∞
i =1 (cid:80) + ∞
j =1 a ij µ β
i ρ 1 − γ
j f j ⊗ e i with coefficient matrix
A {+
 = (+} a ij ) + ∞ i,j =1 such that ∥ A 0 ∥ 2 F ⩽ B 2 .
We fix ( i, j
) ∈
Z 2+ and assume WLOG that y m j − 1 ⩽ j < y m j for some m j ⩾ 0 , where y L N +1 = + ∞ .
It follows from (26) that
[Equation]
Thus we can proceed as in (23) to deduce that
[Equation]
The definition (24) implies that
[Equation]
On the other hand, since x L N < 1 , by (24) implies that y − γ ′− γ q L N ≲ N − η 1 .
Therefore, for the bias term (cid:13)(cid:13)
A
[Equation] − A λ ml (cid:13)(cid:13) 2 β ′ , [Equation] ′ ≲ N − η 1 .
Bounding the variance term.
Repeating the arguments in (24), we can deduce that there exists a constant C > 0 such that
[Equation]
with probability ⩾ 1 − Ne − a .
Combining the bias and variance bounds, we arrive at the desired conclusion.
□
The conclusion of Theorem 5.1 then follows from Theorems C.1 and C.2.
D Auxiliary results
Lemma D.1 We have ∥ T ∥ β,
[Equation]
Proof : We recall from the definition that ∥ T ∥ β,γ =
[Equation]
I
[Equation]
,Q L ) † ◦ T ◦ I ∗ 1 β,P K (cid:13)(cid:13)(cid:13) HS( H βK , H γL ) , so that [Equation]
as desired.
□
Lemma D.2 Under Assumption 2.2, we have
[Equation]
Proof : By Assumption 2.2 we have [Equation] C
[Equation]
as desired.
□
D.1 Concentration inequalities
Theorem D.1 [19, Theorem 27] Let (Ω , B , P ) be a probability space, H be a separable Hilbert space and X : Ω → HS( H ; H ) be a random variable with self-adjoint values.
Furthermore, assume that ∥ X ∥ F ⩽ B, P − a.s.
and V be a positive semi-definite matrix with E P (cid:0)
X 2 (cid:1) ≼ V , i.e.
V − E P (cid:0) X 2 (cid:1) is positive semi-definite.
Then, for g ( V ) := log (cid:0) 2 e tr( V ) ∥ V ∥ − 1 (cid:1) , τ ⩾ 1 , and n ⩾ 1 , the following concentration inequality is satisfied
P n
[Equation]
Theorem D.2 [19, Theorem 26] Let (Ω , B , P ) be a probability space, H be a separable Hilbert space, and ξ : Ω → H be a random variable with
[Equation]
for all m ⩾ 2 .
Then, for τ ⩾ 1 and n ⩾ 1 , the following concentration inequality is satisfied P n
[Equation]
The following theorem shows that the regularized covariance C KK + λI can be estimated with small error when λ is above a certain threshold.
Although it is well-known [19, 21], we still recall it below for completeness.
Theorem D.3 Recall that C KK = E P K u ⊗ u and ˆ C KK = 1 N (cid:80) Ni =1 u i ⊗ u i where u i i .
i .
d . ∼ P K .
Suppose that Assumption 2.2 holds and N ≳
A 21 τg λ λ − α , where g λ = log (cid:16) 2 e N P K ( λ ) ∥C KK ∥ + λ ∥C KK ∥ (cid:17) and N P K ( λ ) = tr (cid:0) ( C KK + λI ) − 1 C KK (cid:1) is the effective dimension, then with probability at least 1 − e − τ , we have
[Equation]
Proof : Let X ( u ) =
( C KK + λI ) − 12 u ⊗ u ( C KK + λI ) − 12 where u ∈ H K , then the LHS of (27) can be expressed as (cid:13)(cid:13)(cid:13) 1 N (cid:80)
Ni =1 X ( u i ) − E u ∼ P K X ( u ) (cid:13)(cid:13)(cid:13) .
We hope to apply Theorem D.1 and start with verifying the assumptions.
Since E P K X = ( C KK + λI ) − 12 C KK ( C KK + λI ) − 12 and ∥ X ( u ) ∥ =
∥ X ( u ) ∥
F = (cid:13)(cid:13)(cid:13) ( C KK + λI ) − 12 u (cid:13)(cid:13)(cid:13) 2 ⩽
A 21 (cid:13)(cid:13)(cid:13) ( C KK + λI ) − 1 − α 2 (cid:13)(cid:13)(cid:13) 2 ≲
A 21 λ − α , so that there exists V =
O (cid:16) λ − α
( C KK + λI ) − 12 C KK ( C KK + λI )
− 12 (cid:17) such that E P K X 2 ≼ V .
It’s easy to see that ∥ V ∥ ≲ λ − α and tr( V ) ≲ N P K ( λ ) .
The conclusion then follows from Theorem D.1 with
B =
O ( λ − α ) and g ( V ) =
g λ .
□
Corollary D.1 Under the notations and assumptions of Theorem D.3, there exists a constant C 1 >with probability ⩾ 1 − e − τ we have (cid:13)(cid:13)(cid:13)(cid:13)
( C KK + λI ) 12 (cid:16) ˆ C KK + λI (cid:17) − 1 ( C KK + λI ) 12 (cid:13)(cid:13)(cid:13)(cid:13) ⩽ C
[Equation]
Proof : By Theorem D.3 we have
[Equation]
⩽ [Equation] with probability ⩾ 1 − e − λ , as desired.
□