Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descent
Abstract
In this paper, we study the statistical limits in terms of Sobolev norms of gradient descent for solving inverse problems from randomly sampled noisy observations using a general class of objective functions.
Our class of objective functions includes Sobolev training for kernel regression, Deep Ritz Methods (DRM), and
Physics Informed Neural Networks (PINN) for solving elliptic partial differential equations (PDEs) as special cases.
We consider a potentially infinite-dimensional parameterization of our model using a suitable Reproducing Kernel Hilbert Space and a continuous parameterization of problem hardness through the definition of kernel integral operators.
We prove that gradient descent over this objective function can also achieve statistical optimality and the optimal number of passes over the data increases with sample size.
Based on our theory, we explain an implicit acceleration of using a Sobolev norm as the objective function for training, inferring that the optimal number of epochs of DRM becomes larger than the number of PINN when both the data size and the hardness of tasks increase, although both DRM and PINN can achieve statistical optimality.
Introduction
Several learning based methods for solving inverse problems have been proposed recently with state-of-the-art performance across a wide range of tasks, including medical image reconstruction [1], inverse scattering [2] and 3D reconstruction [3].
In this paper, we study the statistical limit of machine learning methods for solving inverse problems.
To be specific, we consider the problem of reconstructing a function from random sampled observations with statistical noise in measurements.
We apply gradient descent to a general class of objective functions for the reconstruction.
When the observations are the direct observations of the function, the problem is non-parametric function estimation [4, 5].
The observations may also come from certain physical laws described by a partial differential equation (PDE)[6, 7].
Formally, we aim to reconstruct a function f ∗ based on independently sampled data set D = { ( x i , y i ) } ni =1 from an unknown distribution P on X ×Y , where y i is the noisy measurement of u ∗ though a measurement procedure A .
For simplicity, we assume A is self-adjoint in this paper.
The conditional mean function f ∗ ( x ) =
E P ( Y | X = x ) is the ground truth function for observation of u ∗ through the measurement procedure A , i.e. f ∗ =
A u ∗ .
To solve this problem, we consider gradient descending over the following general class of objective function
[Equation]
where P n = 1 n (cid:80) n i =1 δ ( x i , y i ) is the empirical distribution, H is a reproducing kernel Hilbert space (RKHS) and
A i , i = 1 , 2 are two self-adjoint operators that satisfy
A 1 = AA 2 .
In Section 2, we show that several algorithms, including kernel regression [4, 8] via Sobolev training [9, 10, 11] and solving PDEs via machine learning based algorithm, [12, 13, 2, 14] can be considered as special cases of this formulation.
Recent works [15, 16] have considered the statistical limit of learning of elliptic inverse problem, i.e. how many observation of the right hand side function of an elliptic PDE are needed to reach a prescribed performance level.
However, none of these papers consider computationally feasible methods for constructing such optimal estimators.
In this paper, we consider the statistical optimality of gradient descent [17, 18, 19, 20], a successful and widely used algorithm in machine learning.
We show that proper early stopped gradient descent can achieve information theoretical optimal convergence rate according to a continuous scale of suitable Hilbert norm ( i.e. Sobolev norms[21, 22], detailed definition see Section 2).
We first prove that a properly early stopped gradient descent algorithm over the class of objective functions can achieve statistical optimality.
At the same time, although any suitably early stopped gradient flow of the class of loss function can achieve statistical optimality according to our theory, we discover an acceleration effect of using a Sobolev norm as the loss function for kernel based machine learning algorithms.
The implicit acceleration of the Sobolev loss function arises because a differential operator can enlarge the small eigenvalue of the kernel integral operator for high frequency functions, leading to better condition numbers and faster convergence in these eigenspaces while maintaining the statistical optimality.
We justify our theoretical finding with several numerical experiments.
1.1 Related Works
Machine Learning Based PDE Solver.
Partial differential equations (PDEs) are widely used in many disciplines of science and engineering and play a prominent role in modeling and forecasting the dynamics of multiphysics and multiscale systems.
The recent breakthroughs in deep learning and the rapid development of sensors, computational power, and data storage in the past decade have drawn attention to numerically solving PDEs via machine learning methods [23, 24, 12, 25, 13, 2], especially in high dimensions where conventional methods become impractical.
Based on the natural idea of representing solutions of PDEs by (deep) neural networks, different loss functions for solving
PDEs are proposed.
[25, 26] utilize the Feynman-Kac formulation which turns solving a PDE into a stochastic control problem.
The work of [27] solves the weak formulations of PDEs via an adversarial network.
In this paper, we focus on the convergence rate of the Deep Ritz Method (DRM) [14, 2] and the Physics-Informed neural network (PINN) [12, 13].
DRM [14, 2] utilizes the variational structure of the PDE, which is similar to the Ritz-Galerkin method in classical numerical analysis of PDEs, and trains a neural network to minimize the variational objective.
PINN [12, 13] trains a neural network directly to minimize the residual of the PDE, i.e., using the strong form of the PDE.
Theoretical convergence results for deep learning based PDE solvers has also received considerable attention recently.
Specifically, [28, 29, 30, 31, 32, 33, 34] investigated the regularity of PDEs approximated by a neural network and [28, 35, 36, 37, 38] further provided generalization analyses.
[15, 16, 39, 40] provided information theoretical optimal lower and upper bounds for solving PDEs from random samples.
However, all these papers assume accessibility of the global solution of empirical loss minimization.
In contrast, here we consider the gradient descent algorithm for learning the estimator.
The most relevant work in connection to is [41], which considers a polynomial-time Langevintype algorithms to sample from the posterior measure of the Bayesian inverse methods.
Instead of considering the Bayesian setting, here we optimize on the un-regularized objective.
However, the estimator is regularized via early stopping [42, 43, 44], i.e. we consider the statistical optimality of the implicit regularization effect of optimization algorithm.
A concurrent paper [45] considered similar stochastic gradient descent approach for statistical inverse problem.
Learning with kernel.
Supervised least square regression in RKHS has a long history and its generalization ability and mini-max optimality has been thoroughly studied [8, 46, 4, 47, 48].
Statistical optimality of early stopped (stochastic) gradient descent has been widely discussed in [42, 49, 50, 18, 17, 51, 52].
The convergence of least square regression in Sobolev norm has been discussed recently in [21, 22].
Recently, training neural networks with stochastic gradient descent in certain regimes has been found to be equivalent to kernel regression [53, 54, 55].
Gradient descent training of neural network in the kernel regime has been found optimal for a wide class of non-parametric functions with both early stopping regularization and ridge regression [56, 57].
1.2 Contribution
• We provide information theoretical lower bounds (Theorem 3.1) for a wide class of inverse problems, including the Sobolev learning rate [21] for the solution of elliptic inverse problems.
We also show that the previous lower bound [15, 16] for machine learning solving elliptic equations can be considered as a special case of our lower bound.
• We provide a proof of statistical optimality of the gradient descent algorithm of a general class of objective functions (Theorem 3.2), including PINN [12, 13] and Deep Ritz Methods [14, 2] for solving PDEs as well as Sobolev training [11, 9, 58] of kernel methods.
We provide [16] a computational feasible estimator and generalize the previous statistical optimality results of gradient descent [42, 18, 19] to general Sobolev norm.
• We also characterize the acceleration effect of Sobolev loss function for learning with kernel.
The acceleration happens because differential operator can enlarge the small eigenvalues for high frequency functions, leading to better condition number and faster convergence in these eigenspaces while keeping the statistical optimality.
Thus when the target function have more high frequency component, the lead of PINN will become larger (Figure 3).
We justify our theoretical finding with several numerical experiments (Figure 2 and Figure 4)
Problem Formulation
In this section, we formulate the problem of learning inverse problem using the kernelized gradient descent.
As described previously, we aim to reconstruct a function f ∗ ∈ R X from random observations of u ∗ =
A f ∗ , where A is an observation process which is modeled by an operator maps from R X to R X .
To solve this problem, we write the operator A in terms of two operators
A i ( i = 1 , 2) with A 1 = AA 2 and build our objective function as
[Equation]
where P is the joint distribution of x and y with x sampled from the uniform distribution on X for simplicity and y as the noisy observation of f ( x )
=
( A u )( x ) .
In other words, E ( y | x ) = f
( x ) .
The minimizer of objective function (1) is the ground truth function u ∗ =
A − 1 f that we are interested in.
Learning with Kernel Consider the case that u is parameterized by a Reproducing Kernel Hilbert
Space u θ ( x ) = ⟨ θ, K x ⟩ (we provide standard notations of RKHS in Appendix A).
At the same time, the kernel function has the following representation K ( s, t )
= (cid:80) ∞
i =
λ i e i ( s ) e j ( t ) , where e i are orthogonal basis of L 2 ( ρ X ) with ρ X being the uniform distribution over X , where L 2 denotes the space of all the square integrable functions.
Then e i is also the eigenvector of the covariance operator Σ
= E x ∼ P K x ⊗ K x with eigenvalue
λ i > 0 , i.e. Σ e i
= λ i e i .
Here g ⊗ h = gh ⊤ is an operator from H to H defined as g ⊗ h : f → ⟨ f, h ⟩ H g. The covariance matrix Σ is the core of the integral operator technique [46, 8] for kernel regression.
For any f ∈ H , the reproducing property gives (Σ f )( z ) =
⟨ K z , Σ f ⟩ H =
E [ f ( X ) k
( X, z )]
=
E [ f ( X ) K x ( X )] .
If we consider the mapping S : H → L 2 ( dx ) defined as a parameterization of a vast class of functions in R X via H through the mapping ( Sg )( x ) = ⟨ g, K x ⟩ ( Φ( x ) =
K x = K ( · , x ) ).
Its adjoint operator
S ∗ : L 2 → H then can be defined as g → (cid:82) X g ( x ) K x
ρ X ( dx ) .
Σ is the same as the self-adjoint operator S ∗ S and the self-adjoint operator L = SS ∗ : L 2 ( dx )
→ L 2 ( dx ) can be defined as ( L f )( x ) = (cid:82) X K ( x, z ) f ( z ) ρ X ( dz ) .
Based on this notation, we present all our assumptions on the underlying kernel.
Assumption 2.1 (Assumptions on Kernel) .
We assume the standard capacity condition on kernel covariance operator with a source condition about the regularity of the target function following [8].
We further assume a regularity condition for our kernel k ( · , · ) via a ℓ ∞ embedding property follows [59, 60, 18, 21].
These conditions are stated explicitly below.
• (a) Standard assumptions.
The kernel feature are bounded almost surely, i e. | k ( x, y ) | ≤ R and the observation y is also bounded by M almost surely.
• (b) Capacity condition.
Consider the spectral representation of the kernel covariance operator σ = (cid:80)
λ i e i ⊗ e i , we assume polynomial decay of eigenvalues of the covariance matrix
λ
i ∝ i − α for some α > 1 .
As a result Q = tr (Σ 1 /α ) < ∞ .
• (c) Source condition.
We also impose an assumption on the smoothness of the true function.
There exists β ∈ (0 , 1] such that u ∗ =
L β/ 2 ϕ for some ϕ ∈ L 2 .
If u ∗ ( x ) = ⟨ θ ∗ , K x ⟩ H , the source condition can also be written as
[Equation]
• (d) Capacity conditions on A i .
For theoretical simplicity, we assume that the self-adjoint operators
A i are diagonalizable in the same orthonormal basis e i .
Thus we can assume
A
[Equation]
e
[Equation]
for positive constants p i , q i > 0 .
We further assume p i ∝ i − p and q
i ∝ i − q .
This commuting assumptions also made in [61, 62].
due to the Bochner’s theorem.
We further assume p < 0 , q < 0 , α + p > 0 .
We refer the detailed discussion to Remark 1.
• (e) Regularity results on RKHS.
For µ ∈ [0 , 1] , there exists κ µ ≥ 0 such that Φ( x ) ⊗ Φ( x ) ≤ k 2 µ R 2 µ Σ 1 − µ holds almost surely.
The regularity assumption here is equivalent to ∥ g ∥ 2 L ∞ ≤ κ 2 µ R 2 µ ∥ Σ 1 / 2 − µ/ 2 g ∥ 2 H and implies ∥ g ∥ L ∞ ≤ κ µ R µ ∥ g ∥
µ H ∥ g ∥ 1 − µ L 2 for every g ∈ H .
As a consequence, we know that ∥ Σ µ/ 2 − 1 / 2 Φ( x ) ∥
H ≤ κ µ R µ holds almost surely.
Remark 1 .
To simplify the technical exposition, we assume that operator A
i ( i = 1 , 2) commute with the kernel covariance operator Σ .
This assumption is also made in [62, 61].
Here we provide several examples that satisfy this assumption.
The simplest case is A 1 =
A 2 = id , which gives rise to the function regression setting.
[63] assumes the operator
A i to be bounded operator in operator norm, which can be consider as a special case of ours.
At the same time, for numerically solving a PDE/elliptic inverse problem, we take A i to become the power of the Laplace operator ∆ , which contradicts with [63]’s assumption.
If the domain is a sphere, the eigen-functions are spherical harmonics which are also the eigen-functions of a wide class of kernels, examples includes the dot product kernels [64] and the Neural Tangent Kernel [65, 66], when the data distribution is uniform distribution.
When the domain is the torus, the eigen-functions are Fourier modes.
If we consider a shift invariant kernel K ( x, y )
=
ψ ( x − y ) , from Bochner’s Theorem K ( x, y )
= (cid:80) ni
=1 ˜ ψ ( w ) e iws e −
iwt
we know that the eigen-functions are also Fourier modes.
There are also works that use Green function as the kernel [67, 68], where the three operators will automatically commute with each other.
In this paper, we consider the convergence of the estimator in Sobolev norm class.
We define the different Sobolev spaces via the power space approaches used in [69, 21].
Definition 2.2 (Sobolev Norm) .
For γ > 0 , the γ -power space is
[Equation]
equipped with the γ -power norm via ∥ (cid:80) i ≥ 1 a
i λ γ/
i
[Equation]
It is obvious that ∥L γ/ 2 f ∥ γ = ∥
f ∥ L 2 and ∥ f ∥ γ ≤ ∥ Σ 1 − γ 2 f ∥ H [21].
The source condition can also be understood as the target function u ∗ lies in the β -power Sobolev space.
The regularity condition of the kernel function implies a continuously embedding from H γ → L ∞ .
Throughout this paper, we consider the convergence rate of ˆ u − u ∗ in γ -power Sobolev norm ( γ > 0 ).
2.1 Examples
Sobolev Training [70, 9, 11] introduce the idea of training using Sobolev spaces via matching not only the function value but also the derivative of the classifier.
Using different Sobolev norms as loss function has also been used widely in image processing, inverse problems, and graphics applications [71, 72, 10, 73, 74, 75].
The work of [71] discovered that different Sobolev loss functions would lead to different implicit bias and that the proper Sobolev preconditioned gradient descent can accelerate the optimization of geometry objectives [73, 74, 75].
In this paper, we discover that stochastic gradient descent over Sobolev norm loss class functions can achieve statistical optimal but proper selection of the Sobolev norm loss function can accelerate training.
We call this phenomenon Sobolev
Implicit Acceleration and discuss it in Section 4.
Machine Learning Based PDE Solver.
To simplify the exposition, we focus on a prototype elliptic PDE: Poisson’s equation on a torus, i.e. Ω = T d = [0 , 1] d per .
Our focus is on the analysis of deep-learning-based numerical methods for the elliptic equations
[Equation]
We mainly focus on analyzing Deep Ritz Method (DRM) [14] and Physics Informed Neural Network (PINN) [12, 13].
DRM solves the equation (2) via minimizing the following variational form
[Equation]
while PINNs solves the equation (4) via minimizing the following strong formula, i.e the residual of the PDE,
[Equation]
where u is minimized over a parameterized function class F (for example neural network).
Here we consider the function class to be the RKHS space [76, 77].
[16] showed that empirical risk minimization of both objectives can achieve information theoretical optimal bounds.
The objective function in 3 and 4 can be considered as special case of objective function (1).
For DRM, A 1 u = ∆ u and A 2 u = u for all function u ∈ R X .
For PINN, A 1 u = ∆ 2 u and A 2 u = ∆ u for all function u ∈ R X .
We discover that PINN convergences faster than DRM consistently due to the implicit Sobolev acceleration, matching the observation made in [78].
[61] considered semi-supervised learning using
Laplacian regularization with kernel parameterization.
However, this paper does not consider training with stochastic gradient descent and also does not introduce the source condition assumption that leads to different convergence rate for a hierarchical parameterization of task difficulty.
Main Theorem
We present our main results in this section, including an information theoretical lower bound and a matching upper bound with proper selected early stopping time.
3.1 Lower Bounds
This subsection investigates the statistical optimality of the Sobolev convergence rate of solving elliptic problem using stochastic gradient descent.
We provide the information theoretical lower bound of learning the elliptic problems.
Different from [15, 16], we formulate the problem in an
RKHS.
This leads to a different construction of hypothesis and show that [15, 16] is a special case of our lower bound using specific kernel and operator
A i ( i = 1 , 2) in Section 3.3.
Theorem 3.1 (Lower Bound) .
Let ( X, B ) be a measurable space, H be a separable RKHS on X with respect to a bounded and measurable kernel k and operator A
=
( A − 1 2 A 1 ) satisfies Assumption 2.1.
We have n i.i.d.
random observations { ( x i , y i ) ∈
X × Y} ni
=1 of f ∗ =
A u, u ∈ H γ ∩ L ∞ , i.e. y i =
f ∗ ( x i ) + η i where η i is a mean zero random noise satisfies the momentum assumption E | η | m ≤ 12 m !
σ 2 L m − 2 for some constants σ, L > 0 .
Then for all estimators H :
( X × Y ) ⊗ n → H γ satisfies
[Equation]
3.2 Upper Bounds
This subsection, we consider the (multiple pass) gradient descent over the empirical data of objective function (1).
We aim to construct our estimator via optimizing the empirical loss function (cid:80)
[Equation]
x i ) , where x i is sampled randomly and y i is the associated noisy observation introduced in Section 1.
We consider a parameterization u ( x ) = ⟨ u, K x ⟩ and
A i u ( x ) = [Equation]
i θ, K x ⟩ H = ⟨ θ, A i K x ⟩ H and express our empirical objective function as
[Equation]
Then the gradient descent algorithm can be written as the following procedure:
• Initialization : θ 0 =
¯ θ 0 = 0 , γ is a constant to be determined later which is used as the learning rate in the algorithm.
• Iteration : For the t − th iteration, we perform the following gradient descent step
[Equation]
with an averaging step ¯ θ [Equation]
Remark.
Note that the optimizing dynamics considered here is not the exact gradient descent dynamics over the empirical objective.
The gradient of the quadratic termn (cid:80) ni
=1 u ( x i )
A 1 u ( x i ) should be 1 n (cid:80) ni =1 ( ⟨ θ t − 1 , A 1 K x i
⟩ H K x
i + ⟨ θ t − 1 , K x i ⟩ H
A 1 K x i )
but we take instead 1 n (cid:80) ni
=1 ⟨ θ t − 1 , A 1 K x i ⟩ H K x
i in our dynamics.
In the population expectation, the two dynamics are the same due to the commuting assumption between the kernel integral operator and operator A 1 .
Without our modification, the statistical rate will become sub-optimal in some cases due to the fact that the variance in the empirical covariance matrix dominates the statistical rate.
This observation matches the reason behind the sub-optimality of the Deep Ritz Method discovered in [16].
The following theorem is the main result for upper bounds with the proof details given in the appendix.
Theorem 3.2.
Under Assumption 2.1, we have the following three regimes shown in Figure 1.
• For β > α +2 q − p − 1 α , if we take t = n and γ = n
[Equation]
1 , we obtain the following rate
[Equation]
• For α +2 q − p − 1 α ≤ β ≤ µα +2 q − p +
α , if we take t = n α + p βα +2( p − q )
+1 and γ a small enough constant, we obtain the following rate
[Equation]
• For β > µα +2 q − p +
α , if we take t = n α + p µα + p and γ a small enough constant, we obtain the following rate
[Equation]
which is not an optimal converging rate.
Sketch of the Proof.
We first rewrite the averaged gradient descent in a more compact formula as η 0 = 0 , η u = η u − 1 + γ ( A ⊤ 2 ˆ S ∗ n ˆ y − ˆΣ Id, A 1 η t − 1 )
where ˆ S n : H → R n is defined as ˆ S n g = 1 √ n ( g ( x 1 ) , · · · , g ( x n )) , ˆΣ O 1 , O 2 = 1 n (cid:80) ni
=
O 1 K x ⊗ O 2 K x
and Id is the identity operator.
For the error of GD, we consider early stopping of gradient descent algorithm as a spectral filtering [79, 18, 63, 19].
Our proof is based on standard bias-variance decomposition.
For t iteration, GD will behave similarly to ridge regression with γt regularization strength [42, 18] and this result in bias of ( 1 γt ) ( β − γ )
α α + p .
For the variance, we provide a bound which is related to the effective dimension given by tr ((Σ Id, A 1 + ( 1 γt ) I ) − 1 Σ
A ⊤ 2 A 2 ) and obtain a final variance of the form
[Equation]
α
[Equation]
p .
If we only have the first term of variance, we shall achieve information theoretical optimal bound when t = n α + p
[Equation]
+
[Equation]
( p −
[Equation]
+1 .
For the section term in the variance is from the convergence of empirical covariance matrix ˆΣ Id, Ato the population one Σ Id, A 1 .
This term can be reduced using semi-supervised learning techniques as in [80, 16].
3.3 Discussion and Implications of Our Theory
Relationship with [15, 16].
[15, 16] provided a lower bound of the form n − 2 α − 2 s
α − 4t + d for a 2 t − th order linear PDE ∆ t u = f with solution in H α , evaluated in H s norm.
We shall discuss the relationship between their bound with our n − ( β − γ ) α βα +2 (p-q) +1 lower bound based on the kernel representation of
Sobolev spaces.
The numerator ( β − γ ) matches the α − s term in [15, 16]’s lower bound and the q − p term is the order of the linear PDE which matches the t term in the denominator in [15, 16]’s lower bound.
The spectral decay speed of kernel α is always relative to the dimension d .
To understand this problem, we consider the following two examples.
For the first example, the kernel is defined on the torus T d = [0 , 1] d per .
We consider the space of square integrable functions on T d with mean 0 and the Matérn kernel K σ,l,v ( x, y )
=
σ 2 2 1 − v Γ( v ) (cid:16) | x − y | l (cid:17) v B v ( | x − y | l ) , where B v is the modified Bessel function of section kind.
The covariance operator is C θ = σ 2 ( − ∆ + τ 2 I ) − s with orthonormal eigenfunctions ϕ m ( x ) =
e 2 πi ⟨ m,x ⟩ and corresponding eigenvalues λ m =
σ 2 (4 π 2 | m | 2 + τ 2 ) − s for every m ∈ Z d \{ 0 } [81].
[Figure]
// For the second example, we consider the Mercer’s decomposition of a translation invariant kernel via Fourier series K ( s − t ) =
12 π (cid:80) w ˜ K
( w ) e iw ( s )
e iw ( − t ) dw .
The eigenfunctions of the translation invariant kernel is the Fourier modes and the eigenvalues are the Fourier coefficients.
As an example, for Neural Tangent Kernel, [82, 66, 65, 56] proved that the corresponding α = dd − 1 and the eigenfunctions are spherical harmonics that diagonalize the differential equation.
For the upper bound, [16] established the convergence rate based on the empirical process technique [48, 59], while our paper switches to the integral operator/inverse problem technique [4, 46, 8].
An advantage of the integral operator/inverse problem technique is that it can provide convergence results with respect to a continuous scale of Sobolev norms while the empirical process technique can only be used for the Sobolev norm equivalent to the objective function.
Relationship with [70] [70] also considered learning from data involving function value and gradients under the framework of least-square regularized regression in reproducing kernel Hilbert spaces.
In this paper, we only have access to the noisy observation of the function values but still aim to know about the convergence rate with respect to the Sobolev norm.
At the same time, we further consider an inverse problem setting with an early stopping regularization, which is not discussed in [70].
However, we introduce a commuting assumption over the differential operator with the kernel integral operator that facilitates our analysis.
Sobolev Implicit Acceleration Below we discuss the implication of the choice of early stopping time t = n
[Equation]
[Equation]
First of all, the best early stopping time here does not depend on γ , which means the best model in different Sobolev is the same over the stochastic gradient descent path asymptotically.
Secondly, all the components in an iteration step depend on the problem itself except the numerator α + p .
For differential operators, the p is actually negative (differential operators have large eigenvalues over high-frequency basis).
Thus we can accelerate the training via letting p more negative, i.e. using a higher order Sobolev norm as loss can lead to earlier stopping.
As an implication, the PINN achieves the statistical optimal solution faster than DRM.
Relationship with implicit bias of frequency Recent work credit the success of deep learning to the fast training in low frequency components [83, 84, 85].
However, in our work, with Sobolev preconditioning, the training speed of high frequency part increases, yet achieving statistical optimality in the class of Sobolev norm.
This suggests that the implicit bias of frequency is not necessary for good generalization results.
We also would like refer to [86, 87] Theorem 8 for the extreme case, where the authors directly invert the population covariance matrix which leads to the same training speed in every eigen-spaces while still maintaining the statistical optimality in ℓ 2 norm.
However the preconditioning matrix in [86] is the population Fisher information matrix, which requires further sampling of unlabeled data that is not accessible in our setting.
Discussion of the Sub-Optimal Regime In the sub-optimal regime, the concentration error between the empirical covariance matrix ˆΣ Id, A 1 and the population one Σ Id, A 1 dominates.
With the observation that these concentrations have no relationship with the supervision signal, [16, 80] proposed to utilize the semi-supervised learning to reduce the error in this regime.
In [16], Deep Ritz method requires semi-supervised learning while PINN does not for the exact empirical risk minimization solution.
In our formulation, if | p | is larger, the sub-optimal regime will become smaller, which contradict with the observation in [16].
However [16] only considers the statistical generalization bound but doesn’t take optimization into consideration.
We leave designing algorithm with smaller sub-optimal regime as future work.
Sobolev implicit acceleration
[Figure]
The Sobolev norm has already been proposed as loss function for training neural network [9] and solving PDEs [11, 58].
However, all these papers need a further gradient information of the supervision signal.
This does not fit the theoretical framework considered here and hence it is also not fair to compare their algorithms with methods without gradient supervision signal.
Thus in this section, we proposed an alternative objective that can perform Sobolev training without gradient supervision loss function.
The basic idea is to using an integration by parts
[Equation]
(6) which leads to an objective function without the gradient of the target function.
In this section, we shall show how this idea is applied to different machine learning examples.
4.1 Predicting a Toy Function on Torus
In this section, we conduct experiments to illustrate the Sobolev implicit acceleration for function regression.
Different from the Sobolev training [9], the objective that we are interested in does not involve the gradient of the target function.
As a result, we do not need to train a teacher network to provide the gradient supervision information as done in [9].
In the toy example, for simplicity we ignore the boundary terms introduced by the integral by part.
Here consider estimating a function on the torus, i.e. a periodic function.
We consider using (cid:82) λ ∥
u − f ∥ 2 + ∥∇ u ∥ 22 + 2∆ u · f + ∥∇ f ∥ 22 dx as our objective function.
The goal is to fit function y
=
(cid:80) di
=1 sin(2 πx i ) using Gaussian Kernel and a simple three layer feed-forward network with tanh activation function.
We randomly sampled 1000 data in 10 dimension as our dataset and run a gradient descent algorithm.
Figure 2 presents our convergence result of the validation error, where the Sobolev norm have shown an acceleration effect for training.
4.2 Solving Partial Differential Equations
[Figure]
We show the convergence result of PINN and Deep Ritz Method for smooth problem (cid:80) di
=1 sin(2 πx ) and harder problem (cid:80) di =1 sin(4 πx ) .
PINN convergence faster than
DRM for online stream input which also matches our theory and the empirical observation in [78].
The Sobolev Implicit Acceleration will becomes more significant for harder problem as our theory shows.
In this section, we conduct experiments to illustrate the Sobolev implicit acceleration for solving partial differential equation using PINNs [12, 58] in 3 dimensions.
The example is a simple Poisson equation (static schrödinger equation) on the torus ∆ u + u = f in T d = [0 , 1]
[Equation]
per .
We first compare the Physics Informed
Neural Network [12] and Deep Ritz Method [14, 2] with online random inputs.
To enforce the periodic boundary conditions, we add a penalty term L b = (cid:82) ( x,y ) ∈ [0 , 1] 2 ( u ( x, y, 0) − u ( x, y, 1))
+ ( u (0 , x, y ) − u (1 , x, y )) 
 + ( u ( x, 0 , y ) − u ( x, 1 , y ))
2 dxdy to match the periodic condition of the function value and another term L b,grad = (cid:82) ( x,y ) ∈ [0 , 1] 2 ( ∇ u ( x, y, 0) − ∇ u ( x, y, 1))
+( ∇ u (0 , x, y ) −∇ u (1 , x, y ))
+ ( ∇ u ( x, 0 , y ) − ∇ u ( x, 1 , y )) 2 dxdy to match the periodic condition of the function value.
We tested
PINN and Deep Ritz on both u ( x ) =
(cid:80) di
=1 sin(2 x i ) and u ( x ) = (cid:80) di =1 sin(4 x i ) .
We use the same experiment setting as [78] and keep the learning rate constantly to 1 e − 3 to match our theory.data points are randomly sampled in every batch.
The results are shown in Figure 3.
PINN converges faster than DRM consistently in terms of iteration number and the lead seems to become significant for more oscillatory problems.
To solve equation (7), we consider minimizing the following Sobolev norm objective function
[Equation]
[11, 58] also considered using Sobolev norms as the loss function.
[11] showed that the Sobolev norms exhibit an acceleration effect.
However, in our setting, we cannot have random samples of ∇ f .
To avoid information of ∇ f appearing in the objective function, we perform an integration by parts that leads to the following objective function
[Equation]
We conduct the Sobolev training with the objective function L pinn + λ L grad + λ 1 L b + λ L b,grad and compare it with PINN and DRM.
Following mostly the experiment setting in [78], we fixrandom samples as the dataset and run stochastic gradient descent with batchsize 50.
The result presented in Figure 4 show the Sobolev implicit acceleration, i.e., the gradient dynamic of higher order Sobolev norm convergence faster.
We do not scale the Sobolev training to online setting as under large batch size the Sobolev training consume too much memory at this point.
(a) Solution by Sobolev Training.
[Figure]
Iteration
R e l e t i v e
E
[Figure]
Conclusion and Discussion
In this paper, we consider the statistical optimality of gradient descent for solving elliptic inverse problem using a general class of objective functions.
Although we can achieve statistical optimality of gradient descent using all the objective functions with proper early stopping time, the early stopping iteration strategy for the optimal solution behaves differently as a function of the sample size.
For instance, we observed that PINN convergences faster than the DRM method.
Generally speaking, by using a higher order Sobolev norm as loss function, one can accelerate training.
The reason is that the differential operator can counteract the kernel integral operator, leading to better condition number for optimization.
We call this phenomena Sobolev implicit acceleration .
Although we have shown the Sobolev implicit acceleration on several simple examples, the ∆ s u term is hard to compute in high dimensions, scalable Sobolev training without gradient supervision in higher dimension remains as future work.
However, we believe that this direction is promising.
For example, we can use MIM method [88, 89] to accelerate the training.
It is also interesting to generalize our results beyond GD, for example to mirror descent [90] and accelerated gradient descent [91].
In this paper, we did not consider operators with continuous spectrum and it will be interesting to extend our results using the techniques in [92].
Due to technical issue, we have not considered the batch stochastic gradient descent.
It will be interesting to characterize the condition under which the stochastic noise in gradient does not degrade the optimal bounds that we obtain.
At the same time, we also want to investigate more complex nonlinear inverse problems as [93, 94] considered.
It is also interesting to consider inverse problem arising from integral equation where p > 0 .
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[N/A] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[TODO] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix.
The appendix is constructed as follows:
• In Appendix A, we introduce the basic notations of Reproducing Kernel Hilbert space and the associated kernel integral operator.
We also put a discussion of how differential operators and Sobolev spaces relates to the kernel setting we considered as a preliminary.
• In Appendix B.1, we consider the statistical optimality of the early stopped gradient descent algorithm.
We bound the difference of the gradient descent.
• In Appendix C, we provide our proof for the lower bound in Section 3.1 using the Fano method.
A Preliminaries and Notations
This section starts with an overview of reproducing kernel Hilbert space, including Mercer’s decomposition, the integral operator techniques[46, 4, 8, 21] and the relationship between RKHS and the
Sobolev space[95].
In order to fit the objective function we considered, we did a slight modification to the original integral operator technique[46, 4, 8].
A.1 Reproducing Kernel Hilbert Space
We consider a Hilbert space H with inner product ⟨ , ⟩ H is a separable Hilbert space of functions
H ⊂ R X .
We call this space a Reproducing Kernel Hilbert space if f ( x ) = ⟨ f, K x ⟩ H for all
K x ∈ H : t → K ( x, t ) , x ∈ X .
Now we consider a distribution ρ on X × Y
( Y ⊂ R ) and denote ρ X as the margin distribution of ρ on X .
We further assume E
[ K ( x, x )] < ∞ and E [ Y 2 ] < ∞ .
We define g ⊗ h = gh ⊤ is an operator from H to H defined as g ⊗ h : f → ⟨ f, h ⟩ H g.
At the same time, we knows that
[Equation]
holds for all f, g ∈ H .
The integral operator technique[46, 8] consider the covariance operator on the Hilbert space H defined as Σ = E ρ X K x ⊗ K x .
Then for all f ∈ H , using the reproducing property, we know that
[Equation]
[ f ( X ) K x ( X )] .
If we consider the mapping S : H → L 2 ( dx ) defined as a parameterization of a vast class of functions in R X via H through the mapping ( Sg )( x ) = ⟨ g, K x ⟩ ( Φ( x ) =
K x = K ( · , x ) ).
Its adjoint operator S ∗ then can be defined as S ∗ :
L 2 → H : g → (cid:82) X g ( x ) K x ρ X ( dx ) and at the same time Σ is the same as the self-adjoint operator S ∗ S and the self-adjoint operator L = SS ∗ : L 2 → L 2 can be defined as
[Equation]
Next we consider the eigen-decomposition of the integral operator L via Mecer’s Theorem.
There exists an orthonormal basis { ψ i } of L 2 ( X ) consisting of eigenfunctions of kernel integral operator L .
At the same time, the kernel function have the following representation K ( s, t )
= (cid:80) ∞
i =
λ i e i ( s ) e j ( t ) where e i are orthogonal basis of L 2 ( ρ X ) .
Then e i is also the eigenvector of the covariance operator Σ with eigenvalue λ i > 0 , i.e. Σ e i
= λ i e i .
B Proof of the Upper Bound
In this section, we consider the convergence of the gradient descent algorithm to the target function 1.
In particular, we consider the gradient descent as a special case of a wider class of spectral filter algorithms [49, 79, 18, 19].
In our inverse problem setting, the spectral filter is defined as the estimator of the following form for λ > 0 ,
[Equation]
where ˆ S n g = ( g ( x 1 ) , · , g ( x n )) (leads to ˆ S ∗ n maps from R n to H via ˆ S ∗ n ( a 1 , a 2 , · · · , a n ) =n (cid:80) ni
=
a n K x n ), ˆΣ O 1 , O 2 = 1 n (cid:80) ni
=
O 1 K x ⊗ O 2 K x
and Id is the identity operator.
The function q λ : R + → R + is a function known as filter , which is an approximation of x − 1 controlled by λ .
We further define the error of approximation via r λ ( x ) = 1 − xq λ ( x ) .
Spectral Filters need the function q λ further satisfies
[Equation]
for some positive c q > 0 .
Next we show that the averaged gradient descent can be considered as spectral filter algorithm with filter q η ( x ) =
[Equation]
1 x . Let us consider the gradient descent η
[Equation] [Equation]
) , then
[Equation]
) and
[Equation]
Thus if we take the filter q t ( x ) = 1 x
[Equation]
1 −
[Equation]
(cid:17) , we can have ¯ η = q t (Σ Id, A 1 )
A ⊤ 2 ˆ S ∗ n ˆ y .
At the same time x u r t ( x ) = x u (1 − xq t ( x )) =
x
[Equation]
γt ) u .
Thus we can consider the gradient descent algorithm for the inverse problem as a spectral filtering algorithm.
Next, we compare the spectral filter of early stopped gradient descent with ridge regression and decompose the risk to bias and variance terms.
Via bounding the bias and variance separately, we can achieve information theoretical optimal upper bound for such problems.
B.1 Convergence Of the Gradient Descent Algorithm
To conduct our proof of the upper bound, we consider g λ
= (Σ Id, A 1 + λI ) − 1 A 2 S ∗ f ρ and decompose the error as ( g λ − u ∗ )
[Equation]
We first bound the bias in the general Sobolev norm then come to bound the variance.
B.1.1 Auxiliary Lemmas
We first introduce several auxiliary lemmas which aims to bound different quantities according to the effective dimension/capacity of the kernel covariance operator.
We define N ( λ ) =
E x ∥ (Σ A 1 + λ ) −
[Equation]
A
[Equation]
K
[Equation]
H and N 2 ∞ ( λ ) =
sup x ∈ ρ ( x ) ∥ (Σ Id, A 1 + λ ) − 1 / 2 K x ∥ 2 H which are important important quantities used to bound the variance of our estimator.
Lemma B.1.
There exists a constant D such that the following inequality is satisfied for λ > 0 ,
[Equation]
Proof.
We use the spectral representation to bound the effective dimension N ( λ ) as
[Equation]
Lemma B
[Equation]
There exists a constant D such that the following inequality is satisfied for λ > 0 ,
[Equation]
Proof.
Similarly we use the spectral representation to bound the LHS as
[Equation]
Lemma B.3.
We denote the following quantity by N 1 ∞ , N 2 ∞ and N 3 ∞ can be bounded by
• N 1 ∞ ( λ ) =
sup x ∈ ρ ( x ) ∥ (Σ Id, A 1 + λ ) − 1 / 2 K x ∥ 2 H ≤ ∥ k αv ∥ 2 ∞ λ − µα + p α + p
[Equation]
• N 2 ∞ ( λ ) =
sup x ∈ ρ ( x ) ∥ (Σ Id, A 1 + λ ) − 1 / 2 A 2 K x ∥ 2 H ≤ ∥ k αv ∥ 2 ∞ λ − µα + p +
q α + p
[Equation]
• N 3 ∞ ( λ ) =
sup x ∈ ρ ( x ) ∥ (Σ Id, A 1 + λ ) − 1 / 2 A 1 K x ∥ 2 H ≤ ∥
k αv ∥ 2 ∞ λ − µα +3 p
α
+ p .
Proof.
We can also prove the bound from the spectral formulation and the l ∞ embedding property of the kernel function
[Equation] [Equation]
and
[Equation]
Similarly we have
[Equation]
Lemma B.4.
For all λ > 0 , we have
[Equation]
Proof.
We first bound ∥ Σ 1 − γ 2 (Σ Id, A
[Equation]
+ λ
[Equation]
−
[Equation]
∥
[Equation]
Lemma B.5.
With probability 1 − e − τ , we have
[Equation]
n and as a consequence once n ≳ τλ − µα +2 p
α + p , we’ll have
[Equation]
Proof.
We utilize the concentration result for Hilbert space valued random variable [96] to prove the bound here.
Now, we consider the operator C x
: H → H the operator defined by
[Equation]
and consider the random variable ξ x := (Σ Id, A 1 + λ ) 1 / 2 C x (Σ Id, A 1 + λ ) − 1 / 2 .
From definition, we know that
[Equation] [Equation]
At the same time, we know that ∥ f ⊗ g ∥ =
∥ f ∥ H ∥ g ∥ H for all f, g ∈ H , thus utilizing the concentration results for Hilbert space valued random variable, we have
[Equation]
n .
From Lemma B.3, we know that N 1 ∞ ( λ ) =
sup x ∈ ρ ( x ) ∥ (Σ Id, A 1 + λ ) − 1 / 2 K x ∥ 2 H ≤ ∥ k αv ∥ 2 ∞ λ − µα + p α + p , and N 3 ∞ ( λ ) =
sup x ∈ ρ ( x ) ∥ (Σ Id, A 1 + λ ) − 1 / 2 A 1 K x ∥ 2 H ≤ ∥
k αv ∥ 2 ∞ λ − µα +3 p
α
+ p .
Thus once n ≳ τλ − µα +2 p α + p , we’ll have
[Equation]
Theorem B.6 (Bernstein’s Inequality) .
Let (Ω , B , P ) be a probability space, H be a separable
Hilbert space, and ξ : Ω → H with
[Equation]
for all m ≥ 2 .
Then, for τ ≥ 1 and n ≥ 1 , the following concentration inequality is satisfied
[Equation]
−
[Equation]
Lemma B.7 (Lemma 25 in [21]) .
For λ > 0 and 0 ≤ α ≤ 1 , the function f λ,α : [0 , ∞ )
→ R be defined by f λ,α ( t ) :
=
t α λ + t .
In the case α =
the function is decreasing and for α = 1 the function is increasing.
Furthermore
[Equation]
for 0 < α < 1 the function attain its supremum at t ∗ =
λα 1 − α
Proof.
For completeness, we provide the proof here.
For function f λ,α ( t ) :
=
t α λ + t with 0 < α < 1 , we know the derivative of it is f ′ λ,α ( t ) =
αt
[Equation]
−
[Equation]
+
[Equation]
−
[Equation]
α
[Equation]
The derivative f ′ λ,α has a unique root at t ∗ = αλ/
(1 − α ) .
f λ,α attains global maximum at t ∗ and
[Equation]
At the same time, ( α α (1 − α ) 1 − α ) ′ =
α α (1 − α ) 1 − α log (cid:16) α 1 − α (cid:17) thus α α (1 − α ) 1 − α achieves minimum 12 when α = 12 .
Thus we know λ
α − 1 / 2 ≤ sup t ≥
f
λ,α ( t ) .
B.1.2 Bias
In this section, we consider the bias introduced by the regularization factor, i.e. the difference between g λ =
(Σ Id, A 1 + λI ) − 1 A 2 S ∗ f ρ and the ground truth solution A − 1
A 2 f ρ .
Lemma B.8.
If u ∗ =
A − 1
A 2 f ρ ∈
[ H ] β holds, then for all 0 ≤ γ ≤ β and λ > 0 , the following bounds holds
[Equation]
Here g λ
= (Σ Id, A 1 + λI ) − 1 A 2 S ∗ f ρ .
Proof.
Since u ∗ =
A − 1
A 2 f ρ ∈
[ H ] β , we can use the spectral representation u ∗ =
(cid:80) ni
=
a
i e i with ∥ u ∗ ∥
[ H ] β = (cid:80) ∞
i =1 µ − β i a i .
At the same time A 2 f ρ =
A 1 u ∗ = (cid:80) ni
=1 a i p i e i .
We also observe that the matrix (Σ Id, A 1 + λI ) − 1 have the spectral representation (Σ
[Equation]
e i and leads to the spectral representation of the solution
[Equation]
Then we can bound the bias via the spectral representation
[Equation]
In this section
[Equation]
we also bound a bias over the energy function ∥A 1 g λ −
A 2 f ρ ∥ 22 , which will be used in bounding the variance term.
Lemma B.9.
If u ∗ =
A − 1
A 2 f ρ ∈
[ H ] β holds, then for all 0 ≤ γ ≤ β and λ > 0 , the following bounds holds
[Equation]
Here g λ
= (Σ Id, A 1 + λI ) − 1 A 2 S ∗ f ρ .
Proof.
As discussed in the proof of Lemma B.8, we have the spectral representation of g λ as
[Equation]
Thus A 1 g λ − A 2 f
[Equation]
= (cid:80) ∞
[Equation]
=
[Equation]
µ
[Equation]
p 2 i µ i
[Equation]
+ λ − p
i
[Equation]
a i e i = − (cid:80) ∞
i =1 (cid:16) p i λ µ i p i + λ (cid:17) a i e i and we can have the bound of the bias in the energy norm as
[Equation]
B.1.3 Variance
In this section
[Equation]
we bound the variance which defined as the difference between between g λ =
[Equation]
Σ Id, A 1 + λI )
[Equation]
A
[Equation]
S ∗ f ρ and ˆ g λ = q λ (ˆΣ Id, A 1 )
A 2 ˆ S ∗ y at the scale
O
[Equation]
o
[Equation]
We first did the following decomposition
[Equation]
where we take the random variable ξ ( x, y ) as ξ ( x, y )
=
(Σ Id, A 1 + λ ) − 1 / 2 ( y A 2 K x − A 1 g λ ( x ) K x ) which satisfies E Q ξ 2 =
(Σ Id, A 1 + λ ) − 1 / 2 ( A 2 f Q − Σ QId, A 1 g λ ) where f Q = E Q
f
( x ) K x and Σ QId,
A 1 = E
Q K x ⊗A 1 K x for arbitrary distribution Q and E P ξ ( x, y ) =
(Σ Id, A 1 + λ ) − 1 / 2 ( A 2 S ∗ f ρ − Σ Id, A 1 g λ ).
We bound different terms (I), (II) and (III) separately and combine them to get the final upper bound.
We show that (I) is the mean variance term and is at the scale N ( λ ) n =
Tr
[Equation]
Σ Id,
[Equation]
Σ
[Equation]
A 2 n when the problem is regular.
Term (II) and (III) is smaller than the bias.
Our bound of term (III) bounds tighter than [18] (the second term, Lemma 10) via the spectral representation.
Bounding term (I).
The term (I) is the concentration error of the random variable ξ ( x, y ) and can be bounded via a Bernstein Inequality.
We first bound term (I) via the following decomposition
[Equation]
∥
[Equation]
where Σ λId, A 1 = Σ Id, A 1 + λI and ˆΣ λId, A 1 = ˆΣ Id, A 1 + λI .
At the same time, we knows ∥ Σ 1 −
[Equation]
2 (Σ Id, A 1 + λ ) − 1 / 2 ∥ 2 ≤ λ − γα + p α + p
From lemma B.4)
and ∥ (Σ λId, A 1 ) 1 / 2 (ˆΣ λId, A 1 ) − 1 / 2 ∥ 2 ≤(From lemma B.5) with high probability.
At the same time, we have
[Equation]
Thus we only need to focus on bounding the concentration error 1 n (cid:80) ni =1 ( ξ ( x i , y i ) − E P ξ ) .
We recall the moment condition to control the noise of the observations.
There are constants σ, L > 0 such that
[Equation]
is satisfied for µ -almost all x ∈ X and all m > 2 .
Note that the moment condition is satisfied for
Gaussian noise with bounded variance or have a bounded observation noise.
Then we can bound the second order momentum of the random variable ξ ( x, y )
=
(Σ Id, A 1 + λ ) − 1 /
[Equation]
λ
[Equation]
x ) K x ) via decomposing the random into three parts (Σ Id, A 1 + λ ) − 1 / 2 ( y A
[Equation]
K x
[Equation]
f ∗
[Equation]
x
[Equation]
A
[Equation]
K x ) , (Σ Id, A 1 + λ ) − 1 / 2 ( f
[Equation]
( x ) A
[Equation]
K
[Equation]
−
[Equation]
f ∗ ( x ) K x ) and (Σ Id, A 1 + λ ) − 1 /
A 2 f ∗ ( x ) K x − A 1 g λ ( x ) K x ) .
Base on the decomposition, we can bound the moments of random variable ξ ( x, y ) as
[Equation]
≲
[Equation]
where L λ = ∥A 2 f ∗ ( x ) − A 1 g λ ∥
L ∞ , h 1 x =
(Σ Id, A 1 + λ ) − 1 / 2 A 2 K x and h 2 x = (Σ Id, A 1 + λ ) − 1 / 2 K x .
The two vectors’ norms are bounded in Lemma B.3 as N 1 ∞ ( λ ) =
sup x ∈ ρ ( x ) ∥ (Σ Id, A 1 + λ ) − 1 / 2 K x ∥ 2 H ≤ ∥ k αv ∥ 2 ∞ λ − µα + p α + p , and N 2 ∞ ( λ ) = sup x ∈ ρ ( x ) ∥ (Σ Id, A 1 + λ ) − 1 / 2 K x ∥ 2 H ≤ ∥ k αv ∥ 2 ∞ λ − µα + p +
q α + p .
At the same time, we know that ∥A 1 g λ − A 2 f ρ ∥ 2 ≲ λ βα − 2 p 2( α + p ) ∥
u ∗ ∥
[ H ] β from
Lemma B.9 and Trace ((Σ A 1 + λ ) − 1 Σ Id,Id )
≤ Dλ p − 1 p + α from Lemma B.2.
Then using Bernstein
Inequality (Theorem B.6), we knows that with probability 1 − 2 e − τ ∥n n (cid:88) i =
[Equation]
≲ 32 τn
[Equation]
(cid:19) ≲
[Equation]
Thus we have the final bound
[Equation]
Remark 2 .
In this remark, we’ll bound the L λ = ∥A 2 f ∗ ( x ) −
A 1 g λ ∥
L ∞ here.
For the embedding theorem of the ℓ ∞ , L λ ≤ ∥A 2 f ∗ ( x ) − A 1 g λ ∥ µ ≲ λ − ( µ − β )+ α α + p .
From Lemma B.3, we know that ∥ h 1 x ∥ 2 H ≲ λ − µα + p +
q α + p and ∥
h 2 x ∥ 2 H ≲ λ − µα + p α + p .
Bounding term (III).
At last we bound the term Σ 1 − γ 2 r
(ˆΣ Id, A 1 ) g λ via the following decomposition ∥
Σ 1 − γ 2 r
(ˆΣ Id, A 1 ) g λ ∥
H = ∥ Σ 1 − γ 2 (Σ λId, A 1 ) − 1 / 2 (Σ λId, A 1 ) 1 /(cid:124) (cid:123)(cid:122) (cid:125) I
d (ˆΣ λId, A 1 ) − 1 / 2 (ˆΣ λId
[Equation]
, where we use Σ λId, A 1 to denote Σ Id, A 1 + λI .
From Lemma B.4 we now that ∥ Σ
[Equation]
γ 2 (
[Equation]
λId,
[Equation]
Then we bound the term ∥ Σ 1 − γ 2 r (ˆΣ Id, A 1 ) g λ
∥
H using r λ ( x ) x u ≲ λ u and get
[Equation]
At the same time, we can bound ∥ (Σ λId, A 1 ) −
[Equation]
2 S ∗ f ρ ∥ using the spectral representation
[Equation]
Thus we know that
[Equation]
where the last inequality is because p < 0 in our assumption.
Bounding term (II).
In this paragraph, we demonstrate the proof to bound the term
[Equation]
Note that Σ Id, A 1 (Σ λId, A 1 ) −= I − λ
(Σ λId, A 1 ) − 1 , thus we knows that Σ 1 − γ 2 q λ (ˆΣ Id, A 1 )
[Equation]
At the same time
[Equation]
according to our assumption on the spectral filter q λ
[Equation]
we know that
[Equation]
Thus we can bound Σ 1 − γ 2 q λ (ˆΣ Id, A 1 ) E P ξ ( x, y ) via the following decomposition
[Equation]
The last line is because of ∥ Σ 1 − γ 2 (Σ λId, A 1 ) − 1 / 2 ∥≤ λ − γα + p α + p (Lemma B.4), ∥ (Σ λId, A 1 ) 1 / 2 (ˆΣ λId, A 1 ) − 1 / 2 ∥ ≤ 2 with high probability (Lemma B.5), ∥ (Σ λId, A 1 ) − 1 ∥ ≤ λ − 1 , ∥ (Σ λId, A 1 ) − 1 / 2 A 2 S ∗ f ρ ∥ ≤ λ β 2( α + p ) − 12 (proved while bounding term (III)) and p < 0 .
B.2 Final Bound
At this time we can have our final bound in Theorem 3.2 via combining the bound for bias (Appendix
B.1.2) and (Appendix B.1.3)
[Equation]
Case 1.
β ≤ µα +2 q − p +1 α
In this situation, λ − γα +
( p − q )+1 p + α n is larger than λ ( β − γ )
α − µα − p α + p n .
Thus λ ( β − γ )
α α + p +
[Equation]
n is the dominating term of the loss upper bound.
Thus we can take λ = n − α + p βα +
( p − q )+1 and leads to n − ( β − γ )
α β +2( p − q )+1 upper bound.
At the same time, the third term is dominated by the second term.
Case 2.
β > µα +2 q − p +1 α
In this situation, λ ( β − γ ) α − µα − p α + p n is larger than λ − γα +
( p − q )+1 p + α n .
Thus λ ( β − γ )
α α + p + λ ( β − γ )
α − µα − p α + p n is the dominating term of the loss upper bound.
Thus we can take λ = n − α + p µα + p and leads to n − ( β − γ )
α µα + p upper bound.
At the same time, the third term is also dominated by the second term.
C Proof of the Lower Bound C.1 Preliminaries on Tools for Lower Bounds
In this section, we repeat the standard tools we use to establish the lower bound.
The main tool we use is the Fano’s inequality and the Varshamov-Gilber Lemma.
Lemma C.1 (Fano’s methods) .
Assume that V is a uniform random variable over set V , then for any
Markov chain V → X → ˆ V , we always have
[Equation]
Lemma C.2 (Varshamov-Gillbert Lemma,[97] Theorem 2.9) .
Let D ≥ 8 .
There exists a subset V = { τ (0) , · · · , τ (2 D/ 8 ) } of D − dimensional hypercube H D
= { 0 , 1 } D such that τ (0) =
(0 , 0 , · · · , 0) and the ℓ 1 distance between every two elements is larger than D
D (cid:88)
[Equation]
all
[Equation]
C.2 Proof of the Lower Bound
Theorem C.3.
Let ( X, B ) be a measurable space, H be a separable RKHS on X respect to a bounded and measurable kernel k and operator A
=
( A − 1 2 A 1 ) satisfies Assumption 2.1.
We have n random observations { ( x i , y i ) ∈
X × Y} ni
=1 of f ∗ =
A u, u ∈ H γ ∩ L ∞ , i.e. y i =
f ∗ ( x i ) + η i where η i is a random noise satisfies the momentum assumption E | η | m ≤ 12 m !
σ 2 L m − 2 for some constant σ, L > 0 .
Then for all estimators H :
( X × Y ) ⊗ n → H γ satisfies
[Equation]
Proof.
To proof the lower bound, we use the standard Fano methods via reducing the lower bound to multiple hypothesis testing.
We construct our hypothesis using binary strings ω = ( ω 1 , · · · , ω m ) ∈ { 0 , 1 } m ( m to be determined later) by defining
[Equation]
If we control m ≲ ϵ − 1 αβ − αγ , then we can always keep u ω ∈ H β for ∥ u ω ∥ 2 β
= ϵm (cid:80) mi =1 ω 2 i µ − ( β − γ ) i + m ≲ ϵµ − ( β − γ ) 2 m ≲ m α ( β − γ )
ϵ
= O (1) .
Similarly, we can select m ≲ ϵ − 1 αµ − αγ to control ∥ u ω ∥
L ∞ ≤ ∥ u ω ∥ µ ≤ O (1) .
At the same time, the associated PDE right hand side function f ω =
A
[Equation] [Equation]
A 1 u ω = (cid:0) ϵm (cid:1) 1 / 2 (cid:80) mi =
q
i ω
i p i µ γ/ 2 i + m e i + m .
Using Gilbert-Varshamov Lemma we know that there exists M ≥ 2 m/ 8 binary strings ω (1) , · · · , ω ( k ) ∈ { 0 , 1 } m with ω (0) =
(0 , · · · , 0) subject to
[Equation]
holds for all j ̸ = k .
As consequence, the distance between f ω and f ω ′ can be lower bounded as ∥ u ω − u ω ′ ∥ 2 γ = ϵm (cid:80) mi =1 ( ω i − ω ′ i ) 2 ≥ ϵ/ 8 .
To apply the Fano method, we still need to bound the mutual information between the uniform distribution over all the hypothesis and the distribution of the observed data.
We take η i is sampled form N (0 , min { σ, L } 2 ) which satisfies the momentum condition.
Then we know that this mutual information can be bounded by the following average of
KL divergence[97] via
[Equation]
Then we apply the Fano’s inequality
[Equation]
Take ϵ = n −
[Equation]
, we know that with constant probability we have
[Equation]