Original: M INIMAX O PTIMAL K ERNEL O PERATOR L EARNING VIA M ULTILEVEL T RAINING
Revised: M INIMAX O PTIMAL K ERNEL O PERATOR L EARNING VIA M ULTILEVEL T RAINING

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning mappings between infinite-dimensional function spaces have achieved empirical success in many disciplines of machine learning, including generative modeling, functional data analysis, causal inference, and multi-agent reinforcement learning.
Revised: Learning mappings between infinite-dimensional function spaces have achieved empirical success in many disciplines of machine learning, including generative modeling, functional data analysis, causal inference, and multi-agent reinforcement learning.

Original: In this paper, we study the statistical limit of learning a HilbertSchmidt operator between two infinite-dimensional Sobolev reproducing kernel
Revised: In this paper, we study the statistical limit of learning a HilbertSchmidt operator between two infinite-dimensional Sobolev reproducing kernel

Original: Hilbert spaces (RKHSs).
Revised: Hilbert spaces (RKHSs).

Original: We establish the information-theoretic lower bound in terms of the Sobolev Hilbert-Schmidt norm and show that a regularization that learns the spectral components below the bias contour and ignores the ones above the variance contour can achieve the optimal learning rate.
Revised: We establish the information-theoretic lower bound in terms of the Sobolev Hilbert-Schmidt norm and show that a regularization that learns the spectral components below the bias contour and ignores the ones above the variance contour can achieve the optimal learning rate.

Original: At the same time, the spectral components between the bias and variance contours give us flexibility in designing computationally feasible machine learning algorithms.
Revised: At the same time, the spectral components between the bias and variance contours give us flexibility in designing computationally feasible machine learning algorithms.

Original: Based on this observation, we develop a multilevel kernel operator learning algorithm that is optimal when learning linear operators between infinite-dimensional function spaces.
Revised: Based on this observation, we develop a multilevel kernel operator learning algorithm that is optimal when learning linear operators between infinite-dimensional function spaces.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Supervised learning of operators between two infinite-dimensional spaces has attracted attention in several areas of application of machine learning, including, scientific computing (Lu et al., 2019;
Revised: Supervised learning of operators between two infinite-dimensional spaces has attracted attention in several areas of application of machine learning, including, scientific computing (Lu et al., 2019;

Original: Li et al., 2020; de Hoop et al., 2021; Li et al., 2018; 2021b), functional data analysis (Crambes & Mas, 2013; H¨ormann & Kidzi´nski, 2015; Wang et al., 2020a), mean-field games (Guo et al., 2019; Wang et al., 2020b), conditional kernel mean embedding (Song et al., 2009; 2013; Muandet et al., 2017) and econometrics (Singh et al., 2019; Muandet et al., 2020; Dikkala et al., 2020; Singh et al., 2020).
Revised: Li et al., 2020; de Hoop et al., 2021; Li et al., 2018; 2021b), functional data analysis (Crambes & Mas, 2013; H¨ormann & Kidzi´nski, 2015; Wang et al., 2020a), mean-field games (Guo et al., 2019; Wang et al., 2020b), conditional kernel mean embedding (Song et al., 2009; 2013; Muandet et al., 2017) and econometrics (Singh et al., 2019; Muandet et al., 2020; Dikkala et al., 2020; Singh et al., 2020).

Original: Despite the empirical success of operator learning, the statistical limit of learning an infinite-dimensional operator has not been investigated studied.
Revised: Despite the empirical success of operator learning, the statistical limit of learning an infinite-dimensional operator has not been investigated studied.

Original: In this paper, we study the problem of learning Hilbert Schmidt operators between infinite-dimensional Sobolev RKHSs H βK and H γL with given kernels k and l , respectively with β, γ ∈ [0 , 1) (Adams & Fournier, 2003; Christmann &
Revised: In this paper, we study the problem of learning Hilbert Schmidt operators between infinite-dimensional Sobolev RKHSs H βK and H γL with given kernels k and l , respectively with β, γ ∈ [0 , 1) (Adams & Fournier, 2003; Christmann &

Original: Steinwart, 2008; Fischer & Steinwart, 2020).
Revised: Steinwart, 2008; Fischer & Steinwart, 2020).

Original: Our goal is to derive the optimal sample complexity for linear operator learning, i.e. how much data is required to achieve a certain performance level.
Revised: Our goal is to derive the optimal sample complexity for linear operator learning, i.e. how much data is required to achieve a certain performance level.

Original: We first establish an information-theoretic lower bound for learning a Hilbert-Schmidt operator between Sobolev spaces with respect to a general Sobolev norm.
Revised: We first establish an information-theoretic lower bound for learning a Hilbert-Schmidt operator between Sobolev spaces with respect to a general Sobolev norm.

Original: Our information-theoretic lower bound indicates that the optimal learning rate is determined by the minimum of two polynomial rates: one is purely decided by the input Sobolev reproducing kernel Hilbert space and its evaluating norm, while the other one is purely determined by the output space along with its evaluating norm.
Revised: Our information-theoretic lower bound indicates that the optimal learning rate is determined by the minimum of two polynomial rates: one is purely decided by the input Sobolev reproducing kernel Hilbert space and its evaluating norm, while the other one is purely determined by the output space along with its evaluating norm.

Original: The rate is novel in that all existing results (Fischer & Steinwart, 2020; Li et al., 2022; de Hoop et al., 2021) only establish rates that depend on the parameter of input space.
Revised: The rate is novel in that all existing results (Fischer & Steinwart, 2020; Li et al., 2022; de Hoop et al., 2021) only establish rates that depend on the parameter of input space.

Original: The reason is all previous works (Talwai et al., 2022; Li et al., 2022; de Hoop et al., 2021) only consider the case of the output space as a subspace of a trace bounded reproducing kernel Hilbert space but not a general Sobolev space.
Revised: The reason is all previous works (Talwai et al., 2022; Li et al., 2022; de Hoop et al., 2021) only consider the case of the output space as a subspace of a trace bounded reproducing kernel Hilbert space but not a general Sobolev space.

Original: We refer to Remark 2.1 for detailed comparisons.
Revised: We refer to Remark 2.1 for detailed comparisons.

Original: To design a learning algorithm for approximating an infinite-dimensional operator, we need to learn a finite-dimensional restriction instead of the whole operator, as the latter would result in infinite variance.
Revised: To design a learning algorithm for approximating an infinite-dimensional operator, we need to learn a finite-dimensional restriction instead of the whole operator, as the latter would result in infinite variance.

Original: The finite-dimensional selection leads to bias error but decreases the variance.
Revised: The finite-dimensional selection leads to bias error but decreases the variance.

Original: A natural task is then to study the shape of regularization that can lead to the optimal bias-variance trade-off and achieve the optimal learning rate.
Revised: A natural task is then to study the shape of regularization that can lead to the optimal bias-variance trade-off and achieve the optimal learning rate.

Original: In this paper, we consider the bias and variance contour at the scale of optimal learning.
Revised: In this paper, we consider the bias and variance contour at the scale of optimal learning.

Original: Once the regularization enables one to learn all the spectral parts above the bias contour and below the variance contour, the learning is optimal.
Revised: Once the regularization enables one to learn all the spectral parts above the bias contour and below the variance contour, the learning is optimal.

Original: Finally, utilizing the region between the bias contour and variance contour, we developed a multilevel training algorithm
Revised: Finally, utilizing the region between the bias contour and variance contour, we developed a multilevel training algorithm

Original: (Lye et al., 2021;
Revised: (Lye et al., 2021;

Original: Li et al., 2021a) which first learns the mapping on low frequency and then successively fine-tunes the machine learning models to fit the high-frequency output.
Revised: Li et al., 2021a) which first learns the mapping on low frequency and then successively fine-tunes the machine learning models to fit the high-frequency output.

Original: The intuition of our algorithm aligns with the original motivation of multilevel Monte Carlo (Giles, 2008; 2015): we use the next level to reduce bias while keeping the variance at the same scale.
Revised: The intuition of our algorithm aligns with the original motivation of multilevel Monte Carlo (Giles, 2008; 2015): we use the next level to reduce bias while keeping the variance at the same scale.

Original: We demonstrate that such a multilevel algorithm can achieve an optimal non-parametric rate for linear operator learning.
Revised: We demonstrate that such a multilevel algorithm can achieve an optimal non-parametric rate for linear operator learning.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Machine Learning Based PDE
Revised: Machine Learning Based PDE

Original: Solver Solving partial differential equations (PDEs) plays a prominent role in many scientific and engineering disciplines, such as physics, chemistry, operation management, macro-economy, etc.
Revised: Solver Solving partial differential equations (PDEs) plays a prominent role in many scientific and engineering disciplines, such as physics, chemistry, operation management, macro-economy, etc.

Original: The recent deep learning breakthrough has drawn attention to solving PDEs via machine learning methods (Raissi et al., 2019; Han et al., 2018; Sirignano &
Revised: The recent deep learning breakthrough has drawn attention to solving PDEs via machine learning methods (Raissi et al., 2019; Han et al., 2018; Sirignano &

Original: Spiliopoulos, 2018; Yu et al., 2018; Khoo et al., 2019; Chen et al., 2021).
Revised: Spiliopoulos, 2018; Yu et al., 2018; Khoo et al., 2019; Chen et al., 2021).

Original: The statistical power and computational cost of these problem is well-studied by recent papers (Lu et al., 2021; 2022; Nickl et al., 2020; Nickl & Wang, 2020).
Revised: The statistical power and computational cost of these problem is well-studied by recent papers (Lu et al., 2021; 2022; Nickl et al., 2020; Nickl & Wang, 2020).

Original: This paper focuses on operator learning (Chen & Chen, 1995;
Revised: This paper focuses on operator learning (Chen & Chen, 1995;

Original: Long et al., 2018; 2019; Feliu-Faba et al., 2020; Khoo et al., 2021;
Revised: Long et al., 2018; 2019; Feliu-Faba et al., 2020; Khoo et al., 2021;

Original: Lu et al., 2019; Li et al., 2020;
Revised: Lu et al., 2019; Li et al., 2020;

Original: Kovachki et al., 2021; Stepaniants, 2021), i.e. learning a map between two infinite-dimensional function spaces.
Revised: Kovachki et al., 2021; Stepaniants, 2021), i.e. learning a map between two infinite-dimensional function spaces.

Original: For example, one can learn a PDE solver that maps from the boundary condition to the solution or an inverse problem that maps from the boundary measurement to the coefficient field.
Revised: For example, one can learn a PDE solver that maps from the boundary condition to the solution or an inverse problem that maps from the boundary measurement to the coefficient field.

Original: Regarding the mathematical foundation of operator learning, (Liu et al., 2022) considers the learning rate of non-parametric operator learning.
Revised: Regarding the mathematical foundation of operator learning, (Liu et al., 2022) considers the learning rate of non-parametric operator learning.

Original: However, non-parametric functional data analysis often suffers from slower-than-polynomial convergence rates (Mas, 2012) due to the small ball probability problem for the probability distributions in infinite dimensional spaces (Delaigle & Hall, 2010).
Revised: However, non-parametric functional data analysis often suffers from slower-than-polynomial convergence rates (Mas, 2012) due to the small ball probability problem for the probability distributions in infinite dimensional spaces (Delaigle & Hall, 2010).

Original: The most relevant works are (Lin et al., 2011; Reimherr, 2015; de Hoop et al., 2021), which consider the rates for learning a linear operator.
Revised: The most relevant works are (Lin et al., 2011; Reimherr, 2015; de Hoop et al., 2021), which consider the rates for learning a linear operator.

Original: For the comparison between our work and (de Hoop et al., 2021), see Remark 2.1.
Revised: For the comparison between our work and (de Hoop et al., 2021), see Remark 2.1.

Original: Learning with Kernel.
Revised: Learning with Kernel.

Original: Supervised least square regression in RKHS and its generalization capability have been thoroughly studied (Caponnetto & De Vito, 2007; Smale & Zhou, 2007; De Vito et al., 2005; Rosasco et al., 2010; Mendelson & Neeman, 2010).
Revised: Supervised least square regression in RKHS and its generalization capability have been thoroughly studied (Caponnetto & De Vito, 2007; Smale & Zhou, 2007; De Vito et al., 2005; Rosasco et al., 2010; Mendelson & Neeman, 2010).

Original: The minimax optimality with respect to the Sobolev norm has been discussed recently in (Fischer & Steinwart, 2020;
Revised: The minimax optimality with respect to the Sobolev norm has been discussed recently in (Fischer & Steinwart, 2020;

Original: Liu & Li, 2020; Lu et al., 2022).
Revised: Liu & Li, 2020; Lu et al., 2022).

Original: Our paper is highly related to recent works (Schuster et al., 2020; Mollenhauer & Koltai, 2020; Talwai et al., 2022;
Revised: Our paper is highly related to recent works (Schuster et al., 2020; Mollenhauer & Koltai, 2020; Talwai et al., 2022;

Original: Li et al., 2022; Park & Muandet, 2020; Singh et al., 2019; 2020) on identifying the Sobolev norm learning rate for the kernel mean embedding(Song et al., 2009; 2013; Muandet et al., 2017), which can also formulated as learning an operator.
Revised: Li et al., 2022; Park & Muandet, 2020; Singh et al., 2019; 2020) on identifying the Sobolev norm learning rate for the kernel mean embedding(Song et al., 2009; 2013; Muandet et al., 2017), which can also formulated as learning an operator.

Original: The difference between our work and (Talwai et al., 2022; Li et al., 2022) sees Remark 2.1.
Revised: The difference between our work and (Talwai et al., 2022; Li et al., 2022) sees Remark 2.1.

Original: A concurrent paper (Balasubramanian et al., 2022) considers a unified RKHS methodology for functional data analysis.
Revised: A concurrent paper (Balasubramanian et al., 2022) considers a unified RKHS methodology for functional data analysis.

Original: Our paper provided a refined analysis and provided information theortical optimal rates for this problem.
Revised: Our paper provided a refined analysis and provided information theortical optimal rates for this problem.

Original: Multilevel Monte Carlo By combining biased estimators with multiple stepsizes, multilevel
Revised: Multilevel Monte Carlo By combining biased estimators with multiple stepsizes, multilevel

Original: Monte Carlo (MLMC) (Giles, 2008; 2015) dramatically improves the rate of convergence and achieves in many settings the canonical square root convergence rate associated with unbiased Monte
Revised: Monte Carlo (MLMC) (Giles, 2008; 2015) dramatically improves the rate of convergence and achieves in many settings the canonical square root convergence rate associated with unbiased Monte

Original: Carlo (Rhee & Glynn, 2015; Blanchet & Glynn, 2015).
Revised: Carlo (Rhee & Glynn, 2015; Blanchet & Glynn, 2015).

Original: Multilevel Monte Carlo can also be used for a random variable with infinite variance (Blanchet & Liu, 2016; Chen et al., 2020).
Revised: Multilevel Monte Carlo can also be used for a random variable with infinite variance (Blanchet & Liu, 2016; Chen et al., 2020).

Original: To the best of our knowledge, this is the first paper that provides optimal sample complexity for multilevel Monte
Revised: To the best of our knowledge, this is the first paper that provides optimal sample complexity for multilevel Monte

Original: Carlo type algorithm for infinite variance problems in the non-parametric regime.
Revised: Carlo type algorithm for infinite variance problems in the non-parametric regime.

Original: Very recently,
Revised: Very recently,

Original: (Lye et al., 2021; Li et al., 2021a) developed a multilevel machine learning Monte Carlo algorithm (ML2MC) /
Revised: (Lye et al., 2021; Li et al., 2021a) developed a multilevel machine learning Monte Carlo algorithm (ML2MC) /

Original: multilevel fine-tuning algorithm for learning solution maps by first learning the map on the coarsest grid and then successively fine-tuning the network on samples generated at finer grids.
Revised: multilevel fine-tuning algorithm for learning solution maps by first learning the map on the coarsest grid and then successively fine-tuning the network on samples generated at finer grids.

Original: The authors also showed that, following the telescoping in MLMC, the multilevel training procedure could reduce the generalization error without spending more time on generating training samples.
Revised: The authors also showed that, following the telescoping in MLMC, the multilevel training procedure could reduce the generalization error without spending more time on generating training samples.

Original: (Sch¨afer & Owhadi, 2021; Boull´e et al., 2022) consider such multi-scale algorithm for learning Green’s function.
Revised: (Sch¨afer & Owhadi, 2021; Boull´e et al., 2022) consider such multi-scale algorithm for learning Green’s function.

Original: However, the statistical power of such an algorithm is still under investigation.
Revised: However, the statistical power of such an algorithm is still under investigation.

Original: Another difference with (Boull´e et al., 2022) is that we consider the Green function in H −norm rather than the ℓ 1 norm used in (Boull´e et al., 2022).
Revised: Another difference with (Boull´e et al., 2022) is that we consider the Green function in H −norm rather than the ℓ 1 norm used in (Boull´e et al., 2022).

Original: In this paper, we qualify a specific setting where this multilevel procedure can and is necessary to achieve the minimax optimal learning rate.
Revised: In this paper, we qualify a specific setting where this multilevel procedure can and is necessary to achieve the minimax optimal learning rate.

Original: 1. C ONTRIBUTION
Revised: 1. C ONTRIBUTION

Original: • We derive a novel information-theoretic lower bound of learning a linear operator between two infinite-dimensional Sobolev reproducing kernel Hilbert spaces.
Revised: • We derive a novel information-theoretic lower bound of learning a linear operator between two infinite-dimensional Sobolev reproducing kernel Hilbert spaces.

Original: The optimal learning rate is a minimum of two polynomial rates, one only dependent on the parameters of the input space while the other only on the parameters of the output space.
Revised: The optimal learning rate is a minimum of two polynomial rates, one only dependent on the parameters of the input space while the other only on the parameters of the output space.

Original: The first rate aligns with the previous works (Li et al., 2022), while the second is novel to the literature.
Revised: The first rate aligns with the previous works (Li et al., 2022), while the second is novel to the literature.

Original: • We study the shape of regularization that can lead to the optimal learning rate.
Revised: • We study the shape of regularization that can lead to the optimal learning rate.

Original: One should learn all the spectral parts under the bias contour at the level of the optimal learning rate but not the spectral components above the variance contour at the level of the learning rate.
Revised: One should learn all the spectral parts under the bias contour at the level of the optimal learning rate but not the spectral components above the variance contour at the level of the learning rate.

Original: This enables the estimator to enjoy an optimal balance of bias-variance.
Revised: This enables the estimator to enjoy an optimal balance of bias-variance.

Original: • We qualify a specific setting where a multilevel training procedure (Lye et al., 2021;
Revised: • We qualify a specific setting where a multilevel training procedure (Lye et al., 2021;

Original: Li et al., 2021a) is necessary and capable of achieving a minimax optimal learning rate for learning a linear operator.
Revised: Li et al., 2021a) is necessary and capable of achieving a minimax optimal learning rate for learning a linear operator.

Original: We perform the optimal learning rate via O (ln ln n ) ensemble
Revised: We perform the optimal learning rate via O (ln ln n ) ensemble

Original: of ridge regression models.
Revised: of ridge regression models.

Original: This differs from finite-dimensional operator learning, where a single-level estimator can be optimal.
Revised: This differs from finite-dimensional operator learning, where a single-level estimator can be optimal.

Original:  P ROBLEM F ORMULATION 2. P RELIMINARY
Revised:  P ROBLEM F ORMULATION 2. P RELIMINARY

Original: Let P K be a distribution over the input space H K and define covariance operator C KK
Revised: Let P K be a distribution over the input space H K and define covariance operator C KK

Original: = E u ∼ P K u ⊗ u .
Revised: = E u ∼ P K u ⊗ u .

Original: Consider its spectral decomposition C KK = (cid:80) + ∞ i =1 µ 2 i e i ⊗ e i , where { µ 12 i e i } + ∞ i =1 is an orthogonal eigenbasis and { µ i } is the corresponding eigenvalues of C KK (here the g ⊗ h is an operator defined as g ⊗ h = gh ∗ : f → ⟨ f, h ⟩ g ).
Revised: Consider its spectral decomposition C KK = (cid:80) + ∞ i =1 µ 2 i e i ⊗ e i , where { µ 12 i e i } + ∞ i =1 is an orthogonal eigenbasis and { µ i } is the corresponding eigenvalues of C KK (here the g ⊗ h is an operator defined as g ⊗ h = gh ∗ : f → ⟨ f, h ⟩ g ).

Original: In the typical machine learning applications, the test distribution is the same as the training distribution, so we can assume that H K
Revised: In the typical machine learning applications, the test distribution is the same as the training distribution, so we can assume that H K

Original: =
Revised: =

Original: (cid:110)(cid:80)
Revised: (cid:110)(cid:80)

Original: i a i µ 12 i e i : { a i } ∞ i =1 ∈ ℓ 2 (cid:111) without loss of generality.
Revised: i a i µ 12 i e i : { a i } ∞ i =1 ∈ ℓ 2 (cid:111) without loss of generality.

Original: Note that this automatically holds in the context of learning the conditional mean embedding (CME)
Revised: Note that this automatically holds in the context of learning the conditional mean embedding (CME)

Original: (Fischer & Steinwart, 2020; Talwai et al., 2022; Li et al., 2022).
Revised: (Fischer & Steinwart, 2020; Talwai et al., 2022; Li et al., 2022).

Original: Following Christmann & Steinwart (2008); Fischer & Steinwart (2020), we define the interpolation
Revised: Following Christmann & Steinwart (2008); Fischer & Steinwart (2020), we define the interpolation

Original: Sobolev space H βK = (cid:26) f
Revised: Sobolev space H βK = (cid:26) f

Original: = (cid:80)
Revised: = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: ( µ
Revised: ( µ

Original: [Equation]
Revised: [Equation]

Original: e i ) : { a i } ∞ i =1 ∈ l(cid:27) for any β > 0 , equipped with Sobolev norm defined by the inner product (cid:68)(cid:80) i
Revised: e i ) : { a i } ∞ i =1 ∈ l(cid:27) for any β > 0 , equipped with Sobolev norm defined by the inner product (cid:68)(cid:80) i

Original: a i ( µ β/ 2 i e i ) , (cid:80) i b i ( µ β/ 2 i e i ) (cid:69) H βK
Revised: a i ( µ β/ 2 i e i ) , (cid:80) i b i ( µ β/ 2 i e i ) (cid:69) H βK

Original: =
Revised: =

Original: (cid:80) i a
Revised: (cid:80) i a

Original: i b i .
Revised: i b i .

Original: For the output space, we fix a user-specified distribution Q L and a reproducing Kernel Hilbert Space.
Revised: For the output space, we fix a user-specified distribution Q L and a reproducing Kernel Hilbert Space.

Original: We can similarly define the covariance operator C Q L and the Sobolev space H γL .
Revised: We can similarly define the covariance operator C Q L and the Sobolev space H γL .

Original: Natural choices of Q L include some distribution on kernel functions { ℓ ( y, · ) : y ∈ Y } of H L induced by some distribution
Revised: Natural choices of Q L include some distribution on kernel functions { ℓ ( y, · ) : y ∈ Y } of H L induced by some distribution

Original: Q L on Y , so that C Q L is a kernel integral operator with respect to Q L and H γL is an interpolation space between H L and L 2 ( Q L ) ; see Example 2.1 for a specific example.
Revised: Q L on Y , so that C Q L is a kernel integral operator with respect to Q L and H γL is an interpolation space between H L and L 2 ( Q L ) ; see Example 2.1 for a specific example.

Original: Following Li et al.
Revised: Following Li et al.

Original: (2022), in this paper we consider the Hilbert-Schmidt norm between two Sobolev
Revised: (2022), in this paper we consider the Hilbert-Schmidt norm between two Sobolev

Original: Spaces for all the operators, which is defined as following.
Revised: Spaces for all the operators, which is defined as following.

Original: Definition 2.1 ( ( β, γ ) -norm) Let T : H K (cid:55)→ H L be a possibly unbounded linear operator.
Revised: Definition 2.1 ( ( β, γ ) -norm) Let T : H K (cid:55)→ H L be a possibly unbounded linear operator.

Original: I 1 ,β,P K : H K (cid:55)→ H βK , β ∈ (0 , 1) is the canonical embedding mapping that takes u ∈ H K to the same element u in the larger space H β K , and I 1 ,γ,Q L : H L (cid:55)→ H γ L , γ < 1 is similarly defined.
Revised: I 1 ,β,P K : H K (cid:55)→ H βK , β ∈ (0 , 1) is the canonical embedding mapping that takes u ∈ H K to the same element u in the larger space H β K , and I 1 ,γ,Q L : H L (cid:55)→ H γ L , γ < 1 is similarly defined.

Original: Then the ( β, γ ) -norm of T is defined as
Revised: Then the ( β, γ ) -norm of T is defined as

Original: [Equation]
Revised: [Equation]

Original: where we omit the dependence of ∥·∥ β,γ on P K and Q L since it will always be clear from context.
Revised: where we omit the dependence of ∥·∥ β,γ on P K and Q L since it will always be clear from context.

Original: 2. P ROBLEM F ORMULATION
Revised: 2. P ROBLEM F ORMULATION

Original: We consider the problem of learning an unknown linear operator A 0 : H K (cid:55)→
Revised: We consider the problem of learning an unknown linear operator A 0 : H K (cid:55)→

Original: H L between two reproducing kernel Hilbert spaces corresponding to kernl k and l respectively.
Revised: H L between two reproducing kernel Hilbert spaces corresponding to kernl k and l respectively.

Original: We are given N noisy data pairs ( u i , v i ) , 1 ⩽
Revised: We are given N noisy data pairs ( u i , v i ) , 1 ⩽

Original: i ⩽ N related by
Revised: i ⩽ N related by

Original: [Equation]
Revised: [Equation]

Original: where u i i .
Revised: where u i i .

Original: i .
Revised: i .

Original: d . ∼ P K for some unknown distribution P K and ε i is the noise drawn from some distribution with zero mean that may depend on u i .
Revised: d . ∼ P K for some unknown distribution P K and ε i is the noise drawn from some distribution with zero mean that may depend on u i .

Original: We use P KL for the joint distribution of ( u i , v i ) .
Revised: We use P KL for the joint distribution of ( u i , v i ) .

Original: Denote C KK = E u ∼ P K u ⊗ u , C KL =
Revised: Denote C KK = E u ∼ P K u ⊗ u , C KL =

Original: E ( u,v ) ∼ P KL u ⊗ v and its adjoint C LK
Revised: E ( u,v ) ∼ P KL u ⊗ v and its adjoint C LK

Original: = C ∗ KL be uncentered cross-covariance operators associated with P KL .
Revised: = C ∗ KL be uncentered cross-covariance operators associated with P KL .

Original: Then we can reformulate the ground turth operator as A 0 =
Revised: Then we can reformulate the ground turth operator as A 0 =

Original: C LK C † KK , where † is the pseudo-inverse (Talwai et al., 2022; Li et al., 2022).
Revised: C LK C † KK , where † is the pseudo-inverse (Talwai et al., 2022; Li et al., 2022).

Original: With the goal of understanding the relative difficulty of learning different types of linear operators, we investigate the sample efficiency of learning A 0 under certain source assumptions imposed on the data model (1).
Revised: With the goal of understanding the relative difficulty of learning different types of linear operators, we investigate the sample efficiency of learning A 0 under certain source assumptions imposed on the data model (1).

Original: Source condition (Caponnetto & De Vito, 2007; Mendelson & Neeman, 2010;
Revised: Source condition (Caponnetto & De Vito, 2007; Mendelson & Neeman, 2010;

Original: Steinwart et al., 2009; Rosasco et al., 2010; Fischer & Steinwart, 2020) assumes that the learning target lies in a parameterized function class and study the learning rate for different problems with different hardness.
Revised: Steinwart et al., 2009; Rosasco et al., 2010; Fischer & Steinwart, 2020) assumes that the learning target lies in a parameterized function class and study the learning rate for different problems with different hardness.

Original: Specifically, the source condition assume that the learning target is bounded in certain Sobolev norm.
Revised: Specifically, the source condition assume that the learning target is bounded in certain Sobolev norm.

Original: In this paper, we consider learning an operator with bounded ( β, γ ) -norm, which is the Hilbert-Schmidt norm that maps from H βK to H γL .
Revised: In this paper, we consider learning an operator with bounded ( β, γ ) -norm, which is the Hilbert-Schmidt norm that maps from H βK to H γL .

Original: We consider the generalization error/convergence rate under another ( β ′ , γ ′ )
Revised: We consider the generalization error/convergence rate under another ( β ′ , γ ′ )

Original: -norm as in Fischer & Steinwart (2020)
Revised: -norm as in Fischer & Steinwart (2020)

Original: ; Lu et al.
Revised: ; Lu et al.

Original: Talwai et al.
Revised: Talwai et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: Remark 2.

Revised: Remark 2.

Original: Although recent works have considered similar problems in the context of conditional mean embedding (Talwai et al., 2022; Li et al., 2022) and functional data analysis (de Hoop et al., 2021), in all these papers, the output space is a trace bounded RKHS (de Hoop et al., 2021, Assumption 2.14 (vi)) rather than the general parameterized Sobolev space in our paper.
Revised: Although recent works have considered similar problems in the context of conditional mean embedding (Talwai et al., 2022; Li et al., 2022) and functional data analysis (de Hoop et al., 2021), in all these papers, the output space is a trace bounded RKHS (de Hoop et al., 2021, Assumption 2.14 (vi)) rather than the general parameterized Sobolev space in our paper.

Original: We then list all the assumptions imposed on the underlying kernel for our theoretical results.
Revised: We then list all the assumptions imposed on the underlying kernel for our theoretical results.

Original: We follow the standard capacity assumptions and embedding properties used in kernel regression (Fischer & Steinwart, 2020; Talwai et al., 2022;
Revised: We follow the standard capacity assumptions and embedding properties used in kernel regression (Fischer & Steinwart, 2020; Talwai et al., 2022;

Original: Li et al., 2022).
Revised: Li et al., 2022).

Original: Assumption 2.1 (Capacity Condition of the Covariance)
Revised: Assumption 2.1 (Capacity Condition of the Covariance)

Original: The eigenvalues { µ i } i ⩾ 1 of the covariance operator C KK = E u ∼ P K u ⊗ u satisfies µ i ∝ i − 1 p for some p ∈ (0 , 1) .
Revised: The eigenvalues { µ i } i ⩾ 1 of the covariance operator C KK = E u ∼ P K u ⊗ u satisfies µ i ∝ i − 1 p for some p ∈ (0 , 1) .

Original: Similarly, the eigenvalues { ρ i } i ⩾ 1 of the covariance operator C Q L = E v ∼ Q L v ⊗ v satisfies ρ i ∝ i − 1 q for some q ∈ (0 , 1) .
Revised: Similarly, the eigenvalues { ρ i } i ⩾ 1 of the covariance operator C Q L = E v ∼ Q L v ⊗ v satisfies ρ i ∝ i − 1 q for some q ∈ (0 , 1) .

Original: Assumption 2.2 ( ℓ ∞ Embedding Property of the Input RKHS)
Revised: Assumption 2.2 ( ℓ ∞ Embedding Property of the Input RKHS)

Original: There exists a smallest α ∈ (0 , 1) such that (cid:13)(cid:13)(cid:13)(cid:0)
Revised: There exists a smallest α ∈ (0 , 1) such that (cid:13)(cid:13)(cid:13)(cid:0)

Original: I ∗
Revised: I ∗

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: .s.
Revised: .s.

Original: under P K for some A 1 < + ∞ .
Revised: under P K for some A 1 < + ∞ .

Original: Assumption 2.3 ( ℓ ∞ Embedding Property of the Output RKHS)
Revised: Assumption 2.3 ( ℓ ∞ Embedding Property of the Output RKHS)

Original: There exists A 2 < + ∞ such that ∥ g ∥
Revised: There exists A 2 < + ∞ such that ∥ g ∥

Original: H L ⩽
Revised: H L ⩽

Original: A 2 holds for all g in the range of A 0 , except from a Q L -null set.
Revised: A 2 holds for all g in the range of A 0 , except from a Q L -null set.

Original: Assumption 2.4 (Moment Condition)
Revised: Assumption 2.4 (Moment Condition)

Original: There exists an operator V : H L (cid:55)→ H L with tr ( V ) ⩽ σsuch that for every u ∈ H K , We have E v ∼ P KL ( ·| u ) (cid:104) (( v − A 0 u ) ⊗ ( v − A 0 u ))
Revised: There exists an operator V : H L (cid:55)→ H L with tr ( V ) ⩽ σsuch that for every u ∈ H K , We have E v ∼ P KL ( ·| u ) (cid:104) (( v − A 0 u ) ⊗ ( v − A 0 u ))

Original: k (cid:105) ⪯ 12 (2 k )!
Revised: k (cid:105) ⪯ 12 (2 k )!

Original: R 2 k − 2 V. holds for all k ≥ 2 .
Revised: R 2 k − 2 V. holds for all k ≥ 2 .

Original: Assumption 2.5 (Source Condition)
Revised: Assumption 2.5 (Source Condition)

Original: A 0 is bounded under ( β, γ ) -norm i.e. ∥A 0 ∥ β,γ ⩽ B .
Revised: A 0 is bounded under ( β, γ ) -norm i.e. ∥A 0 ∥ β,γ ⩽ B .

Original: 2. E XAMPLES
Revised: 2. E XAMPLES

Original: In this section, we will introduce two examples of our theory.
Revised: In this section, we will introduce two examples of our theory.

Original: The first one is about learning a differential operator, for example inferring an advection-diffusion model Portone & Moser (2022) from observations or predicting the future Long et al.
Revised: The first one is about learning a differential operator, for example inferring an advection-diffusion model Portone & Moser (2022) from observations or predicting the future Long et al.

Original: ; Lu et al.
Revised: ; Lu et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Feliu-Faba et al.
Revised: ; Feliu-Faba et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: The second example is about learning conditional mean embedding Song et al.
Revised: The second example is about learning conditional mean embedding Song et al.

Original: ; Muandet et al.
Revised: ; Muandet et al.

Original: (2017), which represents a conditional distribution as an RKHS element.
Revised: (2017), which represents a conditional distribution as an RKHS element.

Original: Thus conditional distribution regression can be reduced to kernel operator learning.
Revised: Thus conditional distribution regression can be reduced to kernel operator learning.

Original: Our theory can also be used for linear inverse problems such as radial electrical impedance tomography (EIT) Mueller & Siltanen (2012), and the severely ill-posed inverse boundary problem for the Helmholtz equation with unknown wave-number parameter Agapiou et al.
Revised: Our theory can also be used for linear inverse problems such as radial electrical impedance tomography (EIT) Mueller & Siltanen (2012), and the severely ill-posed inverse boundary problem for the Helmholtz equation with unknown wave-number parameter Agapiou et al.

Original: For detailed discussion, we refer to (de Hoop et al., 2021, Section 1.3)
Revised: For detailed discussion, we refer to (de Hoop et al., 2021, Section 1.3)

Original: Example 2.1 (Learning differential operators) Suppose that the ground-truth operator A 0 =
Revised: Example 2.1 (Learning differential operators) Suppose that the ground-truth operator A 0 =

Original: ∆ t where ∆ is the Laplacian and t ∈ Z .
Revised: ∆ t where ∆ is the Laplacian and t ∈ Z .

Original: Let H K
Revised: Let H K

Original: = H m +2 t ([0 , 1]) be the Sobolev space with smoothness m + 2 t on [0 , 1] and H L
Revised: = H m +2 t ([0 , 1]) be the Sobolev space with smoothness m + 2 t on [0 , 1] and H L

Original: = H m ([0 , 1]) , then A 0 is a bounded operator from H K to H L which corresponds to the β = γ = 1 case.
Revised: = H m ([0 , 1]) , then A 0 is a bounded operator from H K to H L which corresponds to the β = γ = 1 case.

Original: However, we will see below that we can obtain a better characterization of the learning error using our theory.
Revised: However, we will see below that we can obtain a better characterization of the learning error using our theory.

Original: Consider for example that the input has mean zero and the Mat´ern-type covariance operator C KK = σ 2 (cid:0) − ∆ + τ 2 I (cid:1) − s .
Revised: Consider for example that the input has mean zero and the Mat´ern-type covariance operator C KK = σ 2 (cid:0) − ∆ + τ 2 I (cid:1) − s .

Original: Its eigenvalues satisfy µ n ∝ n − 2 s .
Revised: Its eigenvalues satisfy µ n ∝ n − 2 s .

Original: On the other hand, we choose Q Y to be a distribution supported on { ℓ ( y, · ) : y ∈ [0 , 1] } induced by a uniform distribution on [0 , 1] , where ℓ is the kernel function of H L .
Revised: On the other hand, we choose Q Y to be a distribution supported on { ℓ ( y, · ) : y ∈ [0 , 1] } induced by a uniform distribution on [0 , 1] , where ℓ is the kernel function of H L .

Original: Then C Q Y is essentially the kernel integral operator on H L w.r.t.
Revised: Then C Q Y is essentially the kernel integral operator on H L w.r.t.

Original: the uniform distribution, and its eigenvalues are ρ n ∝ n − 2 m .
Revised: the uniform distribution, and its eigenvalues are ρ n ∝ n − 2 m .

Original: The assumption ∥A 0 ∥ β,γ < + ∞ is satisfied if and only if (1 − γ ) m < (1 − β )
Revised: The assumption ∥A 0 ∥ β,γ < + ∞ is satisfied if and only if (1 − γ ) m < (1 − β )

Original: s − 12 ( i.e. γ > 1 − 2(1 − β )
Revised: s − 12 ( i.e. γ > 1 − 2(1 − β )

Original: s − 1 2 m ).
Revised: s − 1 2 m ).

Original: Example 2.2 (Conditional mean embedding) Suppose that we would like to learn the conditional distribution P ( y | x ) from a data set { ( x i , y i ) : 1 ⩽ i ⩽ N } ⊂ X × Y where x i i .
Revised: Example 2.2 (Conditional mean embedding) Suppose that we would like to learn the conditional distribution P ( y | x ) from a data set { ( x i , y i ) : 1 ⩽ i ⩽ N } ⊂ X × Y where x i i .

Original: i .
Revised: i .

Original: d . ∼ P K .
Revised: d . ∼ P K .

Original: Let H K and H L be two RKHSs on X and Y respectively, with measurable kernel k ( · , · ) and ℓ ( · , · ) .
Revised: Let H K and H L be two RKHSs on X and Y respectively, with measurable kernel k ( · , · ) and ℓ ( · , · ) .

Original: Then we can define a conditional mean embedding (CME) operator C Y |
Revised: Then we can define a conditional mean embedding (CME) operator C Y |

Original: X that satisfies
Revised: X that satisfies

Original: [Equation]
Revised: [Equation]

Original: We choose A 0 = C Y | x .
Revised: We choose A 0 = C Y | x .

Original: In this case, C KK = E P K k ( X, · ) ⊗ k ( X, · ) .
Revised: In this case, C KK = E P K k ( X, · ) ⊗ k ( X, · ) .

Original: Assumption 2.2 states that sup x ∈ X k α
Revised: Assumption 2.2 states that sup x ∈ X k α

Original: ( x, x ) =
Revised: ( x, x ) =

Original: A 1 , while Assumption 2.3 is equivalent to sup x ∈ X (cid:13)(cid:13) µ Y | x (cid:13)(cid:13) ⩽
Revised: A 1 , while Assumption 2.3 is equivalent to sup x ∈ X (cid:13)(cid:13) µ Y | x (cid:13)(cid:13) ⩽

Original: A 2 (for simplicity we only focus on the case ζ = 1 ).
Revised: A 2 (for simplicity we only focus on the case ζ = 1 ).

Original: According to Assumption 2.5, we assume that (cid:13)(cid:13) C Y | X (cid:13)(cid:13) β,γ ⩽ B .
Revised: According to Assumption 2.5, we assume that (cid:13)(cid:13) C Y | X (cid:13)(cid:13) β,γ ⩽ B .

Original: The mis-specified setting where β < 1 has been studied in previous work (Fischer & Steinwart, 2020; Talwai et al., 2022; Li et al., 2022).
Revised: The mis-specified setting where β < 1 has been studied in previous work (Fischer & Steinwart, 2020; Talwai et al., 2022; Li et al., 2022).

Original: However, they only consider the case γ = 1 .
Revised: However, they only consider the case γ = 1 .

Original: Our results also cover the case γ < 1 , which allows us to obtain theoretical guarantee for computing conditional expectation of the larger function class H γL .
Revised: Our results also cover the case γ < 1 , which allows us to obtain theoretical guarantee for computing conditional expectation of the larger function class H γL .

Original:  I NFORMATION T HEORETIC L OWER B OUND
Revised:  I NFORMATION T HEORETIC L OWER B OUND

Original: In this section, we provide an information-theoretic lower bound via the Fano method for the convergence rate of the operator learning problem formulated in Section 2.
Revised: In this section, we provide an information-theoretic lower bound via the Fano method for the convergence rate of the operator learning problem formulated in Section 2.

Original: Theorem 3.1 Suppose that P K and Q L are probability distributions on Hilbert spaces H K and H L respectively such that Assumptions 2.1 and 2.2 hold.
Revised: Theorem 3.1 Suppose that P K and Q L are probability distributions on Hilbert spaces H K and H L respectively such that Assumptions 2.1 and 2.2 hold.

Original: Then for any estimator L : ( H K × H L ) ⊗ N (cid:55)→
Revised: Then for any estimator L : ( H K × H L ) ⊗ N (cid:55)→

Original: HS (cid:16) H βK , H γL (cid:17) , there exists a linear operator A 0 and a joint data distribution P KL with marginal distribution P K on H K satisfying Assumptions 2.3 to 2.5, such that with probability ⩾ 0 .
Revised: HS (cid:16) H βK , H γL (cid:17) , there exists a linear operator A 0 and a joint data distribution P KL with marginal distribution P K on H K satisfying Assumptions 2.3 to 2.5, such that with probability ⩾ 0 .

Original: 99 over ( u i , v i ) i .
Revised: 99 over ( u i , v i ) i .

Original: i .
Revised: i .

Original: d . ∼ P KL we have
Revised: d . ∼ P KL we have

Original: [Equation]
Revised: [Equation]

Original: Remark 3.1 Our lower bound is composed of a minimum of two parts.
Revised: Remark 3.1 Our lower bound is composed of a minimum of two parts.

Original: The first rate N − max { α,β }− β ′ max { α,β } + p is the minimax optimal Sobolev learning rate for kernel regression (Fischer & Steinwart, 2020; Talwai et al., 2022;
Revised: The first rate N − max { α,β }− β ′ max { α,β } + p is the minimax optimal Sobolev learning rate for kernel regression (Fischer & Steinwart, 2020; Talwai et al., 2022;

Original: Li et al., 2022; Lu et al., 2022) and is fully determined by the parameter of the input Sobolev reproducing kernel Hilbert space.
Revised: Li et al., 2022; Lu et al., 2022) and is fully determined by the parameter of the input Sobolev reproducing kernel Hilbert space.

Original: Our second rate N − γ ′− γ 1 − γ is novel to the literature.
Revised: Our second rate N − γ ′− γ 1 − γ is novel to the literature.

Original: This bound shows how the infinite-dimensional problem is different from the finite-dimensional regression problem and is fully determined by the output Sobolev reproducing kernel Hilbert space parameter.
Revised: This bound shows how the infinite-dimensional problem is different from the finite-dimensional regression problem and is fully determined by the output Sobolev reproducing kernel Hilbert space parameter.

Original: Our lower bound shows that the hardness of learning a linear operator is determined by the harder part between the input and output spaces.
Revised: Our lower bound shows that the hardness of learning a linear operator is determined by the harder part between the input and output spaces.

Original: We will explain why the lower bound has such a structure in Remark 4.2 and Figure 2.
Revised: We will explain why the lower bound has such a structure in Remark 4.2 and Figure 2.

Original:  O N THE S HAPE OF R EGULARIZATION
Revised:  O N THE S HAPE OF R EGULARIZATION

Original: In this section, we aim to understand the shape of regularization so that the constructed estimator ˆ A based on N i.i.d.
Revised: In this section, we aim to understand the shape of regularization so that the constructed estimator ˆ A based on N i.i.d.

Original: data { ( u i , v i ) } ni
Revised: data { ( u i , v i ) } ni

Original: =1 ∼ P ⊗ nKL for 1 ⩽
Revised: =1 ∼ P ⊗ nKL for 1 ⩽

Original: i ⩽ N enjoys an optimal learning rate.
Revised: i ⩽ N enjoys an optimal learning rate.

Original: Compared with existing approaches where a regularized least-squares estimator can achieve statistical optimality (Fischer & Steinwart, 2020; Talwai et al., 2022; Li et al., 2022; de Hoop et al., 2021) under ( β, 1) -norm, we study the learning rate under the ( β, γ ) -norm ( β ′ ∈ (0 , β ) , γ ′ ∈ ( γ, 1) ) which is defined in Definition 2.1 as (cid:13)(cid:13)(cid:13)
Revised: Compared with existing approaches where a regularized least-squares estimator can achieve statistical optimality (Fischer & Steinwart, 2020; Talwai et al., 2022; Li et al., 2022; de Hoop et al., 2021) under ( β, 1) -norm, we study the learning rate under the ( β, γ ) -norm ( β ′ ∈ (0 , β ) , γ ′ ∈ ( γ, 1) ) which is defined in Definition 2.1 as (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: The norm of the additional C − 1 − γ
Revised: The norm of the additional C − 1 − γ

Original: [Equation]
Revised: [Equation]

Original: 2 Q L term is unbounded which make our setting harder than the convergence in ( β, 1) -norm in existing works.
Revised: 2 Q L term is unbounded which make our setting harder than the convergence in ( β, 1) -norm in existing works.

Original: Since C − 1 − γ ′ 2 Q L is bounded when restricted to the finite-dimensional space span (cid:16) ρ 12 i f i : 1 ⩽ i ⩽ n (cid:17) , we should also include another bias-variance trade-off via regularizing in the output shape.
Revised: Since C − 1 − γ ′ 2 Q L is bounded when restricted to the finite-dimensional space span (cid:16) ρ 12 i f i : 1 ⩽ i ⩽ n (cid:17) , we should also include another bias-variance trade-off via regularizing in the output shape.

Original: As a result, we are interested in answering the following question
Revised: As a result, we are interested in answering the following question

Original: What is the optimal way to combine the regularization in the input space and regularization in the output space?
Revised: What is the optimal way to combine the regularization in the input space and regularization in the output space?

Original: i.e. What is the optimal shape of regularization?
Revised: i.e. What is the optimal shape of regularization?

Original: To answer this question, we investigate the problem in the spectral space, i.e. considering the spectral representation of operator
Revised: To answer this question, we investigate the problem in the spectral space, i.e. considering the spectral representation of operator

Original: A 

Revised: A

Original: = (cid:80)
Revised: = (cid:80)

Original: +
Revised: +

Original: ∞ i,j =1 a ij µ β 2 i e i ⊗ ρ 1 −
Revised: ∞ i,j =1 a ij µ β 2 i e i ⊗ ρ 1 −

Original: [Equation]
Revised: [Equation]

Original: 2 j f j .
Revised: 2 j f j .

Original: The problem of estimating A then reduces to learning the coefficients “matrix” ( a ij ) ∞ i,j =1 .
Revised: The problem of estimating A then reduces to learning the coefficients “matrix” ( a ij ) ∞ i,j =1 .

Original: The source condition Assumption 2.
Revised: The source condition Assumption 2.

Original: enforces (cid:80) ∞ i,j =1 a 2 ij ≤ B .
Revised: enforces (cid:80) ∞ i,j =1 a 2 ij ≤ B .

Original: We show in Appendix B.1.1 that regularizing the basis e i ⊗ f j will introduce a bias of order (cid:13)(cid:13)(cid:13)(cid:13)
Revised: We show in Appendix B.1.1 that regularizing the basis e i ⊗ f j will introduce a bias of order (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: under the ( β ′ , γ ′ ) -norm.
Revised: under the ( β ′ , γ ′ ) -norm.

Original: On the other hand, when α ⩽ β + p , we show in Appendix B.1.2 that the variance of learning ( i, j ) from noisy data scales as 1 N µ − β ′
Revised: On the other hand, when α ⩽ β + p , we show in Appendix B.1.2 that the variance of learning ( i, j ) from noisy data scales as 1 N µ − β ′

Original: i ρ
Revised: i ρ

Original: [Equation]
Revised: [Equation]

Original: ∝
Revised: ∝

Original: [Equation]
Revised: [Equation]

Original: N i
Revised: N i

Original: [Equation]
Revised: [Equation]

Original: p j
Revised: p j

Original: [Equation]
Revised: [Equation]

Original: q .
Revised: q .

Original: Since the variance would accumulate for a fixed j , learning ( i, j ) for i ⩽ i max results in a variance of ∝ 1 N i β
Revised: Since the variance would accumulate for a fixed j , learning ( i, j ) for i ⩽ i max results in a variance of ∝ 1 N i β

Original: [Equation]
Revised: [Equation]

Original: (Similar analysis can be carried out for the α > β + p case as well, but the variance now scales as 1 N i
Revised: (Similar analysis can be carried out for the α > β + p case as well, but the variance now scales as 1 N i

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: p max j 1 − γ ′ q ; see Appendix B for detailed derivations.)
Revised: p max j 1 − γ ′ q ; see Appendix B for detailed derivations.)

Original: In summary, we need to make biasvariance trade off in the ( i, j ) − plane, i.e. decide whether we should learn or regularize over the basis e i ⊗ f j .
Revised: In summary, we need to make biasvariance trade off in the ( i, j ) − plane, i.e. decide whether we should learn or regularize over the basis e i ⊗ f j .

Original: 4. R EGULARIZATION VIA VARIANCE CONTOUR
Revised: 4. R EGULARIZATION VIA VARIANCE CONTOUR

Original: The underlying idea of regularization is that some components are intrinsically hard to learn due to large variance; these components are then neglected by adding regularization and are counted as bias.
Revised: The underlying idea of regularization is that some components are intrinsically hard to learn due to large variance; these components are then neglected by adding regularization and are counted as bias.

Original: Thus, the remaining components are easy to learn due to controllable variance.
Revised: Thus, the remaining components are easy to learn due to controllable variance.

Original: This intuition works well when the estimation error results from the noise of the data and is well-studied in a line of works (Fischer & Steinwart, 2020; de Hoop et al., 2021;
Revised: This intuition works well when the estimation error results from the noise of the data and is well-studied in a line of works (Fischer & Steinwart, 2020; de Hoop et al., 2021;

Original: Talwai et al., 2022; Li et al., 2022).
Revised: Talwai et al., 2022; Li et al., 2022).

Original: This idea still works in our setting, but we need to re-evaluate the bias and variance of each component.
Revised: This idea still works in our setting, but we need to re-evaluate the bias and variance of each component.

Original: Since we work with the Hilbert-Schmidt norm, this can be done in a coordinate-wise manner, meaning that we can look at each a ij separately and decide whether to neglect it (contribute to bias) or to learn it from data (contribute to variance).
Revised: Since we work with the Hilbert-Schmidt norm, this can be done in a coordinate-wise manner, meaning that we can look at each a ij separately and decide whether to neglect it (contribute to bias) or to learn it from data (contribute to variance).

Original: Since the variance term measures the hardness of learning, we naturally introduce the notion of variance contour , which is a curve on the R 2+ plane on which all points induce the same order of variance (here we work with real coordinates for convenience, although we only care about integer points).
Revised: Since the variance term measures the hardness of learning, we naturally introduce the notion of variance contour , which is a curve on the R 2+ plane on which all points induce the same order of variance (here we work with real coordinates for convenience, although we only care about integer points).

Original: Formally, we fix an arbitrary constant C > 0 and define
Revised: Formally, we fix an arbitrary constant C > 0 and define

Original: [Equation]
Revised: [Equation]

Original: A reasonable regularization scheme is then to learn all coordinates ( i,
Revised: A reasonable regularization scheme is then to learn all coordinates ( i,

Original: j ) ∈
Revised: j ) ∈

Original: Z 2+ below the curve ℓ C, var and ‘regularize out’ the remaining coordinates that are difficult to learn due to large variances.
Revised: Z 2+ below the curve ℓ C, var and ‘regularize out’ the remaining coordinates that are difficult to learn due to large variances.

Original: This can gives us the estimator with smallest estimator at give variance level.
Revised: This can gives us the estimator with smallest estimator at give variance level.

Original: This observation motivates us to construct our estimator as
Revised: This observation motivates us to construct our estimator as

Original: [Equation]
Revised: [Equation]

Original: where ˆ C LK = 1 N (cid:80) Ni
Revised: where ˆ C LK = 1 N (cid:80) Ni

Original: =1 v i ⊗ u i , λ j (1 ⩽ j ⩽
Revised: =1 v i ⊗ u i , λ j (1 ⩽ j ⩽

Original: y N = C q 1 − γ ′ ) are the regularization coefficients imposed on different dimensions of the output space.
Revised: y N = C q 1 − γ ′ ) are the regularization coefficients imposed on different dimensions of the output space.

Original: According to (2) and noting that µ i ∝ i − 1 p , we define
Revised: According to (2) and noting that µ i ∝ i − 1 p , we define

Original: [Equation]
Revised: [Equation]

Original: with C = N max (cid:110) 1 − β − β ′ max { α,β + p } , 1 − γ ′ 1 − γ (cid:111) in (2).
Revised: with C = N max (cid:110) 1 − β − β ′ max { α,β + p } , 1 − γ ′ 1 − γ (cid:111) in (2).

Original: The additional N − 1 α term in (4) is needed for controlling the error of approximating C KK via ˆ C KK (cf.
Revised: The additional N − 1 α term in (4) is needed for controlling the error of approximating C KK via ˆ C KK (cf.

Original: Theorem D.3) which is standard in the Sobolev learning literature Fischer & Steinwart (2020); Talwai et al.
Revised: Theorem D.3) which is standard in the Sobolev learning literature Fischer & Steinwart (2020); Talwai et al.

Original: ; Lu et al.
Revised: ; Lu et al.

Original: The following theorem describes the convergence rate of our estimator defined by (3) and (4).
Revised: The following theorem describes the convergence rate of our estimator defined by (3) and (4).

Original: Theorem 4.1 Consider the estimator ˆ A defined by (3) and (4) .
Revised: Theorem 4.1 Consider the estimator ˆ A defined by (3) and (4) .

Original: Suppose that Assumptions 2.
Revised: Suppose that Assumptions 2.

Original: to 2.5 hold, then there exists a universal constant C such that with probability ⩾ 1 − e − τ , we have
Revised: to 2.5 hold, then there exists a universal constant C such that with probability ⩾ 1 − e − τ , we have

Original: [Equation]
Revised: [Equation]

Original: Remark 4.1 Compared with Theorem 3.1, our upper bound is optimal up to logarithmic factors when α ⩽ β .
Revised: Remark 4.1 Compared with Theorem 3.1, our upper bound is optimal up to logarithmic factors when α ⩽ β .

Original: The optimal learning rate in the α > β regime is an outstanding problem for decades, even without the additional problem-dependent parameters γ, γ ′ (see e.g. the discussions following (Fischer & Steinwart, 2020, Theorem 2)).
Revised: The optimal learning rate in the α > β regime is an outstanding problem for decades, even without the additional problem-dependent parameters γ, γ ′ (see e.g. the discussions following (Fischer & Steinwart, 2020, Theorem 2)).

Original: In this paper, we do not address this problem either.
Revised: In this paper, we do not address this problem either.

Original: 4. R EGULARIZATION VIA B IAS C ONTOUR
Revised: 4. R EGULARIZATION VIA B IAS C ONTOUR

Original: We have showed that if we learn all the spectral components under certain variance contour and regularize all other component can achieve optimal rate.
Revised: We have showed that if we learn all the spectral components under certain variance contour and regularize all other component can achieve optimal rate.

Original: In this section, we introduce another scheme to design the optimal estimator via learning all the spectral component under a certain bias contour.
Revised: In this section, we introduce another scheme to design the optimal estimator via learning all the spectral component under a certain bias contour.

Original: Specifically, we consider deciding the regularization strength according to the spectral elements induce a certain level of bias i.e. the bias contour ℓ C ′ , bias = (cid:110) (
Revised: Specifically, we consider deciding the regularization strength according to the spectral elements induce a certain level of bias i.e. the bias contour ℓ C ′ , bias = (cid:110) (

Original: x, y ) ∈
Revised: x, y ) ∈

Original: R 2+ : x β
Revised: R 2+ : x β

Original: [Equation]
Revised: [Equation]

Original: does not coincide with ℓ C, var for any C ′ up to constant scaling.
Revised: does not coincide with ℓ C, var for any C ′ up to constant scaling.

Original: Thus, there exists a point
Revised: Thus, there exists a point

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ( x ∗ , y ∗ ) the variance contour with maximal contribution to bias.
Revised: ( x ∗ , y ∗ ) the variance contour with maximal contribution to bias.

Original: Naturally, we can also construct our estimator using a bias contour that passes through ( x ∗ , y ∗ ) .
Revised: Naturally, we can also construct our estimator using a bias contour that passes through ( x ∗ , y ∗ ) .

Original: In this case, we may define λ j = max (cid:40)(cid:18)
Revised: In this case, we may define λ j = max (cid:40)(cid:18)

Original: [Figure]
Revised: [Figure]

Original: for similar reasons as Section 4.1, which also yields optimal rate as stated in Theorem 4.2 below.
Revised: for similar reasons as Section 4.1, which also yields optimal rate as stated in Theorem 4.2 below.

Original: Remark 4.2 (On the optimal shape of regularization)
Revised: Remark 4.2 (On the optimal shape of regularization)

Original: The discussion in Sections 4.1 and 4. reveals another understanding of our information theoretic lower bound.
Revised: The discussion in Sections 4.1 and 4. reveals another understanding of our information theoretic lower bound.

Original: Firstly, we should learn all the spectral components under the bias contour otherwise the bias will exceed the lower bound.
Revised: Firstly, we should learn all the spectral components under the bias contour otherwise the bias will exceed the lower bound.

Original: Secondly, we should not learn any spectral component over the variance contour since otherwise the variance will exceed the lower bound.
Revised: Secondly, we should not learn any spectral component over the variance contour since otherwise the variance will exceed the lower bound.

Original: Thus the bias contour should always be under the variance contour, otherwise no estimator can be designed.
Revised: Thus the bias contour should always be under the variance contour, otherwise no estimator can be designed.

Original: The bias and variance contours at the level of optimal learning rate are plotted in Figure 2.
Revised: The bias and variance contours at the level of optimal learning rate are plotted in Figure 2.

Original: They only meet at ( x ∗ , y ∗ ) with x ∗ = 1 or y ∗ = 1 , which has the largest contribution to the bias (resp. variance) among all points on the variance (resp. bias) contour, thus dominating the estimation error.
Revised: They only meet at ( x ∗ , y ∗ ) with x ∗ = 1 or y ∗ = 1 , which has the largest contribution to the bias (resp. variance) among all points on the variance (resp. bias) contour, thus dominating the estimation error.

Original: When the two curves meet at y ∗ = 1 , it reduces to the original kernel regression case.
Revised: When the two curves meet at y ∗ = 1 , it reduces to the original kernel regression case.

Original: When the two curves meet at x ∗ = 1 , it leads to our new rate that depends on the output space.
Revised: When the two curves meet at x ∗ = 1 , it leads to our new rate that depends on the output space.

Original: Theorem 4.2 Consider the estimator ˆ A defined by (3) with λ j defined above.
Revised: Theorem 4.2 Consider the estimator ˆ A defined by (3) with λ j defined above.

Original: Suppose that Assumptions 2.1 to 2.5 hold, then there exists a universal constant C , such that (cid:13)(cid:13)(cid:13) ˆ A − A 0 (cid:13)(cid:13)(cid:13) 2 β ′ ,γ ′ ⩽
Revised: Suppose that Assumptions 2.1 to 2.5 hold, then there exists a universal constant C , such that (cid:13)(cid:13)(cid:13) ˆ A − A 0 (cid:13)(cid:13)(cid:13) 2 β ′ ,γ ′ ⩽

Original: [Equation]
Revised: [Equation]

Original: with probability ⩾
Revised: with probability ⩾

Original: [Equation]
Revised: [Equation]

Original:  M ULTI L EVEL K ERNEL O PERATOR L EARNING
Revised:  M ULTI L EVEL K ERNEL O PERATOR L EARNING

Original: In this section, we study a multilevel machine learning algorithm Lye et al.
Revised: In this section, we study a multilevel machine learning algorithm Lye et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2021a);
Revised: (2021a);

Original: Boull´e et al.
Revised: Boull´e et al.

Original: but at each level we consider a cost-accuracy trade-off De Hoop et al.
Revised: but at each level we consider a cost-accuracy trade-off De Hoop et al.

Original: (2022) to control the variance at a proper scale.
Revised: (2022) to control the variance at a proper scale.

Original: We show that the multilevel level algorithm can cover all the spectral component below the bias contour and achieve the optimal learning rate.
Revised: We show that the multilevel level algorithm can cover all the spectral component below the bias contour and achieve the optimal learning rate.

Original: Our idea is similar to the multilevel Monte Carlo Giles (2008; 2015), which reduces bias from multilevel algorithm.
Revised: Our idea is similar to the multilevel Monte Carlo Giles (2008; 2015), which reduces bias from multilevel algorithm.

Original: Our multilevel estimator differs from the DeepONet (Lu et al., 2019) and the PCA-Net (Bhattacharya et al., 2020) since we add different regularizations for each level.
Revised: Our multilevel estimator differs from the DeepONet (Lu et al., 2019) and the PCA-Net (Bhattacharya et al., 2020) since we add different regularizations for each level.

Original: Our theory indicates that the multilevel approach outperforms previous ones and achieves the optimal learning rate.
Revised: Our theory indicates that the multilevel approach outperforms previous ones and achieves the optimal learning rate.

Original: The basic idea is to design a minimum number of machine learning estimators that cover all the spectral elements under the bias contour but do not exceed the variance contour at the same time.
Revised: The basic idea is to design a minimum number of machine learning estimators that cover all the spectral elements under the bias contour but do not exceed the variance contour at the same time.

Original: To achieve this, we choose sequences { x i } and { y i } for 1 ⩽
Revised: To achieve this, we choose sequences { x i } and { y i } for 1 ⩽

Original: i ⩽ L N where y i denotes the i -th level and x i controls the corresponding regularization via the regularization coefficient
Revised: i ⩽ L N where y i denotes the i -th level and x i controls the corresponding regularization via the regularization coefficient

Original: λ ( K ) i
Revised: λ ( K ) i

Original: [Equation]
Revised: [Equation]

Original: The sequences are chosen in a staircase manner as plotted in Figure 3 (for formal definitions see
Revised: The sequences are chosen in a staircase manner as plotted in Figure 3 (for formal definitions see

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , where the bias and variance contours overlap and we set x n +1 = 12 x n .
Revised: , where the bias and variance contours overlap and we set x n +1 = 12 x n .

Original: Each rectangular represents a certain level of regularization.
Revised: Each rectangular represents a certain level of regularization.

Original: Appendix C).
Revised: Appendix C).

Original: The eigenbasis (cid:110) ρ 12 j f j (cid:111) of the output space is divided into defferent levels by { y i } .
Revised: The eigenbasis (cid:110) ρ 12 j f j (cid:111) of the output space is divided into defferent levels by { y i } .

Original: The main idea behind our multilevel method is that different levels of the output need to be learned with different regularization.
Revised: The main idea behind our multilevel method is that different levels of the output need to be learned with different regularization.

Original: Formally, we define our multilevel estimator as
Revised: Formally, we define our multilevel estimator as

Original: [Equation]
Revised: [Equation]

Original: The following theorem shows that the estimator (5) can achieve the optimal convergence rate with
Revised: The following theorem shows that the estimator (5) can achieve the optimal convergence rate with

Original: L N =
Revised: L N =

Original: O
Revised: O

Original: (ln ln N )
Revised: (ln ln N )

Original: when
Revised: when

Original: [Equation]
Revised: [Equation]

Original: We also show that O (ln N ) estimator is needed for the case when
Revised: We also show that O (ln N ) estimator is needed for the case when

Original: [Equation]
Revised: [Equation]

Original: 3 Right) in Appendix C.
Revised: 3 Right) in Appendix C.

Original: Theorem 5.1 Suppose that Assumptions 2.1 to 2.5 hold, then there exists a sequence { y i } 1 ⩽
Revised: Theorem 5.1 Suppose that Assumptions 2.1 to 2.5 hold, then there exists a sequence { y i } 1 ⩽

Original: i ⩽ L N with L N
Revised: i ⩽ L N with L N

Original: =
Revised: =

Original: O
Revised: O

Original: (ln N )
Revised: (ln N )

Original: when
Revised: when

Original: [Equation]
Revised: [Equation]

Original: and O (ln ln N ) otherwise, such that the estimator ˆ A ml satisfies
Revised: and O (ln ln N ) otherwise, such that the estimator ˆ A ml satisfies

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: 1 − e − τ , where C is a universal constant.
Revised: 1 − e − τ , where C is a universal constant.

Original: Remark 5.1 Our multilevel algorithm first apply the regression algorithm on low-frequency projections of the output samples with small regularization and then successively fine-tune the regression model on high-frequency projections of the output samples with stronger regularization, which matches the empirical use (Li et al., 2021a; Lye et al., 2021).
Revised: Remark 5.1 Our multilevel algorithm first apply the regression algorithm on low-frequency projections of the output samples with small regularization and then successively fine-tune the regression model on high-frequency projections of the output samples with stronger regularization, which matches the empirical use (Li et al., 2021a; Lye et al., 2021).

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: We considered the sample complexity of learning an operator between two infinite-dimensional
Revised: We considered the sample complexity of learning an operator between two infinite-dimensional

Original: Sobolev kernel Hilbert spaces.
Revised: Sobolev kernel Hilbert spaces.

Original: We provided an information theoretical lower bound for this problem along with a multi-level machine learning algorithm.
Revised: We provided an information theoretical lower bound for this problem along with a multi-level machine learning algorithm.

Original: Our lower bound is determined by the harder rate of two polynomial rates: one is fully determined by the hardness of the input space, while the other is fully controlled by the hardness of the output space.
Revised: Our lower bound is determined by the harder rate of two polynomial rates: one is fully determined by the hardness of the input space, while the other is fully controlled by the hardness of the output space.

Original: The second rate is new to the literature.
Revised: The second rate is new to the literature.

Original: We explained our bound from the viewpoint of variance and bias counters in Remark 4.
Revised: We explained our bound from the viewpoint of variance and bias counters in Remark 4.

Original: and Figure 2.
Revised: and Figure 2.

Original: The optimal estimator should learn all the spectral elements under the bias contour but learn no information above the variance contour.
Revised: The optimal estimator should learn all the spectral elements under the bias contour but learn no information above the variance contour.

Original: To meet this requirement, we combined the idea of multi-level Monte Carlo with kernel operator learning, using successive levels to fit higher frequency information while keeping the variance at the same scale to reduce the bias.
Revised: To meet this requirement, we combined the idea of multi-level Monte Carlo with kernel operator learning, using successive levels to fit higher frequency information while keeping the variance at the same scale to reduce the bias.

Original: Our paper is the first on the non-parametric statistical optimality for multi-level algorithms.
Revised: Our paper is the first on the non-parametric statistical optimality for multi-level algorithms.

Original: We leave estimation from discretely observed functional covariates with noise as future work (Zhou et al.; 2022).
Revised: We leave estimation from discretely observed functional covariates with noise as future work (Zhou et al.; 2022).

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Jikai Jin is partially supported by the elite undergraduate training program of School of Mathematical
Revised: Jikai Jin is partially supported by the elite undergraduate training program of School of Mathematical

Original: Sciences in Peking University.
Revised: Sciences in Peking University.

Original: Yiping Lu is supported by the Stanford Interdisciplinary Graduate
Revised: Yiping Lu is supported by the Stanford Interdisciplinary Graduate

Original: Fellowship (SIGF).
Revised: Fellowship (SIGF).

Original: Jose Blanchet is supported in part by the Air Force Office of Scientific Research under award number FA9550-20-1-0397.
Revised: Jose Blanchet is supported in part by the Air Force Office of Scientific Research under award number FA9550-20-1-0397.

Original: Lexing Ying is supported is supported by National Science
Revised: Lexing Ying is supported is supported by National Science

Original: Foundation under award DMS-2208163.
Revised: Foundation under award DMS-2208163.

Original: A P ROOF OF THE LOWER BOUND
Revised: A P ROOF OF THE LOWER BOUND

Original: In this section, we follow the lower bound proof in Fischer & Steinwart (2020) to give a lower bound of the convergence rate in our operator learning setting.
Revised: In this section, we follow the lower bound proof in Fischer & Steinwart (2020) to give a lower bound of the convergence rate in our operator learning setting.

Original: A. P RELIMINARIES ON T OOLS FOR L OWER B OUNDS
Revised: A. P RELIMINARIES ON T OOLS FOR L OWER B OUNDS

Original: In this section, we repeat the standard tools we use to establish the lower bound.
Revised: In this section, we repeat the standard tools we use to establish the lower bound.

Original: The main tool we use is the Fano’s inequality and the Varshamov-Gilber Lemma.
Revised: The main tool we use is the Fano’s inequality and the Varshamov-Gilber Lemma.

Original: Lemma A.1 (Fano’s methods) Assume that V is a uniform random variable over set V , then for any Markov chain V
Revised: Lemma A.1 (Fano’s methods) Assume that V is a uniform random variable over set V , then for any Markov chain V

Original: → X → ˆ V , we always have
Revised: → X → ˆ V , we always have

Original: [Equation]
Revised: [Equation]

Original: In our proof we will use a version from Fischer & Steinwart (2020).
Revised: In our proof we will use a version from Fischer & Steinwart (2020).

Original: Lemma A.2 (Fischer & Steinwart, 2020, Theorem 20) Let M ⩾ 2 , (Ω , A ) be a measurable space,
Revised: Lemma A.2 (Fischer & Steinwart, 2020, Theorem 20) Let M ⩾ 2 , (Ω , A ) be a measurable space,

Original: P 0 , P 1 , . .
Revised: P 0 , P 1 , . .

Original: , P M be probability measures on (Ω , A ) with P j ≪ P 0 for all j = 1 , . . .
Revised: , P M be probability measures on (Ω , A ) with P j ≪ P 0 for all j = 1 , . . .

Original: , M , and 0 < α ∗ < ∞ with
Revised: , M , and 0 < α ∗ < ∞ with

Original: [Equation]
Revised: [Equation]

Original: Then, for all measurable functions Ψ : Ω → { 0 , 1 , . . .
Revised: Then, for all measurable functions Ψ : Ω → { 0 , 1 , . . .

Original: , M } , the following bound is satisfied
Revised: , M } , the following bound is satisfied

Original: [Equation]
Revised: [Equation]

Original: Lemma A.3 (Varshamov-Gillbert Lemma,Tsybakov (2008) Theorem 2.9) Let D ≥ 8 .
Revised: Lemma A.3 (Varshamov-Gillbert Lemma,Tsybakov (2008) Theorem 2.9) Let D ≥ 8 .

Original: There exists a subset V = { τ (0) , · · · , τ (2 D/ 8 ) } of D − dimensional hypercube H D
Revised: There exists a subset V = { τ (0) , · · · , τ (2 D/ 8 ) } of D − dimensional hypercube H D

Original: = { 0 , 1 } D such that τ (0) =
Revised: = { 0 , 1 } D such that τ (0) =

Original: (0 , 0 , · · · , 0) and the ℓ 1 distance between every two elements is larger than D
Revised: (0 , 0 , · · · , 0) and the ℓ 1 distance between every two elements is larger than D

Original: D (cid:88)
Revised: D (cid:88)

Original: [Equation]
Revised: [Equation]

Original: , for all
Revised: , for all

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF THE L OWER B OUND
Revised: A. P ROOF OF THE L OWER B OUND

Original: To prove our lower bound, we construct a sequence of linear operators as follows:
Revised: To prove our lower bound, we construct a sequence of linear operators as follows:

Original: [Equation]
Revised: [Equation]

Original: where m 1 and m 2 are hyper-parameters (scale as poly( N ) and will be selected later) and K is a constant that will be specified afterwards.
Revised: where m 1 and m 2 are hyper-parameters (scale as poly( N ) and will be selected later) and K is a constant that will be specified afterwards.

Original: It’s easy to check that
Revised: It’s easy to check that

Original: [Equation]
Revised: [Equation]

Original: By Gilbert-Varshamov Lemma it is possible to select M ε ⩾ 2 m 1 K/ 8 binary strings
Revised: By Gilbert-Varshamov Lemma it is possible to select M ε ⩾ 2 m 1 K/ 8 binary strings

Original: [Equation]
Revised: [Equation]

Original: such that (cid:13)(cid:13) ω ( i ) − ω
Revised: such that (cid:13)(cid:13) ω ( i ) − ω

Original: ( j ) (cid:13)(cid:13) 2 2 ⩾ 4 ε .
Revised: ( j ) (cid:13)(cid:13) 2 2 ⩾ 4 ε .

Original: Let Ω be the collection of this strings.
Revised: Let Ω be the collection of this strings.

Original: We now select the hyper-parameters to satisfies the assumptions made in Section 2.
Revised: We now select the hyper-parameters to satisfies the assumptions made in Section 2.

Original: First we have
Revised: First we have

Original: [Equation]
Revised: [Equation]

Original: where the last step follows from Assumption 2.1.
Revised: where the last step follows from Assumption 2.1.

Original: Similarly, we have ∥A ω ∥ 2 α,≲ ε (2 m 1 ) α
Revised: Similarly, we have ∥A ω ∥ 2 α,≲ ε (2 m 1 ) α

Original: [Equation]
Revised: [Equation]

Original: β ′ p (2 m 2 )
Revised: β ′ p (2 m 2 )

Original: [Equation]
Revised: [Equation]

Original: 1 q .
Revised: 1 q .

Original: To the assumptions made in Section 2, we should make
Revised: To the assumptions made in Section 2, we should make

Original: [Equation]
Revised: [Equation]

Original: be satisfied.
Revised: be satisfied.

Original: To be specific, with the previous selection of hyper-parameters, we can have ∥A ω ∥ β,γ =
Revised: To be specific, with the previous selection of hyper-parameters, we can have ∥A ω ∥ β,γ =

Original: O (1) and
Revised: O (1) and

Original: [Equation]
Revised: [Equation]

Original: where the last step follows from our assumption on the input distribution Assumption 2.2.
Revised: where the last step follows from our assumption on the input distribution Assumption 2.2.

Original: This verifies that Assumptions 2.3 and 2.5 hold for A ω , ∀ ω ∈ Ω .
Revised: This verifies that Assumptions 2.3 and 2.5 hold for A ω , ∀ ω ∈ Ω .

Original: We now construct the hypothesis (probability distributions) as follows: for ∀ ω ∈ { 0 , 1 } m 1 , define
Revised: We now construct the hypothesis (probability distributions) as follows: for ∀ ω ∈ { 0 , 1 } m 1 , define

Original: P ω (d f, d g
Revised: P ω (d f, d g

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: d N
Revised: d N

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where the covariance operator Σ
Revised: ) where the covariance operator Σ

Original: =
Revised: =

Original: σ 2 K (cid:80) Kj =1 ρ j + m 2 f j + m 2 ⊗ f j + m 2 for some constant σ > 0 .
Revised: σ 2 K (cid:80) Kj =1 ρ j + m 2 f j + m 2 ⊗ f j + m 2 for some constant σ > 0 .

Original: It’s then easy to see that tr (Σ) =
Revised: It’s then easy to see that tr (Σ) =

Original: σ 2 , which satisfies Assumption 2.4 .Note that the range of A ω is span( f m 2 ) and Σ is non-degenerate on this subspace.
Revised: σ 2 , which satisfies Assumption 2.4 .Note that the range of A ω is span( f m 2 ) and Σ is non-degenerate on this subspace.

Original: As a result, we can view P ω , ω ∈ Ω as distributions on H K × span ( f j + m 2 : 1 ⩽ j ⩽ K ) , and we have for ∀ ω, ω
Revised: As a result, we can view P ω , ω ∈ Ω as distributions on H K × span ( f j + m 2 : 1 ⩽ j ⩽ K ) , and we have for ∀ ω, ω

Original: ′ ∈
Revised: ′ ∈

Original: Ω
Revised: Ω

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: where the last step follows from E P K u ⊗ u = C
Revised: where the last step follows from E P K u ⊗ u = C

Original: KK = (cid:80) ∞
Revised: KK = (cid:80) ∞

Original: i =1 µ 2 i e i ⊗ e i and recall that K is a constant.
Revised: i =1 µ 2 i e i ⊗ e i and recall that K is a constant.

Original: Hence we deduce that
Revised: Hence we deduce that

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma A.2, we find that when
Revised: Applying Lemma A.2, we find that when

Original: [Equation]
Revised: [Equation]

Original: there exists a hypothesis P ω 0 such that for any estimator ˆ C ω 0 ,
Revised: there exists a hypothesis P ω 0 such that for any estimator ˆ A ω 0 ,

Original: [Equation]
Revised: [Equation]

Original: holds with high probability.
Revised: holds with high probability.

Original: Finally, we need to choose optimal m 1 and m 2 under the constraint (6).
Revised: Finally, we need to choose optimal m 1 and m 2 under the constraint (6).

Original: It turns out that either m 1 = 1 or m 2 = 1 , and the resulting lower bound is
Revised: It turns out that either m 1 = 1 or m 2 = 1 , and the resulting lower bound is

Original: [Equation]
Revised: [Equation]

Original: B P ROOF OF THE UPPER BOUND
Revised: B P ROOF OF THE UPPER BOUND

Original: In this section, we upper-bound the learning error of estimator (3) which defined as
Revised: In this section, we upper-bound the learning error of estimator (3) which defined as

Original: [Equation]
Revised: [Equation]

Original: where λ j , 1 ⩽ j ⩽
Revised: where λ j , 1 ⩽ j ⩽

Original: y N = N q
Revised: y N = N q

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: { α,β + p } , 1 − γ ′ 1 − γ } are regularization coefficients that we impose on different dimensions of the output space.
Revised: { α,β + p } , 1 − γ ′ 1 − γ } are regularization coefficients that we impose on different dimensions of the output space.

Original: In this section, we consider the following two ways to select regularization coefficients in Section 4:
Revised: In this section, we consider the following two ways to select regularization coefficients in Section 4:

Original: • We regularize all spectral component below certain variance contour,
Revised: • We regularize all spectral component below certain variance contour,

Original: i
Revised: i

Original: e. we set regularization strength
Revised: e. we set regularization strength

Original: λ j = max
Revised: λ j = max

Original: [Equation]
Revised: [Equation]

Original: • We regularize all spectral component below certain bias contour, i.e. we set regularization strength
Revised: • We regularize all spectral component below certain bias contour, i.e. we set regularization strength

Original: λ j = max
Revised: λ j = max

Original: [Equation]
Revised: [Equation]

Original: To obtain the upper bound for our estimator, we decompose the learning error E ( ˆ A ) =
Revised: To obtain the upper bound for our estimator, we decompose the learning error E ( ˆ A ) =

Original: (cid:13)(cid:13)(cid:13) ˆ A − A
Revised: (cid:13)(cid:13)(cid:13) ˆ A − A

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: γ ′
Revised: γ ′

Original: in
Revised: in

Original: to bias and variance via
Revised: to bias and variance via

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: [Equation]
Revised: [Equation]

Original: B. R EGULARIZATION VIA V ARIANCE C OUNTER
Revised: B. R EGULARIZATION VIA V ARIANCE C OUNTER

Original: In the following, we separately bound the bias term and the variance term.
Revised: In the following, we separately bound the bias term and the variance term.

Original: We first assume α ⩽ β + p in Appendix B.1.1 and Appendix B.1.2, then the case α > β + p is treated in Appendix B.1.3.
Revised: We first assume α ⩽ β + p in Appendix B.1.1 and Appendix B.1.2, then the case α > β + p is treated in Appendix B.1.3.

Original: Finally in Appendix B.2, we establish the same convergence rate for regularization via bias contour.
Revised: Finally in Appendix B.2, we establish the same convergence rate for regularization via bias contour.

Original: B.1. B IAS
Revised: B.1. B IAS

Original: Lemma
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: Proof sketch : Since ∥A 0 ∥ β,γ ⩽ B , we can write A 0 :=
Revised: Proof sketch : Since ∥A 0 ∥ β,γ ⩽ B , we can write A 0 :=

Original: (cid:80) + ∞ i =1 (cid:80) + ∞
Revised: (cid:80) + ∞ i =1 (cid:80) + ∞

Original: j =1 a ij µ β 

Revised: j =1 a ij µ β

Original: i ρ 1 − γ 

Revised: i ρ 1 − γ

Original: j f j ⊗ e i where the coefficient matrix
Revised: j f j ⊗ e i where the coefficient matrix

Original: A 

Revised: A

Original: = ( a ij ) 1 ⩽
Revised: = ( a ij ) 1 ⩽

Original: i,j ⩽ + ∞ satisfies ∥
Revised: i,j ⩽ + ∞ satisfies ∥

Original: A 0 ∥ 2 F ⩽ B 2 .
Revised: A 0 ∥ 2 F ⩽ B 2 .

Original: The definition (8) implies that for 1 ⩽ j ⩽ y N
Revised: The definition (8) implies that for 1 ⩽ j ⩽ y N

Original: and i ⩾ 

Revised: and i ⩾

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: The bias term can be bounded as follows:
Revised: The bias term can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: We now prove that
Revised: We now prove that

Original: [Equation]
Revised: [Equation]

Original: Case 1.
Revised: Case 1.

Original: If λ j = c 0 (cid:16) N log N (cid:17) 1 α , then
Revised: If λ j = c 0 (cid:16) N log N (cid:17) 1 α , then

Original: [Equation]
Revised: [Equation]

Original: where we use α ⩽ β + p in the final step.
Revised: where we use α ⩽ β + p in the final step.

Original: Case 2.
Revised: Case 2.

Original: If λ j = (cid:18)
Revised: If λ j = (cid:18)

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: p , we need to consider two sub-cases:
Revised: p , we need to consider two sub-cases:

Original: • If β ′ + p β + p > 1 − γ ′ 1 − γ , then we have λ j = (cid:16) N β ′
Revised: • If β ′ + p β + p > 1 − γ ′ 1 − γ , then we have λ j = (cid:16) N β ′

Original: [Equation]
Revised: [Equation]

Original: p β + p
Revised: p β + p

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: 1 β ′ + p and thus
Revised: 1 β ′ + p and thus

Original: [Equation]
Revised: [Equation]

Original: • If β ′ + p β + p < 1 − γ ′ 1 − γ , then similarly we have λ j = (cid:16) N 1 − γ
Revised: • If β ′ + p β + p < 1 − γ ′ 1 − γ , then similarly we have λ j = (cid:16) N 1 − γ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: γ j
Revised: γ j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: 1 β ′ + p and
Revised: 1 β ′ + p and

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: in all cases (10) holds and we have that
Revised: in all cases (10) holds and we have that

Original: [Equation]
Revised: [Equation]

Original: B.1. V ARIANCE
Revised: B.1. V ARIANCE

Original: The variance term can be rewritten in the following way:
Revised: The variance term can be rewritten in the following way:

Original: [Equation]
Revised: [Equation]

Original: In (12), (12a) uses the definition of the Hilbert-Schmidt norm; (12b) follows from the definition of ˆ A (cf.(3)) and the fact that for any j ⩾ y N , we have (cid:68) ρ 12 j f j , (cid:16) ˆ A − A λ (cid:17) µ 12 i e i (cid:69) = 0 ; (12c) is obtained from re-arranging and (12d) follows from (cid:13)(cid:13)(cid:13)(cid:13)(cid:80) + ∞ i =1 µ 2 − β ′
Revised: In (12), (12a) uses the definition of the Hilbert-Schmidt norm; (12b) follows from the definition of ˆ A (cf.(3)) and the fact that for any j ⩾ y N , we have (cid:68) ρ 12 j f j , (cid:16) ˆ A − A λ (cid:17) µ 12 i e i (cid:69) = 0 ; (12c) is obtained from re-arranging and (12d) follows from (cid:13)(cid:13)(cid:13)(cid:13)(cid:80) + ∞ i =1 µ 2 − β ′

Original: i µ i + λ j e i ⊗ e i (cid:13)(cid:13)(cid:13)(cid:13) =
Revised: i µ i + λ j e i ⊗ e i (cid:13)(cid:13)(cid:13)(cid:13) =

Original: max i ⩾ 1 µ 1 − β ′ i µ i + λ j ≲ λ − β ′
Revised: max i ⩾ 1 µ 1 − β ′ i µ i + λ j ≲ λ − β ′

Original: j and ρ j ≲ j − 1 q .
Revised: j and ρ j ≲ j − 1 q .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: The U 1 j term is the variance of observational noise and U 2 j term is the variance of regularized bias.
Revised: The U 1 j term is the variance of observational noise and U 2 j term is the variance of regularized bias.

Original: Thus the U 1 j term is the dominating term.
Revised: Thus the U 1 j term is the dominating term.

Original: Plugging the above decomposition into (12), we deduce that V ⩽ 2 ( V 1 + V 2 ) where
Revised: Plugging the above decomposition into (12), we deduce that V ⩽ 2 ( V 1 + V 2 ) where

Original: [Equation]
Revised: [Equation]

Original: where ˆ E [ X ] = 1 N (cid:80) Nk
Revised: where ˆ E [ X ] = 1 N (cid:80) Nk

Original: =1 X k denotes the empirical mean.
Revised: =1 X k denotes the empirical mean.

Original: Define the event
Revised: Define the event

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: ⩽
Revised: ⩽

Original: [Equation]
Revised: [Equation]

Original: a 1 . (cid:27) .
Revised: a 1 . (cid:27) .

Original: Recall that m N ⩽ c 0 (cid:16) N log N (cid:17) pα , by Theorem D.3, we know that E 1 ,j holds with probability ⩾ 1 − 2 e − a 1 .
Revised: Recall that m N ⩽ c 0 (cid:16) N log N (cid:17) pα , by Theorem D.3, we know that E 1 ,j holds with probability ⩾ 1 − 2 e − a 1 .

Original: As a result E 1 = ∩ n N j =1 E 1 ,j holds with probability ⩾ 1 − 2 n N e − a 1 .
Revised: As a result E 1 = ∩ n N j =1 E 1 ,j holds with probability ⩾ 1 − 2 n N e − a 1 .

Original: We assume event
Revised: We assume event

Original: E 1 holds in all the following proof.
Revised: E 1 holds in all the following proof.

Original: Bounding V 1 .
Revised: Bounding V 1 .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: and X k = ( X j,k : 1 ⩽ j ⩽ n N ) ∈
Revised: and X k = ( X j,k : 1 ⩽ j ⩽ n N ) ∈

Original: H y N K .
Revised: H y N K .

Original: Then we have V 1 ≲ (cid:13)(cid:13)(cid:13) 1 N (cid:80)
Revised: Then we have V 1 ≲ (cid:13)(cid:13)(cid:13) 1 N (cid:80)

Original: Nk
Revised: Nk

Original: =1 X k (cid:13)(cid:13)(cid:13) 2 where the norm here defined for H ⊗ y N K is induced by ⟨ a, b ⟩ = (cid:80) n N i =1 ⟨ a i , b i ⟩ H K .
Revised: =1 X k (cid:13)(cid:13)(cid:13) 2 where the norm here defined for H ⊗ y N K is induced by ⟨ a, b ⟩ = (cid:80) n N i =1 ⟨ a i , b i ⟩ H K .

Original: Note that X k , k = 1 , 2 , · · · , N are i.i.d.
Revised: Note that X k , k = 1 , 2 , · · · , N are i.i.d.

Original: random variables with mean zero, and
Revised: random variables with mean zero, and

Original: [Equation]
Revised: [Equation]

Original: N sup u ∈ supp( P K )   j 1 − γ ′ q i β ′ p
Revised: N sup u ∈ supp( P K )   j 1 − γ ′ q i β ′ p

Original: j (cid:13)(cid:13)(cid:13) ( C KK + λ j I ) − 12 u (cid:13)(cid:13)(cid:13)(cid:124) (cid:123)(cid:122) (cid:125)
Revised: j (cid:13)(cid:13)(cid:13) ( C KK + λ j I ) − 12 u (cid:13)(cid:13)(cid:13)(cid:124) (cid:123)(cid:122) (cid:125)

Original: =: G  t −·
Revised: =: G  t −·

Original: E ( u,v ) ∼ P KL  
Revised: E ( u,v ) ∼ P KL  

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: t −  n N (cid:88)
Revised: t −  n N (cid:88)

Original: j = j 1 − γ ′
Revised: j = j 1 − γ ′

Original: q λ
Revised: q λ

Original: [Equation]
Revised: [Equation]

Original: β ′ j (cid:68) v − A 0 u, ρ 12 j f j (cid:69) 2 (cid:13)(cid:13)(cid:13) ( C KK + λ j I ) − 12 u (cid:13)(cid:13)(cid:13)  
Revised: β ′ j (cid:68) v − A 0 u, ρ 12 j f j (cid:69) 2 (cid:13)(cid:13)(cid:13) ( C KK + λ j I ) − 12 u (cid:13)(cid:13)(cid:13)  

Original:  (cid:124) (cid:123)(cid:122) (cid:125) =:
Revised:  (cid:124) (cid:123)(cid:122) (cid:125) =:

Original: G
Revised: G

Original: By Lemma D.2 we have
Revised: By Lemma D.2 we have

Original: G 1 ≲ j 1 − γ ′ q λ − ( β ′ + α ) j .
Revised: G 1 ≲ j 1 − γ ′ q λ − ( β ′ + α ) j .

Original: For G 2 , note that for fixed u , Assumption 2.4 implies that
Revised: For G 2 , note that for fixed u , Assumption 2.4 implies that

Original: E v | u 
Revised: E v | u 

Original: [Equation]
Revised: [Equation]

Original: −  n N (cid:88)
Revised: −  n N (cid:88)

Original: j = j 1 − γ ′ q i β ′ p
Revised: j = j 1 − γ ′ q i β ′ p

Original: j (cid:68) v − A 0 u, ρ 12 j f j (cid:69) 2 (cid:13)(cid:13)(cid:13) ( C KK + λ j I ) − 12 u (cid:13)(cid:13)(cid:13)  
Revised: j (cid:68) v − A 0 u, ρ 12 j f j (cid:69) 2 (cid:13)(cid:13)(cid:13) ( C KK + λ j I ) − 12 u (cid:13)(cid:13)(cid:13)  

Original:  ⩽2 (2 t )!
Revised:  ⩽2 (2 t )!

Original: R 2 t −n N (cid:88)
Revised: R 2 t −n N (cid:88)

Original: j = σ 2 j j 1 − γ ′
Revised: j = σ 2 j j 1 − γ ′

Original: q λ − β ′ j
Revised: q λ − β ′ j

Original: (cid:13)(cid:13)(cid:13) ( C KK + λ j I ) − 12 u (cid:13)(cid:13)(cid:13) 2 .
Revised: (cid:13)(cid:13)(cid:13) ( C KK + λ j I ) − 12 u (cid:13)(cid:13)(cid:13) 2 .

Original: where σ 2 j = (cid:68) ρ 12 j f j , V ρ 12 f j (cid:69) .
Revised: where σ 2 j = (cid:68) ρ 12 j f j , V ρ 12 f j (cid:69) .

Original: As a result, we have
Revised: As a result, we have

Original: [Equation]
Revised: [Equation]

Original: where in the second step we use (cid:80) + ∞ j =1 σ 2 j = tr ( V ) =
Revised: where in the second step we use (cid:80) + ∞ j =1 σ 2 j = tr ( V ) =

Original: σ 2 and
Revised: σ 2 and

Original: [Equation]
Revised: [Equation]

Original: We have shown that for some constant c 1 > 0 ,
Revised: We have shown that for some constant c 1 > 0 ,

Original: [Equation]
Revised: [Equation]

Original: By Bernstein’s inequality
Revised: By Bernstein’s inequality

Original: [Equation]
Revised: [Equation]

Original: the event
Revised: the event

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: holds with probability ⩾ 1 − 2 e − a 2 .
Revised: holds with probability ⩾ 1 − 2 e − a 2 .

Original: By our definition of λ j , we have
Revised: By our definition of λ j , we have

Original: [Equation]
Revised: [Equation]

Original: and λ j ≳ N − 1 α (which implies that the 1 N 2 term is dominated by the 1 N term).
Revised: and λ j ≳ N − 1 α (which implies that the 1 N 2 term is dominated by the 1 N term).

Original: Hence, under E 1 ∩ Ewe have
Revised: Hence, under E 1 ∩ Ewe have

Original: [Equation]
Revised: [Equation]

Original: with probability ⩾ 1 − 2 n N e − a 2 .
Revised: with probability ⩾ 1 − 2 n N e − a 2 .

Original: Bounding V 2 .
Revised: Bounding V 2 .

Original: For any j ∈ Z + we have
Revised: For any j ∈ Z + we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 15a)
Revised: 15a)

Original: =
Revised: =

Original: ρ
Revised: ρ

Original: [Equation]
Revised: [Equation]

Original: (cid:28)(cid:18)
Revised: (cid:28)(cid:18)

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: ρ
Revised: ρ

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: C
Revised: C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:18) C
Revised: (cid:18) C

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: Q K
Revised: Q K

Original: A 0 C 1 − β
Revised: A 0 C 1 − β

Original: [Equation]
Revised: [Equation]

Original: KK (cid:19) ∗ ρ 12 j f j (cid:29) (15b) ≲ j − 1 − γ q λ − (2 − β )
Revised: KK (cid:19) ∗ ρ 12 j f j (cid:29) (15b) ≲ j − 1 − γ q λ − (2 − β )

Original: j (cid:13) (cid:13)(cid:13)(cid:13) (cid:18)
Revised: j (cid:13) (cid:13)(cid:13)(cid:13) (cid:18)

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125) =: D j,(15c) where (15a) follows from E P K u ⊗ u = C KK , (15b) uses the fact that C KK and C KK + λ j I commute, and lastly (15c) follows from (cid:13)(cid:13)(cid:13)
Revised: (cid:124) (cid:123)(cid:122) (cid:125) =: D j,(15c) where (15a) follows from E P K u ⊗ u = C KK , (15b) uses the fact that C KK and C KK + λ j I commute, and lastly (15c) follows from (cid:13)(cid:13)(cid:13)

Original: ( C KK + λ j I ) − 1 C β KK ( C KK + λ j I ) − 1 (cid:13)(cid:13)(cid:13)
Revised: ( C KK + λ j I ) − 1 C β KK ( C KK + λ j I ) − 1 (cid:13)(cid:13)(cid:13)

Original: H K ∝ λ − (2 − β ) j .
Revised: H K ∝ λ − (2 − β ) j .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Note that Y k , k = 1 , 2 , · · · , N are i.i.d.
Revised: Note that Y k , k = 1 , 2 , · · · , N are i.i.d.

Original: random variables, and
Revised: random variables, and

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: ⩽
Revised: ⩽

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ≲
Revised: ≲

Original: [Equation]
Revised: [Equation]

Original: For any j ∈ Z + and u ∈ supp( P K ) we have n N (cid:88) j
Revised: For any j ∈ Z + and u ∈ supp( P K ) we have n N (cid:88) j

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ⩽ n
Revised: ⩽ n

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: 17a) ≲ max 1 ⩽
Revised: 17a) ≲ max 1 ⩽

Original: j ⩽
Revised: j ⩽

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:18)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (17a) uses the AM-GM inequality
Revised: where (17a) uses the AM-GM inequality

Original: [Equation]
Revised: [Equation]

Original: (17b) follows from the assumption that ∥A 0 u
Revised: (17b) follows from the assumption that ∥A 0 u

Original: ∥ ⩽ A 2 is uniformly bounded
Revised: ∥ ⩽ A 2 is uniformly bounded

Original: [Equation]
Revised: [Equation]

Original: and that
Revised: and that

Original: [Equation]
Revised: [Equation]

Original: by Assumption 2.
Revised: by Assumption 2.

Original: [Equation]
Revised: [Equation]

Original: and lastly (17c) follows from ∥A 

Revised: and lastly (17c) follows from ∥A

Original: ∥ β,γ ⩽ B .
Revised: ∥ β,γ ⩽ B .

Original: Plugging into (16), we deduce that
Revised: Plugging into (16), we deduce that

Original: [Equation]
Revised: [Equation]

Original: where the last step follows from (cid:80) + ∞
Revised: where the last step follows from (cid:80) + ∞

Original: j =1 D j, 2 = ∥A 

Revised: j =1 D j, 2 = ∥A

Original: ∥ 2 β,γ .
Revised: ∥ 2 β,γ .

Original: By Bernstein’s inequality, there exists a constant C 3 such that the event
Revised: By Bernstein’s inequality, there exists a constant C 3 such that the event

Original: E 3 =
Revised: E 3 =

Original: [Equation]
Revised: [Equation]

Original: holds with probability ⩾ 1 − 2 e − a 3 .
Revised: holds with probability ⩾ 1 − 2 e − a 3 .

Original: The definition of λ N implies that the 1 N 2 term is dominated by the 1 N term, so
Revised: The definition of λ N implies that the 1 N 2 term is dominated by the 1 N term, so

Original: [Equation]
Revised: [Equation]

Original: holds under E 1 ∩ E 3 .
Revised: holds under E 1 ∩ E 3 .

Original: To summarize, under E 1 ∩ E 2 ∩ E 3 which holds with probability ⩾ 1 − 2 n N e − a 1 − 2 e − a 2 − 2 e − a 3 , we have
Revised: To summarize, under E 1 ∩ E 2 ∩ E 3 which holds with probability ⩾ 1 − 2 n N e − a 1 − 2 e − a 2 − 2 e − a 3 , we have

Original: [Equation]
Revised: [Equation]

Original: Recall that the bias term is upper bounded in (11).
Revised: Recall that the bias term is upper bounded in (11).

Original: This gives the final upper bound
Revised: This gives the final upper bound

Original: [Equation]
Revised: [Equation]

Original: B.1. T HE HARD - LEARNING REGIME
Revised: B.1. T HE HARD - LEARNING REGIME

Original: In the previous sections, we focus on the case where α ⩽ β + p and establish an upper bound for the convergence rate via an optimal bias-variance trade-off.
Revised: In the previous sections, we focus on the case where α ⩽ β + p and establish an upper bound for the convergence rate via an optimal bias-variance trade-off.

Original: The opposite case, α > β + p is referred to as the hard-learning regime, for which the optimal rate is not known for several decades even in the case of γ = 1 (cf.
Revised: The opposite case, α > β + p is referred to as the hard-learning regime, for which the optimal rate is not known for several decades even in the case of γ = 1 (cf.

Original: the discussion following (Fischer & Steinwart, 2020, Theorem 2)).
Revised: the discussion following (Fischer & Steinwart, 2020, Theorem 2)).

Original: In the hard learning regime the V 2 term becomes the leading terms.
Revised: In the hard learning regime the V 2 term becomes the leading terms.

Original: In this section, we use the technique developed in previous sections to obtain an upper bound in the hard-learning regime.
Revised: In this section, we use the technique developed in previous sections to obtain an upper bound in the hard-learning regime.

Original: To do this, we need to re-define the truncation set S N as follows:
Revised: To do this, we need to re-define the truncation set S N as follows:

Original: [Equation]
Revised: [Equation]

Original: The definition implies that the variance can be controlled by N − min
Revised: The definition implies that the variance can be controlled by N − min

Original: [Equation]
Revised: [Equation]

Original: and it remains to focus on the bias term.
Revised: and it remains to focus on the bias term.

Original: Similar to the derivations in Appendix B.1.1, we have
Revised: Similar to the derivations in Appendix B.1.1, we have

Original: [Equation]
Revised: [Equation]

Original: The maximum value of the right hand side can be achieved in either of the following two cases:
Revised: The maximum value of the right hand side can be achieved in either of the following two cases:

Original: • i
Revised: • i

Original: = O (1) .
Revised: = O (1) .

Original: Then we have j ≳ N q 1 − γ ′ (cid:16) 1 − min (cid:110)
Revised: Then we have j ≳ N q 1 − γ ′ (cid:16) 1 − min (cid:110)

Original: β − β ′ α , γ ′−
Revised: β − β ′ α , γ ′−

Original: [Equation]
Revised: [Equation]

Original: 1 − γ (cid:111)(cid:17) so that i − β − β
Revised: 1 − γ (cid:111)(cid:17) so that i − β − β

Original: [Equation]
Revised: [Equation]

Original: − γ ′− γ
Revised: − γ ′− γ

Original: [Equation]
Revised: [Equation]

Original: (cid:110)
Revised: (cid:110)

Original: [Equation]
Revised: [Equation]

Original: − β
Revised: − β

Original: [Equation]
Revised: [Equation]

Original: α
Revised: α

Original: [Equation]
Revised: [Equation]

Original: ′−
Revised: ′−

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: (cid:111)(cid:17) ⩽ N − γ ′− γ 1 − γ
Revised: (cid:111)(cid:17) ⩽ N − γ ′− γ 1 − γ

Original: [Equation]
Revised: [Equation]

Original: • j
Revised: • j

Original: =
Revised: =

Original: O
Revised: O

Original: In this case we must have i ≲ N min (cid:110) pα , p β − β ′ γ ′− γ
Revised: In this case we must have i ≲ N min (cid:110) pα , p β − β ′ γ ′− γ

Original: [Equation]
Revised: [Equation]

Original: γ (cid:111) , otherwise it falls into S N by definition.
Revised: γ (cid:111) , otherwise it falls into S N by definition.

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand, for the variance term we still have V 1 ≲ 1 N max 1 ⩽ j ⩽ n N j 1 − γ
Revised: On the other hand, for the variance term we still have V 1 ≲ 1 N max 1 ⩽ j ⩽ n N j 1 − γ

Original: [Equation]
Revised: [Equation]

Original: q i β
Revised: q i β

Original: [Equation]
Revised: [Equation]

Original: + p p j and V 2 ⩽ 1 N max
Revised: + p p j and V 2 ⩽ 1 N max

Original: [Equation]
Revised: [Equation]

Original: ⩽
Revised: ⩽

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: , so that
Revised: , so that

Original: [Equation]
Revised: [Equation]

Original: As a result
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: we can obtain the following convergence rate:
Revised: we can obtain the following convergence rate:

Original: [Equation]
Revised: [Equation]

Original: B. R EGULARIZATION VIA BIAS CONTOUR
Revised: B. R EGULARIZATION VIA BIAS CONTOUR

Original: In this subsection
Revised: In this subsection

Original: [Equation]
Revised: [Equation]

Original: we analyze the convergence rate of regularization via bias contour (cf.
Revised: we analyze the convergence rate of regularization via bias contour (cf.

Original: Figure 2).
Revised: Figure 2).

Original: Specifically, we consider the estimator (3) with the choice
Revised: Specifically, we consider the estimator (3) with the choice

Original: [Equation]
Revised: [Equation]

Original: It now remains to plug the above λ j into our bounds for bias and variance derived in the previous subsections.
Revised: It now remains to plug the above λ j into our bounds for bias and variance derived in the previous subsections.

Original: Bounding the bias term.
Revised: Bounding the bias term.

Original: It follows from (9) that
Revised: It follows from (9) that

Original: [Equation]
Revised: [Equation]

Original: Bounding the variance term It follows from (14) and (18) that the variance is bounded by
Revised: Bounding the variance term It follows from (14) and (18) that the variance is bounded by

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: As before
Revised: As before

Original: [Equation]
Revised: [Equation]

Original: we consider the cases α ⩽ β + p
Revised: we consider the cases α ⩽ β + p

Original: and α > β + p separately.
Revised: and α > β + p separately.

Original: • If α ⩽ β + p , then it follows that
Revised: • If α ⩽ β + p , then it follows that

Original: [Equation]
Revised: [Equation]

Original: where we use y γ ′− γ q N
Revised: where we use y γ ′− γ q N

Original: [Equation]
Revised: [Equation]

Original: by definition.
Revised: by definition.

Original: • If α > β + p , then similarly we have
Revised: • If α > β + p , then similarly we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Hence we deduce that
Revised: Hence we deduce that

Original: [Equation]
Revised: [Equation]

Original: as desired.
Revised: as desired.

Original: B. I MPLICATION OF THE UPPER BOUND
Revised: B. I MPLICATION OF THE UPPER BOUND

Original: In this section, we discuss the implications of our upper bounds under the ( β ′ , γ ′ ) -norm.
Revised: In this section, we discuss the implications of our upper bounds under the ( β ′ , γ ′ ) -norm.

Original: Note that (cid:13)(cid:13)(cid:13)(cid:13) C − 1 − γ ′ 2 Q K v (cid:13)(cid:13)(cid:13)(cid:13)
Revised: Note that (cid:13)(cid:13)(cid:13)(cid:13) C − 1 − γ ′ 2 Q K v (cid:13)(cid:13)(cid:13)(cid:13)

Original: H L = ∥ v ∥ H 2 − γ ′
Revised: H L = ∥ v ∥ H 2 − γ ′

Original: L for all v ∈ L 2 ( Q K ) (if one side of the equation is + ∞ then so is the other), we have that
Revised: L for all v ∈ L 2 ( Q K ) (if one side of the equation is + ∞ then so is the other), we have that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the last step follows from E u ∼ P K u ⊗ u = C P K .
Revised: where the last step follows from E u ∼ P K u ⊗ u = C P K .

Original: Note that the above derivations hold for any 0 ⩽ β ′ <
Revised: Note that the above derivations hold for any 0 ⩽ β ′ <

Original: β , so choosing β ′ = 

Revised: β , so choosing β ′ =

Original: yields the best upper bound.
Revised: yields the best upper bound.

Original: We can see from (20) that our analysis implies an upper bound of the expected error of the learned solution evaluated under the H 

Revised: We can see from (20) that our analysis implies an upper bound of the expected error of the learned solution evaluated under the H

Original: − γ ′
Revised: − γ ′

Original: L norm.
Revised: L norm.

Original: On the other hand, it is also possible to obtain a uniform convergence rate when β ′ ⩾ α :
Revised: On the other hand, it is also possible to obtain a uniform convergence rate when β ′ ⩾ α :

Original: [Equation]
Revised: [Equation]

Original: C P ROOFS FOR THE MULTI - LEVEL OPERATOR LEARNING ALGORITHM
Revised: C P ROOFS FOR THE MULTI - LEVEL OPERATOR LEARNING ALGORITHM

Original: In this section, we analyze the convergence rate of our multi-level algorithm described in Section 5.
Revised: In this section, we analyze the convergence rate of our multi-level algorithm described in Section 5.

Original: We define η 1 = min (cid:110) β − β ′ max { α,β + p } , γ ′ − γ 1 − γ (cid:111) and η 2 = max (cid:110) 1 − β − β ′ max { α,β + p } , 1 − γ ′ 1 − γ (cid:111)
Revised: We define η 1 = min (cid:110) β − β ′ max { α,β + p } , γ ′ − γ 1 − γ (cid:111) and η 2 = max (cid:110) 1 − β − β ′ max { α,β + p } , 1 − γ ′ 1 − γ (cid:111)

Original: =
Revised: =

Original: 1 − η 1 .
Revised: 1 − η 1 .

Original: We first restrict ourselves to the case when β
Revised: We first restrict ourselves to the case when β

Original: [Equation]
Revised: [Equation]

Original: max {
Revised: max {

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: − γ ; the special case when the two terms are equal will be separately treated in Appendix C.1.
Revised: − γ ; the special case when the two terms are equal will be separately treated in Appendix C.1.

Original: For the optimal bias and variance contours ℓ C 1 , bias and ℓ C 2 , var with C 1 = N η 1 and C 2 = N η 2 , we define a sequence { x n } as follows:
Revised: For the optimal bias and variance contours ℓ C 1 , bias and ℓ C 2 , var with C 1 = N η 1 and C 2 = N η 2 , we define a sequence { x n } as follows:

Original: [Equation]
Revised: [Equation]

Original: = the solution of
Revised: = the solution of

Original: [Equation]
Revised: [Equation]

Original: We first derive an explicit recursive formula for { x n } .
Revised: We first derive an explicit recursive formula for { x n } .

Original: Lemma C.1 Let u = β ′ +max { α − β,p } β − β ′ γ
Revised: Lemma C.1 Let u = β ′ +max { α − β,p } β − β ′ γ

Original: − γ 1 − γ ′ > 0 , then (1).
Revised: − γ 1 − γ ′ > 0 , then (1).

Original: if u > 1 , then
Revised: if u > 1 , then

Original: [Equation]
Revised: [Equation]

Original: if u < 1 , then
Revised: if u < 1 , then

Original: [Equation]
Revised: [Equation]

Original: Proof : (1).
Revised: Proof : (1).

Original: Suppose that u > 1 , then we have η 1 = β − β
Revised: Suppose that u > 1 , then we have η 1 = β − β

Original: [Equation]
Revised: [Equation]

Original: max { α,β + p } and η 2 = 1 − η 1 .
Revised: max { α,β + p } and η 2 = 1 − η 1 .

Original: It follows from (21b) and (21c) that
Revised: It follows from (21b) and (21c) that

Original: [Equation]
Revised: [Equation]

Original: Suppose that u < 1 , then we have η 1 = γ ′ − γ 1 − γ and η 2 = 1 − γ ′ 1 − γ , so that η 1 η 2 = γ ′ − γ 1 − γ ′ , and it follows from (21b) and (21c) that x β
Revised: Suppose that u < 1 , then we have η 1 = γ ′ − γ 1 − γ and η 2 = 1 − γ ′ 1 − γ , so that η 1 η 2 = γ ′ − γ 1 − γ ′ , and it follows from (21b) and (21c) that x β

Original: [Equation]
Revised: [Equation]

Original: □
Revised: □

Original: Lemma C.1 implies that when u ̸ = 1 , the sequence { x n } decreases super-exponentially.
Revised: Lemma C.1 implies that when u ̸ = 1 , the sequence { x n } decreases super-exponentially.

Original: Thus, there exists L N =
Revised: Thus, there exists L N =

Original: O (log log N ) such that x n ⩽ 2 for all n ⩾ L N .
Revised: O (log log N ) such that x n ⩽ 2 for all n ⩾ L N .

Original: Let λ ( K ) i
Revised: Let λ ( K ) i

Original: = x − 1 p i and λ ( L ) i =
Revised: = x − 1 p i and λ ( L ) i =

Original: y − 1 q i , then we construct the following estimator:
Revised: y − 1 q i , then we construct the following estimator:

Original: [Equation]
Revised: [Equation]

Original: where y − 1 := 

Revised: where y − 1 :=

Original: Note that each summand in the above equation is essentially a regularized leastsquares estimator and learns a rectangular region.
Revised: Note that each summand in the above equation is essentially a regularized leastsquares estimator and learns a rectangular region.

Original: The following theorem states that the estimator ˆ
Revised: The following theorem states that the estimator ˆ

Original: A ml can achieve minimax optimal convergence rate.
Revised: A ml can achieve minimax optimal convergence rate.

Original: Theorem C.1 Consider the estimator ˆ A ml defined by (5) .
Revised: Theorem C.1 Consider the estimator ˆ A ml defined by (5) .

Original: Suppose that Assumptions 2.1 to 2.5 hold, then there exists a universal constant C , such that
Revised: Suppose that Assumptions 2.1 to 2.5 hold, then there exists a universal constant C , such that

Original: [Equation]
Revised: [Equation]

Original: holds with probability ⩾ 1 − e − τ .
Revised: holds with probability ⩾ 1 − e − τ .

Original: Proof : The proof of Theorem 5.1 is similar to that of Theorems 4.1 and 4.2.
Revised: Proof : The proof of Theorem 5.1 is similar to that of Theorems 4.1 and 4.2.

Original: We consider the bias-variance decomposition of the estimation error
Revised: We consider the bias-variance decomposition of the estimation error

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Bounding the bias term.
Revised: Bounding the bias term.

Original: Since ∥A 0 ∥ β,γ ⩽ B , we can write
Revised: Since ∥A 0 ∥ β,γ ⩽ B , we can write

Original: [Equation]
Revised: [Equation]

Original: where the coefficient matrix A 

Revised: where the coefficient matrix A

Original: = ( a ij ) 1 ⩽
Revised: = ( a ij ) 1 ⩽

Original: i,j ⩽ + ∞ satisfies ∥
Revised: i,j ⩽ + ∞ satisfies ∥

Original: A 0 ∥ 2 F ⩽ B 2 .
Revised: A 0 ∥ 2 F ⩽ B 2 .

Original: We fix ( i, j
Revised: We fix ( i, j

Original: ) ∈
Revised: ) ∈

Original: Z 2+ and assume WLOG that y m j − 1 ⩽ j < y m j for some m ⩾ 0 , where y L N +1 = + ∞ .
Revised: Z 2+ and assume WLOG that y m j − 1 ⩽ j < y m j for some m ⩾ 0 , where y L N +1 = + ∞ .

Original: It follows from (23) that
Revised: It follows from (23) that

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where we recall that η 1 = min (cid:110)
Revised: where we recall that η 1 = min (cid:110)

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: and the last step follows from (21c).
Revised: and the last step follows from (21c).

Original: Bounding the variance term.
Revised: Bounding the variance term.

Original: The variance term can be rewritten in the following way:
Revised: The variance term can be rewritten in the following way:

Original: [Equation]
Revised: [Equation]

Original: for reasons similar to (12).
Revised: for reasons similar to (12).

Original: It now remains to bound (cid:13)(cid:13) G m j (cid:13)(cid:13) and (cid:13)(cid:13)(cid:13) U m j ρ 1 2 j f j (cid:13)(cid:13)(cid:13) for 1 ⩽ j ⩽ L N .
Revised: It now remains to bound (cid:13)(cid:13) G m j (cid:13)(cid:13) and (cid:13)(cid:13)(cid:13) U m j ρ 1 2 j f j (cid:13)(cid:13)(cid:13) for 1 ⩽ j ⩽ L N .

Original: Note that these quantities have already been bounded in Appendix B.1.2 with λ m j replaced with λ j (there we use a different regularization for each j ).
Revised: Note that these quantities have already been bounded in Appendix B.1.2 with λ m j replaced with λ j (there we use a different regularization for each j ).

Original: Hence, those bounds can be directly applied here, so there exists a constant C > 0 such that
Revised: Hence, those bounds can be directly applied here, so there exists a constant C > 0 such that

Original: [Equation]
Revised: [Equation]

Original: with probability ⩾ 1 − Ne − a .
Revised: with probability ⩾ 1 − Ne − a .

Original: Since j ⩽ y m j , by (21b) we have
Revised: Since j ⩽ y m j , by (21b) we have

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Combining the bias and variance bounds, the conclusion directly follows.
Revised: Combining the bias and variance bounds, the conclusion directly follows.

Original: □
Revised: □

Original: C. 
Revised: C.

Original: S PECIAL CASE :
Revised: S PECIAL CASE :

Original: [Equation]
Revised: [Equation]

Original: Note that Lemma C.1 does not cover the case u = 1 , or equivalently
Revised: Note that Lemma C.1 does not cover the case u = 1 , or equivalently

Original: [Equation]
Revised: [Equation]

Original: This case is special since the bias contour coincides with the variance contour, and we need to modify our construction of the multilevel estimator.
Revised: This case is special since the bias contour coincides with the variance contour, and we need to modify our construction of the multilevel estimator.

Original: We define two sequences { x n } , { y n } as follows:
Revised: We define two sequences { x n } , { y n } as follows:

Original: [Equation]
Revised: [Equation]

Original: y n = the solution of x
Revised: y n = the solution of x

Original: [Equation]
Revised: [Equation]

Original: where we recall that η 1 = β − β ′ max { α,β + p } =
Revised: where we recall that η 1 = β − β ′ max { α,β + p } =

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: − γ 1 − γ .
Revised: − γ 1 − γ .

Original: In this case, there exists L N =
Revised: In this case, there exists L N =

Original: O (ln N ) such that x n < 1 for all n ⩾ L N .
Revised: O (ln N ) such that x n < 1 for all n ⩾ L N .

Original: Let λ ( K ) i
Revised: Let λ ( K ) i

Original: = x − 1 p i , then we construct the following estimator:
Revised: = x − 1 p i , then we construct the following estimator:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similar to Theorem C.1, we can establish the following result:
Revised: Similar to Theorem C.1, we can establish the following result:

Original: Theorem C.2 Consider the estimator ˆ A ml defined by (26) .
Revised: Theorem C.2 Consider the estimator ˆ A ml defined by (26) .

Original: Suppose that Assumptions 2.1 to 2.
Revised: Suppose that Assumptions 2.1 to 2.

Original: hold, then there exists a universal constant C , such that
Revised: hold, then there exists a universal constant C , such that

Original: [Equation]
Revised: [Equation]

Original: holds with probability ⩾ 1 − e − τ .
Revised: holds with probability ⩾ 1 − e − τ .

Original: Proof : The proof of Theorem C.2 is similar to that of Theorems 4.1 and 4.2.
Revised: Proof : The proof of Theorem C.2 is similar to that of Theorems 4.1 and 4.2.

Original: We consider the bias-variance decomposition
Revised: We consider the bias-variance decomposition

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: A
Revised: A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: as defined in (26).
Revised: as defined in (26).

Original: Bounding the bias term.
Revised: Bounding the bias term.

Original: Let A 0 :
Revised: Let A 0 :

Original: = (cid:80) + ∞
Revised: = (cid:80) + ∞

Original: i =1 (cid:80) + ∞
Revised: i =1 (cid:80) + ∞

Original: j =1 a ij µ β 

Revised: j =1 a ij µ β

Original: i ρ 1 − γ 

Revised: i ρ 1 − γ

Original: j f j ⊗ e i with coefficient matrix
Revised: j f j ⊗ e i with coefficient matrix

Original: A 

Revised: A

Original: = ( a ij ) + ∞ i,j =1 such that ∥ A 0 ∥ 2 F ⩽ B 2 .
Revised: = ( a ij ) + ∞ i,j =1 such that ∥ A 0 ∥ 2 F ⩽ B 2 .

Original: We fix ( i, j
Revised: We fix ( i, j

Original: ) ∈
Revised: ) ∈

Original: Z 2+ and assume WLOG that y m j − 1 ⩽ j < y m j for some m j ⩾ 0 , where y L N +1 = + ∞ .
Revised: Z 2+ and assume WLOG that y m j − 1 ⩽ j < y m j for some m j ⩾ 0 , where y L N +1 = + ∞ .

Original: It follows from (27) that
Revised: It follows from (27) that

Original: [Equation]
Revised: [Equation]

Original: Thus we can proceed as in (24) to deduce that
Revised: Thus we can proceed as in (24) to deduce that

Original: [Equation]
Revised: [Equation]

Original: The definition (25) implies that
Revised: The definition (25) implies that

Original: [Equation]
Revised: [Equation]

Original: On the other hand, since x L N < 1 , by (25) implies that y − γ ′− γ q L N ≲ N − η 1 .
Revised: On the other hand, since x L N < 1 , by (25) implies that y − γ ′− γ q L N ≲ N − η 1 .

Original: Therefore, for the bias term (cid:13)(cid:13)
Revised: Therefore, for the bias term (cid:13)(cid:13)

Original: A 0 − A λ ml (cid:13)(cid:13) 2 β ′ ,γ ′ ≲ N − η 1 .
Revised: A 0 − A λ ml (cid:13)(cid:13) 2 β ′ ,γ ′ ≲ N − η 1 .

Original: Bounding the variance term.
Revised: Bounding the variance term.

Original: Repeating the arguments in (25), we can deduce that there exists a constant C > 0 such that
Revised: Repeating the arguments in (25), we can deduce that there exists a constant C > 0 such that

Original: [Equation]
Revised: [Equation]

Original: with probability ⩾ 1 − Ne − a .
Revised: with probability ⩾ 1 − Ne − a .

Original: Combining the bias and variance bounds, we arrive at the desired conclusion.
Revised: Combining the bias and variance bounds, we arrive at the desired conclusion.

Original: □
Revised: □

Original: The conclusion of Theorem 5.1 then follows from Theorems C.1 and C.2.
Revised: The conclusion of Theorem 5.1 then follows from Theorems C.1 and C.2.

Original: D A UXILIARY RESULTS
Revised: D A UXILIARY RESULTS

Original: Lemma D.1 We have ∥ T ∥ β,γ = (cid:13)(cid:13)(cid:13) C − (1 − γ ) / 2 Q L ◦ T ◦ C (1 − β ) /
Revised: Lemma D.1 We have ∥ T ∥ β,γ = (cid:13)(cid:13)(cid:13) C − (1 − γ ) / 2 Q L ◦ T ◦ C (1 − β ) /

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13) HS( H K , H L ) .
Revised: (cid:13)(cid:13)(cid:13) HS( H K , H L ) .

Original: Proof : We recall from the definition that ∥ T ∥ β,γ = (cid:13)(cid:13)(cid:13) ( I 1 ,γ,Q L ) † ◦ T ◦ I ∗ 1 β,P K (cid:13)(cid:13)(cid:13) HS( H βK , H γL ) , so that ∥ T ∥ 2 β,γ = (cid:13)(cid:13)(cid:13) ( I 1 ,γ,
Revised: Proof : We recall from the definition that ∥ T ∥ β,γ = (cid:13)(cid:13)(cid:13) ( I 1 ,γ,Q L ) † ◦ T ◦ I ∗ 1 β,P K (cid:13)(cid:13)(cid:13) HS( H βK , H γL ) , so that ∥ T ∥ 2 β,γ = (cid:13)(cid:13)(cid:13) ( I 1 ,γ,

Original: [Equation]
Revised: [Equation]

Original: L )
Revised: L )

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i,j = (cid:28) ρ
Revised: (cid:88) i,j = (cid:28) ρ

Original: [Equation]
Revised: [Equation]

Original: (cid:0)
Revised: (cid:0)

Original: I
Revised: I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:1)
Revised: (cid:1)

Original: [Equation]
Revised: [Equation]

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: (cid:29)
Revised: (cid:29)

Original: [Equation]
Revised: [Equation]

Original: = + ∞ (cid:88) i,j = (cid:28) ρ
Revised: = + ∞ (cid:88) i,j = (cid:28) ρ

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: , (cid:0)
Revised: , (cid:0)

Original: I
Revised: I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:1)
Revised: (cid:1)

Original: [Equation]
Revised: [Equation]

Original: − β 2 i e
Revised: − β 2 i e

Original: [Equation]
Revised: [Equation]

Original: (cid:29)
Revised: (cid:29)

Original: [Equation]
Revised: [Equation]

Original: = + ∞ (cid:88) i,j = (cid:28) ρ
Revised: = + ∞ (cid:88) i,j = (cid:28) ρ

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: − β 2 i e
Revised: − β 2 i e

Original: [Equation]
Revised: [Equation]

Original: (cid:29)
Revised: (cid:29)

Original: [Equation]
Revised: [Equation]

Original: H L = +
Revised: H L = +

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i,j = (cid:68)
Revised: (cid:88) i,j = (cid:68)

Original: [Equation]
Revised: [Equation]

Original: 12 j f
Revised: 12 j f

Original: [Equation]
Revised: [Equation]

Original: , C
Revised: , C

Original: [Equation]
Revised: [Equation]

Original: (1 − γ ) / 2 Q L ◦ T ◦ C (1 − β ) /
Revised: (1 − γ ) / 2 Q L ◦ T ◦ C (1 − β ) /

Original: [Equation]
Revised: [Equation]

Original: KK µ 12 i e i (cid:69) 2 H
Revised: KK µ 12 i e i (cid:69) 2 H

Original: L = (cid:13) (cid:13) (cid:13) C −
Revised: L = (cid:13) (cid:13) (cid:13) C −

Original: [Equation]
Revised: [Equation]

Original: 1 − γ
Revised: 1 − γ

Original: [Equation]
Revised: [Equation]

Original: (cid:13) (cid:13) (cid:13)
Revised: (cid:13) (cid:13) (cid:13)

Original: HS as desired.
Revised: HS as desired.

Original: □
Revised: □

Original: Lemma D.2 Under Assumption 2.2, we have
Revised: Lemma D.2 Under Assumption 2.2, we have

Original: [Equation]
Revised: [Equation]

Original: Proof : By Assumption 2.2 we have
Revised: Proof : By Assumption 2.2 we have

Original: [Equation]
Revised: [Equation]

Original: , so that
Revised: , so that

Original: [Equation]
Revised: [Equation]

Original: as desired.
Revised: as desired.

Original: □
Revised: □

Original: D. C ONCENTRATION INEQUALITIES
Revised: D. C ONCENTRATION INEQUALITIES

Original: Theorem D.1 (Fischer & Steinwart, 2020, Theorem 27) Let (Ω , B , P ) be a probability space,
Revised: Theorem D.1 (Fischer & Steinwart, 2020, Theorem 27) Let (Ω , B , P ) be a probability space,

Original: H be a separable Hilbert space and X : Ω → HS( H ; H ) be a random variable with selfadjoint values.
Revised: H be a separable Hilbert space and X : Ω → HS( H ; H ) be a random variable with selfadjoint values.

Original: Furthermore, assume that ∥ X ∥ F ⩽ B, P − a.s.
Revised: Furthermore, assume that ∥ X ∥ F ⩽ B, P − a.s.

Original: and V be a positive semidefinite matrix with E P (cid:0)
Revised: and V be a positive semidefinite matrix with E P (cid:0)

Original: X 2 (cid:1) ≼ V , i.e. V − E P (cid:0)
Revised: X 2 (cid:1) ≼ V , i.e. V − E P (cid:0)

Original: X 2 (cid:1) is positive semi-definite.
Revised: X 2 (cid:1) is positive semi-definite.

Original: Then, for g ( V ) := log (cid:0) 2 e tr( V ) ∥ V ∥ − 1 (cid:1) , τ ⩾ 1 , and n ⩾ 1 , the following concentration inequality is satisfied
Revised: Then, for g ( V ) := log (cid:0) 2 e tr( V ) ∥ V ∥ − 1 (cid:1) , τ ⩾ 1 , and n ⩾ 1 , the following concentration inequality is satisfied

Original: [Equation]
Revised: [Equation]

Original: Theorem D.2 (Fischer & Steinwart, 2020, Theorem 26) Let (Ω , B , P ) be a probability space, H be a separable Hilbert space, and ξ : Ω → H be a random variable with
Revised: Theorem D.2 (Fischer & Steinwart, 2020, Theorem 26) Let (Ω , B , P ) be a probability space, H be a separable Hilbert space, and ξ : Ω → H be a random variable with

Original: [Equation]
Revised: [Equation]

Original: for all m ⩾ 2 .
Revised: for all m ⩾ 2 .

Original: Then, for τ ⩾ 1 and n ⩾ 1 , the following concentration inequality is satisfied
Revised: Then, for τ ⩾ 1 and n ⩾ 1 , the following concentration inequality is satisfied

Original: [Equation]
Revised: [Equation]

Original: The following theorem shows that the regularized covariance C KK + λI can be estimated with small error when λ is above a certain threshold.
Revised: The following theorem shows that the regularized covariance C KK + λI can be estimated with small error when λ is above a certain threshold.

Original: Although it is well-known (Fischer & Steinwart, 2020;
Revised: Although it is well-known (Fischer & Steinwart, 2020;

Original: Talwai et al., 2022), we still recall it below for completeness.
Revised: Talwai et al., 2022), we still recall it below for completeness.

Original: Theorem D.3 Recall that C KK = E P K u ⊗ u and ˆ C KK = 1 N (cid:80) Ni =1 u i ⊗ u i where u i i .
Revised: Theorem D.3 Recall that C KK = E P K u ⊗ u and ˆ C KK = 1 N (cid:80) Ni =1 u i ⊗ u i where u i i .

Original: i .
Revised: i .

Original: d . ∼ P K .
Revised: d . ∼ P K .

Original: Suppose that Assumption 2.2 holds and N ≳
Revised: Suppose that Assumption 2.2 holds and N ≳

Original: A 21 τg λ λ − α , where g λ = log (cid:16) 2 e N P K ( λ ) ∥C KK ∥ + λ ∥C KK ∥ (cid:17) and N P K ( λ ) =
Revised: A 21 τg λ λ − α , where g λ = log (cid:16) 2 e N P K ( λ ) ∥C KK ∥ + λ ∥C KK ∥ (cid:17) and N P K ( λ ) =

Original: tr (cid:0) ( C KK + λI ) − 1 C KK (cid:1) is the effective dimension, then with probability at least 1 − e − τ , we have
Revised: tr (cid:0) ( C KK + λI ) − 1 C KK (cid:1) is the effective dimension, then with probability at least 1 − e − τ , we have

Original: [Equation]
Revised: [Equation]

Original: + λI ) − 12 (cid:16) C KK − ˆ C KK (cid:17)
Revised: + λI ) − 12 (cid:16) C KK − ˆ C KK (cid:17)

Original: ( C KK + λI ) − 12 (cid:13)(cid:13)(cid:13) ≲ (cid:114)
Revised: ( C KK + λI ) − 12 (cid:13)(cid:13)(cid:13) ≲ (cid:114)

Original: A 21 τg λ
Revised: A 21 τg λ

Original: Nλ α ⩽ 0 . 1 .
Revised: Nλ α ⩽ 0 . 1 .

Original: Proof : Let X ( u ) =
Revised: Proof : Let X ( u ) =

Original: ( C KK + λI ) − 12 u ⊗ u ( C KK + λI ) − 12 where u ∈ H K , then the LHS of (28) can be expressed as (cid:13)(cid:13)(cid:13) 1 N (cid:80)
Revised: ( C KK + λI ) − 12 u ⊗ u ( C KK + λI ) − 12 where u ∈ H K , then the LHS of (28) can be expressed as (cid:13)(cid:13)(cid:13) 1 N (cid:80)

Original: Ni =1 X ( u i ) − E u ∼ P K X ( u ) (cid:13)(cid:13)(cid:13) .
Revised: Ni =1 X ( u i ) − E u ∼ P K X ( u ) (cid:13)(cid:13)(cid:13) .

Original: We hope to apply Theorem D.1 and start with verifying the assumptions.
Revised: We hope to apply Theorem D.1 and start with verifying the assumptions.

Original: Since E P K X = ( C KK + λI ) − 12 C KK ( C KK + λI ) − 12 and ∥ X ( u ) ∥ =
Revised: Since E P K X = ( C KK + λI ) − 12 C KK ( C KK + λI ) − 12 and ∥ X ( u ) ∥ =

Original: ∥ X ( u ) ∥
Revised: ∥ X ( u ) ∥

Original: F = (cid:13)(cid:13)(cid:13) ( C KK + λI ) − 12 u (cid:13)(cid:13)(cid:13) 2 ⩽
Revised: F = (cid:13)(cid:13)(cid:13) ( C KK + λI ) − 12 u (cid:13)(cid:13)(cid:13) 2 ⩽

Original: A 21 (cid:13)(cid:13)(cid:13) ( C KK + λI ) − 1 − α 2 (cid:13)(cid:13)(cid:13) 2 ≲
Revised: A 21 (cid:13)(cid:13)(cid:13) ( C KK + λI ) − 1 − α 2 (cid:13)(cid:13)(cid:13) 2 ≲

Original: A 21 λ − α , so that there exists V =
Revised: A 21 λ − α , so that there exists V =

Original: O (cid:16) λ − α
Revised: O (cid:16) λ − α

Original: ( C KK + λI ) − 12 C KK ( C KK + λI )
Revised: ( C KK + λI ) − 12 C KK ( C KK + λI )

Original: − 12 (cid:17) such that E P K X 2 ≼ V .
Revised: − 12 (cid:17) such that E P K X 2 ≼ V .

Original: It’s easy to see that ∥ V ∥ ≲ λ − α and tr( V ) ≲ N P K ( λ ) .
Revised: It’s easy to see that ∥ V ∥ ≲ λ − α and tr( V ) ≲ N P K ( λ ) .

Original: The conclusion then follows from Theorem D.1 with
Revised: The conclusion then follows from Theorem D.1 with

Original: B =
Revised: B =

Original: O ( λ − α ) and g ( V ) =
Revised: O ( λ − α ) and g ( V ) =

Original: g λ .
Revised: g λ .

Original: □
Revised: □

Original: Corollary D.1 Under the notations and assumptions of Theorem D.3, there exists a constant C 1 >with probability ⩾ 1 − e − τ we have (cid:13)(cid:13)(cid:13)(cid:13)
Revised: Corollary D.1 Under the notations and assumptions of Theorem D.3, there exists a constant C 1 >with probability ⩾ 1 − e − τ we have (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof : By Theorem D.3 we have
Revised: Proof : By Theorem D.3 we have

Original: [Equation]
Revised: [Equation]

Original: ⩽with probability ⩾ 1 − e − λ , as desired.
Revised: ⩽with probability ⩾ 1 − e − λ , as desired.

Original: □
Revised: □