Optimal Learning Rates for the Conditional Mean Embedding
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We address the consistency of a ridge regression estimate of the conditional meanembedding (CME), which is an embedding of the conditional distribution of Ygiven X into a target reproducing kernel Hilbert space H Y .
The CME allows us totake conditional expectations of target RKHS functions, and has been employed innonparametric causal and Bayesian inference.
Our results focus on the misspeciﬁedsetting, where the target CME is in the space of Hilbert-Schmidt operators actingfrom an input interpolation space between H X and L 2 , to H Y .
We derive a noveland adaptive statistical learning rate for the empirical CME estimator under thissetting.
We further establish a lower bound on the learning rate, which reveals thatwhen the target CME operator is smooth, the obtained upper bound is optimal.
Introduction
Approximation of the conditional expectation operator is a central issue in the statistical learning community, and many approaches have been proposed [41, 16, 17, 18].
Given random variables X and Y with a joint distribution P , the conditional expectation operator for a function f is deﬁned
[Equation]
Conventional parametric models to approximate P often involve density estimation and expensivenumerical analysis.
Hence, recent studies attempt to explore a new framework to approximate P viakernel methods.
Speciﬁcally, given kernels k X and k Y with corresponding reproducing kernel Hilbertspace H X and H Y for X and Y respectively, we may deﬁne the conditional meaning embedding(CME) operator as F ⇤ ( x ) := E P ( k Y ( · , Y ) | X = x ) , and we may employ the reproducing property toobtain
[ Pf ]( x ) =
h f, F ⇤ ( x ) i H Y for any f 2 H Y .
The advantage of the CME framework is that itallows the straightforward evaluation of conditional expectations of any function in H Y .
The CMEframework has been applied successfully to many learning problems such as probabilistic inference[32], reinforcement learning [25, 12] and causal inference [22, 28].
Despite these successful applications, there have been two main challenges in establishing a rigoroustheory of conditional mean embeddings.
The ﬁrst challenge, remarkably, has been in establishing aprincipled and sufﬁciently general deﬁnition of the conditional mean embedding itself.
The CME wasoriginally introduced as an operator mapping from H X to H Y [9, 34].
This deﬁnition has the beneﬁtof elegance, and of a straightforward expression in terms of feature covariances and cross-covariances.
A disadvantage is that the deﬁnition requires the conditional mean
[Equation] :we call this the well-speciﬁed scenario .
This strong assumption may be violated in practice (see [14] and [10, Section 3.1] for illustrations), and signiﬁcantly restricts the class of distributions on whichwe can deﬁne a CME.
An alternative approach, due to [11], is to express the conditional mean embedding as the solution ofa least-squares regression problem in a vector valued RKHS [4, 5].
In subsequent work, [26] establisha rigorous measure-theoretic deﬁnition of the conditional mean embedding as a square integrablefunction on H Y , which is the deﬁnition we will use in the present work.
Both [11, 26] connect this CME deﬁnition to the original operator-mapping deﬁnition by means of a surrogate loss, which upperbounds the regression loss.
A direct connection remained elusive until the work of [23], who showthat in the well-speciﬁed case, the CME can be arbitrarily well approximated by a Hilbert-Schmidtoperator from H X to H Y , thus connecting the operator-theoretic and measure-theoretic deﬁnitions.
The connection in the misspeciﬁed case remains to be established, however.
The second challenge has been in obtaining consistency results and the optimal learning rates forempirical estimates of the CME.
An early consistency analysis of the sample estimator, due to[33], requires very strong smoothness assumptions.
A more reﬁned analysis, due to [11], attainsthe optimal O (log n/n ) learning rate for the sample estimator, but only in the case where H Y isﬁnite dimensional.
For the inﬁnite dimensional RKHS, [28] and [26] establish consistency in thewell-speciﬁed case, with learning rates of O ( n  1 / 6 ) and O ( n  1 / 4 ) .
Subsequent work [39] yieldsa further improvement in the statistical learning rate, and consistency in the misspeciﬁed setting,building on recent developments in regularized regression under misspeciﬁcation [8].
A limitation of[39] is that it requires an explicit relation between the smoothness of the target CME and the size ofthe RKHS.
In particular, when the kernel has slow eigenvalue decay (as in the case of Matérn kernels,for example), the setting of their result is very close to the well-speciﬁed scenario.
Finally, to ourknowledge, there is presently no result establishing a matching lower bound for the CME learningrate.
Hence, whether the obtained upper rate is optimal remains unknown.
In the present work, we address the above mentioned challenges.
Building on [26, 23] and theinterpolation space theory results of [38], we establish an operator-theoretic deﬁnition of the CMEin the misspeciﬁed case , and show that the target CME can be well approximated by Hilbert-Schmidtoperators. In doing so, we avoid the well-speciﬁed requirement, and simply assume the target CMEoperator to be deﬁned in respect of an intermediate interpolation space, strictly larger than the original
RKHS. Moreover, we show that the space of Hilbert-Schmidt operators deﬁned on the interpolationspace can approximate the CME operator with arbitrary accuracy in operator norm (see Theorem 2). Next, we establish consistency and convergence rates of the CME sample estimator in themisspeciﬁed setting , building on [8, 39].
In particular, under certain benign conditions, we obtainthe optimal O (log n/n ) learning rate up to logarithmic factor. This matches with the current optimalanalysis from [11] without the restrictive assumption of ﬁnite dimensional H Y .
Thanks to ouroperator-theoretic deﬁnition of the CME, and unlike [39], we do not require an a-priori relationbetween the rate of kernel eigenvalue decay and the smoothness of the conditional mean operator(i.e., our results apply generally in the misspeciﬁed setting).
Finally, in Theorem 5, we provide anovel lower bound on the CME learning rate , which reveals that the obtained upper rate is optimalin the setting of a smooth CME operator.
Background
Throughout the paper, we consider two random variables X Y deﬁned on the measurable space( E, F E ) where E is a second countable locally compact Hausdorff space and F E its Borel  -ﬁeld.
We let ( ⌦ , F , P ) be the underlying probability space with expectation operator E .
Let ⇡ and ⌫ be thepushforward of P under X and Y respectively, i.e., X ⇠ ⇡ and Y ⇠ ⌫ .
To be rigorous, we use the Markov kernel p : E ⇥ F E !
R + to deﬁne the conditional distribution introduced before as
P [ Y 2 A | X = x ] =
Z A p ( x, dy ) , for all x 2 E and the events A 2 F E .
In addition, we denote the space of real-valued Lebesguesquare integrable functions on ( E, F E ) with respect to ⇡ as L 2 ( E, F E , ⇡ ; R ) = L 2 ( ⇡ ) and similarly to ⌫ as L 2 ( E, F E , ⌫ ; R ) = L 2 ( ⌫ ) .
We denote B for a separable Banach space with norm k · k Band H for a separable real Hilbert space with inner product h · , · i H .
We write L ( B, B 0 ) as the Banach space of bounded linear operators from B to another Banach space B 0 , equipped with theoperator norm k · k B !
B 0 .
When B = B 0 , we simply write L ( B ) instead.
We also let L p ( ⌦ , F , ⇡ ; B )denote the space of strongly F  F B measurable and Bochner p -integrable functions f : ⌦ !
Bfor 1 < p  1 .
H ⌦ H 0 denotes the tensor product of Hilbert spaces H , H 0 .
[-S 2 ( H, H 0 ) is the Hilbert space of Hilbert-Schmidt operators from H to H 0 . Tensor Product of Hilbert Spaces ([1], Section 12): Denote H ⌦ H 
 the tensor product of Hilbertspaces H , H 0 .-] The Hilbertspace H ⌦ H 0 is the completion of the algebraic tensor product with respect to the inner producth x 1 ⌦ x 0 1 , x 2 ⌦ x 0
i H ⌦
H
= h x 1 , x 2 i H h x 0 1 , x 0
i H 0 for x 1 , x 2 2 H and x 0 1 , x 0 2 2 H 0 .
Finally, we 83 denote the p -Schatten class S p ( H, H 0 ) to be the space of all compact operators C from H to H 0such that k C k S p ( H,H 0 ) :
[-h y 0 , x 
-]=  (  i ( C ))
i 2 J  ` p is ﬁnite.
Here k (  i ( C ))
i 2 J k ` p is the ` p sequence spacenorm of the sequence of the strictly positive singular values of C indexed by the countable set J .
Forp = 2 , S 2 ( H, H 0 ) is the Hilbert space of Hilbert-Schmidt operators from H to H 0 .
Reproducing Kernel Hilbert Spaces, Covariance Operators: Consider a random variable Xdeﬁned on ( E, F E ) . We let k X : E ⇥ E !
R be a symmetric and positive deﬁnite kernel functionand H X be a vector space of E !
R functions, endowed with a Hilbert space structure via aninner product h · , · i H X .
k X is a reproducing kernel of H X if and only if: 1. 8 x 2 X, k X ( · , x ) {+
+} H X ; 2 . 8 x 2 X and 8 f 2 H X , f ( x ) =
h f, k X ( x, · ) i H X .
A space H X which possesses a reproducingkernel is called a reproducing kernel Hilbert space (RKHS)[1].
We denote the canonical feature mapof H X as  X ( x ) =
k X ( · , x ) .
Similarly for H Y , we have  Y ( y ) .
We now introduce some facts about the interplay between H X and L 2 ( ⇡ ) which has been extensively studied by [29, 30],[7] and [38].
We ﬁrst deﬁne the not necessarily injective embedding I ⇡ : H X !
L 2 ( ⇡ ) , mapping a function f 2 H X to its ⇡ -equivalence class [ f ] ⇡ .
This is well-deﬁned, HilbertSchmidt, and the Hilbert-Schmidt norm satisﬁes
[Equation]
Moreover, the adjoint operator S ⇡ := I ⇤ ⇡ : L 2 ( ⇡ ) !
H X is an integral operator with respect to the kernel k X , i.e. for f 2 L 2 ( ⇡ ) and x 2 E we have
[Equation]
Next, we deﬁne the self-adjoint and positive semi-deﬁnite integral operators
[Equation]
and
[Equation]
These operators are trace class and their trace norms satisfy
[Equation]
X . Vector-valued RKHS We also give a brief overview of the vector-valued reproducing kernel Hilbertspace (vRKHS).
Since the construction is very technical, we only introduce the basic properties andrefer the reader to [4] and [5] for more details.
Deﬁnition 1.
Let E be a nonempty set and H be a real Hilbert space. Let function K : E ⇥ E !
L ( H ) be an operator valued positive-semideﬁnite (psd) kernel such that K ( x, x 0 )
=
K ( x 0 , x ) ⇤ forall x, x 0 2 E , and for all x 1 , . . .
, x n 2 E and h i , h j 2 H ,
[Equation]
Fix K , x 2 E , and h 2 H , [ K x h ] ( · ) :=
K ( · , x ) h deﬁnes a function from E to H .
We now consider
[Equation] with inner product on G pre by linearly extending the expression
[Equation]
Let G be the completion of G pre with respect to this inner product.
We call G the vRKHS induced by the kernel K .
The space
G is a Hilbert space consisting of functions from E to H with the reproducing property
[Equation]
for all F 2 G , h 2 H and x 2 E .
For all F 2 G we obtain
[Equation]
Since the inner product given by Eq.
(1) implies that K x is a bounded operator for all x 2 E .
For all F 2 G and x 2 E , we can rewrite the reproducing property Eq.
(2) as F ( x ) =
K ⇤ x F .
The linear operators K x : H !
G and K ⇤ x : G !
H are bounded with
[Equation]
and we have K ⇤ x K x
= K ( x, x 0 ) , x, x 0 2 E .
Recall X and Y are two random variables deﬁned on ( E, F E ) with marginal distribution ⇡ and ⌫ .
We let H X and H Y be the RKHS with kernel k X and k Y respectively.
In the following, we willdenote G as the vRKHS induced by the kernel K : E ⇥ E !
L ( H Y ) with
[Equation]
An important property of G is that elements in G with kernel K are isometric to Hilbert-Schmidtoperators between H X and H Y .
Theorem 1.
Let H X
and H Y be real-valued RKHS with kernel k X and k Y respectively.
Recall wedenote the tensor product space between H X and H Y as H Y ⌦ H X .
For f Y 2 H Y and g X 2 H X , 108 deﬁne the map ⇥ as
[Equation]
We then have that ⇥ deﬁnes an isometric isomorphism between H Y ⌦ H X and G .
A proof of Theorem 1 can be found in [23, Theorem 4.4].
The isometric isomorphism ⇥ induces theoperator reproducing property stated below
Corollary 1.
Recall we denote S 2 ( H X , H Y ) as the 2 -Schatten class of compact operators from H Xto H Y .
For every function F 2 G there exists an operator C := ⇥
1 ( F ) 2 S 2 ( H X , H Y ) such that
[Equation]
for all x 2 E with k C k S 2 ( H X , H Y ) =
k F k G and vice versa.
Conversely, for any pair F 2 G and 115 C 2 S 2 ( H X , H Y ) , we have C = ⇥  1 ( F ) as long as F ( x ) =
C  X ( x ) .
The proof of Corollary 1 is a simple extension of Lemma 15 in [6] and Corollary 4 . 5 in [23].
Corollary1 shows that the vRKHS G is generated via the space of Hilbert-Schmidt operators S 2 ( H X , H Y )
[Equation]
Conditional Mean Embedding: A particular advantage of kernel methods is its convenience ofoperating probability distributions, see [24, 27] for examples.
This is through the so called kernelmean embedding [1, 31, 13].
Suppose X has distribution ⇡ on E , assuming the integrability condition
R
X p k X
( x, x ) d ⇡ ( x ) < 1 , we deﬁne the kernel mean embedding µ X ( · ) =
R
X k X ( · , x ) d ⇡ ( x ) .
It iseasy to show that for each f 2 H X ,
R
X f ( x ) d ⇡ ( x ) =
h f, µ X i H X .
Replacing ⇡ with the conditionaldistribution, we obtain the kernel conditional mean embedding as deﬁned in [26].
Deﬁnition 2.
Given p ( x, dy ) , the conditional distribution of Y given X , we deﬁne the H Y -valued conditional mean embedding (CME) as
[Equation]
By reproducing property, we have E [ f Y ( Y ) | X
= x ]
= h f Y , F ⇤ ( x )
i H Y , 8 f Y 2 H Y and x 2 E .
The approximation of F ⇤ is a key concept in kernel methods.
Assuming C [-
 ( E X ) where C 0 ( E X ) is the space of continuous functions vanishing at inﬁnity 1 ; and ii) H X-]XX is injective, [34, 9, 10]show that F ⇤ has a closed form expression via
[Equation]
where C Y X = E (  Y ⌦  X ) and C † denotes the pseudoinverse of C .
In addition, estimation of F ⇤ can also be interpreted as the following optimization problem [11, 26, 39]
[Equation]
ˆ F  ( x ) :
= ˆ C Y | X,   X ( x ) , where  is the regularization parameter and k · k HS is the Hilbert-Schmidt operator norm.
Implicit in the construction, however, is the assumption that
[Equation]
This is a strong assumption [10, 14] and constitutes the so-called well-speciﬁed case F ⇤ ( · ) 2 G .
Ournext task is to characterize the Hilbert spaces used to deﬁne the misspeciﬁed case, and to deﬁne the
CME in this setting.
Interpolation Space, misspeciﬁed CME: We now review the results of [38, 8] that set out theeigendecompositions of L X and C XX , and apply these in constructing the interpolation spaces usedfor the misspeciﬁed setting.
We deﬁne an index set I , a non-increasing sequence ( µ i ) i 2 I > 0 , and afamily ( e i ) i
I 2 H X , such that ([ e i ] ⇡ ) i 2 I is an orthonormal basis (ONB) of L 2 ( ⇡ ) and ( µ 1 / 2 i e i )
i 2 Iis an ONB of H X , we have
[Equation]
For ↵  0 , we deﬁne the ↵ -interpolation space [38] by[ H ]
[Equation]
, equipped with the ↵ -power norm
[Equation]
I [Equation] For ( a i ) i 2 I 2 ` 2 ( N ) , the ↵ -interpolation space becomes a Hilbert space with inner product deﬁned as
[Equation]
Moreover, ⇣ µ ↵ / 2 i [ e i ] ⇡ ⌘ i forms an ONB of [ H ] ↵ X and consequently [ H ] ↵ X is a separable Hilbertspace.
In the following, we use the abbreviation k · k ↵ :=
k · k [ H ] ↵ X .
For ↵ =
we have [ H ] 0 X =ran I ⇡ ✓ L 2 ( ⇡ ) with k · k 0 =
k · k L 2 ( ⇡ ) .
Moreover, for ↵ = 1 we have [ H ] 1 X = ran I ⇡ and [ H ] 1 X isisometrically isomorphic to the closed subspace (ker I ⇡ ) ?
of H X via I ⇡ , i.e. k [ f ] ⇡ k 1 = k f k H X forf 2 (ker I ⇡ ) ? .
For 0 <  < ↵ < 1 , and assuming I ⇡ is injective, we have
[Equation]
Finally, if
P
[Equation]
[Equation]
1 , H ↵ X is an RKHS with corresponding kernel deﬁned as
[Equation]
For a function f ↵ X = P i a i µ ↵ / 2 i e i 2 H ↵ X , we recover the reproducing property via
[Equation]
Remark 1. We point out the distinction between H ↵ X and [ H ] ↵ X .
In particular, H ↵ X denotes theinterpolating RKHS consisting of continuous functions, while [ H ] ↵ X is the interpolating Hilbert space,where elements are deﬁned through equivalence class.
Under our construction, we have H ↵ X ✓ [ H ] ↵ X .
We use the interpolation space to address the CME in the context of misspeciﬁed case.
In particular,we consider the vRKHS G ↵ induced by the kernel K ↵ : E ⇥ E !
L ( H Y ) via [Equation]
Assuming H ↵ X is an RKHS and utilizing the same analysis from Theorem 1 and Corollary 1, we havethe following result.
Corollary 2. For every function F ↵ 2 G ↵ , there exists an operator C ↵ :=
( ⇥ ↵ )  1 ( F ↵ ) {+
+}
S 2 ( H ↵ X , H Y ) such that
[Equation]
for all x 2 E with k C ↵ k S 2 ( H ↵ X , H Y ) = k F ↵ k G ↵ and vice versa.
Conversely, for any pair F ↵ 2 G ↵and C ↵ 2 S 2 ( H ↵ X , H Y ) , we have C ↵ = ( ⇥ ↵ )  1 ( F ↵ ) as long as F ↵ ( x ) =
C ↵  ↵ X ( x ) .
We have that⇥ ↵ deﬁnes an isometric isomorphism between H Y ⌦ H ↵ X and G ↵ .
Corollary 2 shows that the vRKHS G ↵ is generated via S 2 ( H ↵ X , H Y )
[Equation]
Throughout the paper, we assume that F ⇤ 2 G ↵ , indicating E ( f Y ( Y ) | X = · ) 2 H ↵ X , 8 f Y 2 H Y .
Comparing to Eq.
(5), we can see that our assumption is strictly weaker in the sense that H X ⇢ H ↵ X ,for some ↵ < 1 .  Approximation of CME with Interpolation Space In this section, we demonstrate that the operator F ⇤ can be arbitrarily well approximated in L 2 normby vectors in the vRKHS G ↵ , without imposing any smoothness assumption on F ⇤ . Before we present our theoretical analysis, we ﬁrst state some technical assumptions on the previouslydeﬁned RKHS and kernels.
The RKHS H X and H Y are separable. This is satisﬁed if E is a Polish space and k X and k Y are continuous [36]. k X ( · , x ) and k Y ( · , y ) are F R measurable for all x, y 2 E .
The feature maps of k X and k Y satisfy E
[Equation]
< 1 ,
respectively. This can be trivially satisﬁed if sup x,y 2 E k X ( x, x ) {+, k Y ( y, y ) < 1 . Let C 
+}
( E ) be the space of continuous functions vanishing at inﬁnity.
{+Then H X , H Y 
+}
C 0 ( E ) . This can be satisﬁed if k X ( x, x ) and k Y ( y, y ) are bounded, k X ( · , x ) 2 C 0 ( E ) andk Y ( · , y ) 2 C 0 ( E ) . H X and H Y
are dense in L 2 ( ⇡ ) and L ⌫ ) respectively,
i
e., k X and k Y
are universal.
Remark 2.
Assumptions 1 - 3 ensure that the inclusion mappings I ⇡ , I µ are bounded and compact, andthat H X
and H Y can be continuously embedded into L 2 ( ⇡ ) and L 2 ( ⌫ ) , respectively.
Assumptions4 - 5 implies that the RKHS H X and H Y are dense in L 2 ( ⇡ ) and L 2 ( ⌫ ) respectively.
In addition,
Assumptions 1 - 5 together imply that the RKHS G induced by kernel K = k Id H Y is dense in
L 2 ( E, F E , ⌫ , H Y ) .
We also remark that the above assumptions are not restrictive in practice, sincewell-known kernels such as the Gaussian, Laplacian and Matérn kernels satisfy all of the aboveassumptions for E ✓ R d and arbitrary probability measures ⇡ and ⌫ on ( E, F E ) .
With the above assumptions made, our Theorem 2 demonstrates that the CME operator F ⇤ can bearbitrarily approximated by operators in S 2 ( H ↵ X , H Y ) .
Theorem 2.
Let Assumptions 1 - 5 be satisﬁed, for every  > 0 , there exists a Hilbert-Schmidt operator
C ↵ 2 S 2 ( H ↵ X , H Y ) such that [Equation] In Eq (7) we prove that for any operator C ↵ in S 2 ( H ↵ X , H Y ) , there is a corresponding vector F ↵ inthe vRKHS G ↵ induced by the kernel K ↵ = k ↵ Id.
As a result, Theorem 2 shows that the constructedinterpolation space G ↵ is rich enough to approximate the CME operator F ⇤ up to arbitrary accuracyunder mild assumptions.
However, there are inﬁnitely many operators in S 2 ( H ↵ X , H Y ) , and how toﬁnd the appropriate C ↵ remains unclear.
The following lemma provides a way to ﬁnd C ↵ .
Lemma 1.
Let ↵ ,  2 (0 , 2] and  < ↵ , the following two holdsi).
For any f ↵ 2 H ↵ X and C ↵ 2 S 2 ( H ↵ X , H Y ) , there is a corresponding f  2 H  X , C  : H  X !
H Yand C
=
C ↵ C ↵   XX such that
[Equation]
ii).
We have C  2 S 2 ( H  X , H Y ) .
By Lemma 1, we can see that if for some C Y | X 2 S 2 ( H X , H Y ) such that C Y |
X k ( · , x ) =
E [  Y ( Y ) | X = x ] = F ⇤ ( x ) , we will have that C  Y |
X k  ( · , x ) =
C Y | X C 1   XX k  ( · , x ) =
C Y |
X k ( · , x ) =
F ⇤ ( x ) .
Previous result [34] shows that C Y | X
= C Y X C † XX .
However, this requires that h f Y , C Y | X  X ( · ) i H Y 2 H X , for all f Y 2 H Y .
In contrast, if we use C  Y | X instead,this would only require h f Y , C  Y | X   X ( · ) i H Y 2 H  X , which is strictly weaker than the previousrequirement.
Remark 3.
We point out that a simpler version of part i) in Lemma 1 in the case of ↵ = 1 hasbeen proved in [39]. Part ii) in Lemma 1 is novel and is a crucial result that enables us to study thelearning rate in the misspeciﬁed case. Our following result establishes the connection between the CME and the Hilbert-Schmidt operatorspace S 2 ( H ↵ X , H Y ) and points out the condition when the approximation error can be zero. Theorem 3.
Under Assumptions 1 - 5 , the following statements are equivalent:
i) F ⇤ ( · ) =
E [  Y ( Y ) | X = · ] 2 G ↵ ;
ii) There exists an operator C ↵ 2 S ( H ↵ X , H Y )
such that [( C ↵ )
⇤ f Y ]( x ) =
h ( C ↵ )
⇤ f Y ,  ↵ X ( x ) i H ↵ X
= h f Y ,
C ↵  ↵ X ( x ) i H Y
=
E [ f Y ( Y ) | X = x ] for all x 2 E and f Y 2 H Y . Both i) and ii) imply the following iii) There exists an operator C ↵ such that k F ⇤  C ↵  ↵ X ( · ) k 2 L 2 ( E, F E , ⇡ , H Y )
= 0 . By comparing point ii) in Theorem 3 to the CME property [34], one can see that the operator C ↵ isexactly the CME operator introduced in [34]. In particular, in the well-speciﬁed case where ↵ = 1 ,we have the closed form of the CME operator as C =
C Y X C † XX [34, 15].  Optimal Learning Rate for CME In this section, we derive the learning rate for    ˆ F  ( X )  F ⇤ ( X )    L 2 ( E, F E , ⇡ , H Y ) .
We ﬁrst stateadditional assumptions that are needed in our derivation.
As our assumptions match those of [8], weinclude the corresponding labels from [8] for ease of reference.6 .
(EVD+) Recall ( µ i )
i 2 I are the eigenvalues of C XX , for some constant c 1 , c 2 > 0 andp 2 (0 , 1) , we assume that
[Equation]
7 . (EMB)
We assume that for ↵ 2 ( p, 1] , the inclusion map I ↵⇡ : H ↵ X , !
L 1 ( ⇡ ) is continuousand has bounded norm A > 0 .
(SRC) There exists <
 2 such that F ⇤ = C  Y | X k  X ( · , x ) 2 G
(MOM)
We assume that there are constants  , R > 0 such that
[Equation]
Assumption 6 is a standard assumption on the eigenvalue decays (see more details in [3, 8, 39]).
Assumption 7 is referred as the embedding property in [8] and can be loosely understood as requiringthat
P i µ ↵ i < 1 (see [8]). Therefore, the smaller ↵ is, the smoother of the corresponding RKHS H X .
Since we assume k X to be bounded, the embedding property always hold true when ↵  1 .
Assumption 8 is often referred as the source condition in literature ([3, 8, 19, 20]).
It imposes thesmoothness assumption on the target CME operator F ⇤ .
In particular, when   1 , the sourcecondition implies that F ⇤ has a representative from G , indicating the well speciﬁed scenario.
However,once we let  < 1 , we are in the hard learning scenario, which is the main interest in this manuscript.
Finally, Assumption 9 is a typical noise condition [-is employed. In particular, insteadof assuming F ⇤ 
-]that we impose on the underlying distribution P(see [3, 8, 39] for more details).
Under these assumptions, our next result provide an upper bound on the learning rate.
Theorem 4.
Let Assumption 1 - 3 and 6 - 9 hold, for some constant c and r > 1 , deﬁne the parameter
[Equation]
then we have for some constant c , with probability greater than 1   ,
[Equation]
Theorem 4 provides the ﬁnite sample learning rate for the empirical CME estimator deﬁned in Eq.
It states that the learning rate for ˆ F  is governed by the interplay between p , ↵ , and  .
Intuitively,p describes the decay rate of the eigenvalues ( µ i )
i 2 I , and ↵ determines the boundedness of theinterpolation kernel and has maximum value of 1 according to our assumption.  characterizes thesmoothness of the target CME operator.
The exponent  / max { ↵ ,  + p } explicitly provides the learning rate for the CME operator.
Forexample, if we have ↵   , we obtain a learning rate of  / (  + p ) .
In particular, if Gaussian kernelis used where p is arbitrarily close to 0 , we can see that our learning rate can even achieve log r n/n .
In addition, if a kernel with slow eigenvalue decay is used, such as the Matérn kernel, we can obtainthe minimax optimal learning rate n  1 / 2 up to logarithmic factors if we have p   .
Finally, in theworst case where  is close to 0 , the learning rate can be arbitrarily slow.
We point out that our obtained upper bound on the learning rate has a similar form to [39].
Inparticular, by considering the  -norm for some 0 <  <  , [39] obtain an upper bound of
O ⇣ n log r n ⌘     max { ↵ ,  + p } .
There are two main differences between their result and ours.
A ﬁrstdifference is that we consider the L 2 difference between ˆ F  and F ⇤ , while [39] study the differencebetween ˆ C Y | X,  and C  Y | X .
A second difference, which is arguably the more important of the two,is that [39] address the misspeciﬁed case by assuming that the smoothness of the target CME F ⇤ isrelated to the size of the RKHS H X , i.e.,  > p . This can be limiting, especially when non-smoothkernels such as Matérn kernels are used, since in these cases p is close to 1 . In contrast, by realizingthat the vRKHS induced by the kernel K  can be identiﬁed with the Hilbert-Schmidt operator space,i.e., C  2 S 2 ( H  X , H Y ) , we can avoid making this strong assumption (with implications for theproof, as brieﬂy discussed next). Sketch of Proof
We ﬁrst notice that
[Equation]
where we deﬁne F Y | X,  ( x ) = C Y X ( C XX +  )  1 k X ( · , x ) . It is established that ˆ F  ( x ) = ˆ
C Y X ( ˆ C XX +  )  1 k X ( · , x ) .
Hence we can see that the error for the ﬁrst term is mainly due to the sample approximation.
We therefore refer the ﬁrst term as the Variance , for which [39] establish consistency.
Since we assume F ⇤ ( x ) = C  Y | X k  X ( · , x ) , the second term can be written as
[Equation]
We refer to this term as the Bias .
Our proof of convergence of the bias adapts the proof in [28,
Theorem 6 ], and utilizes the fact that C  Y | X is Hilbert-Schmidt to obtain a sharp rate.
Our ﬁnal theorem provides a lower convergence rate, which allows us to conﬁrm the optimality ofour learning rate.
Theorem 5.
Suppose we are given a dataset D := { ( x i , y i ) } ni =1 . Let Assumptions 1 - 3 hold.
Let ⇡be the marginal distribution deﬁned on E such that Assumption 6 and 7 hold for kernel k X with 0 <p  ↵  1 .
Then for all 0 <   2 and all constants  , R, c 1 , there exists constants c 2 , c 3 , c 4 > 0such that for all learning methods D !
ˆ F D and all constant ⌧ > 0 , there is a distribution P 0 deﬁnedon E ⇥ H Y with its marginal distribution as ⇡ such that k F ⇤ k L 2 ( E, F E , ⇡ ; H Y )  c 1 and Assumption8 and 9 are satisﬁed with  and R and with probability not less than 1  c 2 ⌧ 1 /c 3 ,
[Equation]
Theorem 5 states that under Assumption 1 - 3 and 6 - 9 , there is no learning method can achieve alearning rate faster than n  max { ↵ ,  } max { ↵ ,  } + p .
To our knowledge, this is the ﬁrst analysis that demonstratesthe lower rate for CME learning.
In the context of regularized regression, [3, 37, 2] provide a similarlower bound on the learning rate.
However, a key difference in our analysis is that the output of theregression learning now lives in an potentially inﬁnite dimensional RKHS, rather than in R .
Ouranalysis reveals that in the case where ↵   , the obtained upper rate in Theorem 4 is optimal.
However, the optimal rate for  < ↵ still remains a challenge (even when the output is R ).
Conclusion
In this paper, we provide a rigorous theoretical foundation for approximating the CME operator,as well as studying the statistical learning rate.
Utilizing recently developed interpolation spacetechniques, we ﬁrst demonstrate that the Hilbert-Schmidt operator space S 2 ( H ↵ X , H Y ) deﬁned onthe interpolation space H ↵ X and the target Hilbert space H Y can approximate the CME operator F ⇤with arbitrary accuracy.
This then allows us to deﬁne the target CME operator to live in the largerinterpolation space G ↵ , in contrast to the well-speciﬁed setting where F ⇤ 2 G .
By doing so, we areable to study the convergence rate of the empirical CME operator in the misspeciﬁed scenario.
In themost benign case, we derive the optimal learning rate O (log n/n ) up to logarithmic factors withoutrequiring ﬁnite dimensionality on H Y . We also show that when less smooth kernels are used, thelearning rate can achieve the minimax optimal rate O ( n  1 / 2 ) .
Finally, we provide a matching lowerbound on the learning rate, which reveals that the obtained upper rate is optimal when  > ↵ .
Looking beyond, our current interpolation space setting indicates that the convergence rate canbe arbitrarily slow if  ! 0 .
This prevents the learning of the constant function, which plays acrucial role in completing the theory of CME as pointed out by [14].
Addressing this challenge is animportant future direction of research.