Optimal Learning Rates for Regularized Conditional Mean Embedding
Abstract
We address the consistency of a kernel ridge regression estimate of the conditional mean embedding (CME), which is an embedding of the conditional distribution of
Y given X into a target reproducing kernel Hilbert space H Y .
The CME allows us to take conditional expectations of target RKHS functions, and has been employed in nonparametric causal and Bayesian inference.
We address the misspeciﬁed setting, where the target CME is in the space of Hilbert-Schmidt operators acting from an input interpolation space between H X and L 2 , to H Y .
This space of operators is shown to be isomorphic to a newly deﬁned vector-valued interpolation space.
Using this isomorphism, we derive a novel and adaptive statistical learning rate for the empirical CME estimator under the misspeciﬁed setting.
Our analysis reveals that our rates match the optimal O (log n/n ) rates without assuming H Y to be ﬁnite dimensional.
We further establish a lower bound on the learning rate, which shows that the obtained upper bound is optimal.
Introduction
Approximation of the conditional expectation operator is a central issue in the statistical learning community, and many approaches have been proposed [42, 20, 21, 22].
Given random variables X and Y , the conditional expectation operator for a function f is deﬁned
[Equation]
Conventional parametric models to approximate P often involve density estimation and expensive numerical analysis.
Hence, recent studies attempt to explore a new framework to approximate
P via kernel methods.
Speciﬁcally, given kernels k X and k Y with corresponding reproducing kernel Hilbert space H X and H Y for X and Y respectively, we may deﬁne the conditional mean embedding (CME) as F ⇤ ( x ) :=
E [ k Y ( · , Y )
| X = x ] , and we may employ the reproducing property to obtain [ Pf ]( x )
=
h f, F ⇤ ( x ) i H Y for any f 2 H Y .
The advantage of the CME framework is that it allows the straightforward evaluation of conditional expectations of any function in H Y .
The CME framework has been applied successfully to many learning problems such as probabilistic inference [35], reinforcement learning [28, 16] and causal inference [25, 31].
Despite these successful applications, there have been two main challenges in establishing a rigorous theory of CMEs.
The ﬁrst challenge, remarkably, has been in establishing a principled and sufﬁciently general deﬁnition of the conditional mean embedding itself.
The CME was originally introduced as an operator mapping from H X to H Y [12, 37].
This deﬁnition has the beneﬁt of elegance, and of a straightforward expression in terms of feature covariances and cross-covariances.
A disadvantage is that the deﬁnition requires the conditional mean E [ g ( Y )
| X = · ] 2 H X , 8 g 2 H Y .
This strong assumption may be violated in practice (see [18, 19] and [13, Section 3.1] for illustrations and alternative requirements), and signiﬁcantly restricts the class of distributions on which we can deﬁne a CME.
An alternative approach, due to [15], is to express the conditional mean embedding as the solution of a least-squares regression problem in a vector-valued RKHS [5, 6].
In subsequent work, a rigorous measure-theoretic deﬁnition of the conditional mean embedding as the H Y -valued square integrable function F ⇤ is established in [29, 18], which is the deﬁnition we will use in the present work.
Both [15, 29] connect this CME deﬁnition to the original operator-mapping deﬁnition by means of a surrogate loss, which upper bounds the regression loss.
A direct connection remained elusive until the work of [26, 19], which show that under denseness assumptions, the CME can be arbitrarily well approximated by a Hilbert-Schmidt operator from H X to H Y , thus connecting the operator-theoretic and measure-theoretic deﬁnitions.
The second challenge has been in obtaining consistency results and the optimal learning rates for empirical estimates of the CME.
An early consistency analysis of the sample estimator, due to [36], requires very strong smoothness assumptions.
A more reﬁned analysis, due to [15], attains the minimax optimal learning rate O (log n/n ) for the sample estimator, but only in the case where H Y is ﬁnite dimensional.
For the inﬁnite dimensional RKHS, [31] and [29] establish consistency in the well-speciﬁed case, with learning rates of O ( n  1 / 6 ) and O ( n  1 / 4 ) .
Nevertheless, the obtained rates are far from optimal and consistency under misspeciﬁed setting was not established.
Recently, [41] obtains a sharper rate under the misspeciﬁed case using the interpolation RKHS.
The results of [41] impose assumptions, however, which strongly limit their applicability (refer to Remark 5 for a rigorous discussion):
They require an explicit relation between the smoothness of the target CME and the size of the RKHS.
In particular, when the kernel has slow eigenvalue decay (as in the case of Matérn kernels, for example), the setting is very close to the well-speciﬁed scenario.
They rely on the explicit construction of an interpolation RKHS.
Unlike in [11], where a similar approach is based only on equivalence classes of functions (i e., Sobolev-like spaces), this concept requires the embedding of the RKHS into the corresponding L 2 -space (or equivalently the integral operator) to be injective—which is generally not the case (see [40] for details).
Counterexamples can easily be constructed when one considers degenerate pushforward measures on the RKHS in one or more coordinate directions (for example point masses).
By contrast, the authors of [11] do not explicitly require the injectivity in the real-valued learning scenario.
Moreover, in case where the chosen kernel has slow eigenvalue decay, the constructed interpolation RKHS is not well-deﬁned.
Finally, to our knowledge, there is presently no result establishing a matching lower bound for the
CME learning rate in the case where H Y may be inﬁnite dimensional.
Hence , whether the obtained upper rate is optimal remains unknown.
In the present work, we address the challenges mentioned above.
Building on [29, 26] and the interpolation space theory results of [40, 11], we introduce an interpolation space consisting of vector-valued functions via a natural tensor product construction.
This concept is compatible with the recent measure-theoretic deﬁnition of the CME due to [29] and allows to prove convergence in the misspeciﬁed setting without the limitations of prior work.
Based on this novel vector-valued interpolation space, we establish consistency and convergence rates of the CME sample estimator in the misspeciﬁed setting .
In particular, under certain benign conditions, we obtain the optimal
O (1 /n ) learning rate up to a logarithmic factor. This matches with the current optimal analysis from [15] without the restrictive assumption of ﬁnite dimensional H Y .
Thanks to our operator-theoretic deﬁnition of the CME, and unlike [41], we do not require an a-priori relation between the rate of kernel eigenvalue decay and the smoothness of the conditional mean operator (i.e., our results apply generally in the misspeciﬁed setting).
Finally, in Theorem 3, we provide a novel lower bound on the CME learning rate , which demonstrates that the obtained upper rate is optimal in the setting of a smooth CME operator.
Background
Throughout the paper, we consider two random variables X , Y deﬁned respectively on the second countable locally compact Hausdorff spaces E X and E Y endowed with their respective Borel  -ﬁeld
F E X and F E Y .
We let ( ⌦ , F , P ) be the underlying probability space with expectation operator E .
Let ⇡ and ⌫ be the pushforward of P under X and Y respectively, i.e., X ⇠ ⇡ and Y ⇠ ⌫ .
We use the Markov kernel p : E X ⇥ F E Y !
R + to deﬁne the conditional distribution:
[Equation]
, for all x 2 E X and events A 2 F E Y .
We denote the space of real-valued Lebesgue square integrable functions on ( E X , F E X ) with respect to ⇡ as L 2 ( E X , F E X , ⇡ ) abbreviated L 2 ( ⇡ ) and similarly for ⌫ we use L 2 ( E Y , F E Y , ⌫ ) abbreviated L 2 ( ⌫ ) .
Let B be a separable Banach space with norm k · k B and H a separable real Hilbert space with inner product h · , · i H .
We write L ( B, B 0 ) as the Banach space of bounded linear operators from B to another Banach space B 0 , equipped with the operator norm k · k B !
B 0 .
When B = B 0 , we simply write L ( B ) instead.
We also let L p ( E X , F E X , ⇡ ; B ) , abbreviated L p ( ⇡ ; B ) , the space of strongly F E X  F B measurable and Bochner p -integrable functions f : E X !
B for 1  p  1 .
Finally, we denote the p -Schatten class S p ( H, H 0 ) to be the space of all compact operators C from H to another Hilbert space H {+
+} such that k C k S p ( H,H 0 ) :
=  (  i ( C ))
i 2 J  ` p is ﬁnite.
Here k (  i ( C ))
i 2 J k ` p is the ` p sequence space norm of the sequence of the strictly positive singular values of C indexed by the countable set J .
For p = 2 ,
S 2 ( H, H 0 ) is the Hilbert space of Hilbert-Schmidt operators from H to H 0 .
Tensor Product of Hilbert Spaces ([1], Section 12): Denote H ⌦ H
the tensor product of Hilbert spaces H , H 0 .
The Hilbert space H ⌦ H 0 is the completion of the algebraic tensor product with respect to the norm induced by the inner product h x 1 ⌦ x 0 1 , x 2 ⌦ x 0
i H ⌦
H
= h x 1 , x 2 i H h x 0 1 , x 0
i H 0 for x 1 , x 2 2 H and x 0 1 , x 0 2 2 H 0 deﬁned on the elementary tensors of H ⌦ H 0 .
This deﬁnition extends to span { x ⌦ x 0 | x 2 H, x 0 2 H 0 } and ﬁnally to its completion.
The space H ⌦ H 0 is separable whenever both H and H 0 are separable.
The element x ⌦ x 0 2 H ⌦ H 0 is treated as the linear rank-one operator x ⌦ x 0 : H 0 !
H deﬁned by y 0 !
h y 0 , x
i H 0 x for y 0 2 H 0 .
Based on this identiﬁcation, the tensor product space H ⌦ H 0 is isometrically isomorphic to the space of Hilbert-Schmidt operators from
H 0 to H , i.e., H ⌦ H 0 ' S 2 ( H 0 , H ) .
We will hereafter not make the distinction between those two spaces and see them as identical.
If { e i } i 2 I and { e 0 j } j 2 J are orthonormal basis in H and H 0 , { e i ⌦ e 0 j } i 2 I,j 2 J is an orthonormal basis in H ⌦ H 0 .
Remark 1 ([1], Theorem 12.6.1) .
Consider the Bochner space L 2 ( ⇡ ; H ) where H is a separable Hilbert space.
One can show that L 2 ( ⇡ ; H ) is isometrically identiﬁed with the tensor product space H ⌦
L 2 ( ⇡ ) .
Reproducing Kernel Hilbert Spaces, Covariance Operators: We let k X : E X ⇥ E X !
R be a symmetric and positive deﬁnite kernel function and H X be a vector space of E X !
R functions, endowed with a Hilbert space structure via an inner product h · , · i H X .
k X is a reproducing kernel of H X if and only if: 1. 8 x 2 E X , k X ( · , x ) 2 H X ; 2 . 8 x 2 E X and 8 f 2 H X , f ( x ) =
h f, k X ( x, · ) i H X .
A space H X which possesses a reproducing kernel is called a reproducing kernel
Hilbert space (RKHS)[2].
We denote the canonical feature map of H X as  X ( x ) =
k X ( · , x ) .
Similarly for E Y , we consider a RKHS H Y with symmetric and positive deﬁnite kernel k Y :
E Y ⇥ E Y !
R and canonical feature map denoted as  Y .
We require some technical assumptions on the previously deﬁned RKHSs and kernels:
H X and H Y are separable, this is satisﬁed if E X and E Y are Polish spaces and k X , k Y are continuous [38]; 2 .
k X ( · , x ) and k Y ( · , y ) are measurable for ⇡ -almost all x 2 E X and ⌫ -almost all y 2 E Y ; 3 .
k X ( x, x )   2 X for ⇡ -almost all x 2 E X and k Y ( y, y )
  2 Y for ⌫ -almost all y 2 E Y .
Note that the above assumptions are not restrictive in practice, as well-known kernels such as the Gaussian, Laplacian and Matérn kernels satisfy all of the above assumptions on R d .
We now introduce some facts about the interplay between H X and L 2 ( ⇡ ) , which has been extensively studied by [32, 33],[9] and [40].
We ﬁrst deﬁne the (not necessarily injective) embedding I ⇡ : H X !
L 2 ( ⇡ ) , mapping a function f 2 H X to its ⇡ -equivalence class [ f ] .
The embedding is a well-deﬁned compact operator as long as its Hilbert-Schmidt norm is ﬁnite.
In fact, this requirement is satisﬁed since its
Hilbert-Schmidt norm can be computed as
[Equation]
The adjoint operator S ⇡ := I ⇤ ⇡ : L 2 ( ⇡ ) !
H X is an integral operator with respect to the kernel k X , i.e. for f 2 L 2 ( ⇡ ) and x 2 E X
we have
[Equation]
Next, we deﬁne the self-adjoint and positive semi-deﬁnite integral operators
[Equation]
and
[Equation]
These operators are trace class and their trace norms satisfy
[Equation]
Vector-valued RKHS We also give a brief overview of the vector-valued reproducing kernel Hilbert space (vRKHS).
We refer the reader to [5] and [6] for more detail.
Deﬁnition 1.
Let H be a real Hilbert space and K : E X ⇥ E X !
L ( H ) be an operator valued positive-semideﬁnite (psd) kernel such that K ( x, x 0 )
=
K ( x 0 , x ) ⇤ for all x, x2 E X , and for all x 1 , . . .
, x n 2 E X
and h i , h j 2 H ,
[Equation]
Fix K , x 2 E X , and h 2 H , [ K x h ] ( · ) :=
K ( · , x ) h deﬁnes a function from E X to H .
We now consider
G pre := span { K [Equation] [Equation] } with inner product on G pre by linearly extending the expression
[Equation]
Let G be the completion of G pre with respect to this inner product.
We call G the vRKHS induced by the kernel K .
The space G is a Hilbert space consisting of functions from E X to H with the reproducing property
[Equation]
for all F 2 G , h 2 H and x 2 E X .
For all F 2 G we obtain
[Equation]
Since the inner product given by Eq.
(1) implies that K x is a bounded operator for all x 2 E X .
For all F 2 G and x 2 E X , Eq.
(2) can be written as F ( x ) =
K ⇤ x F .
The linear operators K x : H !
G and K ⇤ x : G !
H are bounded with
[Equation]
and we have K ⇤ x K x
= K ( x, x 0 ) , x, x 0 2 E X .
In the following, we will denote G as the vRKHS induced by the kernel K : E X ⇥ E X !
L ( H Y ) with
[Equation]
An important property of G is that elements
Theorem 1 (Theorem 4 . 4 in [26]) .
Let H X
and H Y be real-valued RKHS with kernel k X and k Y respectively.
For f Y 2 H Y and g X 2 H X , deﬁne the map ¯ on the elementary tensors as
[Equation]
We then have that ¯ deﬁnes an isometric isomorphism between S 2 ( H X , H Y ) and G through linearity and completion.
More details regarding Theorem 1 can be found in [26, Theorem 4.4].
The isometric isomorphism ¯ induces the operator reproducing property stated below.
Corollary 1.
For every function F 2 G there exists an operator C :
= ¯  1 ( F ) 2 S 2 ( H X , H Y ) such that
[Equation]
for all x 2 E X with k C k S 2 ( H X , H Y )
=
k F k G and vice versa.
Conversely, for any pair F 2 G and C 2 S 2 ( H X , H Y ) , we have C = ¯  1 ( F ) as long as F ( x ) =
C  X ( x ) .
The proof of Corollary 1 is a simple extension of Lemma 15 in [7] and Corollary 4 . 5 in [26].
Corollary 1 shows that the vRKHS G is generated via the space of Hilbert-Schmidt operators S 2 ( H X , H Y ) G = { F : E X ! H Y | F = C  X ( ·
[Equation]
, C 2 S 2 ( H X , H Y ) } .
Conditional Mean Embedding: A particular advantage of kernel methods is its convenience of operating probability distributions, see [27, 30] for examples.
This is through the so called kernel mean embedding [2, 34, 17].
Assuming the integrability condition
R
E X p k X
( x, x ) d ⇡ ( x ) < 1 (which is satisﬁed when the kernel is almost surely bounded), we deﬁne the kernel mean embedding µ X ( · ) =
R
E X k X ( · , x ) d ⇡ ( x ) .
It is easy to show that for each f 2 H X ,
[Equation]
Replacing ⇡ with the conditional distribution, we obtain the kernel conditional mean embedding as deﬁned in [29, 18].
Deﬁnition 2.
The H Y -valued conditional mean embedding (CME) for the Markov kernel p ( x, dy ) is deﬁned as
[Equation]
By the reproducing property, we have E [ f Y ( Y ) | X
= x ]
= h f Y , F ⇤ ( x )
i H Y , 8 f Y 2 H Y and x 2 E X .
The approximation of F ⇤ is a key concept in kernel methods.
By [26], suppose we impose Assumptions 1 - 3 together with two additional assumptions: i) H X ✓ C
( E X ) where C 0 ( E X ) is the space of continuous functions vanishing at inﬁnity 2 ; and ii) H X is dense in L 2 ( ⇡ ) , then we have that G is dense in L 2 ( E X , F E X , ⇡ ; H Y ) .
As a result, for any  > 0 , there is an F 2 G such that k F  F ⇤ k L 2 <  .
Hence, in the literature, we often assume the so-called well-speciﬁed case to obtain a closed-form solution,
[Equation]
It is shown in [18, Theorem 5 . 3 ] and [26, Corollary 5 . 6 and Remark 5 . 8 ] that F ⇤ admits a closed form expression under Eq.
(4) via
[Equation]
where C Y X =
E [  Y ( Y ) ⌦  X
( X )] and C † denotes the pseudoinverse of C .
Remark 2.
We point out that in the original derivations, the CME is written as F ⇤ ( x ) =
C Y X C † XX  X ( x ) [37, 12, 13].
However, C † XX is not globally deﬁned if H X is inﬁnite-dimensional.
Hence the expression C Y X C † XX  X ( x ) is problematic, as we expect F ⇤ to be deﬁned for all x 2 E X based on the Markov kernel p .
In the well-speciﬁed scenario, [18] corrected this issue by deﬁning the CME as ( C † XX C XY )
⇤  X ( x ) .
It is shown that in this case, ( C † XX C XY )
⇤ is bounded (actually Hilbert-Schmidt, see also [19]), and hence globally deﬁned.
The connection of this corrected operator-theoretic perspective to the well-speciﬁed regression scenario was established in [26].
Once we have the closed-form solution, a natural question to ask is how to estimate the CME.
Indeed, this has been extensively studied in [15, 29, 41].
Given a data set D = { ( x i , y i ) } ni =1 independently and identically sampled from the joint distribution of X and Y , a regularized estimate of F ⇤ is the solution of the following optimization problem:
[Equation]
) , where  is the regularization parameter.
Implicit in the construction, however, is the assumption F ⇤
G that the solution is well-speciﬁed.
We provide a few remarks regarding this assumption:
Remark 3.
i)
In the literature, the prevalent deﬁnition of well-speciﬁedness is through
[Equation]
see e.g. [12, 37, 18] for details.
However, this deﬁnition is not equivalent to that in Eq.
Speciﬁcally, assuming F ⇤ 2 G implies that Eq.
(6) holds.
Nonetheless, the reverse is not true.
In particular, there exist concrete examples satisfying Eq.
(6), but the corresponding operator representative of the CME is not Hilbert–Schmidt (see Section D in Appendix for details).
To avoid confusion, we refer to Eq.
(4) as the well-speciﬁed case hereafter. ii)
The conventional assumption Eq.
(6) can actually be reﬁned via the inclusion map I ⇡ .
In particular, since I ⇡ is an inclusion map from H X to L 2 ( ⇡ ) and E [ f Y ( Y ) | X = · ] 2 H X , we can apply I ⇡ to
E [ f Y ( Y ) | X = · ] .
In addition, we are only interested in the case where I ⇡ ( E [ f Y ( Y ) | X = · ]) 6 = 0 .
As a result, the reﬁned deﬁnition should be
[Equation]
We now characterize the Hilbert spaces used to deﬁne the CME in the misspeciﬁed setting.
Real-valued Interpolation Space
: We review the results of [40, 11] that set out the eigendecompositions of L X and C XX , and apply these in constructing the interpolation spaces used for the misspeciﬁed setting.
By the spectral theorem for self-adjoint compact operators, there exists an at most countable index set I , a non-increasing sequence ( µ i ) i 2 I > 0 , and a family ( e i )
i
I 2 H X , such that ([ e i ])
i 2 I is an orthonormal basis (ONB) of ran I ⇡ ✓ L 2 ( ⇡ ) and ( µ 1 / 2 i e i )
i 2 I is an ONB of (ker I ⇡ ) ?
✓ H X , and we have
[Equation]
For ↵  0 , we deﬁne the ↵ -interpolation space [40] by [
[Equation]
equipped with the ↵ -power norm
[Equation]
More broadly, we sometimes need to deal with function of the form f + c where f 2 [ H ] ↵ X and c 2 R .
For this, we follow the classical deﬁnition of the direct sum of two Hilbert spaces [8] and deﬁne the ↵ -power norm for f
+ c as
[Equation]
For ( a i ) i 2 I 2 ` 2 ( I ) , the ↵ -interpolation space becomes a Hilbert space with inner product deﬁned as
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure] Moreover, [Equation] forms an ONB of [ H ] ↵ X and consequently [ H ] ↵ X is a separable Hilbert space.
In the following, we use the abbreviation k · k ↵ :=
k · k [ H ] ↵ X .
For ↵ =
we have [ H ] 0 X = ran I ⇡ ✓ L 2 ( ⇡ ) with k · k 0 =
k · k L 2 ( ⇡ ) .
Moreover, for ↵ = 1 we have [ H ] 1 X = ran I ⇡ and [ H ] 1 X is isometrically isomorphic to the closed subspace (ker I ⇡ ) ?
of H X via I ⇡ , i.e. k [ f ] k 1 = k f k H X for f 2 (ker I ⇡ ) ? .
For 0 <  < ↵ , we have
[Equation]
Moreover, under Assumptions 1 - 3 , if we further assume that supp ( ⇡ ) = E X and H X is dense in
L 2 ( ⇡ ) , we have [ H ] 0 X
=
L 2 ( ⇡ ) .
Approximation of CME with Vector-valued Interpolation Space
In this section, we deal with the misspeciﬁed setting where F ⇤ / 2 G .
To do this, we ﬁrst deﬁne the vector-valued interpolation space via the tensor product space.
We now recall from Remarkthat L 2 ( ⇡ ; H Y ) is isomorphic to S 2 ( L 2 ( ⇡ ) , H Y ) and we denote by the isomorphism between the two spaces.
Similarly, we have G ' S 2 ( H X , H Y ) and we denote by ¯ the isomorphism between both spaces in accordance with Theorem 1.
This is summarized in Figure 1.
The second chain of spaces is not isometric to the ﬁrst but can be naturally embedded into the ﬁrst as follows.
Recall that we denote by I ⇡ : H X !
L 2 ( ⇡ ) the embedding that maps each function to its equivalent class, I ⇡ ( f ) = [ f ] .
We therefore naturally deﬁne the embedding I ⇡ : S 2 ( H X , H Y ) !
S 2 ( L 2 ( ⇡ ) , H Y ) through I ⇡ ( g ⌦ f )
=
g ⌦ I ⇡ ( f ) =
g ⌦ [ f ] for all f 2 H X , g 2 H Y , and obtain the extension to the whole space by linearity and continuity.
Therefore, for F 2 G we deﬁne [ F ] :=
I ⇡  ¯  1 ( F ) .
In the rest of the paper, every embedding will be denoted using the notation [ · ] .
Strict notation would require us to write [ · ] ⇡ due to dependence on the measure ⇡ , but we omit the subscript for ease of notation.
Deﬁnition 3.
Suppose that we are given real-valued kernels k X and k Y with associated RKHS H X and H Y and let [ H ] ↵ X be the real-valued interpolation space associated to H X with some ↵  0 .
Since [ H ] ↵ X ✓ L 2 ( ⇡ ) , it is natural to deﬁne the vector-valued interpolation space
[ G ] ↵ as [ G
[Equation]
[ G ] ↵ is a Hilbert space equipped with the norm
[Equation]
where C =  1 ( F )
For ↵ = 0 , we retrieve,
[Equation]
Remark 4.
The vector-valued interpolation space [ G ] ↵ allows us to study the CME in the misspeciﬁed case.
To see this, we note that by Eq.
(3), we have F ⇤ 2 L 2 ( E X , F E X , ⇡ ; H Y ) .
In light of Eq.
(9), for 0 <  < ↵ we have
[Equation]
Again, once we further assume that supp ( ⇡ ) =
E X
and H X is dense in L 2 ( ⇡ ) , we have [ G ] 0 =
L 2 ( E X , F E X , ⇡ ; H Y ) .
Hence, while the well-speciﬁed case corresponds to F ⇤ 2 G , the misspeciﬁed case amounts to assuming that F ⇤ 2 [ G ] ↵ for some 0 < ↵ < 1 , and relaxes the well-speciﬁed assumption in Eq.
In this section, we derive the learning rate for the difference between [ ˆ F  ] and F ⇤ in the interpolation norm.
We ﬁrst state additional assumptions that are needed in our derivations.
As our assumptions match those of [11], we include the corresponding labels from [11] for ease of reference.
Recall that ( µ i )
i 2 I are the eigenvalues of C XX .
For some constants c 2 > 0 and p 2 (0 , 1] and for all i 2 I ,
[Equation]
[Equation]
For ↵ 2 ( p, 1] , the inclusion map I ↵ , 1 ⇡
[Equation]
[Equation]
L 1 ( ⇡ ) is continuous, and there is a constant A > 0 such that
[Equation]
There exists 0 <   2 such that
[Equation]
We let C Y | X :=
1 ( F ⇤ ) 2 S 2 ([ H ]  X , H Y ) and we call C Y | X the conditional mean embedding operator. (EVD) is a standard assumption on the eigenvalue decay of the integral operator (see more details in [4, 11, 41]).
(EMB) is referred as the embedding property in [11] and it can be shown that it implies
P
[Equation] e [Equation] i ( x )  A 2 for ⇡ -almost all x 2 E X ([11] Theorem 9).
Since we assume k X to be bounded, the embedding property always hold true
when ↵ = 1 .
Furthermore, (EMB) implies a polynomial eigenvalue decay of order 1 / ↵ , which is why we take ↵  p .
(SRC) is justiﬁed by
Remark 4 and is often referred as the source condition in literature ([4, 11, 23, 24]).
It imposes the smoothness assumption on the target CME operator F ⇤ .
In particular, when   1 , the source condition implies that F ⇤ has a representative from G , indicating the well speciﬁed scenario.
However, once we let  < 1 , we are in the misspeciﬁed learning setting, which is the main interest in this manuscript.
Finally, in computing the learning rate for real-valued regression, one often needs the so-called (MOM) condition on the Markov kernel p ( x, dy ) (see [4, 11, 41] for more details).
The generalization to our setting would amount to assume that there exists constants  , R > 0 such that
[Equation]
for ⇡ -almost surely all x 2 E X and all q  2 .
The reason for requiring (MOM) in the scalar regression setting is that we do not usually have | Y | < 1 almost surely.
In our setting, however,
Assumption 3 implies k  Y ( y )
F ⇤ ( x ) k H Y  2  Y for ⇡ -almost all x 2 E X and ⌫ -almost all y 2 E Y .
Therefore, (MOM) is automatically satisﬁed with  = R = 2  Y .
Remark 5.
We remark that in [41], a variant of SRC condition is employed.
In particular, instead of assuming F ⇤
[ G ]  , they impose the assumption that F ⇤ 2 G  ' S 2 ( H  X , H Y ) , where H  X is an RKHS with corresponding kernel deﬁned as k  X ( · , x ) = P
[Equation]
) e i ( x ) . Comparing to [ G ]  ' S 2 ([ H ]  X , H Y ) , there are two shortcomings that arise when working with G  .
First, H  X denotes the interpolating RKHS consisting of continuous functions only, while [ H ]  X is the interpolating Hilbert space, where elements are deﬁned through an equivalence classes.
Hence, by working with G  , the implicit assumption is that F ⇤ is a continuous function.
On the other hand, assuming F ⇤ 2 [ G ]  avoids the continuity requirement.
In particular, we have H  X ✓ [ H ]  X for any  > 0 .
Therefore, our (SRC) condition applies to a more general setting.
Second, and more importantly, the construction of H  X relies on the condition that
[Equation] as pointed out in [40].
Failing this, the kernel k  X associated with the interpolating RKHS H  X is unbounded, indicating H  X is not well-deﬁned.
Under (EVD) , this effectively amounts to require that   p .
When kernel has slow eigenvalue decay (as for the Matérn kernel), p can be close to 1 .
Results obtained using G  while requiring  > p are very close to the well-speciﬁed case.
By contrast, [ H ]  X is always well-deﬁned as a subspace of L 2 ( ⇡ ) for any  > 0 , even if
[Equation]
not hold.
Remark 6.
It is important to assume  > 0 in (SRC) , as our results do not apply when  = 0 .
The  = 0 setting arises for instance when Y ?
X , or when both Y = X and H Y
= H X .
In the former case, it is easy to see that the CME is the constant function (w.r.t X ) F ⇤ ( x ) =
µ Y =
R
E Y  Y ( y ) d ⌫ ( y ) .
In the latter case, F ⇤ ( x ) =
X ( x ) , and the CME is the identity operator.
These functions are not covered by [ G ]  for any  > 0 (see Appendix D for details).
We now provide an upper bound on the learning rate.
Theorem 2.
Let Assumptions 1 - 3 , (EVD) , (EMB) and (SRC) with 0 <   2 hold, and let 0   with  <  ,
In the case  + p  ↵ and  n = ⇥ ⇣ ( n/ log r ( n ))
1 ↵ ⌘ , for some r > 1 , there is a constant K > 0 independent of n  1 and ⌧  1 such that
[Equation]
is satisﬁed for sufﬁciently large n  1 with P n -probability not less than 1  4 e  ⌧ . 2.
In the case  + p > ↵ and  n = ⇥ ⇣ n  1  + p ⌘ , there is a constant K > 0 independent of n  1 and ⌧  1 such that
[Equation]
is satisﬁed for sufﬁciently large n  1 with P n -probability not less than 1  4 e  ⌧ .
Theorem 2 provides the ﬁnite sample  -norm learning rate for the empirical CME estimator deﬁned in Eq.
It states that the learning rate for [ ˆ F  ] is governed by the interplay between p , ↵ , and  .
Intuitively, p describes the decay rate of the eigenvalues ( µ i )
i 2 I , ↵ determines the boundedness of the interpolation kernel (and has maximum value of 1 according to our assumption),  characterizes the smoothness of the target CME operator.
To simplify the discussion, we may focus on the L 2 ( E X , F E X , ⇡ ; H Y ) learning rate, corresponding to  = 0 .
The exponent  / max { ↵ ,  + p } explicitly provides the learning rate for the CME operator.
For example, if we have ↵   , we obtain a learning rate of  / (  + p ) .
In particular, for a Gaussian kernel, p and ↵ are arbitrarily close to 0 , and our learning rate can achieve a fast O (1 /n ) rate up to a logarithmic factor.
If a kernel with slow eigenvalue decay is used, such as the Matérn kernel, we can obtain the minimax optimal learning rate n  1 / 2 up to logarithmic factors if we have p   .
Finally, in the worst case where  is close to 0 , the learning rate can be arbitrarily slow.
Lower Bound
Our ﬁnal theorem provides a lower bound for the convergence rate, which allows us to conﬁrm the optimality of our learning rate.
In deriving the lower bound, we need an extra assumption 8 .
For some constants c 1 , c 2 > 0 and p 2 (0 , 1] and for all i 2 I ,
[Equation]
Theorem 3.
Let k X be a kernel on E X such that Assumptions 1 - 3 hold and ⇡ be a probability distribution on E X such that (EVD+) and (EMB) hold 0 <
p  ↵  1 .
Then for all 0 <   2 , 0    1 with  <  there exist constants K 0 , K, s > 0 such that for all learning methods D !
ˆ F D
( D := { ( x i , y i ) } ni =1 ), all ⌧ > 0 , and all sufﬁciently large n  1 there is a distribution P deﬁned on E X ⇥ E Y used to sample D , with marginal distribution ⇡ on E X , such that (SRC) with respect to  is satisﬁed, and with P n -probability not less than 1  K 0 ⌧ 1 /s ,
[Equation]
Theorem 3 states that under the assumtions of Theorem 2 and (EVD+), no learning method can achieve a learning rate faster than n  max { ↵ ,  } max { ↵ ,  } + p in L 2 norm.
To our knowledge, this is the ﬁrst analysis that demonstrates the lower rate for CME learning.
In the context of regularized regression, [4, 39, 3] provide a similar lower bound on the learning rate.
However, a key difference in our analysis is that the output of the regression learning now lives in an inﬁnite dimensional RKHS H Y , rather than in R .
Our analysis reveals that in the case where ↵   , the obtained upper rate in Theorem 2 is optimal, i.e., O ( n   + p ) .
In particular, when k X is an exponentiated quadratic kernel on a compact set E X ⇢ R d with Lipschitz boundary, (EMB) is satisﬁed with any ↵ 2 (0 , 1) [17, see Corollary 4 . 13 ].
As a result, the optimal rate is attained as long as  > 0 .
We point out that ﬁnding the optimal rate for  <
↵ remains a challenge, and is an open problem when the output is R . Remark 7.
Theorem 3 states that the upper bound and the lower bound match when  > ↵ .
In particular, for exponential kernels such as Gaussian and Laplacian kernels with subgaussian distributions, the eigenvalues for the covariance operator have geometric decay rate.
In these cases, ↵ is arbitrarily close to 0 . Hence, as long as  > 0 , we will have ↵ <  . In other words, for commonly used kernels, CME learning can always obtain the optimal fast rate n   + p .  Conclusion
In this paper, we provide a rigorous theoretical foundation for approximating the CME operator, and study the statistical learning rate.
Utilizing recently developed interpolation space techniques, we ﬁrst deﬁne the vector-valued interpolation space [ G ] ↵ .
This allows to deﬁne the target CME operator in the larger interpolation space [ G ] ↵ , in contrast to the well-speciﬁed setting where F ⇤ 2 G .
By doing so, we are able to study the convergence rate of the empirical CME operator in the misspeciﬁed scenario.
We then provide a  -norm learning rate for the CME without any assumption on the interplay between  and p , with matching lower bound.
Our analysis shows that under appropriate conditions, we can obtain a fast O (log n/n ) convergence rate, which matches the rate obtained in the existing literature for ﬁnite dimensional H Y . In more challenging settings, we still obtain the minimax optimal rate O ( n  1 / 2 ) .
Looking beyond the present work, our current interpolation space setting indicates that the convergence rate can be arbitrarily slow if  ! 0 .
This prevents learning the constant function, which plays a crucial role in completing the theory of the CME, as pointed out by [18].
Addressing this challenge is an important direction of future research.
Acknowledgement: The authors wish to thank Peter Orbanz and Bharath Sriperumbudur for fruitful discussions and proofreading.
Zhu Li, Dimitri Meunier and Arthur Gretton were supported by the Gatsby Charitable Foundation.
Mattes Mollenhauer was supported by the Deutsche Forschungsgemeinschaft (DFG) through grant EXC 2046 “MATH+”; Project Number 390685689, Project IN- “Inﬁnite-Dimensional Supervised Least Squares Learning as a Noncompact Regularized Inverse
Problem”.