First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domains
Abstract
Real-world machine learning applications often involve deploying neural networks to domains that are not seen in the training time.
Hence, we need to understand the extrapolation of nonlinear models—under what conditions on the distributions and function class, models can be guaranteed to extrapolate to new test distributions.
The question is very challenging because even two-layer neural networks cannot be guaranteed to extrapolate outside the support of the training distribution without further assumptions on the domain shift.
This paper makes some initial steps towards analyzing the extrapolation of nonlinear models for structured domain shift.
We primarily consider settings where the marginal distribution of each coordinate of the data (or subset of coordinates) does not shift signiﬁcantly across the training and test distributions, but the joint distribution may have a much bigger shift.
We prove that the family of nonlinear models of the form f ( x ) = (cid:80)
f i ( x i ) , where f i is an arbitrary function on the subset of features x i , can extrapolate to unseen distributions, if the covariance of the features is well-conditioned.
To the best of our knowledge, this is the ﬁrst result that goes beyond linear models and the bounded density ratio assumption, even though the assumptions on the distribution shift and function class are stylized.
Introduction
In real-world applications machine learning models are often deployed on domains that are not seen in the training time.
For example, we may train machine learning models for medical diagnosis on data from hospitals in Europe and then deploy them to hospitals in Asia.
Thus we need to understand the extrapolation of models to new test distributions — how the model trained on one distribution behaves on another unseen distribution.
This extrapolation of neural networks is central to various robustness questions such as domain generalization (Gulrajani and
Lopez-Paz
[2020], Ganin et al.
[2016], Peters et al.
[2016] and references therein) and adversarial robustness [Goodfellow et al., 2014, Kurakin et al., 2018] and also plays a critical role in nonlinear bandits and reinforcement learning where the distribution is constantly changing during training [Dong et al., 2021, Agarwal et al., 2019, Lattimore and Szepesv´ari, 2020, Sutton and Barto, 2018].
This paper focuses on the following mathematical abstraction of this extrapolation question:
Under what conditions on the source distribution P , target distribution Q , and function class F do we have that any functions f, g ∈ F that agree on P are also guaranteed to agree on Q ?
Here we can measure the agreement of two functions on P by the (cid:96) 2 distance between f and g under distribution P , that is, (cid:107)
f − g (cid:107) P (cid:44) E x ∼ P
[( f ( x ) − g ( x )) 2 ] 1 / 2 .
The function f can be thought of as the learned model, g as the ground-truth function, and thus (cid:107)
f − g (cid:107) P as the error on the source distribution P .
This question is well-understood for linear function class F .
Essentially, if the covariance of Q can be bounded from above by the covariance of P (in any direction), then the error on Q is guaranteed to be bounded above from the error on P .
We refer the reader to Lei et al.
[2021], Mousavi Kalan et al.
[2020] and references therein for more recent advances along this line.
By contrast, theoretical results for extrapolation of nonlinear models is rather limited.
Classical results have long settled the case where P and Q have bounded density ratios [Ben-David and Urner, 2014, Sugiyama et al., 2007].
Bounded density ratio implies that the support of Q must be a subset of the support of P , and thus arguably these results do not capture the extrapolation behavior of models outside the training domain.
Without the bounded density ratio assumption, there was limited prior positive result for characterizing the extrapolation power of neural networks.
Ben-David et al.
[2010] show that the model can extrapolate when the H ∆ H -distance between training and test distribution is small.
However, it remains unclear for what distributions and function class, the H ∆ H -distance can be bounded.
1 In general, the question is challenging partly because of the existence of such a strong impossibility result.
As soon as the support of Q is not contained in the support of P (and they satisfy some non-degeneracy condition), it turns out that even two-layer neural networks cannot extrapolate—there are two-layer neural networks f and g that agree on P perfectly but behave very differently on Q (See Proposition 6 for a formal statement.)
The impossibility result suggests that any positive results on the extrapolation of nonlinear models require more ﬁne-grained structures on the relationship between P and Q (which are common in practice Koh et al.
[2021], Sagawa et al.
[2022]) as well as the function class F .
The structure in the domain shift between P and Q may also need to be compatible with the assumption on the function class F .
This paper makes some ﬁrst steps towards proving certain family of nonlinear models can extrapolate to a new test domain with structured shift.
We consider a setting where the joint distribution of the data can does not have much overlap across P and Q (and thus bounded density ratio assumption does not hold), whereas the marginal distributions for each coordinate of the data does overlap.
Such a scenario may practically happen when the features (coordinates of the data) exhibit different correlations on the source and target distribution.
For example, consider the task of predicting the probability of a lightning storm from basic meteorological information such as precipitation, temperature, etc.
We learn models from some cities on the west coast of United States and deploy them to the east coast.
In this case, the joint test distribution of the features may not necessarily have much overlap with the training distribution— correlation between precipitation and temperature could be vastly different across regions, e.g., the rainy season coincides with the winter’s low temperature on the west coast, but not so much on the east coast.
However, the individual feature’s marginal distributions are much more likely to overlap between the source and target—the possible ranges of temperature on east and west coasts are similar.
Concretely, we assume that the features x ∈ R s 1 + s 2 have Gaussian distributions and can be divided into two subsets x 1 ∈ R s 1 and x 2 ∈ R s 2 such that each set of feature x i ( i ∈ { 1 , 2 } ) has the same marginal distributions on P and Q .
Moreover, we assume that x 1 and x 2 are not exactly correlated on P —the covariance of features x on distribution P has a strictly positive minimum eigenvalue.
As argued before, restricted assumptions on the function class F are still necessary (for almost any P and Q without the bounded density ratio property).
Here, we assume that F consists of all functions of the form f ( x ) = f 1 ( x 1 ) + f 2 ( x 2 ) for arbitrary functions f 1 : R s 1 → R and f 2 : R s 2 → R .
The function class F does not contain all two-layer neural networks (so that the impossibility result does not apply), but still consists of a rich set of functions where each subset of features independently contribute to the prediction with arbitrary nonlinear transformations.
We show that under these assumptions, if any two models approximately agree on P , they must also approximately agree on Q — formally speaking, ∀ f, g ∈ F , (cid:107)
f − g (cid:107) Q (cid:46) (cid:107) f g (cid:107) P (Theorem 5). We also prove a variant of the result above where we divide features vector x ∈ R d into d coordinate, denoted by x = ( x 1 , . . .
, x d ) where x i ∈ R .
The function class consists of all combinations of nonlinear transformations of x i ’s, that is, F = { (cid:80) di
=1 f i ( x i ) } .
Assuming coordinates of x 𝑥 ! 𝑥 " 1 2 3 4 5 624 𝑥 ! 𝑥 " 1 2 3 4 5 624 𝑥 !
𝑥 " 1 2 3 4 5 624 𝑥 !
𝑥 " 1 2 3 4 5 624 test distribution training distributions no marginal coverage clusterable 𝑃 ! 𝑃 " 𝑃 # 𝑄 marginal coverage + non-clusterable (a) (b) (c) (d) are pairwise Gaussian and a non-degenerate covariance matrix, the nonlinear model f ∈ F can extrapolate to any distribution Q that has the same marginals as P (Theorem 4).
These results can be viewed as ﬁrst steps for analyzing extrapolation beyond linear models.
Compared the works of Lei et al.
[2021] on linear models, our assumptions on the covariance of P are qualitatively similar.
We additionally require P and Q have overlapping marginal distributions because it is even necessary for the extrapolation of one-dimensional functions on a single coordinate.
Our results work for a more expressive family of nonlinear functions, that is, F = { (cid:80) di
=1 f i ( x i ) } , than the linear functions.
We also present a result on the case where x i ’s are discrete variables, which demonstrates the key intuition and also may be of its own interest.
Suppose we have two discrete random variable x 1 and x 2 .
In this case, the joint distribution of P and Q can be both presented by a matrix (as visualized in
Figure 1), and the marginal distributions are the column and row sums of this joint probability matrix.
We prove that extrapolation occurs when (1) the support of the marginals of Q is covered by P , and (2) the density matrix of P is non-clusterable — we cannot ﬁnd a subset of rows and columns such the support of P in these rows and columns lies within their intersections.
In Figure 1, we visualize a few interesting cases.
First, distributions P 1 , P 2 visualized in Figures 1a and 1b, respectively, do not satisfy our conditions.
Fundamentally, there are two models that agreed on the support of distribution P 1 (or P 2 ), but still differ much on the non-support.
In contrast, our result predict that models trained on distribution P 3 in Figure 1c can extrapolate to the distribution Q in Figure 1d, despite that the support of P 3 is sparse and their support have very little overlap.
We also note that the failure of P 1 and P 2 demonstrate the non-triviality of our results.
The overlapping marginal assumption by itself does not guarantees extrapolation, and condition (2), or analogously the minimal eigenvalue condition for the Gaussian cases, is critical for extrapolation.
Our proof techniques for the theorems above is generally viewing (cid:107) h (cid:107) 2 P as K P ( h, h ) for some kernel K P : F × F → R .
Here h is a shorthand for the error function f − g .
Note that the kernel takes in two functions in F as inputs and captures relationship between the functions.
Hence, the extrapolation of F (i.e., proving (cid:107) h (cid:107) Q (cid:46) (cid:107) h (cid:107) P for all h ∈ F ) reduces to the relationship of the kernels (i.e., whether K Q ( h, h ) (cid:46) K P ( h, h ) for all h ∈ F ), which is then governed by properties of the eigenspaces of kernel K P , K Q .
Thanks to the special structure of our model class F = (cid:80) f
i ( x i ) , we can analytically relate the eigenspace of the kernels K P , K Q to more explicitly and interpretable property of the data distribution of P and Q .
Problem Setup and Preliminaries
We use P and Q to denote the source and target distribution over the space of features X =
X 1 × · · · × X k respectively.
We measure the extrapolation of a model class F ⊆ R X from P to Q by the the following quantity:[-
-]
[Equation]
When τ ( P, Q, F ) is small, if two models f, g ∈ F approximately agree on P (meaning that E
P [( f ( x ) − g ( x )) 2 ] is small), they must approximately agree on Q because E Q [( f ( x ) − g ( x )) 2 ] ≤ τ E P [( f ( x ) − g ( x )) 2 ] .
If the model class F is expressive enough to include the ground-truth labeling function, τ becomes an upper bound of the ratio between the loss on distribution Q and the loss on distribution P (formally stated in Proposition 1), which provides the robustness guarantee of the trained model.
This is because when g corresponds to the ground-truth label, E P [( f ( x ) − g ( x )) 2 ] becomes the (cid:96) 2 loss of model f .
Proposition 1.
Let τ the quantity deﬁned in Eq.
For any distribution P, Q and model class F , if there exists a model in F that can represent the true labeling y :
X → R on both P and Q : ∃ f (cid:63) ∈ F such that E 12 ( P + Q ) [( y ( x
[Equation]
then we have
[Equation]
Proof of this proposition is deferred to Appendix [-F.
-]F.
Relationship to the H ∆ H -distance.
The quantity τ is closely related to the H ∆ H -distance
[BenDavid et al., 2010]:
[Equation]
with the differences that (1) we consider (cid:96) 2 loss instead of classiﬁcation loss, and (2) τ focuses on the ratio of losses whereas d H ∆ H focuses on the absolute difference.
As we will see later, these differences bring the mathematical simplicity to prove concrete conditions for model extrapolation.
In this paper, we focus on the model class F = { (cid:80) ki
=
f i ( x i ) : E P [ f i ( x i ) 2 ] < ∞ , ∀ i ∈ [ d ] } where f i : X i
→ R is an arbitrary function.
Since F is closed under addition, we can simplify Eq.
(1) to τ ( P, Q, F ) =
sup f ∈F E Q [ f ( x ) 2 ] E P [ f ( x ) 2 ] .
For simplicity, we omit the dependency on P, Q, F when the context is clear.
Summary of Main Results 3.1 Features with Discrete Values
In this section, we relate the extrapolation power of a training distribution P to the clusterablity of its density matrix. Without loss of generality, we assume that x i takes the value in { 1 , 2 , · · · , r i } . We measure the (approximate) clusterablity by eigenvalues of the Laplacian matrix of a bipartite graph associated with the density matrix P ∈ R r 1 × r 2 .
Concretely, let G P be the bipartite graph where P ( x 1 = i, x 2 = j ) corresponds to the weight of the edge between the i -th vertex on the left part of the graph and the j -th vertex on the right part. The Laplacian of G P is computed as follows. Let d 1 ∈ R r 1 and d 2 ∈ R r 2 be the row and column sums of the weight matrix P (in other words, degree of the vertices).
Deﬁne diagonal matrices D 1 = diag( d 1 ) ∈
R r 1 × r 1 and D 2 = diag
( d 2 ) ∈ R r 2 × r 2 .
Then the (signless) Laplacian K
P and normalized Laplacian of ¯ K P is:
[Equation]
And our main theorem in this section is stated as follows.
Theorem 2.
For any two dimensional distribution P, Q over discrete random variables x 1 , x 2 , and the model class F = { f 1 ( x 1 ) + f 2 ( x 2 ) } where f i : X i → R is an arbitrary function, we have
[Equation]
Remarks.
Cheeger’s inequality provides upper and lower bounds of λ 2 ( ¯ K P ) by the sparest cut φ ( G P ) of the bipartite graph G P
[Chung, 1996, Alon, 1986]:
[Equation]
When the bipartite graph is not connected (e.g., the distribution in Figure 1b), we get φ ( G P ) =
and therefore λ 2 ( ¯ K P ) = 0 .
In this case, Theorem 2 is vacuous because the upper bound in Eq.
(6) becomes inﬁnity, and therefore we have no extrapolation guarantees.
{+In fact, we can even ﬁnd two models f, g and a input x † such that (cid:107) f − g (cid:107) P = 
+}
but f ( x † ) (cid:54) =
g ( x † ) (see Proposition 7).
The following corollary proves sufﬁcient conditions for τ ( P, Q, F ) < ∞ , which are visualized in
Figure 1.
Corollary 3.
In the setting of Theorem 2, τ ( P, Q, F ) < ∞ if
the marginals of Q has a smaller support than that of P , and 2.
we cannot ﬁnd a subset of rows and columns such that the support of P in these rows and columns lies within their intersections.
The proof of Corollary 3 is deferred to Appendix F.
Compared with prior works that assumes a bounded density ratio on the entire distribution (e.g., Ben-David and Urner [2014], Sugiyama et al. [2007]), we only require a bounded density ratio of the marginal distributions. This is because the model class f ( x ) =
f 1 ( x 1 ) + f 2 ( x 2 ) can extrapolate to distributions with a larger support (see Figure 1c).
In contrast, for an unstructured model considered in prior works (in this case, f ( x ) is an arbitrary function of the entire input x ) we do not have any control of the model’s behavior on data points outside the support of P . The proof sketch of Theorem 2 is deferred to Appendix B.1. 3.2 Features with Real Values We also consider the case when x 1 , x 2 , · · · , x k are real-valued random variables.
Let Σ P = E P [ xx (cid:62) ] , Σ Q = E Q [ xx (cid:62) ] be the covariance matrix of x on distributions P, Q .
Recall that our model has the structure f ( x )
=
(cid:80) ki
=
f i ( x i ) where f i is an arbitrary one-dimensional function.
Without loss of generality, we assume diag(Σ P ) =
diag(Σ Q )
=
I (a detailed discussion is deferred to Appendix B.2), then we have the following theorem.
Theorem 4.
For any distributions P, Q over variables x = ( x 1 , · · · , x d ) with matching marginals, if ( x i , x j ) has the distribution of a two-dimensional Gaussian random variable for every i, j ∈ [ d ] on both P and Q , and diag(Σ P ) =
diag(Σ Q )
=
I , then
[Equation]
Compared with linear models F linear (cid:44) { v (cid:62) x : v ∈ R d } , Theorem 4 proves comparable conditions for the structured nonlinear model class f ( x )
= (cid:80) di
=1 f i ( x i ) to extrapolate because linear model requires τ ( P, Q, F linear ) = sup [Equation] P = sup [Equation]
R
[Equation]
Σ P v (cid:46) λ min (Σ P ) − 1 to be small, which is the requirement of Eq.
(8) (up to factors that is independent of the distributions P, Q ).
We can extend to the case where x 1 ∈ R d 1 , x 2 ∈ R d 2 are two subsets of the input x , and the input x = ( x 1 , x 2 ) has Gaussian distribution (Theorem 5). Acknowledgment
The authors would like to thank Yuanhao Wang, Yuhao Zhou, Hong Liu, Ananya Kumar, Jason D. Lee for helpful discussions. The authors would also like to thank the support from NSF CIF 2212263. G Hermite polynomial and Gaussian kernel
H Helper Lemmas
A Useful Notations
In this section, we introduce some useful notations.
Let I [ E ] be the indicator function that equals 1 if the condition E is true, and 0 otherwise.
For an integer n , let [ n ] be the set { 1 , 2 , · · · , n } .
For a vector x ∈ R d , we use [ x ] i to denote its i -th coordinate.
Similarly, [ M ] i,j denotes the ( i, j ) -th element of a matrix M .
We use M (cid:12) n to represent the elementwise n -th power of the matrix M (i.e., [ M (cid:12) n ] i,j =
[ M ] i,j ) n ).
Let I d ∈ R d
×
d be the identity matrix, 1 d ∈ R d the all-1 vector and e i,d the i -th base vector.
We omit the subscript d when the context is clear.
For a square matrix P ∈ R d × d , we use diag( P ) ∈
R
d × d to denote the matrix generated by masking out all non-diagonal terms of P .
For list σ 1 , · · · , σ d , let diag( { σ 1 , · · · , σ d } ) ∈
R
d × d be the diagonal matrix whose diagonal terms are σ 1 , · · · , σ d .
For a symmetric matrix M ∈ R d × d , let λ 1 ( M ) ≤ λ 2 ( M ) ≤ · · · ≤ λ d ( M ) be its eigenvalues in ascending order, and λ max ( M ) , λ min ( M ) the maximum and minimum eigenvalue, respectively.
Similarly, we use σ 1 ( M ) ≤ · · · ≤ σ min( d 1 ,d 2 ) ( M ) to denote the singular values of M ∈ R d 1 × d 2 .
B Main Results
In this section, we present our additional main results.
[-(cid:107) P = 
-]
B.1 Proof Sketch of Theorem 2 Values
[-The proof of Corollary 4 is deferred to Appendix F.
-]
Proof sketch of Theorem 2.
In the following we present a proof sketch of Theorem 2, and defer the complete proof to Appendix F.3.
We start with a high-level proof strategy and then instantiate the proof on the setting of Theorem 2.
Suppose we can ﬁnd a set of (not necessarily orthogonal) basis { b 1 , · · · , b r } where b i : X → R , such that any model f ∈ F can be represented as a linear combination of basis (cid:80) ri =1 v i b i .
Since the model family F is closed under subtraction, we have
[Equation]
If we deﬁne the kernel matrix [ K P ] i,j = E P [ b i ( x ) b j ( x )] , it follows that
[Equation]
Hence, upper bounding τ reduces to bounding the eigenvalues of kernel matrices K P , K Q .
Since the model has the structure f ( x ) = f 1 ( x 1 ) + f 2 ( x 2 ) , we can construct the basis { b t } rt = explicitly.
For any i ∈ [2] , t ∈ [ r i ] , with little abuse of notation, let
[Equation]
i
[Equation]
We can verify that the set { b i,t } i ∈ [ r ] ,t ∈ [ r i ] is indeed a complete set of basis.
As a result, the kernel matrices K
P can be computed directly using its deﬁnition:
[Equation]
when i (cid:54) = j
[Equation]
[Equation] = [Equation]
[Equation]
when i =
j, (12) which is exactly the Laplacian matrix deﬁned in Eq.
Let D P = diag
( K P ) and D Q = diag( K Q ) .
To prove Eq.
(6), the high level intuition is that
[Equation]
However, this naive bound is vacuous because for any P we have λ min ( ¯ K P )
= 0 .
In fact, K P and
K Q share the eigenvalue 0 and the corresponding eigenvector.
Therefore we can ignore this directions ﬁrst, and then the desired result directly follows.
Theorem 2 can be easily extend to the case when k > 2 .
We choose to present the current version because of simplicity, and because the kernel matrix K P coincides with the Laplacian of a bipartite graph when k = 2 , which provides further insights and better intuitions.
B.2 Features with Real Values
In this section we extend our analysis to the case where x 1 , x 2 , · · · , x d are real-valued random variables.
Let Σ P = E P
[ xx (cid:62) ] , Σ Q = E Q
[ xx (cid:62) ] be the covariance matrix of x on distributions P, Q .
Recall that our model has the structure f ( x ) =
(cid:80) di
=1 f i ( x i ) where f i is an arbitrary one-dimensional function.
We ﬁrst focus on the case where diag(Σ P ) =
diag(Σ Q )
=
I for simplicity, and then extend to general cases.
When d = 2 , this setting is almost the same as the one in Section 3.1, and the only difference is that the Laplacian “matrix” will be inﬁnite dimensional (indexed by a pair of real numbers) because the feature x i is a real number.
However, we can still compute the eigenvalues of the kernel with some
Gaussianity assumptions on the distribution of features, as stated in the following theorem (restate of
Theorem 4).
Theorem 4.
For any distributions P, Q over variables x = ( x 1 , · · · , x d ) with matching marginals, if ( x i , x j ) has the distribution of a two-dimensional Gaussian random variable for every i, j ∈ [ d ] on both P and Q , and diag(Σ P ) =
diag(Σ Q )
=
I , then
[Equation]
Remarks.
For the general case where diag(Σ P ) (cid:54) =
I , we can normalize the input x
i to t i (cid:44) var P ( x i ) − 1 / 2 ( x i − E P
[ x i ])
ﬁrst.
Let P (cid:48) , Σ (cid:48)
P be the density and covariance of t when x ∼ P (and similarly Q (cid:48) ).
Then we have diag(Σ (cid:48) P ) =
diag(Σ (cid:48) Q )
=
I when P, Q have matching marginals.
Hence, we can prove that τ ( P, Q, F ) =
τ ( P (cid:48) , Q (cid:48) , F ) ≤ d λ min (Σ (cid:48) P ) (Lemma 8).
Compared with linear models, Theorem 4 proves comparable results for the structured nonlinear model class f ( x ) = (cid:80) di
=1 f i ( x i ) to extrapolate.
For linear models F linear (cid:44) { v (cid:62) x : v ∈ R d } we have
[Equation]
which is the RHS of Eq.
(8) (up to factors that is independent of the distributions P, Q ).
We emphasize that we only assume the marginals on every pair of features x i , x j is Gaussian, which does not imply the Gaussianity of the joint distribution of x .
In fact, there exists a non-Gaussian distribution that satisﬁes our assumption.
Proof sketch of Theorem 4.
On a high level, the proof intuition is to pretend the real numbers are discrete, and follow the same proof strategy as Theorem 2.
We present this non-rigorous proof sketch below for its simplicity, and the rigorous proof is deferred to Appendix F.5.
First we consider a simpliﬁed case when d = 2 .
Following the same argument as Theorem 2, we can compute the normalized kernel ¯ K P explicitly.
Because x 1 , x 2 are continuous random variables, the kernel ¯ K P will be inﬁnite dimensional, and have the form ¯ K P = (cid:18)
I
A
A (cid:62)
I (cid:19) , where A is an inﬁnite dimensional “matrix” indexed by real numbers x 1 , x 2 ∈ R , with values [ A ] x 1 ,x 2 = P ( x 1 , x 2 ) / (cid:112) P ( x 1 ) P ( x 2 ) , and I is the “identity matrix” (rigorously speaking they are kernel functions that map R × R to R .)
Similar to Theorem 2, we need to lower bound the second smallest eigenvalue of ¯ K P .
To this end, we ﬁrst decompose A using singular value decomposition
A =
U Λ V (cid:62) , where UU (cid:62) =
I, V V (cid:62)
= I and Λ = diag( { σ n } n ≥ 0 ) with σ 0 ≥ σ 1 ≥ · · · .
Then we get
[Equation]
Since the matrix ˆ K P [Equation] I [Equation]
consists of four diagonal sub-matrices, we can shufﬂe the rows and columns of ˆ K P to form a block-diagonal matrix with blocks
[Equation]
As a result, λ 2 ( ¯ K P ) =
λ 2 ( ˆ K P ) = 1 − σ 1 .
By the assumption that ( x 1 , x 2 ) follows from Gaussian distribution, the “matrix” A is actually a Gaussian kernel, whose eigenvalues and eigenfunctions can be computed analytically — Theorem 11 proves that σ 1 = | E P [ x 1 x 2 ] | .
Consequently, λ 2 ( ¯ K P ) = 1 − σ [Equation] = [Equation]
− | E P [ x
[Equation]
x
[Equation]
] | = [Equation] min (Σ P ) . Now we brieﬂy discuss the case when d = 3 , and the most general cases (i.e., d > 3 ) are proved similarly.
When d = 3 , the normalized kernel will have the following form
[Equation]
By the assumption that x 1 , x 2 , · · · , x d have zero mean and unit variance, matrices A, B, C are symmetric.
In addition, Theorem 11 shows that the eigenfunctions of the Gaussian kernel is independent of the value E P [ x i x j ] .
Hence, the matrices A, B, C shares the same eigenspace and can be diagonalized simultaneously:
[Equation]
[Equation]
By reshufﬂing the columns and rows, the eigenvalues of ¯ K P are the union of the eigenvalues of following matrices
[Equation]
Theorem 11 implies that σ n ( A ) =
([Σ P ] 1 , 2 ) n , σ n ( B ) =
([Σ P ] 1 , 3 ) n and σ n ( C ) =
([Σ P ] 2 , 3 ) n .
Consequently we get ¯ K ( n ) P = Σ (cid:12) nP .
Finally, this theorem follows directly from Lemma 13 which shows λ min (Σ (cid:12) nP ) ≥ λ min (Σ P ) for n ≥ 1 .
B.3 Two Features with Multi-dimensional Gaussian Distribution
Now we extend Theorem 4 to the case where x 1 ∈ R d 1 , x 2 ∈ R d 2 are two subsets of the input x , and the input x = ( x 1 , x 2 ) has Gaussian distribution.
Since f 1 , f 2 are arbitrary functions and P, Q have the same marginal distribution, we can assume without loss of generality that E P [ x i x (cid:62) i ] =
E Q [ x i x (cid:62) i ] = I d i , ∀ i ∈ [2] .
The main theorem for this case is given below, whose proof is deferred to Appendix F.6.
Theorem 5.
For any distributions P, Q over variables x = ( x 1 , x 2 ) where x 1 ∈ R d 1 , x 2 ∈ R d 2 , let Σ P = E P [ xx (cid:62) ] .
If ( x 1 , x 2 ) follows from Gaussian distribution on P and Q with matching marginals and E P [ x 1 x (cid:62) 1 ] = I d 1 , E P [ x 2 x (cid:62) 2 ]
=
I d 2 , then
[Equation]
Compared with Theorem 4 where the features are not grouped, our condition for the covariance is almost the same — that is, λ min (Σ P ) is bounded away from zero.
However, the model class considered Theorem 5 is more powerful because it captures nonlinear interactions between features within the same group.
As a compromise, the assumption on the marginals of P and Q is stronger because Theorem 5 requires matching marginals on each group of the features, whereas Theoremonly requires matching marginals on each individual feature.
In the following, we show the proof sketch of Theorem 5.
Proof sketch of Theorem 5.
We start by considering a simpler case when d 1 = d 2 and Σ 12 (cid:44) E
P [ x 1 x (cid:62) 2 ] ∈ R d 1 × d 2 has only diagonal terms.
In other words, ([ x 1 ] i , [ x 2 ] i ) are independent with every other coordinates in the input.
In this case, we can decompose the multi-dimensional Gaussian kernel into products of one-dimensional Gaussian kernels in the following way
[Equation]
Consequently, the singular values of A will be the products of singular values of these one-dimensional
Gaussian kernels: { (cid:81) d 1 i =1 | [Σ 12 ] i,i | k i } k 1 ,k 2 , ··· ,k d 1 ≥ 0 .
Therefore, the second largest singular value of
A will be max
i ∈
[ d 1 ] | [Σ 12 ] i,i | .
Following the same reasoning as Theorem 4 we get
[Equation]
where σ max (Σ 12 ) is the largest singular value of Σ 12 .
Combining with Lemma 15 we get Eq.
Now we turn to the general cases where Σ 12 is not diagonal.
Recall that our model is f ( x ) =
f 1 ( x 1 )+ f 2 ( x 2 ) for some arbitrary functions f 1 , f 2 .
Hence, we can rotate the inputs x 1 , x 2 without affecting the model class.
Formally speaking, for any orthogonal matrices
U ∈ R d 1 × d 1 and V ∈ R
d 2 × d 2 ,
[Equation]
If U, V are the orthonormal matrices in the singular value decomposition Σ
=
U (cid:62) Λ 12 V where Λ 12 is a diagonal matrix, we get
[Equation]
As a result, reusing the result from the previous case on inputs Ux 1 , V x 2 proves the desired result.
Remarks.
Our current techniques can only handle the case when the input is divided into k =subsets.
This is because for k ≥ 3 we must diagonalize multiple multi-dimensional Gaussian kernels simultaneously using the same set of eigenfunctions, as required in the proof of Theorem 4.
However, these multi-dimensional Gaussian kernels do not share the same eigenfunctions because the rotation matrix U, V depends on the covariance E P [ x i x (cid:62) j ] .
Hence, the proof strategy for Theorem 4 fails.
C Lower Bound
In this section, we prove a lower bound as a motivation to consider structured distributions shifts.
On the high level, the following proposition shows that models learned on P cannot extrapolate to Q when the support of distribution Q is not contained in the support of P .
Proposition 6.
Let the model class F be the family of two-layer neural networks with ReLU activation: F = (cid:8)(cid:80)
i a i ReLU( w (cid:62) i x + b i ) : w i ∈ R d , a i , b i ∈ R (cid:9) .
Suppose for simplicity that all the inputs have unit norm (i.e., (cid:107) x (cid:107) 2 = 1 ).
If Q has non-zero probability mass on the set of points well-separated from the support of P in the sense that
[Equation]
we can construct a model f ∈ F such that (cid:107) f (cid:107) P =
but (cid:107) f (cid:107) Q can be arbitrarily big.
A complete proof of this proposition is deferred to Appendix F.7.
On a high level, we prove this proposition by construct a two-layer neural network g t that represents a bump function around any given input t ∈ S d − 1 .
As a result, when t is a point in supp( Q ) \ supp( P ) , the model g t ( x ) will have zero (cid:96) 2 norm on P but have a positive (cid:96) 2 norm on Q .
This construction is inspired by Dong et al.
[2021, Theorem 5.1].
D Related Works
The most related work is Ben-David et al.
[2010], where they use the H ∆ H -distance to measure the maximum discrepancy of two models f, g ∈ F on any distributions P, Q .
However, it remains an open question to determine when H ∆ H -distance is small for concrete nonlinear model classes and distributions.
On the technical side, the quantity τ is an analog of the H ∆ H -distance for regression problems, and we provide concrete examples where τ is upper bounded even if the distributions P, Q have signiﬁcantly different support.
Another closely related settings are domain adaptation [Ganin and Lempitsky, 2015, Ghifary et al., 2016, Ganin et al., 2016] and domain generalization [Gulrajani and Lopez-Paz, 2020, Peters et al., 2016], where the algorithm actively improve the extrapolation of learned model either by using unlabeled data from the test domain [Sun and Saenko, 2016, Li et al., 2020a,b, Zhang et al., 2019], or learn an invariant model across different domains [Arjovsky et al., 2019, Peters et al., 2016,
Gulrajani and Lopez-Paz, 2020].
In comparison, this paper studies a more basic question: whether a model trained on one distribution (without any implicit bias and unlabeled data from test domain) extrapolates to new distributions.
There are also prior works that theoretically analyze algorithms that use additional (unlabeled) data from the test distribution, such as self-training [Wei et al., 2020,
Chen et al., 2020], contrastive learning [Shen et al., 2022, HaoChen et al., 2022], label propagation [Cai et al., 2021], etc.
E Conclusions
In this paper, we propose to study domain shifts between P and Q with the structure that each feature’s marginal distribution has good overlap between source and target domain but the joint distribution of the features may have a much bigger shift.
As a ﬁrst step toward understanding the extrapolation of nonlinear models, we prove sufﬁcient conditions for the model f ( x ) = (cid:80) ki
=
f i ( x i ) to extrapolate where f i is an arbitrary function of a single feature.
Even though the assumptions on the shift and function class is stylized, to the best of our knowledge, this is the ﬁrst analysis of how nonlinear models extrapolate when source and target distribution do not have shared support in concrete settings.
There still remain many interesting open questions, which we leave as future works
Our current proof can only deal with a restricted nonlinear model family of the special form f ( x ) = (cid:80) ki
=
f i ( x i ) due to technical reasons.
Can we extend to a more general model class? 2.
In this paper, we focus on regression tasks with (cid:96) 2 loss for mathematical simplicity, whereas majority of the prior works focus on the classiﬁcation problems.
Do similar results also hold for classiﬁcation problem?
F Missing Proofs
In the following, we present the missing proofs.
F.1 Proof of Proposition
In this section, we prove Proposition 1.
Proof of Proposition 1.
By the deﬁnition of τ , for any f ∈ F we get
[Equation]
Or equivalently,
[Equation]
As a result,
E Q [( y − f
[Equation]
x
[Equation] [Equation] ( x [Equation]
x
[Equation]
(26) ≤ 4 (cid:15) F + 2 τ E P
[( f ( x ) − f (cid:63) ( x )) 2 ] ≤ 4 (cid:15) F + 4 τ (cid:0) E P
[( y − f ( x )) 2 ] + E P [( y − f (cid:63) ( x )) 2 ] (cid:1) (27) ≤ (8 τ + 4) (cid:15) F + 4 τ E P
[( y − f ( x )) 2 ] .
F.2 Proof of Proposition
In this section, we state and prove Proposition 7.
Proposition 7.
Let P ( x 1 , x 2 ) be a distribution over discrete random variables x 1 ∈ X 1 , x 2 ∈ X 2 , and recall that F = { f 1 ( x 1 ) + f 2 ( x 2 ) } .
If there exists subsets U 1 ⊆
X 1 , U 2 ⊆
X 2 such that (1) 0 < | U 1 | < |X 1 | and 0 < | U 2 | < |X 2 | , and (2) P ( x 1 , x 2 )
=
when I [ x 1 ∈ U 1 ] (cid:54) =
I [ x 2 ∈ U 2 ] (e.g., the distribution visualized in Figure 1b), then we can construct models f, g ∈ F and x † ∈ X 1 × Xwhere (cid:107)
f − g
(cid:107) P = 0 and f ( x † ) (cid:54) =
g ( x † ) .
Proof.
Since the model class F is closed under addition, we only need to construct h ∈ F such that (cid:107) h (cid:107) P =
but h ( x † ) (cid:54) = 0 .
Let h 1 ( x 1 ) =
( − 1)
I [ x 1 ∈ U 1 ] and h 2 ( x 2 ) = − ( − 1)
I [ x 2 ∈ U 2 ] .
Consider the model h ( x )
=
h 1 ( x 1 ) + h 2 ( x 2 ) .
For any x 1 , x 2 where I [ x 1 ∈ U 1 ] =
I [ x 2 ∈ U 2 ] , we get h ( x ) = 0 .
Combining with condition (2) we get (cid:107) h (cid:107) P = 0 .
On the other hand, let x † = ( x † 1 , x † 2 ) with x † 1 ∈ U 1 and x † 2 ∈ X 2 \ U 2 .
Such x † exists because of condition (1).
Then h ( x † ) = − 2 .
F.3 Proof of Theorem
In this section, we prove Theorem 2.
Proof of Theorem 2.
Let
[Equation]
t ] , when t ≤ r 1 , I [ x [Equation]
= [Equation] ] , when r 1 < t ≤ [Equation]
Then for any f, f (cid:48) ∈ F , we can always ﬁnd v ∈ R r 1 + r 2 such that f ( x 1 , x 2 ) − f (cid:48) ( x 1 , x 2 ) =
(cid:80) r 1 + r {+
+} i
=
v i b i ( x 1 , x 2 ) for all x 1 , x 2 .
Indeed, for any f, f (cid:48) ∈ F the architecture of our model implies ( f − f (cid:48) )
( x 1 , x 2 ) =
( f 1 − f (cid:48) 1 )( x 1 ) + ( f 2 − f (cid:48) 2 )( x 2 ) .
Therefore, we can simply set v t =
( f 1 − f (cid:48) 1 )( t ) for 1 ≤ t ≤ r 1 and v t =
( f 2 − f (cid:48) 2 )( t − r 1 ) for r 1 < t ≤ r 2 .
Consequently,
[Equation]
The deﬁnition of b t ( x 1 , x 2 ) implies that for any distribution P ,
E P [ b i ( x 1 , x
[Equation]
b j ( x 1 , x 2 )] =   
I [ i = j ] P ( x 1 = i ) , when 1 ≤ i, j ≤ r 1 ,
I [ i = j ] P ( x 2 = j ) , when r 1 < i, j ≤ r 1 + r 2 ,
P ( x 1 =
i, x 2 = j ) , when 1 ≤ i ≤ r 1 < j ≤ r 1 + r 2 ,
P ( x 1 = j, x 2 = i ) , when 1 ≤ j ≤ r 1 < i ≤ r 1 + r 2 .
Then we have [ K P ] i,j = E P [ b i ( x 1 , x 2 )
b j ( x 1 , x 2 )] .
Consequently, τ = sup
[Equation]
sup
[Equation]
∈ [Equation] v
[Equation]
Let u ∈ R r 1 + r 2 such that [ u ] i = (cid:26) 1 , when i ≤ r 1 , − 1 , when r 1 < i ≤ r 2 .
We claim that u is a eigenvector to both K Q and K P with eigenvalue 0.
To see this, for any distribution
P and i ∈
[ r 1 ] we have
[Equation]
[Equation]
Similarly for i ∈ [ r 2 ] we have [ K P u ] r
+ i = 0 .
Combining these two cases together we prove
K P u = 0 .
Then, by algebraic manipulation,
[Equation]
As a result
[Equation]
we only need to prove λ max ( ¯ K Q ) ≤ 2 .
To this end [Equation] note that
[Equation]
F.4 Proof of Corollary
In this section, we prove Corollary 3.
Proof of Corollary 3.
Recall that Theorem 2 proves
[Equation]
As a result, we only need to prove conditions (a) and (b) implies the RHS of Eq.
(42) is ﬁnite.
First of all, by deﬁnition condition (a) implies max i ∈ [2] ,t ∈ [ r i ] P ( x i = t ) Q ( x i = t ) < ∞ .
Secondly, we prove that λ 2 ( ¯ K P ) =
implies the negation of condition (b), which is equivalent to the statement ”condition (b) implies λ 2 ( ¯ K P ) > 0 ”.
By Cheeger’s inequality (Eq.
(7)), λ 2 ( ¯ K P ) = {+
+} implies that φ ( G P ) = 0 .
By the deﬁnition of φ ( G P ) , there exists subsets U 1 ⊆
[ r 1 ] , U 2 ⊆
[ r 2 ] such that
P ( x 1 = u 1 , x 2 = u 2 ) = 0 when u 1 ∈ U 1 , u 2 (cid:54)∈ U 2 , and 2.
P ( x 1 = u 1 , x 2 = u 2 ) = 0 when u 1 (cid:54)∈ U , u 2 ∈ U 2 .
Then, we can reorganize P ( x 1 , x 2 ) by listing the rows corresponding to U 1 ﬁrst, and the columns corresponding to U 2 ﬁrst.
As a result, P ( x 1 , x 2 ) will be a block diagonal matrix where the ﬁrst block has dimension | U 1 |×| U 2 | and the second block ( r 1 − | U 1 | ) × ( r 2 − | U 2 | ) .
Consequently, we prove that λ 2 ( ¯ K P ) = {+
+} implies the negation of condition (b).
Hence, with condition (b) we get λ ¯ K P > 0 .
It follows that λ 2 ( ¯ K P ) − 1 < ∞ .
Combining everything together we have τ < ∞ .
F.5 Proof of Theorem
In this section, we prove Theorem 4.
Proof of Theorem 4.
Now consider any ﬁxed model pairs f, f (cid:48) ∈ F .
Let g i ( x i ) =
f i ( x i ) − f (cid:48) i ( x i ) and g = (cid:80) di
=
g i ( x i ) .
Recall that we assume the marginals satisﬁes x i ∼ N (0 , 1) for every i ∈ [ d ] .
As a result, Lemma 9 implies that there exists coefﬁcients { α ( n ) i } i ∈
[ d ] ,n ∈ Z + with
[Equation]
i
[Equation] x [Equation] (43) where { ψ n ( · ) } n ∈ Z + are a set of orthonormal basis of L 2 ( R ) deﬁned by
[Equation]
As a result,
[Equation]
[Equation]
By Theorem 11, we have
[Equation]
Continuing Eq.
(48) we get
[Equation]
[Equation]
= (cid:88) 1 ≤
i
[Equation]
(cid:88) n
[Equation]
α (
[Equation]
[Equation]
( n [Equation]
[Equation]
i
[Equation]
As a result,
[Equation]
Deﬁne κ = sup n ≥ 0 sup v ∈ R
d v (cid:62) (Σ Q ) (cid:12) n v v (cid:62) (Σ P ) (cid:12) n v .
In the following, we prove
[Equation]
For any ﬁxed n ≥ 0 , let Σ ( n ) P ∈ R
d
×
d be the matrix where [Σ ( n ) P ] i,j = ([Σ P ] i,j ) n (deﬁne Σ ( n ) Q similarly), and (cid:126)α ( n ) ∈ R d the vector where [ (cid:126)α ( n ) ]
i = α ( n ) i .
Then
E
[Equation]
By the deﬁnition of κ we get Σ ( n ) Q (cid:22) κ Σ ( n ) P for all n ≥ 0 .
Consequently, (cid:88)
[Equation]
55) which implies Eq.
Now we prove κ ≤ d λ min (Σ P ) .
Since Σ Q is a covariance matrix with diag(Σ Q )
=
I , we have (cid:107) Σ Q (cid:107) 2 ≤ d (cid:107) Σ Q (cid:107) ∞ ≤ d.
In addition, Lemma 13 proves that λ min (Σ (cid:12) nP ) ≥ λ min (Σ P ) for every n ≥ 1 .
As a result, inf n
[Equation]
Since n = 0 = ⇒ (Σ P ) (cid:12) n = (Σ Q ) (cid:12) n , we get
[Equation]
By the fact that λ min (Σ P ) ≤ d (cid:107) Σ P (cid:107) ∞ ≤ d , we prove the desired result.
In the following, we state and proof Lemma 8.
Lemma 8.
For any distributions P, Q with matching marginals, deﬁne P (cid:48) to be the density of the random variable t
= ( t 1 , · · · , t d ) where t i (cid:44) var P ( x i ) − 1 / 2 ( x i − E P [ x i ]) , x ∼ P (and deﬁne Q (cid:48) similarly).
Then we have
[Equation]
Proof.
Let { h i : R → R } i =1 , 2 , ··· ,d be any set of invertible functions.
Since f = (cid:80) i f i ( x i ) ∈ F is equivalent to f ◦ h (cid:44) (cid:80)
i f i ( h i ( x i ))
∈ F , we get sup
[Equation]
E [Equation] [Equation] Let P (cid:48) be the density of the random variable ( h 1 ( x 1 ) , · · · , h d ( x d )) when x ∼ P (and deﬁne Q (cid:48) similarly).
Then we have
[Equation]
Consequently,
[Equation]
Finally, this lemma is proved by taking h i ( x ) = var P ( x i ) − 1 / 2 ( x − E P [ x i ]) .
F.6 Proof of Theorem
In this section, we prove Theorem 5.
Proof of Theorem 5.
Without loss of generality, we assume d 2 ≤ d 1 .
Since F is closed under addition, we have
[Equation]
In the following [Equation] we ﬁrst prove
[Equation]
by considering two cases separately.
Case 1: when g = ( g 1 , g 2 ) ∈ F satisﬁes E P [ g 1 ( x 1 )]
=
E P [ g 2 ( x 2 )]
= 0 .
Since P and Q have matching marginals on x 1 and x 2 respectively, we get
[Equation]
As a result [Equation] by the deﬁnition of τ we have
[Equation]
Hence [Equation] we only need to prove
[Equation]
Let Σ 12 = U Λ V (cid:62) be the singular value decomposition of Σ 12 , where
Λ ∈ R d 1 × d 2 is a diagonal matrix with entries σ 1 , · · · , σ d 2 .
Following Theorem [Equation] we deﬁne t 1 =
U (cid:62) x 1 ∈ R d 1 , t 2 = V (cid:62) x 2 ∈ R d 2 .
Consequently, E P [ t 1 t (cid:62) 2 ] =
Λ and
[Equation]
Let P (cid:48)
( t 1 , t 2 ) be the density of the distribution of ( t 1 , t 2 ) , and P (cid:48)
( t 1 ) , P (cid:48) ( t 2 ) the density of corresponding marginals.
As a result,
[Equation]
Since g i ( t i ) (cid:112) P (cid:48)
( t i ) is square integrable, there exists weights { α ( n 1 , ··· ,n di ) i } n 1 , ··· ,n di ≥ 0 such that
[Equation]
where { (cid:81) d i j =1 ψ n j ([ t i ] j ) } n 1 , ··· ,n d i ≥ 0 forms an orthonormal basis of L 2 ( R d i ) .
Consequently,
[Equation]
Now we turn to the RHS of Eq.
Continuing Eq.
(68) and apply Theorem 12 we get,
[Equation]
and
[Equation]
As a result,
[Equation]
Now for any n 1 , · · · , n d 2 ≥ 0 , consider the matrix
[Equation]
[Equation]
When n 1 + · · · + n d 2 (cid:54) = 0 , we have (cid:81) d
2 i =1 σ n i i ≤ max i ∈
[ d 2 ] σ i because Lemma 14 implies σ i ≤for every i ∈ [ d 2 ] .
Therefore in this case,
[Equation] [Equation]
When n 1 + · · · + n d 2 = 0 , by Eq.
(69) we get α
[Equation]
[Equation]
[Equation]
Similarly, α (0 , ··· , 0)= 0 .
As a result, when n 1 + · · · + n d 2 =
we have (cid:32) α [Equation] [Equation] α
[Equation]
n
[Equation]
[Equation]
Combining these two cases together, we get
[Equation]
Plug in to Eq.
[Equation]
74) we get
[Equation]
which ﬁnished the proof for Eq.
Case 2: when g = ( g 1 , g 2 ) ∈ F satisﬁes E P [ g 1 ( x 1 ) + g 2 ( x 2 )]
= 0 .
Let µ = E P
[ g 1 ( x 1 )] .
Deﬁne g (cid:48) =
( g (cid:48) 1 , g (cid:48) 2 ) where g (cid:48) 1 ( x 1 )
=
g 1 ( x 1 ) − µ, g (cid:48) 2 ( x 2 )
=
g 2 ( x 2 ) + µ. By the deﬁnition of F we get g (cid:48) ∈ F , and g (cid:48) satisﬁes E P [ g 1 ( x 1 )]
= E P [ g 2 ( x 2 )]
= 0 .
Because g 1 ( x 1 )+ g 2 ( x 2 ) =
g (cid:48) 1 ( x 1 )+ g (cid:48) 2 ( x 2 ) for all x 1 ∈ R d 1 , x 2 ∈ R + d 2 , plugging in the result of case 1 we get
E Q [( g 1 ( x 1 ) + g 2 ( x 2 )) 2 ]
[Equation]
Case 3: when g = ( g 1 , g 2 ) ∈ F .
Now we consider the most general case.
Let ν = E P
[ g 1 ( x 1 ) + g 2 ( x 2 )] .
Since P and Q have matching marginals on x 1 and x 2 respectively, we get
[Equation]
Deﬁne g (cid:48)
=
( g (cid:48) 1 , g (cid:48) 2 ) where g (cid:48) 1 ( x 1 )
=
g 1 ( x 1 ) − ν, g (cid:48) 2 ( x 2 )
=
g 2 ( x 2 ) .
It follows that
[Equation]
where the last inequality comes from applying case 2’s result to g (cid:48) .
Combining these three cases together we prove Eq.
Then Eq.
(19) follows directly from Eq.
(62) and Lemma 15.
F.7 Proof of Proposition
In this section, we prove Proposition 6.
Proof of Proposition 6.
First, for any t ∈ S
d − 1 and (cid:15) > 0 we construct a two-layer neural network h t,(cid:15) ( x ) such that
for all x ∈ S d − 1 , h t,(cid:15) ( x ) = 0 if (cid:107) x − t (cid:107) 2 ≥ (cid:15) , 2. for all x ∈ S d − 1 , h t,(cid:15) ( x ) ≥ 1 if (cid:107) x − t (cid:107) 2 ≤ (cid:15)/ 2 , and 3.
for all x ∈ S d − 1 , h t,(cid:15) ( x ) ≥ 0 .
Recall that a two layer neural network is parameterized as (cid:80) i a i ReLU( w (cid:62) i x + b i ) .
Then h t,(cid:15) ( x ) can be constructed using one neuron by setting w 1 = t, b 1 = − 1 + (cid:15) 2 / 2 and w 1 = 8 / (3 (cid:15) 2 ) .
We can verify the construction as follows.
When (cid:107) x − t (cid:107) 2 ≥ (cid:15) , we get
[Equation]
As a result, a
[Equation]
When (cid:107) x − t (cid:107) 2 ≤ (cid:15)/ 2 , we get
[Equation]
As a result, a
[Equation]
Now we construct a two layer neural network f such that (cid:107) f (cid:107) P =
but (cid:107) f (cid:107) Q = 0 .
Let X
= { x : x ∈
S d − 1 , dist( x, supp( P )) ≥ (cid:15) } .
By the condition of this proposition we have Q ( X ) > 0 .
Let C be the minimum (cid:15)/ 2 -covering of the set X , and thus | C | < ∞ .
Consider the 2-layer neural network f ( x ) = (cid:80) c ∈ C h c,(cid:15) .
It follows from the deﬁnition of X that (cid:107) x − c
(cid:107) 2 ≥ (cid:15) for every x ∈ supp( P ) .
Consequently, f ( x ) = (cid:80) c ∈ C h c,(cid:15) = 0 for every x ∈ supp( P ) , which implies that (cid:107) f (cid:107) P = 0 .
Now for every x ∈ X , there exists t ∈ C such that (cid:107) x − t (cid:107) 2 ≤ (cid:15)/ 2 .
As a result, h t,(cid:15) ( x ) ≥ 1 , which implies that
[Equation]
Hence,
E Q [ f ( x ) 2 ] ≥ E Q
[ I [ x ∈
[Equation]
]] =
[Equation]
> [Equation] Since cf ( x ) ∈ F for every c > 0 , we prove the desired result.
G Hermite polynomial and Gaussian kernel
The Hermite polynomial H n ( x ) is an degree n polynomial deﬁned as follows,
[Equation]
with the following orthogonality property [Poularikas, 2018]:
[Equation] [Equation]
As a result, we can construct a set of orthonormal basis of square integrable functions L 2 ( R ) using the Hermite polynomial.
Lemma 9 (see e.g., Celeghini et al.
The set of functions { ψ n ( x ) } n ≥ 0 form an orthonormal basis for L 2 ( R ) , where
[Equation]
Theorem 10 (Chapter 6.2 of Fasshauer [2011], also see Zhu et al.
For any (cid:15) > 0 , α > 0 , the eigenfunction expansion for the Gaussian x, z ∈ R is
[Equation]
where
[Equation]
And the eigenfunctions φ n forms an orthonormal basis under weighted L 2 ( R ) space:
[Equation]
Theorem 11.
For any ρ ∈ [ − 1 , 1] [Equation] let P ( x 1 , x 2 ) be the density of ( x 1 , x 2 ) ∼ N (cid:18) 0 , (cid:18)ρ ρ (cid:19)(cid:19) and
P ( x 1 ) , P ( x 2 ) the density of corresponding marginals.
Then the Gaussian kernel
[Equation]
has the following eigen-decomposition
[Equation]
where ψ k ( · ) is deﬁned in Lemma 9.
Proof.
We prove this theorem by considering the following two cases separately.
Case 1: ρ ≥ 0 .
By algebraic manipulation we get
[Equation]
Let (cid:15) 2 = ρ 2(1 − ρ 2 ) and α 2 = ( ρ − 1) 2 2(1 − ρ 2 ) , we can equivalent write
[Equation]
By Theorem 10 we get
[Equation]
with (cid:82) φ m ( x ) φ n ( x ) α √ π exp( − α 2 x 2 )d x = I [ m = n ] .
Then we can deﬁne the function
[Equation]
(cid:18)
[Equation]
(105) such that (cid:82)
[Equation]
Combining Eqs.
(103),(104), and (105) we get
[Equation]
Now we only need to prove that ψ n ( · ) deﬁned in Eq.
(105) has the same form as those in Lemma 9, and λ n
[Equation]
Recall that
[Equation]
Plugin
[Equation]
get
[Equation]
As a result,
[Equation]
[Equation]
Now we turn to the eigenvalues.
Recall that
[Equation]
Plugin (cid:15) 2 =
ρ 2(1 − ρ ) 2 and α 2 = ( ρ − 1) 2 2(1 − ρ ) 2 we get α (cid:16) 1 + (cid:112) {+
+} + (2 (cid:15)/α ) 2 (cid:17) + (cid:15) 2 = 1 2 (cid:16)
α 2 + {+(cid:112) α 4 + 4 (cid:15) 
 α 2 (cid:17) + (cid:15)(116) = (cid:18)+}
[Equation]
[Equation]
Consequently,
[Equation]
[Equation]
Case 2: ρ < 0 .
Recall that
[Equation]
In this case we reuse the results from Case 1 to get
[Equation]
By deﬁnition, we get K ρ ( x 1 , x 2 ) =
K − ρ ( x 1 , − x 2 ) .
As a result,
[Equation]
where the last equation follows from the fact that ψ k ( x 2 ) =
( − 1) k ψ k ( − x 2 ) .
Theorem 12.
For any Σ ∈ R d 1 × d 2 , let P ( x 1 , x 2 ) be the density of ( x 1 , x 2 ) ∼ N (cid:18) 0 , [Equation] I d
[Equation]
and P ( x 1 ) , P ( x 2 ) the density of corresponding marginals, where x 1 ∈ R d 1 , x 2 ∈ R d 2 .
Without loss of generality, assume d 1 ≥ d 2 .
Let Σ = U Λ V (cid:62) be the singular value decomposition of Σ , where U ∈ R d 1 × d 2 , Λ ∈ R d 1 × d 2 ,
V ∈ R
d 2 × d 2 .
Let σ 1 , · · · , σ d 2 be the singular values of Σ (i.e., [Λ] i,i = σ i ).
Then the Gaussian kernel
[Equation]
122) has the following eigen-decomposition
[Equation]
where t 1 = U (cid:62) x 1 , t 2 = V (cid:62) x 2 , and ψ k ( · ) is deﬁned in Lemma 9.
Proof.
We prove this theorem by decomposing the multi-dimensional kernel into products of several one-dimensional kernels described in Theorem 11.
Recall that t 1 =
U (cid:62) x 1 ∈ R d 1 , t 2 = V (cid:62) x 2 ∈ R d 2 , where Σ = U Λ V (cid:62) is the singular value decomposition of Σ .
Let t ∈ R
d
1 + d 2 be the concatenation of t 1 , t 2 .
Then we have
[Equation]
Let P (cid:48)
( t 1 , t 2 ) be the density of ( t 1 , t 2 ) ∼ N
[Equation]
By the fact that Λ is a diagonal matrix, we know that variables { [ t 1 ] i , [ t 2 ] i } is independent from { [ t 1 ] j } j (cid:54)
=
i ∪ { [ t 2 ] j } j (cid:54) =
i for every 1 ≤ i ≤ d 2 , and variable [ t 1 ]
i is independent from { [ t 1 ] j } j (cid:54)
=
i ∪ { [ t 2 ] j } 1 ≤ j ≤ d 2 for every d 2 + 1 ≤ i ≤ d 1 .
In addition, ([ t 1 ] i , [ t 2 ] i ) ∼ N
[Equation]
As a result,
[Equation]
where Eq.
(127) follows from Theorem 12, and Eq.
(128) follows from the deﬁnition of ψ 0 .
H Helper Lemmas
Lemma 13.
Let Σ ∈ R
d
×
d be a positive semi-deﬁnite matrix with diag(Σ)
=
I.
Then we have
[Equation]
where Σ (cid:12) k denotes the element-wise k -th power of the matrix Σ .
Proof.
Let λ = λ min(Σ) and we have Σ − λI (cid:23) 0 .
As a result, (Σ − λI ) (cid:12) k
= Σ (cid:12) k +(1 − λ ) k
I − I (cid:23) 0 .
Note that 0 ≤ λ ≤ 1 because Tr(Σ − I )
= 0 implies λ min (Σ − I ) ≤ 0 .
It follows that for any k ≥ 1 ,
[Equation]
I [Equation] [Equation] 130) where the last inequality follows from the fact that 1 − λ ≥ (1 − λ ) k when k ≥ 1 .
Consequently, λ min (Σ (cid:12) k ) ≥ λ min (Σ) .
[Equation]
Lemma 14.
Let Σ ∈ R d 1 ×
d 2 be a matrix such that
[Equation]
Then we have λ max (Σ (cid:62) Σ) ≤ 1 .
Proof.
We prove by contradiction.
Suppose otherwise λ max (Σ (cid:62) Σ) > 1 .
Let v ∈ R
d
2 be the eigenvector of Σ (cid:62) Σ corresponds to its maximum eigenvalue.
Then
[Equation]
By the assumption that λ max (Σ
[Equation]
Σ) > 1 , we get
[Equation]
which contradicts to Eq.
Therefore, we must have λ max (Σ (cid:62) Σ) ≤ 1 .
Lemma 15.
Let M be a positive semi-deﬁnite matrix with the following form
[Equation]
for some d 1 , d 2 > 0 .
Then we have
[Equation]
where σ max is the largest singular value of Σ .
Proof.
Without loss of generality, we assume d 1 ≥ d 2 .
Let Σ = U Λ V (cid:62) be the singular value decomposition of Σ , with U ∈ R d 1 × d 1 , V ∈ R
d 2 × d 2 , Λ ∈ R
d 1 × d 2 .
Then we have
[Equation]
(cid:19) .
Let ¯ M = (cid:18) I dΛ Λ (cid:62)
I d(cid:19) .
Because U, V are orthonormal matrices, we get λ min ( M ) =
λ min ( ¯ M ) .
Note that [Λ] i,j =
whenever i (cid:54) =
j and [Λ] i,i =
σ i (Σ) .
Consequently, the eigenvalues of ¯ M is
[Equation]
with multiplicity 1, and 1 with multiplicity d 1 − d 2 .
As a result,
[Equation]