Improved Generalization Bounds for Transfer Learning via Neural Collapse
Abstract
Using representations learned by large, pretrained models, also called foundation models, in new tasks with fewer data has been successful in a wide range of machine learning problems.
Recently, Galanti et al.
(2022) introduced a theoretical framework for studying this transfer learning setting for classiﬁcation.
Their analysis is based on the recently observed phenomenon that the features learned by overparameterized deep classiﬁcation networks show an interesting clustering property, called neural collapse (Papyan et al., 2020).
A cornerstone of their analysis demonstrates that neural collapse generalizes from the source classes to new target classes.
However, this analysis is limited as it relies on several unrealistic assumptions.
In this work, we provide an improved theoretical analysis signiﬁcantly relaxing these modeling assumptions.
1. Introduction
Transfer learning is a prominent approach for dealing with overﬁtting (see, e.g., Caruana, 1995; Bengio, 2012;
Yosinski et al., 2014).
A popular approach for transfer learning suggests to pretrain a large neural network on a large-scale source task (e g., ResNet-50 on ImageNet ILSVRC, Russakovsky et al., 2015), and then to train a relatively small network on top of the penultimate layer of the pretrained model, using the available data in the target task.
In fact, due to the impressive success of transfer learning, large pretrained models that can be effectively adapted to a wide variety of tasks (Brown et al., 2020; Ramesh et al., 2021) have recently been characterized as ‘foundation models’ (Bommasani et al., 2021), emphasizing their adaptive nature.
While foundation models are intended to be generic and widely adaptive to downstream tasks, when some speciﬁcs of the target tasks are known, often special-purpose algorithms are designed to utilize this information.
This is the case, for example, in few-shot learning, where it is known in advance that the target problems come with a very small training set (Vinyals et al., 2016; Ravi & Larochelle, 2017;
Finn et al., 2017; Lee et al., 2019).
Even though these approaches signiﬁcantly improved the state of the art for a long time, surprisingly, recent works have demonstrated that predictors trained on top of pretrained models can achieve better performance on few-shot learning benchmarks (Tian et al., 2020; Dhillon et al., 2020;
Galanti et al., 2022).
Complementing the empirical results, Galanti et al.
(2022) also studied this approach theoretically,and provided an explanation for its success based on the recently discovered phenomenon of neural collapse (Papyan et al., 2020).
Informally, neural collapse (see Section 3) identiﬁes training dynamics of deep networks for standard classiﬁcation tasks, where the features (the output of the penultimate layer) associated with training samples belonging to the same class concentrate around their class feature mean.
Galanti et al.
(2022) showed that this property generalizes to new data points and new classes (e.g., the target classes), when the model is trained on several classes with many samples for each.
In addition, they showed that in the presence of neural collapse, training a linear classiﬁer on top of the learned feature map can generalize well, even if trained with few samples only.
However, their analysis demonstrating that neural collapse generalizes to new classes relies on some hard-to-justify assumptions regarding the feature maps, making their results less relevant in practical situations.
In this paper we provide a stronger theoretical analysis for this problem without relying on this kind of assumptions.
2. Problem Setup
We consider a transfer learning setting, where a model is pretrained on some source task and is adapted to solve many downstream tasks.
To model this problem, we assume that a ﬁnal downstream task is a k -class classiﬁcation problem (a target problem), and the auxiliary task where the feature representation is learned on is an l -class classiﬁcation problem, called the source problem.
Formally, a target task is deﬁned by a distribution P over samples ( x, y ) ∈
X × Y k , where
X ⊂ R d is the instance space, and Y k =
[ k ] := { 1 , . . .
, k } is a label space.
For a pair ( x, y ) with distribution P , we denote by P c the class-conditional distribution of x given y =
c (i.e., P c ( · ) :=
P [ x ∈ · | y = c ] ).
For a given target task, our goal is to learn a classiﬁer h :
X → Y k that minimizes the target test error
[Equation]
where I [ E ] denotes the indicator function of an event E (equals 1 if the event holds and 0 otherwise).
As usual, P is unknown and the algorithm is provided with some training data S
=
( x i , y i ) } nki =1 .
In this work we assume that P is a balanced distribution (i.e., P [ y = c ] = 1 /k ) with classconditionals { P c } kc =1 and that the data S is also a balanced set over the k classes, that is, S = ∪ kc =1 { ( x ci , c ) } ni =1 , where the sets S c
= { x ci } ni =1 ∼ P nc are drawn independently.
When n is small, training a classiﬁer on S may not be sufﬁcient to achieve reasonable performance.
To facilitate ﬁnding a good solution, we aim to ﬁnd a classiﬁer of the form h
= g ◦ f , where f ∈ F ⊂ { f (cid:48) : R d → R p } is a feature map and g ∈ G ⊂ { g (cid:48) : R p
→
R k } is a classiﬁer used on the feature space R p .
The idea is that the feature map f can be learned on some other problem where more data is available, and then g is trained to solve the hopefully simpler classiﬁcation problem of ﬁnding y based on f ( x ) , instead of x .
That is, g is actually a function of the modiﬁed training data { ( f ( x i ) , y i ) } nki =1 ; to emphasize this dependence, we use the notation g f,S for the trained classiﬁer based on the features, and h f,S =
g f,S ◦ f for the full classiﬁer (to be speciﬁed in Section 4).
We assume that the source task helping to ﬁnd f is a single l -class classiﬁcation problem over the same sample space
X , given by a distribution ˜ P , and here we are interested in ﬁnding a classiﬁer ˜ h :
X → R l of the form ˜ h
= ˜
g ◦ f , where ˜ g ∈ ˜
G ⊂ { g (cid:48) : R p
→ R l } is a classiﬁer over the feature space f
( X ) :=
{ f ( x ) : x ∈ X} .
Given a training dataset ˜ S
= { (˜ x i , ˜ y i ) } mli =1 , the classiﬁer ˜ h is trained on ˜ S , with the goal of minimizing the source test error, L ˜ P
(˜ h ˜ S ) .
Similarly to the target task, we assume that ˜ P is balanced
(i.e., ˜ P [ y = c ] = 1 /l ) with class-conditionals { ˜ P c } lc =1 , and that the dataset ˜ S = ∪ lc =1 { (˜ x ci , c ) } mi =1 is also balanced with ˜ S c = { ˜ x ci } mi =
∼ ˜ P
mc .
Finally, we assume that the classes in the source and target tasks are selected randomly: that is, the sets of class-conditionals ˜ P
= { ˜ P c } lc =1 and
P =
{ P c } kc =1 for the source and target tasks are sampled i.i.d.
from a distribution D over a set of class-conditional distributions E .
This completes the deﬁnition of the distribution of P and ˜ P (which are random themselves).
In a typical setting, the classiﬁer ˜ h is a deep neural network, f is the representation in the last internal layer of the network (i.e., the penultimate, a.k.a.
embedding layer), and ˜ g , the last layer of the network, is a linear map; similarly, g in the target problem is often taken to be linear.
The learned feature map f is called a foundation model (Bommasani et al., 2021) when it can be effectively used in a wide range of target tasks.
In particular, its effectiveness in dealing with downstream tasks (for any f ) can be measured by its expected transfer error ,
[Equation]
where the expectation is taken over the random choice of P (i.e., averaging over randomly selected target tasks) and the training data S for the target class.
Notice that while the feature map f is evaluated on the distribution of target tasks P determined by D , the training of f in a foundation model, as described above, is fully agnostic of this target.
In the rest of the paper we analyze this setting, and provide an explanation through the recent concept of neural collapse.
Notation.
(cid:107) · (cid:107) denotes the Euclidean norm for vectors and the spectral norm for matrices.
For a distribution Q over X ⊂ R d and u : X → R p , the mean and variance of u ( x ) for x ∼ Q are denoted by µ u ( Q ) :
=
E x ∼ Q [ u ( x )] and by Var u ( Q ) :
= E x ∼ Q [ (cid:107) u ( x ) − µ u ( Q ) (cid:107) 2 ] .
For a ﬁnite set
A = { a i } ni =1 , we denote Avg ni
=
[ a i ] := 1 n (cid:80) ni
=1 a i and by
U
[ A ] the uniform distribution over A .
For two functions f, g : X → [0 , ∞ ) , we denote f × ≤ g if there exists a constant C > 0 , such that, ∀ x ∈ X : f ( x ) ≤ C · g ( x ) .
3. Neural Collapse
Neural collapse (NC) is a recently discovered phenomenon in deep learning (Papyan et al., 2020): it has been observed that during the training of deep networks for standard classiﬁcation tasks, the features (the output of the penultimate, a.k.a. embedding layer) associated with training samples belonging to the same class concentrate around the mean feature value for the same class, also satisfying some additional conditions (such as the mean feature vectors being orthogonal to each other).
From these we focus on the class-features variance collapse (called NC1 by Papyan et al., 2020), and used a simpliﬁed version inroduced by Galanti et al.
For a feature map f : R d → R p and two distributions Q 1 , Q 2 over X ⊂ R d , we deﬁne their class-distance normalized variance (CDNV) as
[Equation]
(the deﬁnition can be extended to ﬁnite sets S 1 , S 2 ⊂ X by deﬁning V f
( S 1 , S 2 ) =
V f
( U [ S 1 ] , U [ S 2 ]) ).
This quantity essentially measures to what extent the feature vectors of samples from Q 1 and Q 2 are clustered in space.
We say that a training process satisﬁes neural collapse if
[Equation]
where f ( t ) is the penultimate layer of a neural network h ( t )
=
g ( t ) ◦ f ( t ) that is trained to ﬁt ˜ S for t steps.
It has been observed (Papyan et al., 2020; Galanti et al., 2022) that for several problems the left-hand side of (3) indeed reduces to some small value during training.
4. Generalization Bounds for Transfer Learning
Galanti et al.
(2022) introduced generalization bounds for transfer learning in the setting of Section 2.
Their analysis takes three main steps; ﬁrst, they show that if the CDNV on the training data, Avg i (cid:54) =
j ∈ [ l ] [ V f ( ˜ S i , ˜ S j )] , is small at the end of training, then we expect that the CDNV over unseen samples, Avg i (cid:54) =
j ∈ [ l ] [ V f ( ˜ P i , ˜ P j )] , would also be small for the same classes (see Lemma 4.1 below).
As a second step, they bound the expected CDNV between two new target classes, E P c (cid:54) =
P c (cid:48)
[ V f ( P c , P c (cid:48) )] , using the CDNV between the source classes, Avg i (cid:54) = j ∈ [ l ] [ V f ( ˜ P i , ˜ P j )] .
Finally, they prove that L D ( f ) × ≤ ( k + k/n ) · E P c (cid:54) =
P c (cid:48)
[ V f ( P c , P c (cid:48) )] , with h f,S :=
arg min c ∈ [ C ] (cid:107) f
( x j ) − µ f ( S c ) (cid:107) being the nearest class-center (NCC) classiﬁer.
However, there are multiple limitations to their arguments.
First, the term E P c (cid:54) =
P c (cid:48)
[ V f ( P c , P c (cid:48) )] may be very large even when L D ( f ) is very small.
For example, if there is at least one anomalous pair of classconditionals P 1 , P 2 ∈ E for which µ f ( P 1 ) = µ f ( P 2 ) , then, E
P c (cid:54) =
P c (cid:48)
[ V f ( P c , P c (cid:48) )]
= ∞ (as long as D [ P 1 ] , D
[ P 2 ] > 0 ) even if for all other pairs P (cid:48) 1 , P (cid:48) 2 ∈ E , V f ( P (cid:48) 1 , P (cid:48) 2 ) is tiny.
In fact, if the support E of D is inﬁnite and f is bounded, then clearly inf P 1 (cid:54) =
P 2 ∈E (cid:107) µ f ( P 1 ) − µ f ( P 2 ) (cid:107) =and E P c (cid:54) =
P c (cid:48)
[ V f ( P c , P c (cid:48) )]
= ∞ .
Second, the bound of
Galanti et al.
(2022) on E P c (cid:54) =
P c (cid:48)
[ V f ( P c , P c (cid:48) )] scales with (inf f ∈F inf P 1 (cid:54) =
P 2 ∈E (cid:107) µ f ( P 1 ) − µ f ( P 2 ) (cid:107) ) − 1 , which may be very large even if E is ﬁnite (e.g., if F contains a constant function).
In this paper we aim to circumvent these issue and provide an upper bound on the error of h which does not suffer from these limitations.
Instead of bounding the term E
P c (cid:54) = P c (cid:48) [ V f ( P c , P c (cid:48) )] , we bound L D ( f ) in terms of the averaged margin error of the NCC classiﬁer between pairs of source classes.
As a next step, in Lemma 4.3, we show that each of these error terms can be bounded using the CDNV between pairs of source classes.
Finally, in Theorem 4.4 we combine Lemmas 4.1-4.3 and obtain an upper bound on the transfer error of f in terms of Avg i (cid:54) =
j ∈ [ l ] [ V f ( ˜ S i , ˜ S j )] which is typically small (see Section 3).
4.1. Neural Collapse Generalizes to New Samples
We start by recalling Proposition 1 of Galanti et al.
This proposition provides an upper bound on V f ( ˜ P i , ˜ P j ) , decomposed into V f ( ˜ S i , ˜ S j ) and additional generalization gap terms bounding the difference between expectations and empirical averages for f and its variants, where f is the output of the learning algorithm with access to ˜ S .
For any δ ∈ (0 , 1) , we let (cid:15) c
1 ( δ ) and (cid:15) c
2 ( δ ) be the smallest positive values such that with probability at least 1 − δ , the learning algorithm returns a function f ∈ F that satisﬁes
[Equation]
These quantities are typically bounded using Rademacher complexities (Bartlett & Mendelson, 2002) related to F , scaling usually as O m (1 / √ m ) (as also shown by Galanti et al., 2022).
Next, we present their bound on the CDNV of the source distributions.
Lemma 4.1 (Galanti et al., 2022, Proposition 1) .
Fix two source classes, i and j with distributions ˜ P i and ˜ P j , and let δ ∈ (0 , 1) .
Let ˜ S
c ∼ ˜
P
mc for c ∈ { i, j } .
Let
[Equation]
Then, with probability at least 1 − δ over the selection of ˜
S i , ˜ S j , we have V f ( ˜ P i , ˜ P j ) ≤ ( V f ( ˜ S i , ˜ S j ) + B ij ( δ [Equation]
[Equation] (1 + [Equation]
The bound in the lemma has several terms.
The ﬁrst one is the empirical CDNV, V f ( ˜ S i , ˜ S j ) , which is assumed to be implicitly minimized by the training algorithm
(see Section 3).
The rest of the terms are proportional to the generalization gaps (cid:15)
c 1 ( δ/ 4) and (cid:15) c
2 ( δ/ 4) — as discussed above, typically we expect these terms to scale as O m (1 / √ m ) .
As discussed in (Galanti et al., 2022), the denominators in A ij ( δ/ 4) and
B ij ( δ/ 4) are expected to be lower bounded by positive constants under reasonable conditions.
Therefore, for a pair i (cid:54) =
j , if the CDNV V f ( ˜ S i , ˜ S j ) is small and m is large, then we expect V f ( ˜ P i , ˜ P j ) to be small, as well.
That is, if neural collapse emerges in the training data of two source classes, we should also expect it to emerge in unseen samples of the same classes (for large m ).
For an extensive empirical validation of this argument, see the works of Galanti et al.
; Galanti & Galanti (2022).
4.2. Bounding the Transfer Error
In this section we bound the transfer error of f using the averaged margin error of the NCC classiﬁer between pairs of source classes.
In the analysis, similarly to Galanti et al.
(2022), we treat class-conditional distributions as data points on which the feature map f is trained.
We apply standard techniques to derive generalization bounds to new data points, which, in this case, are class-conditional distributions.
Recall that, in line with this view, we have already assumed that the source and target class-conditionals { ˜ P c } lc = and { P c } kc =1 are i.i.d.
samples from D .
For simplicity, we focus on the case where F , the class of feature maps is a set of ReLU neural networks of the form f ( · ) =
W q σ ( W q − 1 . . .
σ ( W 1 · )) :
R d → R p , where σ ( x ) := max { 0 , x } is the element-wise ReLU function, and W i ∈
R d i +1 ×
d i for i ∈
[ q ] , where d 1 = d and d q +1 = p .
Our bounds depend on the spectral complexity of a network f , which is deﬁned as C ( f ) :
= max j ∈
[ p ] (cid:107) W qj (cid:107) · (cid:81) q − 1 r =1 (cid:107) W r (cid:107) and ˆ C ( f ) : = C ( f ) + 1 .
This quantity upper bounds the Lipschitz constant of f and is similar in fashion to other (slightly different) notions of spectral complexity for neural networks (Golowich et al., 2017; Bartlett et al., 2017).
We also denote B := sup x ∈X (cid:107) x (cid:107) .
We start by a generic result that allows us to upper bound the transfer error of the NCC classiﬁer (on top of the feature map f ) by the margin-error over the source classes (deﬁned below), plus some additional terms, depending on the complexity of f and the number of classes.
The proof, presented in Appendix A, is partially based on Corollary 3 of Maurer & Pontil (2019) and the analysis of Bartlett et al.
Lemma 4.2.
Let δ ∈ (0 , 1) , l ≥ 2 and let F be a class of ReLU neural networks of depth q .
Then, with probability at least 1 − δ over the random selection of the classconditionals ˜ P
= { ˜ P c } lc =1 of the source classes, for any f ∈ F and ∆ > 0 , we have
[Equation]
where L ∆ ij ( f ) :=
Pr ˆ x i , ˆ S i , ˆ S j [ (cid:107) f
(ˆ x i ) − µ f
( ˆ S j ) (cid:107) ≤ (cid:107) f
(ˆ x i ) − µ f
( ˆ S i ) (cid:107) + ∆] , where ˆ S i ∼ ˜ P ni , ˆ S j ∼ ˜ P nj and ˆ x i ∼ ˜ P
i .
The bound above can be decomposed into several parts.
The ﬁrst term is the average expected ∆ -margin error of a NCC classiﬁer, when averaged over all classiﬁcation problems given by the pairs of source classes with n samples each.
For each pair ( i, j ) , the NCC classiﬁer is deﬁned with two datasets of size n , ˆ S i and ˆ S j (which are independent of the training datasets ˜ S
i and ˜ S j ), and its performance is evaluated over a new test sample ˆ x i ∼ ˜ P
i .
The second term, δ , is a free parameter and can be selected to be very small as the third term scales polylogarithmically with respect to 1 /δ .
The third term is proportional to the ratio between the complexity of the selected function f , captured by √ qpkB · ˆ C ( f ) and ∆ √ l .
Therefore, as long as we increase the number of source classes l , we can expect the generalization to new classes to improve.
The next lemma is an extension of Proposition 5 of Galanti et al.
and relates the margin-error to the CDNV.
Lemma 4.3.
Let { ˜ P
i , ˜ P j } be a pair of class-conditional distributions.
Let f : R d → R p be any feature map and denote µ c = µ f ( ˜ P c ) .
Let ∆ ≤ 0 . 12 s ( f, ˜ P ) − 1 / 2 ·(cid:107) µ f ( ˜ P i ) − µ f ( ˜ P j ) (cid:107) .
Then, we have, for some universal constant C > 0 ,
[Equation]
, where s ( f, ˜ P ) = p if { f ◦
˜ P c } lc =1 are spherically symmetric and s ( f, ˜ P )
= 1 otherwise.
Informally, the expected margin error
L ∆ ij ( f ) of f is upper bounded by V f ( ˜ P i , ˜ P j ) , and when f ◦ ˜ P
i and f ◦ ˜ P
j are spherically symmetric then by (1 /p + 1 /n )
· V f ( ˜ P i , ˜ P j ) .
Therefore, in case of neural collapse (i.e., when the V f ( ˜ P i , ˜ P j ) are small on average), L ∆ ij ( f ) is small, explaining the success of foundation models in the low-data regime (such as in few-shot learning problems) in the presence of neural collapse.
Finally, we also derive a much better bound (see
Lemma B.2) which is exponentially small in p/V f ( ˜ P 1 , ˜ P 2 ) if f ◦ ˜ P c are (assumed to be) centered Gaussian distributions as also assumed by Papyan et al.
Putting together Lemmas 4.1-4.3, we obtain the following generalization bound.
Theorem 4.4.
Let δ ∈ (0 , 1) , l ≥ 2 and let F be a class of ReLU neural networks of depth q .
Then, with probability at least 1 − δ over the selection of the source class-conditionals ˜
P = { ˜ P c } lc =1 and the corresponding source training data ˜
S 1 . . .
, ˜ S l , for any f ∈ F and ∆ = 0 . 12 s ( f, ˜ P ) − 1 / 2 · min i (cid:54) = j ∈ [ l ] (cid:107) [Equation]
f
[Equation] ˜ P i ) [Equation] f ( ˜ P j ) (cid:107) , we have
[Equation]
for some universal constant C > 0 .
Namely, the transfer error of f is bounded by the sum of the averaged CDNV over the source training data, V f ( ˜ S i , ˜ S j ) , combinations of the terms A ij ( δ 8 l 2 ) and B ij ( δ 8 l 2 ) , where (cid:15) c 1 ( [Equation]
) = ˜ O (cid:16) [Equation] m (cid:17) and (cid:15)
[Equation] (cid:17) (see Galanti et al., 2022) and a term that scales as ˜ O (cid:16) √ qpkB · ˆ C ( f ) ∆ √
[Equation] (cid:17) . Therefore, if l and m are large and we have neural collapse across the source training samples (i.e., Avg i (cid:54) = j ∈ [ l ] [ V f ( ˜ S i , ˜ S j )] is small) and ˆ C ( f ) is bounded (as a function of m, l ), then the transfer error L D ( f ) is also small.
Interestingly, in the presence of neural collapse in the source data, the dependence on n is very weak.
5. Conclusions
Using pretrained models for transfer learning is a successful approach in the low-data regime.
Recently, Galanti et al.
(2022) suggested a new perspective on analysing this problem via the newly discovered phenomenon of neural collapse.
However, their theoretical analysis relied on several unrealistic assumptions.
In this work we provided better theoretical analysis by signiﬁcantly relaxing their assumptions 6. Acknowledgements
Tomer Galanti was supported by the Center for Minds,
Brains and Machines (CBMM), funded by NSF STC award
CCF-1231216 and by NSF award 213418.
A. Proof of Lemma 4.
To prove Lemma 4. [Equation] we apply Thm.
2 of Maurer & Pontil (2019).
In order to do so [Equation] we use the following deﬁnitions (their
Deﬁnition 1): Let (cid:107) · (cid:107) be the Euclidean norm, U ⊂ R d be any set and A : U
l → R .
For z = ( z 1 , . . .
, z l ) , j ∈ { 1 , . . .
, l } and z (cid:48)
j ∈ U , we deﬁne the j -th partial difference operator on A as
[Equation]
In addition
[Equation]
we denote
[Equation]
and
[Equation]
and the maximal difference
[Equation]
Now we are ready to proceed with the proof.
Lemma A.1.
Let δ ∈ (0 , 1) , l ≥ 2 and let F be a class of ReLU neural networks of depth q .
Then, with probability at least 1 − δ over the random selection of the class-conditionals ˜ P = { ˜ P c } lc =1 of the source classes, for any f ∈ F and ∆ > 0 , we have
[Equation]
 , where L ∆ ij ( f ) : =
Pr ˆ x i , ˆ S i , ˆ S j (cid:104) (cid:107) f
(ˆ x i ) − µ f
( ˆ S j ) (cid:107) ≤ (cid:107) f
(ˆ x i ) − µ f
( ˆ S i ) (cid:107) + ∆ (cid:105) , where ˆ S i ∼ ˜ P ni , ˆ S j ∼ ˜ P nj and ˆ x i ∼ ˜ P
i .
Proof.
In addition, we let U 1 = ( x 1 , S 1 ) and U 2 = ( x 2 , S 2 ) , where x i ∼ P i and S i
∼ P ni
(for i = 1 , 2 ).
We start the proof by reducing the error bound to a two-class classiﬁcation problem:
[Equation]
.,S
k E ( x,y ) ∼ P (cid:2) I
[ ∃ i (cid:54) =
y : (cid:107) f
( x ) − µ f ( P i ) (cid:107) ≤ (cid:107) f ( x ) − µ f ( P y ) (cid:107) ] (cid:3) ≤ E P 1 ,...,P k E S 1 ,...,S k E ( x,y ) ∼ P  (cid:88)
i (cid:54) =
y
I [ (cid:107) f ( x ) − µ f ( P i ) (cid:107) ≤ (cid:107) f
( x ) − µ f ( P y ) (cid:107) ]   = E P 1 ,...,P k E S 1 ,...,S k E y ∼ U
[ k ] E x ∼ P y  (cid:88)
i (cid:54) =
y
I [ (cid:107) f ( x ) − µ f ( P i ) (cid:107) < (cid:107) f ( x ) − µ f ( P y ) (cid:107) ]   =
E y ∼ U
[ k ] (cid:88)
i (cid:54) =
y
E P i ,P y E S i ,S y E x ∼ P y (cid:2)
I [ (cid:107) f
( x ) − µ f ( P y ) (cid:107) <
(cid:107) f
( x ) − µ f ( P i ) (cid:107) ] (cid:3) =
( k − 1) · E P 1 ,P 2 E S 1 ,S 2 E x 1 ∼ P 1 (cid:2) I [ (cid:107) f ( x ) − µ f ( P 2 ) (cid:107) <
(cid:107) f
( x ) − µ f ( P 1 ) (cid:107) ] (cid:3)
To bound the right-hand side above further, we introduce a “soft-margin” relaxation (cid:96) ∆ : R → [0 , ∞ ) of the zero-one loss with margin ∆ as
[Equation]
Then, clearly (cid:96) ∆ ( − r ) ≥ I [ r < 0] for all ∆ > 0 and r ∈ R , and so
I [
[Equation]
implying
[Equation]
By the random selection of P 1 , P 2 , ˜ P c , c ∈ [ l ] , the main term on the right-hand side above can be expressed as
[Equation]
To write this averaging succinctly, we introduce some notation.
Let ˆ U c :=
(ˆ x c , ˆ S c ) for c ∈ [ l ] where ˆ x and ˆ S c are deﬁned in the statement of the lemma ( ˆ x c ∼ ˜ P c
and
ˆ S c ∼ ˜
P
nc , independently for all c ), and and ˆ U :=
{ ˆ U c } lc =1 .
Furthermore, let Q :=
{ U = ( x, S ) | x ∈ X , S ∈ X n } denote the set of all point-sample pairs (note that ˆ U c ∈ Q ).
For any f ∈ F , deﬁne H f : Q → R as H f ( x, S ) =
( f ( x ) , µ f ( S )) and let H F ( Q ) =
{ H f ( x, S ) : ( x, S ) ∈ Q , f ∈ F} .
Finally deﬁne the aggregation function A ∆ : H F ( Q ) l → R as
[Equation]
where z = ( z 1 , . .
, z l ) such that z i =
( u i , v i )
∈ H F ( Q ) for all i ∈
[ l ] .
Then
[Equation]
by deﬁnition, and (5) can be rewritten as E ˆ U (cid:104)
A ∆ ( H f ( ˆ U )) (cid:105) and from (4) we get [Equation] f [Equation]
≤
[Equation]
Our next step is to bound the expectation on the right-hand side above with the randomly selected samples of class-conditional distributions
˜ P c , c ∈ [ l ] for a ﬁxed ∆ .
To this end, we will use the notion of Rademacher complexity: the Rademacher complexity of a set of vectors Y ⊂ R s is deﬁned as R ( Y ) :
=
E (cid:15) sup y ∈ Y
[ (cid:104) (cid:15), y (cid:105) ] , where (cid:15) =
( (cid:15) 1 , . . .
, (cid:15) s ) ∼ U [ {± 1 } s ] .
Let δ (cid:48) ∈ (0 , 1) and F (cid:48)
⊂ F with α = sup f ∈F (cid:48) C ( f ) .
Then, by Corollary 3 of Maurer & Pontil (2019) 1 , with probability at least 1 − δ (cid:48) over the selection of ˆ U , for all f ∈ F (cid:48) , we have
[Equation]
Next we bound separately the terms K ( A ∆ ) , M ( A ∆ ) , J ( A ∆ ) .
Bounding K ( A ∆ ) , M ( A ∆ ) and J ( A ∆ ) .
Let ∆ > 0 , j ∈ [ l ] and z i =
( u i , v i )
∈
R p × R p for i ∈
[ l ] .
Let z
=
( z 1 , . . .
, z l ) and z =
( z (cid:48) 1 , . . .
, z (cid:48) l ) , where z i =
z (cid:48)
i for all i (cid:54) =
j ∈ [ l ] .
We have
[Equation]
Since (cid:96) ∆ is ∆ − 1 -Lipschitz and by the triangle inequality
[Equation]
Similarly,
[Equation]
mean inequality.
Combining (8), (10) and (12), we obtain that
[Equation]
Using that (cid:107) u j − u (cid:48) j (cid:107) and (cid:107) v j − v (cid:48)
j (cid:107) are both bounded by 2 max x ∈X ,f ∈F (cid:48) (cid:107) f ( x ) (cid:107) ≤ 2 pαB for any z , z (cid:48) , we obtain from (8), (9) and (11)
that
[Equation]
Hence,
[Equation]
Next, we bound J ( A ∆ ) .
Let j, r ∈ [ l ] with j (cid:54) =
r and z = ( z 1 , . .
, z l ) ∈ H F ( Q ) l .
We have
[Equation]
where the second equality follows as all terms are 0 for which i (cid:54)
=
r , since the function in that case is constant in u r (since j (cid:54) =
r by our deﬁnition).
Using again that (cid:96) ∆ is ∆ − 1 -Lipschitz, we have
[Equation]
Similarly,
[Equation]
[Table]
[Equation]
Combining (15)–(17) with the arithmetic mean–quadratic mean inequality, we obtain
[Equation]
which gives
[Equation]
Substituting (13), (14), and (18) into (7), we get for and f ∈ F (cid:48) with sup f ∈F (cid:48)
C ( f ) ≤ α that with probability at least 1 − δ (cid:48) over the selection of ˆ U ,
[Equation]
To be able to adapt to the complexity C ( f ) of f and prove a bound for all ∆ , we will apply the above to F (cid:48)
=
F α := { f ∈ F :
C ( f ) ≤ α } with positive integers α and for an exponential grid for ∆ deﬁned as ∆ t = 2 − t B for any integer t and δ (cid:48)
=
δ t,α where δ t,α :=
δ [Equation]
1) | t | ( | t | +1) for all t (cid:54) = 0 and δ 0 ,α :=
δ 3 α ( α +1) .
Note that by deﬁnition, (cid:80) ∞ t =
−∞ (cid:80) ∞
α =1 δ t,α = δ .
Bounding E
ˆ U
[ R ( H F α ( ˆ U ))] .
As a next step, we upper bound E
ˆ U
[ R ( H F α ( ˆ U ))] .
Let (cid:15) be the aggregation of i.i.d.
Rademacher random variables (cid:15) 1 cj and (cid:15) 2 cj for c ∈ [ l ] and j ∈ [ p ] .
Then
E ˆ U
[ R ( H F α ( ˆ U ))]
=
E
[Equation]
U E (cid:15)   sup f ∈F α l (cid:88) c
= p (cid:88)
j = f (ˆ x c ) j (cid:15) 1 cj + µ f ( ˆ S c ) j (cid:15) 2 cj   ≤ p (cid:88) j =
E ˆ
[Equation]
E (cid:15) (cid:34) sup f ∈F α l (cid:88) c
= f (ˆ x c ) j (cid:15) 1 cj (cid:35) + p (cid:88)
j =
E ˆ U E (cid:15) (cid:34) sup f ∈F α l (cid:88) c
= µ f ( ˆ S c ) j (cid:15) 2 cj (cid:35) ≤ p (cid:88)
j =
E
[Equation]
U E (cid:15) (cid:34) sup f ∈F α l (cid:88) c
= f (ˆ x c ) j (cid:15) 1 cj (cid:35) + p (cid:88)
j =
E ˆ U E ˆ x c ∼ ˆ S c E (cid:15) (cid:34)
[Equation]
=
[Equation]
j =
[Equation] E ˆ [Equation] where f (ˆ x ) j denotes the j th coordinate of f (ˆ x ) , and F α | j :=
{ f ( x ) j | f ∈ F} .
Finally, by Theorem 1 of Golowich et al.
(2017), 2 we have
[Equation]
In particular,
[Equation]
Substituting this into (19) and using that l − 1 ≥ l/ 2 for l ≥ 2 and the union bound, we obtain that with probability at least 1 − δ , for all α ∈ N and t ∈ Z , and all f ∈ F α ,
[Equation]
δ (cid:19)(cid:33) .
Now, for any ∆ > 0 there exists a t such that ∆ ≤ ∆ t ≤ 2∆ ; in this case | t | ≤ | log( ∆ B ) | + 1 .
Now, since (cid:96) ∆ , and hence that of A ∆ , monotonically increases in ∆ , the above inequality with α
= (cid:100)C ( f ) (cid:101) ≤ C ( f ) + 1 implies E
[Equation]
≤
[Equation]
(cid:32) [Equation] δ (cid:33)  .
Taking expectation with respect to ˆ U given ˜ P , we have
[Equation]
Combining with (21) and also that E ˆ U (cid:104)
A ∆ ( H f ( ˆ U ))
(cid:105) ≤ 1 then yields
[Equation]
The statement of the lemma now follows by combining this inequality with (6).
B. Proof of Lemma 4.
Lemma B.1.
Let { ˜ P
i , ˜ P j } be a pair of class-conditional distributions for some given i, j ∈ [ l ] , i (cid:54) = j .
Let f : R d → R p be any feature map, and let ∆ ≤ 0 .
12 s ( f, ˜ P ) − 1 / 2 · (cid:107) µ f ( ˜ P i ) − µ f ( ˜ P j ) (cid:107) .
Then, we have
[Equation]
where s ( f, ˜ P ) = p
if { f ◦ ˜ P c } lc =1 are spherically symmetric and s ( f, ˜ P )
= 1 otherwise.
Proof.
For c ∈ { i, j } , let ˆ S c ∼ ˜ P nc and ˆ x c ∼ ˜ P c , ˆ u c = f
(ˆ x c ) , µ c = µ f ( ˜ P c ) , ˆ µ c = µ f ( ˆ S c ) and σ 2 c = Var(ˆ u c ) .
Let d ij := (cid:107) µ i − µ j (cid:107) denote the considered centers’ true distance, which sets the overall scale, and ˆ d ij := (cid:107) ˆ µ i − ˆ µ j (cid:107) its estimate.
Let s :
= s ( f, ˜ P ) and γ 1 , γ 2 , γ 3 , γ 4 ∈ (0 , 1) be constants that satisfy γ 4 − 2 γ 3 − 4 γ 1 − 2 γ 22 /s > 0 (to be selected by the end of the proof).
Moreover, let α ij := d 2 ij γ 1 / ∆ , where ∆ = d ij γ 2 / √ s .
[Figure]
ErrU
[Figure]
The basic proof idea is graphically illustrated in Figure 1: Consider the error event Err : = [ (cid:107) ˆ u
i − ˆ µ j (cid:107) ≤ (cid:107) ˆ u i − ˆ µ i (cid:107) + ∆] , then L ∆ ij ( f ) =
Pr[ Err ] .
For ∆ = 0 , Err is a p -dimensional half-space orthogonal to the unit vector ˆ e ij :=
(ˆ µ j − ˆ µ i ) / ˆ d ij , and to determine its probability, we can simply project all quantities onto the line R · ˆ e ij .
This makes the problem one-dimensional and improves the bound by a factor of p − 1 in the spherically symmetric case.
However, one complication is that ˆ e ij itself is random, so in order to indeed achieve a scale of 1 /p , one has to carefully avoid using triangle inequalities involving ˆ u i , which complicates the analysis, and also gives an extra 1 /n term.
∆ > 0 involves further complications: Consider the error set ErrU (ˆ µ
i , ˆ µ i ) :
= { ˆ u i : (cid:107) ˆ u i − ˆ µ j (cid:107) ≤ (cid:107) ˆ u i − ˆ µ i (cid:107) + ∆ } .
Its boundary ∂ ErrU is now (one sheet of an) elliptic hyperboloid, and direct projection onto ˆ e ij does not work anymore.
If we conﬁne ˆ u i to the hyperball B := { ˆ u i : (cid:107) ˆ u i − ˆ µ i (cid:107) ≤ α ij } , then B cuts ErrU into a ﬁnite moon-shaped set, and the projection method works again, while the probability of ˆ u i (cid:54)∈ B scales with 1 /p .
[-[Equation] (cid:107) / 5 and β ij ≤ (cid:107) µ i − µ j (cid:107) / 
 and obtain-]We derive a sequence of bounds (A)-(H) which we chain all together at the end.
A)
As a ﬁrst step we determine the halfspace (given
ˆ µ
i and
ˆ
µ j ) that covers the moon B
∩
ErrU. Assume that (cid:107) ˆ
u i −
ˆ
µ j (cid:107) ≤ (cid:107)
ˆ
u i −
ˆ
µ i (cid:107) + ∆ and (cid:107)
ˆ
u i −
ˆ µ
i (cid:107)
≤
α
ij
Then,
[Equation]
Therefore, we have 2(ˆ u i − ˆ µ i ) (cid:62) (ˆ µ j − ˆ µ i ) ≥ ˆ d 2 ij − 2 α ij ∆ − ∆ 2 , which can also be written as follows
[Equation]
This is the halfspace H that covers B ∩ ErrU most tightly. (B) We can use this to bound the probability of the event Err in terms of α ij and ∆ by conditioning:
[Equation]
(C) Next we upper bound the term Pr[(ˆ u i − ˆ µ i ) (cid:62) ˆ e ij ≥ ˆ β ij ] .
Namely, [Equation] for γ 3 ∈ ( [Equation]
[Equation]
(D) By applying the union bound and use that a projection onto ˆ e ij decreases the length of the vector µ i − ˆ µ i , we get
[Equation]
(E) Next we analyze each term separately. For a ﬁxed pair ˆ µ i and ˆ µ j , the variance of the random variable (ˆ u i − µ i ) (cid:62) ˆ e ij is at most Var(ˆ u i ) /s , where we have exploited the projection in the spherically symmetric case.
Therefore, by Markov’s inequality,
[Equation]
(F) Again, by Markov’s inequality, and using ∆ ≤ d ij γ 2 / √ s , hence α ij = d 2 ij γ 1 / ∆ ≥ d ij √ s ( γ 1 /γ 2 ) , we have
[Equation]
(G) and
[Equation]
(H)
Next, by conditioning,
[Equation]
where the last inequality follows from solving the triangle inequality
[Equation]
for ˆ d ij .
We can bound the ﬁrst term by
[Equation]
We notice that if ˆ β ij ≤ γ 3 d ij and ˆ d ij > γ 4 d ij , then
[Equation]
which is a contradiction to γ 4 − 2 γ 3 − 4 γ 1 − 2 γ 22 /s > 0 for appropriate choices of γ 1 , . . . , γ 4 ∈ (0 , 1) .
Hence
[Equation]
Combining everything together ( B ) ≤ ( C ) + ( F ) ≤ ( D ) + ( H ) + ( F ) ≤
( G ) + ( E ) + ( F ) + ( H ) and using
Var(ˆ µ i ) = Var(ˆ u i ) /n
we ﬁnally obtain
[Equation]
Finally, by choosing γ 1 = γ 2 = 0 . 12 , γ 3 = γ 4 / 2 .
5 and γ 4 = 0 .
65 (satisfying γ 4 − 2 γ 3 − 4 γ 1 − 2 γ 22 /p > 0 ), we obtain
Pr[
[Equation]
[Equation]
ˆ u i )
[Equation]
Lemma B.2.
Let { ˜ P
i , ˜ P j } be a pair of class-conditional distributions.
Let f : R d → R p be any feature map and denote µ c = µ f ( ˜ P c ) .
Let ∆ < (cid:107) µ i − µ j (cid:107) .
Assume that f ◦ ˜ P c are all spherical Gaussians with means µ c (for all c ∈ [ l ] ).
Assume that V ijf =
Var f ( ˜ P i ) ( (cid:107) µ i − µ j (cid:107)− ∆) 2 ≤ 16 for all i (cid:54) =
j ∈ [ l ] .
Then, we have
[Equation]
Proof.
Let i (cid:54) =
j ∈ [ l ] be a given pair.
Let d ij := (cid:107) µ i − µ j (cid:107) be the considered centers’ distance, which sets the overall scale.
Let ˆ S c ∼ ˜ P n and ˆ x c ∼ ˜ P c .
Let u c
=
f
(ˆ x c ) , µ c = µ f ( ˜ P c ) , ˆ µ c = µ f ( ˆ S c ) and σ 2 c =
Var( u c ) .
By the triangle inequality,
Pr [ (cid:107) u i − ˆ µ j (cid:107) ≤ (cid:107) u
i − ˆ µ i (cid:107) + ∆] ≤ Pr [ d ij ≤ 2 (cid:107) u i − µ i (cid:107) + (cid:107) ˆ µ
i − ˆ µ i (cid:107) + (cid:107) ˆ µ j − µ j (cid:107) + ∆] ≤ Pr [( d ij − ∆) / 4 ≤ (cid:107) u i − µ i (cid:107) ] + Pr [( d ij − ∆) / 4 ≤ (cid:107) ˆ µ
i − ˆ µ i (cid:107)
[Equation]
Since u c ∼ N ( µ c , σ 2 c · I /p ) and ˆ µ c ∼ N ( µ c , σ 2 c · I / ( np )) , the random variables E
i
=
(cid:107) u
i − µ i (cid:107) 2 p σ 2 i and U i = (cid:107) ˆ µ i − µ i (cid:107) 2 p nσ {+
+} i are chi-squared random variables with p components.
We can write
Pr
[Equation]
where F ( x, p ) is the CDF of the chi-squared distribution with p components and z = (cid:16) d ij − ∆ 4 σ c (cid:17) 2 .
By Chernoff’s bound, if z > 1 , we have
[Equation]
Similarly, we have
[Equation]
Finally, by combining (22)-(24) we obtain the desired inequality.
C. Proof of Theorem 4.
Theorem C.1.
Let δ ∈ (0 , 1) , l ≥ 2 and let F be a class of ReLU neural networks of depth q .
Then, with probability at least 1 − δ over the selection of the source class-conditionals ˜ P = { ˜ P c } lc =1 and the corresponding source training data ˜
S 1 . . .
, ˜ S l , for any f ∈ F and ∆ = 0 . 12 s ( f, ˜ P ) − 1 / 2 · min i (cid:54) =
j ∈ [ l ] (cid:107) µ f ( ˜ P i ) − µ f ( ˜ P j ) (cid:107) , we have
[Equation]
for some universal constant C > 0 .
Proof.
By Lemma A.1, with probability at least 1 − δ 2 over the selection of ˜ P , for any f ∈ F and ∆ > 0 , we have
[Equation]
In addition, by Lemma B.1, for any f and for ∆ = 0 .
[Equation]
( f
[Equation]
˜ [Equation] [Equation] ˜ P
[Equation]
˜ P [Equation] , we have,
[Equation]
By Lemma 4.1, for any pair ˜ P i , ˜ P j with probability at least 1 − δ 2 l 2 over the selection of ˜ S
i , ˜ S j , we have V f ( ˜ P i , ˜ P j ) ≤ ( V f ( ˜ S i , ˜ S j ) + B ij ( δ 8 l 2 )) ·
(1 + A ij ( δ 8 l 2 )) 2 .
Therefore, by union bound, with probability at least 1 − δ 2 over the selection of ˜ S 1 , . . .
, ˜ S l , we have
Avg
[Equation]
[Equation]
Again by union bound, with probability at least 1 − δ , over the selection of ˜ P = { ˜ P c } lc =1 and { ˜ S c } lc
=
1 , we can combine (25) and (26) which gives us the desired bound.