O N THE R OLE OF N EURAL C OLLAPSE IN T RANSFER L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the ability of foundation models to learn representations for classiﬁcation that are transferable to new, unseen classes.
Recent results in the literature show that representations learned by a single classiﬁer over many classes are competitive on few-shot learning problems with representations learned by special-purpose algorithms designed for such problems.
In this paper we provide an explanation for this behavior based on the recently observed phenomenon that the features learned by overparameterized classiﬁcation networks show an interesting clustering property, called neural collapse.
We demonstrate both theoretically and empirically that neural collapse generalizes to new samples from the training classes, and – more importantly – to new classes as well, allowing foundation models to provide feature maps that work well in transfer learning and, speciﬁcally, in the few-shot setting
I NTRODUCTION
In a variety of machine learning applications, we have access to a limited amount of data from the task that we would like to solve, as labeled data is oftentimes scarce and/or expensive.
In such scenarios, training directly on the available data is unlikely to produce a hypothesis that generalizes well to new, unseen test samples.
A prominent solution to this problem is to apply transfer learning (see, e.g., Caruana, 1995; Bengio, 2012;
Yosinski et al., 2014).
In transfer learning, we are typically given a large-scale source task (e.g., ImageNet ILSVRC, Russakovsky et al., 2015) and a target task from which we encounter only a limited amount of data.
While there are multiple approaches to transfer knowledge between tasks, a popular approach suggests to train a large neural network on a source classiﬁcation task with a wide range of classes (such as ResNet50,
He et al., 2016,
MobileNet, Howard et al., 2017 or the VGG network, Simonyan & Zisserman, 2014), and then to train a relatively smaller network (e.g., a linear classiﬁer or a shallow MLP) on top of the penultimate layer of the pretrained network, using the data available in the target task.
Due to the effectiveness of this approach, transfer learning has become a central element in the machine learning toolbox.
For instance, using pretrained feature maps is common practice in a variety of applications, including ﬁne-grained classiﬁcation (Chen et al., 2019a; Huang & Li, 2020;
Yang et al., 2018), object detection, (Redmon et al., 2016; Ren et al., 2015; He et al., 2017), semantic segmentation (Long et al., 2015; Chen et al., 2018), or medical imaging (Chen et al., 2019b).
In fact, due to the cumulative success of transfer learning, large pretrained models that can be effectively adapted to a wide variety of tasks (Brown et al., 2020; Ramesh et al., 2021) have recently been characterized as foundation models (Bommasani et al., 2021), emphasizing their central role in solving various learning tasks.
Typically, a foundation model is pretrained on a source task at a time when the concrete description of the target task (or target tasks) is not – or only partially – available to the practitioner.
Therefore, the ability to precondition or design the training regime of the foundation model to match the target task is limited.
As a result, there might be a domain shift between the source and target tasks, such as a different amount of target classes, or a different number of available samples.
Hence, intuitively, a foundation model should be fairly generic and applicable in a wide range of problems.
On the other hand, when some speciﬁcs of the target tasks are known, often special-purpose algorithms are designed to utilize this information.
Such an example is the problem of few-shot learning, when it is known in advance that the target problems come with a very small training set (Vinyals et al., 2016; Ravi & Larochelle, 2017; Finn et al., 2017; Lee et al., 2019).
While these specialized algorithms have signiﬁcantly improved the state of the art, the recent work of Tian et al.
Dhillon et al.
(2020) demonstrated that predictors trained on top of foundation models can also achieve state-of-the-art performance on few-shot learning benchmarks.
Despite the wide range of applications and success of transfer learning and foundation models in particular, only relatively little is known theoretically why transfer learning is possible between two tasks in the setting mentioned above.
Contributions.
In this paper we present a new perspective on transfer learning with foundation models, based on the recently discovered phenomenon of neural collapse (Papyan et al., 2020).
Informally, neural collapse identiﬁes training dynamics of deep networks for standard classiﬁcation tasks, where the features (the output of the penultimate layer) associated with training samples belonging to the same class concentrate around their class feature mean.
We demonstrate that this property generalizes to new data points and new classes (e.g., the target classes), when the model is trained on a large set of classes with many samples for each class.
In addition, we show that in the presence of neural collapse, training a linear classiﬁer on top of the learned penultimate layer can be done using only few samples.
We verify these ﬁndings both theoretically and via experiments.
In particular, our results provide a compelling explanation for the empirical success of foundation models, as observed, e.g., by Tian et al.
; Dhillon et al.
The rest of the paper is organized as follows: Some additional related work is discussed in Section 1.1.
The problem setting, in particular, foundation models, are introduced in Section 2.
Neural collapse is described in Section 3.
Our theoretical analysis is presented in Section 4: generalization to unseen examples is considered in Section 4.1, to new classes in Section 4.2, while the effect of neural collapse on the classiﬁcation performance is discussed in Section 4.3.
Experiments are presented in Section 5, and conclusions are drawn in Section 6.
Proofs and additional experiments are relegated to the appendix.
1. O THER R ELATED W ORK
Various publications, such as (Baxter, 2000; Maurer et al., 2016; Pentina & Lampert, 2014; Galanti et al., 2016; Khodak et al., 2019; Du et al., 2021), suggested different frameworks to theoretically study various multi-task learning settings.
However, all of these works consider learning settings in which the learning algorithm is provided with a sequence of learning problems, and therefore, are unable to characterize the case where only one source task is available.
In contrast, we propose a theoretical framework in which the learning algorithm is provided with one classiﬁcation task and is required to learn a representation of the data that can be adapted to solve new unseen tasks using few samples.
This kind of modeling is aligned with the common practice of transfer learning using deep neural networks.
While in unsupervised domain adaptation (DA) (Ben-David et al., 2006; Mansour et al., 2009) one typically has access to a single source and a single target task, one does not have access to labels from the former, making the problem ill-posed by nature and shaping the algorithms to behave differently than those that are provided with labeled samples from the target task.
In particular, although in transfer learning we typically train the feature map on the source dataset, independently of the target task, in DA this would signiﬁcantly limit the algorithm’s ability to solve the target task.
P ROBLEM S ETUP
We consider the problem of training foundation models, that is, general feature representations which are useful on a wide range of learning tasks, and are trained on some auxiliary task.
To model this problem, we assume that the ﬁnal target task we want to solve is a k -class classiﬁcation problem T (the target problem), coming from an unknown distribution D over such problems, and the auxiliary task where the feature representation is learned on an l -class classiﬁcation problem, called the source problem.
Formally, the target task is deﬁned by a distribution P over samples ( x, y ) ∈
X × Y k , where X ⊂ R d is the instance space, and Y k is a label space with cardinality k .
To simplify the presentation, we use one-hot encoding for the label space, that is, the labels are represented by the unit vectors in R k , and
Y k = { e i : i = 1 , . . . , k } where e i ∈
R k is the i th standard unit vector in R k ; with a slight abuse of notation, sometimes we will also write y
= i instead of y = e i .
For a pair ( x, y ) with distribution P , we denote by P i the class conditional distribution of x given y =
i (i.e., P i ( · ) =
P [ x ∈ · | y
=
i ] ).
A classiﬁer h : X → R k assigns a soft label to an input point x ∈ X , and its performance on the target task T is measured by the risk
[Equation]
where (cid:96) :
R k ×Y k → [0 , ∞ ) is a non-negative loss function (e.g., zero-one or cross-entropy losses).
Our goal is to learn a classiﬁer h from some training data S
=
( x i , y i ) } ni =1 of n independent and identically distributed (i.i.d.)
samples drawn from P .
However, when n is small and the classiﬁcation problem is complicated, this might not be an easy problem.
To facilitate ﬁnding a good solution, we aim to ﬁnd a classiﬁer of the form h
= g ◦ f , where f : R d → R p is a feature map from a family of functions F ⊂ { f (cid:48) :
R d → R p } and g ∈ G = { g (cid:48) : R p
→ R k } is a classiﬁer used on the feature space R p .
The idea is that the feature map f is learned on some other problem where more data is available, and then g is trained to solve the hopefully simpler classiﬁcation problem of ﬁnding y based on f ( x ) , instead of x .
That is, g is actually a function of the modiﬁed training data { ( f ( x i ) , y i ) } ni =1 ; to emphasize this dependence, we denote the trained classiﬁer by g f,S .
We assume that the auxiliary (source) task helping to ﬁnd f is an l -class classiﬁcation problem over the same sample space X , given by a distribution ˜ P , and here we are interested in ﬁnding a classiﬁer ˜
h :
X → R l of the form ˜ h
= ˜
g ◦ f , where ˜ g ∈ ˜
G ⊂ { g (cid:48) : R p
→ R l } is a classiﬁer over the feature space f
( X ) =
{ f ( x ) : x ∈ X} .
Given a training dataset ˜ S
= { (˜ x i , ˜ y i ) } mi =1 , both components of the trained classiﬁer, denoted by f ˜ S and ˜ g ˜ S are trained on ˜ S , with the goal of minimizing the risk in the source task, given by
[Equation]
(note that with a slight abuse of notation we used the same loss function as in (1), although they operate over spaces of different dimensions).
A standard assumption is that ˜ S is drawn i.i.d.
form ˜
P ; however, in this work instead we will use hierarchical schemes where ﬁrst classes are sampled according to their marginals in ˜ P , and then for any selected class c , training samples ˜ S c are chosen from the corresponding class-conditional distribution (or just class conditional, for short)
˜ P c ; the speciﬁc assumptions on the distribution of ˜ S will be given when needed.
Since f, ˜ g and ˜ h always depend on the source data only, to simplify the notation we will often drop the ˜ S subscript whenever this does not cause any confusion.
In a typical setting ˜ h is a deep neural network, f is the representation in the last internal layer of the network (i.e., the penultimate embedding layer), and ˜ g , the last layer of the network, is a linear map; similarly g in the target problem is often taken to be linear.
The learned feature map f is called a foundation model (Bommasani et al., 2021) when it can be effectively used in a wide range of tasks.
In particular, its effectiveness can be measure by its expected performance over target tasks:
[Equation]
(recall that P is the distribution associated with task T ).
Notice that while the feature map f is evaluated on the distribution of target tasks determined by D , the training of f in a foundation model, as described above, is fully agnostic of this target.
In contrast, several transfer learning methods (such as few-shot learning algorithms) have been developed which optimize f not only as a function of its training data ˜ S , but also based on some properties of
D , such as the number of classes and the number of samples per class in S .
Perhaps surprisingly, recent studies (Tian et al., 2020; Dhillon et al., 2020) demonstrate that the target-agnostic training of foundation models (or some slight variation of them, such as transductive learning) are competitve with such special purpose algorithm.
In the rest of the paper we analyze this phenomenon, and provide an explanation through the recent concept of neural collapse.
Notation.
For an integer k ≥ 1 , [ k ] = { 1 , . . . , k } .
For any real vector z , (cid:107) z (cid:107) denotes its Euclidean norm.
For a given set A = { a 1 , . . . , a n } ⊂ B and a function u ∈ U ⊂ { u (cid:48) : B → R } , we deﬁne u ( A ) =
( u ( a 1 ) , . . .
, u ( a n )) and U ( A ) =
{ u ( A ) : u ∈ U} .
Let Q be a distribution over X ⊂ R d and u : X → R p .
We denote by µ u ( Q ) =
E x ∼ Q [ u ( x )] and by Var u ( Q ) =
E x ∼ Q [ (cid:107) u ( x ) − µ u ( Q ) (cid:107) 2 ] the mean and variance of u ( x ) for x ∼ Q .
For A above, we denote by Avg ni =
[ a i ] =
Avg A =n (cid:80) ni
=1 a i the average of A .
For a ﬁnite set A , we denote by U [ A ] the uniform distribution over A .
N EURAL C OLLAPSE
Neural collapse (NC) is a recently discovered phenomenon in deep learning (Papyan et al., 2020): it has been observed that during the training of deep networks for standard classiﬁcation tasks, the features (the output of the penultimate, a.k.a. embedding layer) associated with training samples belonging to the same class concentrate around the mean feature value for the same class.
More concretely, Papyan et al.
(2020) observed essentially that the ratio of the within-class variances and the distances of the class means converge to zero.
They also noticed that asymptotically the classmeans (centered at their global mean) are not only linearly separable, but are actually maximally distant and located on a sphere centered at the origin up to scaling (they form a simplex equiangular tight frame), and furthermore, that the behavior of the last-layer classiﬁer (operating on the features) converges to that of the nearest-class-mean decision rule.
Han et al.
(2021) provided further empirical evidence regarding the presence of neural collapse in the terminal stages of training for both
MSE- and cross-entropy-loss minimization.
On the theoretical side, already Papyan et al.
(2020) showed the emergence of neural collapse for a Gaussian mixture model and linear score-based classiﬁers.
Poggio & Liao (2020b;a); Mixon et al.
(2020) theoretically investigated whether neural collapse occurs when training neural networks under MSE-loss minimization with different settings of regularization.
Furthermore, Zhu et al.
(2021) showed that in certain cases, any global optimum of the classical cross-entropy loss with weight decay in the unconstrained features model satisﬁes neural collapse.
Papyan et al.
(2020) deﬁned neural collapse via identifying that the within-class variance normalized by the intra-class-covariance tends to zero towards the end of the training (see Appendix B.4 for details).
In this paper, we work with a slightly different variation of within-class variation collapse, which will be later connected to the clusterability of the sample feature vectors.
For a feature map f and two (class-conditional) distributions Q 1 , Q 2 over X , we deﬁne their class-distance normalized variance (CDNV) to be
[Equation]
The above deﬁnition can be extended to ﬁnite sets S 1 , S⊂
X by deﬁning
V f ( S 1 , S 2 ) =
V f
( U [ S 1 ] , U [ S 2 ]) .
Our version of neural collapse (at training) asserts that lim t →∞
Avg i (cid:54) = j ∈ [ l ] (cid:104) V f t ( ˜ S
i , ˜ S j ) (cid:105) = 0 .
Intuitively, when decreasing the variance of the features of a given class in comparison to its distance from another class, we expect to be able to classify the features into the classes with a higher accuracy. This deﬁnition is essentially the same as that of Papyan et al.
(2020), making their empirical observations about neural collapse valid for our deﬁnition (as also demonstrated in our experiments),
but our new deﬁnition simpliﬁes the theoretical analysis.
Furthermore, the theoretical results of Papyan et al.
Zhu et al.
Mixon et al. ; Poggio & Liao (2020a;b) also identify settings in which neural collapse holds for training, or the feature map f induced by the global minima of the training loss satisﬁes Avg i (cid:54) = j V f ( ˜ S i , ˜ S
j )
{+= 
+}
(for large enough sample sizes).
In later sections, we also consider slight variations of neural collapse, requiring that some version of the CDNV converges to zero towards the end of the training.
N EURAL C OLLAPSE ON U NSEEN D ATA
In this section we theoretically analyze neural collapse in the setting of Section 2, with particular attention of variance collapse on data not used in the training of the classiﬁer.
First, in Section 4.
we show that if neural collapse happens on the training set ˜ S , then it generalizes to unseen samples from the same task under mild, natural assumptions.
Next we show, in Section 4.2, that one can also expect neural collapse to happen over new classes when the classes in the source and target tasks are selected randomly from the same class distributions.
The signiﬁcance of these results is that they show that the feature representations learned during training are immediately useful in other tasks; we show in Section 4.3 how this leads to low classiﬁcation error in few-shot learning.
4. G ENERALIZATION TO N EW S AMPLES FROM THE S AME C LASS
In this section, we provide a generalization bound on the CDNV, V f ( ˜ P i , ˜ P j ) , between two source class-conditional distributions ˜ P i and ˜ P j in terms of its empirical counterpart, V f ( ˜ S i , ˜ S j ) , and cer- tain generalization gap terms bounding the difference between expectations and empirical averages for f and its variants, where f is the output of the learning algorithm with access to ˜ S .
Assume that for all c ∈ [ l ] , ˜ S c is a set of m c i.i.d.
samples drawn from ˜ P c , and for any δ ∈ (0 , 1) , let (cid:15) 1 ( ˜ P c , m c , δ ) and (cid:15) {+
+} ( ˜ P c , m c , δ ) be the smallest positive values such that with probability at least 1 − δ , the learning algorithm returns a function f ∈ F that satisﬁes
[Equation]
respectively
[Equation] Typically, these quantities can be upper bounded using Rademacher complexities (Bartlett & Mendelson, 2002) related to F , scaling usually as O ( (cid:112) log(1 /δ ) /m c ) (for a ﬁxed c ), as we show in Proposition 3 (in Appendix E) for ReLU neural networks with bounded weights.
Next, we present our bound on the CDNV of the source distributions; the proof is relegated to Appendix C.
Proposition 1.
Fix two source classes, i and j with distributions ˜ P
i and ˜ P j , and let δ ∈ (0 , 1) .
Let ˜ S
c ∼ ˜ P
m c c for c ∈ { i, j } . Let [Equation] ) + (cid:15) j 1 ( δ/ 4) (cid:107) µ f ( ˜ P i ) − µ f ( ˜ P j ) (cid:107) and
B = Avg [Table] [Table] [Table]
[Table]
f [Table] [Table] Then, with probability at least 1 − δ over ˜ S , we have V f ( ˜ P i , ˜ P j ) ≤ ( V f ( ˜ S i , ˜ S j ) + B ) (cid:0) 1 + A
+ A 2 (cid:1) .
Writing out the bracket, the bound in the proposition has several terms.
The ﬁrst one is the empirical CDNV V f ( ˜ S i , ˜ S j ) between the two datasets ˜ S i ∼ ˜ P
m i i and ˜ S
j ∼ ˜ P
m j j .
This term is assumed to be small by the neural collapse phenomenon during training.
The rest of the terms are proportional to the generalization gaps (cid:15) c 1 ( δ/ 4) and (cid:15) c 2 ( δ/ 4) — as discussed above, typically we expect these terms to scale as O ( (cid:112) log(1 /δ ) /m
c ) .
In addition, according to the theoretical analyses of neural collapse available in the literature (Papyan et al., 2020; Mixon et al., 2020; Zhu et al., 2021; Rangamani et al., 2021), under certain conditions the function f converges to a solution for which { µ f ( ˜ S i ) } li =1 form a simplex equiangular tight frame (ETF), that is, after centering their global mean, µ f ( ˜ S i ) are of equal length, and (cid:107) µ f ( ˜ S i ) − µ f ( ˜ S j ) (cid:107) are also equal and maximized for all i (cid:54)
= j . This implies that if f is properly normalized, the distance (cid:107) µ f ( ˜ S i ) − µ f ( ˜ S j ) (cid:107) is lower bounded by a constant, and hence A and B indeed are small (in Appendix B.3 we demonstrate empirically that the minimum distances of the class means are indeed not too small in the scenarios we consider). Finally, we also note that with probability 1 − δ , we have (cid:107) µ f ( ˜ P i ) − µ f ( ˜ P j ) (cid:107) ≥ (cid:107) µ f ( ˜ S i ) − µ f ( ˜ S j ) (cid:107) − (cid:15) i 1 ( δ/ 2) − (cid:15)
j 1 ( δ/ 2) . Hence, if m i and m j are large enough, (cid:107) µ f ( ˜ P i ) − µ f ( ˜ P j ) (cid:107) is larger than a constant with high probability. Therefore, assuming we have a large enough training set for a pair of source classes i, j , if the CDNV V f ( ˜ S i , ˜ S
j ) is small, then we expect it to be small for new, unseen samples as well.
That is, if neural collapse emerges in the training data of two source classes, we should also expect it to emerge in unseen samples of the same classes.
4. N EURAL C OLLAPSE G ENERALIZES TO N EW C LASSES
Previously, we showed that if the CDNV is minimized by the learning algorithm on the training data, we expect it to be small for unseen source samples.
As a next step, we show that if neural collapse emerges in the set of source classes, we can also expect it to emerge in new, unseen target classes.
To analyze this scenario, we essentially treat class-conditional distributions as data points on which the feature map f is trained (in a noisy manner, depending on the actual samples), and apply standard techniques to derive generalization bounds to new data points, which, in this case, are classconditional distributions.
Accordingly, we assume that both the source and the target classes come from a distribution over D C over a set of classes C .
Each class is represented by its class-conditional distribution, and hence, with a slight abuse of notation, we say that the source class-conditional distributions ˜ P = { ˜ P i } li =1 are selected according to D C ( ˜ P 1 , . . .
, ˜ P l | ˜
P i (cid:54) =
˜
P j for all i (cid:54) =
j ∈ [ l ]) .
Below we show that if neural collapse emerges in the source classes, it also emerges in unseen source classes c (cid:54) =
c (cid:48) with class-conditional distributions P c , P c (cid:48) ∼ D C ( P c , P c (cid:48) | P c (cid:54) =
P c (cid:48) ) in the sense that V f ( P c , P c (cid:48) ) is expected to be small.
Our bound below applies a uniform convergence argument based on Rademacher complexities.
The Rademacher complexity of a set Y ⊂ R n is deﬁned as R ( Y ) =
E (cid:15) sup y ∈ Y [ (cid:104) (cid:15), y (cid:105) ] , where (cid:15) = ( (cid:15) 1 , . . .
, (cid:15) n ) ∼ U [ {± 1 } n ] .
For a given feature map f ∈ F , we also deﬁne a mapping H f : C → R p +1 as H f
( P c ) =
( µ f ( P c ) , Var f
( P c )) , and for a class of functions F ∗ , we deﬁne H F ∗ ( ˜ P ) =
{ ( H f ( ˜ P c )) lc
=1 : f ∈ F ∗ } .
Proposition 2.
Let F ∗ ⊂
F be any ﬁnite set of functions with ∆( F ∗ ) =
inf
[Equation]
= P
[Equation]
(cid:107) µ
[Equation]
( P c ) − µ f
( P c (cid:48) )
(cid:107) > 0 .
Then, with probability at least 1 − δ over the selection of source class distributions ˜ P ,
[Equation]
The above proposition, proved in Appendix D (based on Corollary 3 of Maurer & Pontil, 2019), provides an upper bound on the discrepancy between the expected value of V f ( P c , P c (cid:48) ) for two class-conditional distributions P c (cid:54)
=
P c (cid:48) and its averaged value across a set of source distributions ˜ P 1 , . . . ,
˜ P l that were sampled from D C .
In addition, we treat F ∗ as a set of candidate functions from which the learning algorithm selects its candidates.
Similarly to Corollary 3 of Maurer & Pontil (2019), this set is assumed to be ﬁnite only for technical (measurability) reasons.
[-To extend the proposition to the case where F ∗ is inﬁnite, we can simply replace sup f ∈F ∗ with sup F 0 ⊂F ∗ , |F 
-]
The proposition shows that if Avg i (cid:54) =
j [ V f ( ˜ P i , ˜ P j )] is small, we expect V f ( P c , P c (cid:48) ) to be small for two new classes c and c (cid:48) , if the source data is representative enough, as discussed below.
First note that by Proposition 1, each term V f ( ˜ P i , ˜ P j ) is expected to be small in the presence of neural collapse (for large enough training sets ˜ S c ), so their average Avg i (cid:54) = j [ V f ( ˜ P i , ˜ P j )] is also small.
According to
Proposition 4 in Appendix E, the Rademacher complexity in the bound scales as O ( p √ l ) for ReLU networks with bounded weights, so the corresponding term in the bound is O ( p/ ( √ l ∆( F ∗ ) 2 )) , which is of similar order as the last term (neglecting the supremum terms and the exact dependence on p for now). Hence, the bound tends to zero as the number of source classes l increases as long as ∆( F ∗ ) is not too small (in particular, is not zero).
While there is no way to guarantee this (e.g., if two class-conditional distributions are too close, ∆( F ∗ ) can be very small), we can say that if the feature maps f ∈ F ∗ keep the classes apart, the bound may become reasonably small.
If the feature dimension p is large and f ∈ F ∗ are similar to random maps, this assumption typically holds (e.g., if f is not a constant function, which can be expected with a proper training method).
Even when the number of classes is very large, the bound can be reasonable: if the corresponding feature means are more-or-less uniformly distributed in a p -dimensional unit cube for all f ∈ F ∗ , ∆( F ∗ )
=
Ω( √ p · |C| − 2 /p ) (see Lemma 2 in Appendix F), and hence combining with the dependence of the Rademacher complexity on l and p , and the fact that Var f ( P (cid:48) ) ≤ p and sup x ∈X , f ∈F ∗ (cid:107) f ( x ) (cid:107) ≤ √ p , the bound is O ( (cid:112) p/l · |C| 6 /p ) which tends to zero as l increases.
4. CDNV AND C LASSIFICATION E RROR
In this section we study the relationship between the value of the CDNV and the classiﬁcation performance it induces.
Consider a balanced k -class classiﬁcation problem with class distributions
P 1 , . . , P k , datasets S c ∼ P n c c , c ∈ [ k ] and n 1 = . . . = n k .
Let S = (cid:83) c ∈ [ k ] S c denote all the samples and consider the ’nearest empirical mean’ classiﬁer h S,f ( x ) =
arg min c ∈ [ k ] (cid:107) f
( x ) − µ f ( S c ) (cid:107) (note that h S,f is a classiﬁer with convex polytope cells on top of f , and it is a linear classiﬁer when k = 2 ).
Then, as it is shown in Proposition 5 in Appendix G (for γ = 0 . 5 ), the expected classiﬁcation error of h S,f is upper bounded by 16( k − 1)
V f
( P 1 , P 2 )(1 + 2 /n c ) and if f ◦ P 1 and f ◦ P 2 are spherically symmetric, then the classiﬁcation error is upper bounded by 16( k − 1)
V f
( P 1 , P 2 )(1 /p + 2 /n c ) . Therefore, in case of neural collapse (i.e., small V f ( P 1 , P 2 ) ), the error probability is small, explaining the success of foundation models in the low-data regime (such as in few-shot learning problems) in the presence of neural collapse.
Putting together Propositions 1 and 2, and Proposition 5 in Appendix G, it follows in a straightforward way that, with high probability (over the selection of the source, as well as the source samples), the expected error (over the target classes and target samples) of a nearest class-mean classiﬁer over
[Table]
the target classes can be bounded by the average neural collapse over the source training samples plus terms which converge to zero as the the number of source training samples m c , as well as the number of source classes l increase
[Table]
E XPERIMENTS
In this section we experimentally analyze the neural collapse phenomenon and how it generalizes to new data points and new classes.
We use reasonably good classiﬁers to demonstrate that, in addition to the neural collapse observed in training time by Papyan et al.
(2020), it is also observable on test data from the same classes, as well as on data from new classes, as predicted by our theoretical results.
We also show that, as expected intuitively, neural collapse is strongly correlated with accuracy in few-shot learning scenarios.
The experiments are conducted over multiple datasets and multiple architectures, providing strong empirical evidence that neural collapse provides a compelling explanation for the good performance of foundation models in few-shot learning tasks.
5. S ETUP
Method.
Following our theoretical setup, we ﬁrst train a neural network classiﬁer ˜ h
= ˜
g ◦ f on a source task.
Then we evaluate the few-shot performance of f on target classiﬁcation tasks by training a new classiﬁer
h = g ◦ f .
More speciﬁcally, ˜ h is trained by minimizing the cross-entropy loss between the logits of the network and the one-hot encodings of the labels.
The training is conducted using SGD with learning rate η and momentum 0 .
9 with batch size 64 .
Here, ˜ g is the top linear layer of the neural network and f is the mapping implemented by all other layers.
At the second stage, given a target few-shot classiﬁcation task with training data S = { ( x i , y i ) } ni =1 , we train a new top layer g as a solution of ridge regression acting on the dataset { ( f ( x i ) , y i ) } ni = with regularization λ n = α √ n .
Thus, g is a linear transformation with the weight matrix w S,f = ( f ( X ) (cid:62) f
( X ) + λ n I ) − 1 f ( X ) (cid:62) Y , where f ( X ) is the n × d data matrix for the ridge regression problem containing the feature vectors { f ( x i ) }
(i.e., f ( X ) (cid:62) =
[ f ( x 1 ) , . . .
, f ( x n )] ) and Y is the n × k label matrix (i.e., Y (cid:62) =
[ y 1 , . . .
, y n ] ) where X ∈ R n × d , Y ∈ R n × k and f θ ( X ) ∈ R n × p .
We did not apply any form of ﬁne-tuning for f at the second stage. In the experiments we sample 5-class classiﬁcation tasks randomly from the target dataset, with n c training samples for each class (thus, altogether n = 5 n c above), and measure the performance on 100 random test samples from each class.
We report the resulting accuracy rates averaged over 100 randomly chosen tasks.
Architectures and hyperparameters.
We experimented with two types of architectures for ˜ h : wide ResNets (Zagoruyko & Komodakis, 2016) and vanilla convolutional networks of the same structure without the residual connections.
The networks are denoted by WRN- N - M and Conv- NM , where N is the depth and M is the width factor.
We used the following default hyperparameters: η = 2 − 4 , batch size 64 and α = 1 in setting the ridge regression regularization parameter {+[Table] C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 
+}
C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 22 2 C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 0.
A cc u r a c y
WRN
[Figure] [Figure]
{+C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration3 22 1 22 1 
+}
{+C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 
+}
C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 0.
A cc u r a c y
[Figure] experiment we trained source λ n = α/ √ n . In the Figs. 2-3 in the appendix we provide experiments showing that the results are consistent for different η .
We also provide experiments with a standard learning-rate schedule.
See Appendix A for a complete description of the architectures. Datasets.
We consider four different datasets: (i) Mini-ImageNet (Vinyals et al., 2016); (ii) CIFAR-FS (Bertinetto et al., 2019); (iii) FC-100 (Oreshkin et al., 2018); and (iv) EMNIST (balanced) (Cohen et al., 2017). For a complete description of the datasets, see Appendix A.
Experimental results are reported averaged over 20 random initialization together with 95% conﬁdence intervals. 5. R ESULTS While the ridge regression method to train the few-shot classiﬁer g may seem simplistic, it provides reasonably good performance, and hence it is suitable for studying the success of recent transfer/transductive learning methods (Dhillon et al., 2020; Tian et al., 2020) for few-shot learning.
To demonstrate this, we compared the 1 and 5-shot performance of our simplistic method to several few-shot learning algorithms on Mini-ImageNet, CIFAR-FS and FC-100, summarized in Table 1.
On each dataset, we report the average performance of our method on epochs between 90 and 100.
As can be seen, the method we study in this paper is competitive with the rest of the literature on the three benchmarks, especially in the 1-shot case (even achieving the state of the art on FC-100). To improve the performance of our method a bit, we employed a a standard learning rate scheduling with initial learning rate η = 0 . 05 , decayed twice by a factor 0 .
1 after 30 epochs each (accuracy rates are reported averaging over epochs 90–100, as before).
Since the performance of these networks plateaued slightly after the ﬁrst learning rate decay on the source test data, we also applied model selection based on this information, and used the network from the ﬁrst 60 epochs (to avoid the overﬁtting to the source data and classes happening with the smallest learning rate) with the best source test performance.
The combination of these two variations typically resulted in a small improvement of a few percentage points in the problems considered, see the last line of Table 1.
As our main experiment, we validate the theoretical assessments we made in Section 4.
We argued that in the presence of neural collapse on the training classes, the trained feature map can be used for training a classiﬁer with a small number of samples on new unseen classes.
The argument asserts that if we observe neural collapse on a large set of source classes, then we expect to have neural collapse on new unseen classes as well, when assuming that the classes are selected in an i.i.d. manner.
In this section we demonstrate that neural collapse generalizes to new samples from the same classes, and also to new classes, and we show that it is correlated with good few-shot performance.
To validate the above, we trained classiﬁers ˜ h with a varying number of l randomly selected source classes.
For each run, we plot in Figure 1 the CDNV as a function of the epoch for the training and test datasets of the source classes and over the test samples of the target classes.
In addition, we plot the 5-shot accuracy of ridge regression using the learned feature map f .
Similar experiments with different numbers of target samples are reported in Figures 4 and 5 in the appendix.
As it can be seen in Figure 1, the value of the CDNV decreases over the course of training on the training and test datasets of the source classes, showing that neural collapse generalizes to new samples from the training classes.
Since the classiﬁcation tasks with fewer number of source classes are easier to learn, the CDNV tends to be larger when training with a wider set of source classes.
In contrast, we observe that when increasing the number of source classes, the presence of neural collapse in the target classes strengthens.
This is in alignment with our theoretical expectations (the more “training” classes, the better generalization to new classes), and the few-shot performance also consistently improves when the overall number of source classes is increased.
To validate the generality of our results, this phenomenon is demonstrated in several settings, e.g., using differnet network architectures and datasets in Figure 1.
As can be seen, the values of the CDNV on the target classes are relatively large compared to those on the source classes, except for the results on EMNIST.
However, these values still indicate [-[Table] 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 
 C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 2 2 C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 0.-]a reasonable few-shot learning performance, as demonstrated in the experiments.
[-WRN-28-4 on MiniImageNet 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration3 22 1 22 1 
-]
[-C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 22 2 22 0 22 2 
-]
[-C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 
-]
These results consistently validate our theoretical ﬁndings, that is, that neural collapse generalizes to new source samples, it emerges for new classes, and its presence immediately facilitates good performance in few-shot learning. In Appendix B.2, we also show that similar phenomena as described above happens for feature maps obtained from lower layers of the network, as well, although to a lesser extent.
C ONCLUSIONS
Employing foundation models for transfer learning is a successful approach for dealing with overﬁtting in the low-data regime.
However, the reasons for this success are not clear.
In this paper we presented a new perspective on this problem by connecting it to the newly discovered phenomenon of neural collapse.
We showed that the within-class variance collapse tends to emerge in the test data associated with the classes encountered at train time and, more importantly, in new unseen classes when the new classes are drawn from the same distribution as the training classes.
In addition, we showed that when neural collapse emerges in the new classes, then it requires very few samples to train a linear classiﬁer on top of the learned feature representation that accurately predicts the new classes.
These results provide a justiﬁcation to the recent successes of transfer learning in few-shot tasks, as observed by Tian et al. (2020) and Dhillon et al.
A E XPERIMENTAL D ETAILS
Datasets
Throughout the experiments, we consider four different datasets: (i) MiniImageNet (Vinyals et al., 2016) and (ii) CIFAR-FS (Bertinetto et al., 2019), (iii) FC-100 (Oreshkin et al., 2018) and EMNIST (balanced)
(Cohen et al., 2017).
Each dataset is split into meta-train, metavalidation and meta-test classes; we select the data for the source classes from the meta-training, and use similarly the meta-test data for the target tasks (we do not use the meta-validation classes).
Each one of the class splits is also partitioned into train and test samples; we use these for training and evaluating our models.
The Mini-ImageNet dataset contains 100 classes randomly chosen from ImageNet ILSVRC-2012 (Russakovsky et al., 2015) with 600 images of size 84 × 84 pixels per class.
It is split into 64 meta-training classes, 16 meta-validation classes and 20 classes for meta-testing.
CIFAR-FS and FC-100 are two derivatives of the CIFAR-100 dataset (Krizhevsky, 2012).
CIFARFS consists of a random split of the CIFAR-100 classes into 64 classes for meta-training, 14 for meta-validation and 20 for meta-testing.
FC-100 contains 100 classes which are grouped into 20 superclasses.
These classes are partitioned into 60 meta-train classes from 12 superclasses, 20 classes from 4 superclasses for meta-validation, and 20 meta-test classes from 4 superclasses.
We also consider the EMNIST dataset, which is an extension of the original MNIST dataset.
We randomly split its classes into 35 source classes and 12 target classes (which are: 2, 10, 11, 12, 13, 16, 18, 22, 25, 33, 34, 44).
We use random cropping augmentations at training time across all of the experiments.
Architectures. We experimented with two types of architectures for ˜ h : wide ResNets (Zagoruyko & Komodakis, 2016) and vanilla convolutional networks
Wide ResNets start with a convolutional layer (with kernels × and output channels , followed by three groups of layers
Each group of layers includes a convolutional layer , followed by a sequence of N residual layers.
Each residual layer in the i ’th group contains two convolutional layers using 3 × 3 kernels and output 2 i +3 M channels.
Each convolutional layer is followed by a ReLU activations and batch normalization post activations.
The network’s penultimate layer is a mean pooling activation with kernels of size 4 × 4 , returning an output of dimension 256 and followed by a linear layer.
The vanilla convolutional networks have the same architectures as the wide ResNets , except that we omit the residual connections.
The networks are denoted by WRN- N - M and Conv- N - M , respectively.
B A DDITIONAL E XPERIMENTS
In this section we report additional experiments to provide further insights.
B. V ARYING H YPERPARAMETERS
We start by presenting experimental results to validate the consistency of our ﬁndings when varying different hyperparameters. Varying learning rates.
We repeated the experiments in rows 1 and 4 of Figure 1 with learning rates η = 2 − 2 i − 2 for i = 1 , 2 , 3 , 4 .
We also report the train and test accuracy rates on the source data.
The results are reported in Figure 2 and Figure 3.
The results are consistent with the observations we had in Figure 1.
Interestingly, when using smaller learning rates, the CDNV over the source training and test data tends to be smaller, while the CDNV on the target classes tends to be larger.
We attribute this observation to overﬁtting to the source classes and indeed we observe that the few-shot performance is generally worse in those cases. Learning rate scheduling and varying number of target samples.
To further demonstrate the relationship between neural collapse and generalization to new classes, we experimented with the default learning rate and standard learning rate scheduling (also used by Tian et al.
(2020)), with varying number of target samples.
In this experiment, we trained WRN-28-4 using SGD with the default learning rate and learning rate scheduling, starting with learning rate η = 0 . with decay factor 0 . 1 which is applied every 30 epochs twice. For the default training setting, in Figure 4(a-c) we report the dynamics of the CDNV on the source training data, the source test data, and the target classes (resp.) and in Figure 5(d-g) we plot the 1, 5, 10 and 20-shot accuracy rates of the network during training time.
The results of learning rate scheduling are provided in Figure 5.
Similarly to Figure 1, as expected, we can observe that when increasing the number of source classes, the few-shot performance improves, while the CDNV on the target classes tends to decrease.
Also, in line with our theory, the CDNV on the target classes is negatively correlated with the few-shot performance, that is, better neural collapse yields better performance.
For example, in Figure 5(d-g) it is evident that the peak performance on all few-shot experiments for the case of 64 source classes is achieved around the minimal value of the CDNV on the target classes in Figure 5(c).
This is achieved around iteration 16 , 000 for CIFAR-FS and around iteration 20 , 000 for Mini-ImageNet, a little bit after the ﬁrst learning rate decay (at 15 , 000 and 18 , 000 steps, respectively).
As can be seen, the performance slightly decreases after the peak iteration, while the CDNV on the target classes slightly increases, when the training starts to overﬁt to the source data and the source classes.
This effect can be mitigated by selecting the ﬁnal network based on the performance on the source test data, as we show in Table 1.
B. N EURAL C OLLAPSE AND L OWER L AYERS
We conducted a comparison between the behaviour of the penultimate layer (the top embedding layer) and a lower layer in the network, as the feature layer.
We trained a WRN-28-4 on CIFARFS using the default hyperparameters as described in Section 2, but in this experiment we used the second-to-last embedding layer of the network as the feature layer for few-shot learning.
A comparison of the behavior of this layer and the top embedding layer (that we use everywhere else in the paper) is shown in Figure 6 in terms of the CDNV on the source training data, the source test data, and the new classes (the target data), as well as for the 5-shot 5-class classiﬁcation accuracy.
As can be seen, the few-shot performance of the top embedding layer is superior to the performance of the lower layer.
This is in agreement with the evidently smaller values of the CDNV given by the top embedding layer in comparison to the second-to-last embedding layer in all three cases (i.e., source train and test data and the target data).
Nevertheless, we can see that the neural collapse phenomenon and the associated good few-shot performance is preserved in this lower layer of the network, however, less pronounced.
B. D YNAMICS OF THE C LASS -E MBEDDING D ISTANCES
In Section 4.1 we argued that the generalization bound in Proposition 1 is meaningful when the minimal distances min i (cid:54) =
j (cid:107) µ f ( ˜ S i ) − µ f ( ˜ S j ) (cid:107) 2 (between the empirical class means) and min i (cid:54) =
j (cid:107) µ f ( ˜ P i ) − µ f ( ˜ P j ) (cid:107) 2 (between the true class means) are not too small.
Therefore, we empirically investigated their dynamics during training in our standard setting (WRN-28-4 with the default hyperparameters, see Section 2) on CIFAR-FS, considering a varying number l ∈ { 5 , 10 , 20 , 30 , 40 , 50 , 60 } of source classes and learning rates η ∈ { 2 − 2 i − 2 } 4 i =1 .
As can be seen in Figure 7, the values of min i (cid:54) =
j (cid:107) µ f ( ˜ S i ) − µ f ( ˜ S j ) (cid:107) 2 and min i (cid:54) =
j (cid:107) µ f ( ˜ P i ) − µ f ( ˜ P j ) (cid:107) 2 tend to increase during training.
For completeness, we also plotted the values of min i (cid:54) =
j (cid:107) µ
f ( P i ) − µ f ( P j ) (cid:107)for the target classes { P c } kc =1 ( k = 20 for CIFAR-FS).
Interestingly, min i (cid:54) =
j (cid:107) µ
f ( P i ) − µ f ( P j ) (cid:107)tends to grow with the number of source classes, showing improved generalization to new classes.
B. C LASS -C OVARIANCE N ORMALIZED V ARIANCE
In this section we consider the formal deﬁnition of Papyan et al.
(2020) for neural collapse (their ﬁrst deﬁnition), which uses a normalized variance deﬁnition somewhat different from CDNV we used in our analysis.
For completeness, we present some experiments which demonstrate that our ﬁndings based on CDNV also apply to their normalization.
To give the formal deﬁnition, consider a training set ˜ S
= (cid:83) lc
=1 ˜ S c = (cid:83) lc
=1 { (˜ x cj , c ) } m c
j =1 for an l -class classiﬁcation problem, where ˜ S c is a collection of m c samples from class c .
Assume that a neural network is trained to classify the samples in ˜ S , where after t steps of the training we obtain the classiﬁer ˜ h t = ˜
g
t ◦ f t , where, as before, ˜ g t denotes the linear map of the last layer of the network, and f t is the feature map. Class-Covariance Normalized Variance.
Papyan et al. (2020) deﬁne the class-covariance normalized variance (CCNV) to deﬁne neural collapse: For a given feature map f , let ˜ µ fc = µ f ( ˜ S c ) denote the mean feature value of class c , and let ˜ µ fG = Avg lc =1 [˜ µ fc ] denote the global feature mean.
The CCNV is deﬁned to be Tr (cid:16) Σ f W Σ f B † (cid:17) , where Σ f W and Σ f B are the intra- and [Table] [Table] N[-V  4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 
-]
[Table] [Table] C D N V [-4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 
-][Table]
N [-V  4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 
-][Table] [Table] [Table]
C D N V
[Table] [Table]
C D N V
[Table]
N V
[Table] [Table] [Figure]
A cc u r a c y[-
-]
[Figure]
inter-class covariance matrices Σ fW =
Avg [Table] [Table]
{+C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 22 3 22 1 22 1 
+}
{+C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 
+}
[Table]
{+C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 
+}
[Table]
C D N V
[Table]
{+A cc u r a c y 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 
+}
[Table]
[Table]
A cc u r a c y {+
+}
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
A cc u r a c y
[Table]
[-4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 22 3 22 1 22 1 
 C D N V  4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 22 3 22 
 C D N V  4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 
-][Table] [Table]
C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10
[Table] [Table]
C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10
[Table]
[Table]
[-A cc u r a c y 4 0 4 1 4 2 4 3 4 4 4 5 
-]C D N V
[Table]
[-7 4 8 4 9 4 10 Iteration 
-][Table]
C D N V
[Table] [Table]
N
[Table]
{+8 4 9 4 10 4 11 Iteration 
+} C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 
 C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration
C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 0.
A cc u r a c y 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 0. A cc u r a c y 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 0.
A cc u r a c y 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 0. A cc u r a c y
[Figure]
The i ’th column stands for the results when training with learning rate η = 2 − 2 i − 2 .
matrix A .
According to Papyan et al.
(2020), (their ﬁrst version of) neural collapse happens if
[Equation]
In Figures 10 and 11 we plot the CCNV as a function of training iterations on the source training data, source test data and target classes.
As can be seen, the value of the CCNV decreases on the train and test data of the source classes.
In addition, the CCNV on the target classes decreases when training with a larger set of source classes, which is due to better generalization.
In contrast, the CCNV on the source classes increases when training with a larger set of source classes, since it increases the complexity of the optimization problem.
These results are qualitatively very similar to the once presented for CDNV, showing that both deﬁnitions of neural collapse behaves essentially the same way.
The corresponding train/test/few-shot accuracy rates are provided in Figures 2-3.
In
[-4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 
 CC N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 
-]Figures 8-9 we provide the CDNV, CCNV and accuracy rates of training WRN-28-4 and Conv-28on Mini-ImageNet and CIFAR-FS (resp.) with a varying number of source classes. [Figure]
[-CC N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 
-][Table] [Figure] [Table] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[-CC N V 
-][Figure] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Figure] [Figure]
C D N V top layer
[Figure]
4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 {+Iteration 
 C D N V top layer middle layer+} 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 Iteration
C D N V top layer middle layer 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 Iteration [-
-]0. [Figure] A cc u r a c y [Figure]
[-CC N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 
-][Table] [Table] [Table] [Table]
[Table] [Table]
4 10 4 11 Iteration 222d i s t a n c e  4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 2222d i s t a n c e  4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 2222d i s t a n c e  4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 2222d i s t a n c e
[Figure] [Figure]
C D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 22 2 C {+D N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 22 1 22 1 22 3 
 CC N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 
 CC+} N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 
 CC N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 0.
A cc u r a c y 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 0.0 0. A cc u r a c y 4 0 4 1 4 2 [-
-]4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 0.45 0. A cc u r a c y [Figure]
[-C C N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 Iteration 
-][Table] [Figure] [Figure] [Figure]
[Figure]
N V [Figure] 504 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration {+22 2 22 0 22 2 
 CC N V+}
{+[Figure] 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 
 CC N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 
 CC+} N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration
CC N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 
 CC N V [Figure] [Table]
[Table]
{+4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 
+} CC N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration 
 CC N V 4 0 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 Iteration
CC N V
[Figure]
C P ROOF OF P ROPOSITION
Lemma 1.
Let ˜ P c be a class distribution, δ ∈ (0 , 1) and
˜ S c = { ˜ x cj } m c j =1 ∼ ˜ P m c c be a dataset of m c samples.
Let f be the output of the learning algorithm.
If [Equation] then we have
[Equation]
Proof.
Recall that
[Equation]
We would like to upper bound E
x ∼ ˜ P c
[ (cid:107) f
( x ) (cid:107) 2 ] in terms of Avg x ∈ ˜ S
c
[ (cid:107) f
( x ) (cid:107) 2 ] and to lower bound (cid:107) E x ∼ ˜ P c [ f ( x )] (cid:107) 2 using (cid:107) Avg x ∈ ˜ S c [ f ( x )] (cid:107) 2 . By (2),
[Equation]
By the reverse triangle inequality
[Equation]
In particular [Equation] By (2),
[Equation]
Hence,
[Equation]
which implies [Equation] Combining (5) and (6), we obtain
[Equation]
completing the proof.
Proposition 1.
Fix two source classes, i and j with distributions ˜ P i and ˜ P j , and let δ ∈ (0 , 1) .
Let ˜ S c ∼ ˜ P
m c c for c ∈ { i, j } .
Let A = (cid:15) i 1 ( δ/ 4) + (cid:15) j 1 ( δ/ 4) (cid:107) µ f ( ˜ P i ) − µ f ( ˜ P j ) (cid:107) and B = Avg c ∈{ i,j } [Equation]
+ [Equation] [Equation]
Proof. By deﬁnition and the union bound, with probability at least 1 − δ , the following inequalities hold simultaneously for all c ∈ { i, j } : [Equation]
In the rest of the proof, we assume that the above inequalities hold. Let ∆ = (cid:12)(cid:12)(cid:12) (cid:107) µ
[Equation]
( ˜ P i ) − µ [Equation] ( ˜ P j ) (cid:107) 2 − (cid:107) µ f ( ˜ S i ) − µ f ( ˜ S j ) (cid:107) 2 (cid:12)(cid:12)(cid:12) .
By simple algebraic manipulations,
[Equation]
By (7) and Lemma 1 we have
[Equation]
Multiplying both sides of (8) by Var f
( ˜ P c ) , and using (9), we obtain Combining the above inequality with (8), we obtain
[Equation]
Averaging (10) over c ∈ { i, j } gives [Equation] [Equation] By the triangle inequality and (7),
[Equation]
and by a symmetric argument,
[Equation] [Equation]
This and (7) imply ∆ = [Equation] = [Equation]
µ
[Equation]
µ
[Equation]
µ
[Equation] ˜ S i ) − µ [Equation] ˜ S j )
[Equation]
≤ (cid:16) (cid:15) i 1 ( δ/ 4) + (cid:15) j 1 ( δ/ 4) (cid:17) · (cid:16) (cid:107) µ f ( ˜ P i ) − µ f ( ˜ P j ) (cid:107) + (cid:15) i 1 ( δ/ 4) + (cid:15) j 1 ( δ/ 4) (cid:17) . [Table] [Equation] completing the proof
[Equation]
D P ROOF OF P ROPOSITIONS
Proposition 2.
Let F ∗ ⊂ F be any ﬁnite set of functions with ∆( F ∗ ) = inf [Equation]
µ [Equation] P c ) − µ f ( P c
[Equation] ) (cid:107) > 0 . Then, with probability at least 1 − δ over the selection of source class distributions ˜ P , [Equation]
To prove this proposition [Equation] we apply Theorem 2 of Maurer & Pontil (2019).
In order to do so
[Equation]
we need the following deﬁnitions (their Deﬁnition 1): Let U be any set and A : U l → R . For z = ( z 1 , . . . , z l ) , j ∈ { 1 , . . .
, l } and z (cid:48)
j ∈ U , we deﬁne the j -th partial difference operator on A as
[Equation]
In addition [Equation] the Lipschitzness of A is deﬁned as [Equation]
and the second-order Lipschitzness of A as [Equation] and the maximal difference
[Equation]
Now we are ready to proceed with the proof. Proof. Let U =
H F ∗ ( C ) , z = ( u i , v i ) li =1 ∈ U l , I a (cid:54) = b =
I [ a (cid:54) =
b ] and
[Equation]
Furthermore, assume that P = { P 1 , . . , P l } ∼ D C ( P 1 , . . . , P l | ∀ i (cid:54) = j ∈ [ l ] : P i (cid:54) =
P j ) is an independent and identical copy of ˜ P . Then, by Corollary 3 of Maurer & Pontil (2019) and the fact that the Gaussian complexity of any set A ⊂ R l is at most 2 (cid:112) log( l ) times its Rademacher complexity, with probability at least 1 − δ over the selection of ˜ P , [Equation]
Since ˜ P 1 , . . . , ˜ P l are sampled such that ˜ P i (cid:54) =
˜
P j (for all i (cid:54) = j ∈ [ l ] ) and ∆( F ∗ ) > 0 , we have I [ µ f ( ˜ P i ) (cid:54) =
µ f ( ˜ P j )]
= 1 for all [-of these derivations altogether to obtain the following inequality: [Equation] We recall that ( a + b ) 2 ≤ 2 a 
 + 2 b 2 .-]i (cid:54) =
j , and so
[Equation]
Similarly, we have A (( H f ( P )) =
Avg i (cid:54) =
j [ V f ( P i , P j )] , and by the linearity of expectation, E
P [ A ( H f ( P ))]
=
E P c (cid:54) =
P c (cid:48) [ V f ( P c , P c (cid:48) )] .
By substituting these into (12), we obtain that for any f ∈ F ∗ , we have
[Equation]
To ﬁnish the proof, we bound M ( A ) , J ( A ) and K ( A ) .
Let j ∈ [ l ] be some index and z (cid:48) =
( z i ) li = = ( u (cid:48) i , v (cid:48) i ) li =
1 ∈
U l be a vector, such that, z (cid:48) i = z i for all i (cid:54) =
j and z (cid:48) j (cid:54) = z j .
Then,
[Equation]
since only pairs involving j are non-zero in the average deﬁning A . To simplify notation, let a = (cid:107) u i − u j (cid:107) and b = (cid:107) u
i − u (cid:48)
j (cid:107) and c = (cid:107) z
j − z (cid:48) j (cid:107) = (cid:107) ( u j , v j ) − ( u (cid:48)
j , v (cid:48) j ) (cid:107) .
Next we bound the terms in the sum in (14) individually. We divide the analysis into three cases: Case 1. Assume u
i =
u j = u (cid:48)
j .
In this case we simply have
[Equation]
Case 2.
Assume u i (cid:54) =
u j and u i = u (cid:48)
j (a bound can be obtained similarly for u i =
u j and u i (cid:54) = u (cid:48) j ).
In this case, since z i , z j ∈ U , we have v i , v j ≤ sup f ∈F ∗ ,P (cid:48)
∈C Var f ( P (cid:48) ) , implying (cid:12)(cid:12)(cid:12) (cid:12) (cid:12) I u i (cid:54) =
u
j v i + v j (cid:107) u i − u j (cid:107) 2 − I u
i (cid:54) = u
[Equation]
j v i
[Equation]
In addition
[Equation]
since c ≥ (cid:107) u i − u j (cid:107) ≥ ∆( F ∗ )
[Equation]
we have
[Equation]
Case 3.
Assume
(cid:54) = u j and u i (cid:54) = u (cid:48) j . By simple algebraic manipulations, we have
[Equation]
where the last inequality follows from the triangle inequality. Since (cid:107) u [Equation] u j (cid:107) + (cid:107) u (cid:48)
j (cid:107) ≤ 2 sup x ∈X , f ∈F ∗ (cid:107) f
( x ) (cid:107) and v i , v j ≤ sup f ∈F ∗ , P (cid:48)
∈C Var f ( P (cid:48) ) , we have
[Equation]
v i +
[Equation]
≤
[Equation]
We notice that (cid:107) u
j − u (cid:48)
j (cid:107) ≤ c and | v j − v (cid:48)
j | ≤ c .
Therefore, we obtain
[Equation]
Therefore
[Equation]
in each case we have
[Equation]
and also
[Equation]
Hence,
[Equation]
and
[Equation] [Equation] Therefore, K ( A ) and M ( A ) are also bounded by the right hand sides of (17) and (18). Next, we upper bound J ( A ) .
Let r (cid:54) = j and z (cid:48) r ∈ U . We have [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] c ≤
[Equation]
[Table] [Equation] [Table] c
[Table]
+ l ( l − 1) · (cid:12)
[Table]
c
[Table]
Hence, by (18) we have
[Equation]
Substituting the bounds of M ( A ) , K ( A ) and J ( A ) into (13) proves the proposition.
E A NALYSIS FOR S ECTIONS 4.1-4.
In this section we analyze the asymptotic behaviours of (cid:15) c 1 ( δ ) , (cid:15) c
2 ( δ ) (see Section 4.1) and
E ˜ P
[ R ( H F ∗ ( ˜ P ))]
(see Proposition 2) for ReLU neural networks.
Namely, a ReLU neural network is a function of the form f ( x ) =
W q σ ( W q − 1 . . . σ ( W 1 x )) : R d → R p , where σ is the element-wise ReLU function, max(0 , x ) , and W i ∈
R d i +1 ×
d i , where d 1 = d and d q +1 = p .
We deﬁne the spectral complexity of a network f to be C ( f ) = max j ∈
[ p ] (cid:107) W qj (cid:107) · (cid:81) q − 1 r =1 (cid:107) W r (cid:107) .
This quantity upper bounds the Lipschitz constant of f and is similar in fashion to other notions of spectral complexity of neural networks (Golowich et al., 2017; Bartlett et al., 2017).
Throughout the section, for a given function g : A → B k and j ∈ [ k ] , we denote the j ’th coordinate of g ( x ) by g ( x ) j and for a class G ⊂ { g : A → B k } , we denote G | j = { g ( · ) j : g ∈ G} .
The following proposition shows the ﬁrst and second moments of ReLU neural networks with bounded spectral norms generalize.
In particular, if the learning algorithm discussed in 4.1 always returns a neural network f with C ( f ) ≤ M , then, (cid:15) c 1 ( δ ) and (cid:15)
c
2 ( δ ) are given by the right hand sides of (19) and (20) respectively (when replacing C ( f ) with s ).
In particular, these terms scale as
O ( (cid:112) log(1 /δ ) /m
c ) .
The analysis is based on Theorem 1 in (Golowich et al., 2017) and the proof of
Theorem 1.1 in (Bartlett et al., 2017).
Proposition 3.
Let ˜ P
= { ˜ P c } lc =1 be a set of class-conditional distributions over a bounded set X ⊂ R d and let m 1 , . . .
, m l ∈ N .
Let F be a class of ReLU neural networks W q σ ( W q − 1 . . .
σ ( W 1 x )) :
R d → R p .
Then, with probability at least 1 − δ over the selection of ˜ S 1 ∼ ˜ P m 1 1 , . .
˜ S l ∼ ˜ P m l l , for all c ∈ [ l ] and f ∈ F , we have
[Equation]
and
[Equation]
Proof.
Let M ∈ N . We deﬁne [Equation] Let ˜ P c be a class-conditional distribution and
˜ S c = { ˜ x ci } m c i =1 ∼ ˜
P m c c for some c ∈ [ l ] .
Since √ a + b ≤ √ a + √ b holds for any pair a, b ≥ 0 , for any f ∈ F
[Equation]
we have
[Equation]
By (cf.
Mohri et al.
(2018), Theorem 3.3), with probability at least 1 − δ 2 plM ( M +1) over the selection of ˜ S c ∼ ˜
P m c c , we have
[Equation]
By Theorem 1 in (Golowich et al., 2017)
[Equation]
In addition
[Equation]
the functions f ( x ) j are bounded in the following manner | f
[Equation]
x
[Equation]
Substituting these inequalities in (22) implies
[Equation]
By union bound (24) holds for all p with probability at least 1 −
[Equation]
2 lM ( M +1) .
By combining (24) and (21), with probability at least 1 −
[Equation]
lM ( M +1) , for any f ∈ F M , we have (cid:13)(cid:13)(cid:13) E
[Equation]
x
[Equation]
c [ f
[Equation]
˜ x )]
[Equation]
c [ f
[Equation]
x )] [Equation] Next, we prove the second inequality. By (cf.
Mohri et al. (2018), Theorem 3.3), for each c ∈ [ l ] and M ∈ N , with probability at least 1 − δ 2 lM ( M +1) over the selection of ˜ S c , for any f ∈ F M , we have
[Equation]
where G s = {(cid:107) f ( · ) (cid:107) 2 | f ∈ F M } .
Recall that for any pair of functions f, g : A → R , we have sup
a ∈ A [ f ( a ) + g ( a )] ≤ sup a ∈ A f ( a ) + sup a ∈ A g ( a ) .
Hence,
[Equation]
where ˜ S c = { ˜ x ci } m c i =1 and (cid:15) = ( (cid:15) 1 , . . . , (cid:15) m c ) are i.i.d.
Rademacher random variables. We note that for any f ∈ F and j ∈ [ p ] , we have sup x ∈X | f ( x ) j | ≤ M · sup x ∈X (cid:107) x (cid:107) , while the function u 2 is 2 a -Lipschitz over [ − a, a ] . Therefore, by Lemma 1.1 in Lecture 17 in (Kakade & Tewari, 2008),
[Equation] (28) where the last inequality follows from Theorem 1 in (Golowich et al., 2017).
In addition, we note that [Equation] Substituting (28) and (29) in (26) implies that with probability at least 1 − δ 2 lM ( M +1) over the selection of ˜ S c , for any f ∈ F M , we have [Equation]
Finally, by union bound [Equation] with probability at least 1 − δ
[Equation]
the inequalities in (25) and (30) hold uniformly for all c ∈ [ l ] and f ∈ ∪ M ∈ N F M . Next [Equation] we prove the bounds for the networks in F .
Recall that the ReLU function satisﬁes max(0 , ax ) = a max(0 , x ) for all x ∈ R and a ≥ 0 . Therefore, any neural network f = W q σ ( W q − 1 . . .
σ ( W 1 x )) ∈ F can be represented as a modiﬁed network V q σ ( V q − 1 . . . σ ( V 1 x )) , where V q = W q · (cid:81) q − 1 r =1 (cid:107) W r (cid:107) and ∀ r ≤ q − 1 : V r = W r (cid:107) W r (cid:107) . In particular, V q = W qj · (cid:81) q − 1 r =1 (cid:107) W r (cid:107) and (cid:107) V qj (cid:107) = (cid:107) W qj (cid:107)· (cid:81) q − 1 r =1 (cid:107) W r (cid:107) .
Let M ∈ N be the minimal number, such that, C ( f (cid:48) ) ≤ M . Due to the minimality of M , we have M ≤ 1+ C ( f (cid:48) ) . In addition, C ( f (cid:48) )
= max j ∈ [ p ] (cid:107) V qj (cid:107)· (cid:81) q − 1 r =1 (cid:107) V r (cid:107) = max j ∈ [ p ] (cid:107) V qj (cid:107) = max j ∈ [ p ] (cid:107) W qj (cid:107) · (cid:81) q − 1 r =1 (cid:107) W r (cid:107) =
C ( f ) .
By applying inequalities (25) and (30) for f (cid:48) and M , while considering that f and f (cid:48) are equivalent and M ≤ 1 + C ( f ) , we obtain the desired inequalities.
In the following proposition we show that the Rademacher complexity in Proposition 2 scales as O ( √ l ) for ReLU neural networks with bounded spectral complexities. Proposition 4.
Let F ∗ be a class of ReLU neural networks of the form W q σ ( W q − 1 . . .
σ ( W 1 x )) , such that C ( f ) ≤ M .
Then,
[Equation] Proof.
As discussed in the proof of Proposition 3, any function f ∈ F ∗ is equivalent to a function in the following set
[Equation]
Therefore, we have E ˜ P [ R ( H F ∗ ( ˜ P ))] ≤ E ˜ P [ R ( H F M ( ˜ P ))] . We spell out the deﬁnition of
E ˜ P [ R ( H F M ( ˜ P ))]
[Equation]
where (cid:15) aggregates the i.i.d. Rademacher random variables (cid:15) cj for c ∈ [ l ] and j ∈ { 0 } ∪ [ p ] .
Next, we use the fact that for any pair of functions f, g : A → R , we have sup a ∈ A [ f ( a ) + g ( a )] ≤ sup a ∈ A f ( a ) + sup a ∈ A g ( a ) and the linearity of expectation to obtain
[Equation]
We analyze each term separately. Recall that for any given function f : A × B → R along with a distribution P over B , we have sup a ∈ A E b ∼ P [ f ( a, b )] ≤ E b ∼ P [sup a ∈ A f ( a, b )] .
Hence, [Equation] To analyze the second term
[Equation]
we ﬁrst use sup a ∈ A [ g ( a ) + f ( a )] ≤ sup a ∈ A g ( a ) + sup a ∈ A f ( a ) and the linearity of expectations
[Equation]
where the last equation follows from the fact that (cid:15) c 0 are distributed symmetrically around 0 . Again by sup a ∈ A E b ∼ P [ f ( a, b )] ≤ E b ∼ P [sup a ∈ A f ( a, b )]
[Equation]
we have
[Equation]
We note that the functions f ( x ) j satisfy | f ( x ) j | ≤ M · sup x ∈X (cid:107) x (cid:107) and u (cid:55)→ u 2 is 2 a -Lipschitz over [ − a, a ] . Therefore, by Lemma 1.1 in Lecture 17 of (Kakade & Tewari, 2008),
[Equation]
Similarly, [Equation]
We note that
[Equation] [Equation]
Therefore, again by Lemma 1.1 in Lecture 17 of (Kakade & Tewari, 2008)
[Equation]
We would like to bound R ( F M | j ( { ˜ x c } lc =1 )) for all { ˜ x c } lc =1 and j ∈ [ p ] .
We recall the deﬁnition of
Rademacher complexities.
By Theorem 1 in (Golowich et al., 2017) 3 ,
[Equation]
Substituting (38) and (39) in (36) implies
[Equation]
In addition
[Equation]
by substituting (39) in (35) implies
[Equation]
Finally
[Equation]
by substituting (39) in (32)
[Equation]
By combining these inequalities, together with (31) and (33) imply the desired inequality.
F A NALYSIS FOR S ECTION 4.
Lemma 2.
Let X 1 , . . .
, X n be a set of i.i.d.
uniformly distributed points in the p -dimensional unit cube.
There is a positive constant C > 0 (independent of n and p ), such that,
[Equation]
Proof. For any pair i (cid:54) = j ∈ [ n ] , [Equation]
since the volume of the p -dimensional ball of radius D is
[Equation]
p and the volume of the unit cube is 1 .
Hence, by the union bound over all pairs i (cid:54) =
j ∈ [ n ] :
[Equation]
Since h ( n, D ) > 0 if and only if D < D ∗ =
M − 1 /p , where M = n ( n − 1)
[Equation]
) , we obtain
[Equation]
where the last inequality follows from Ramanujan’s approximations of the Gamma function (Karatsuba, 2001) for some C > 0 independent of n, p .
G A NALYSIS FOR S ECTION 4.
In the following proposition we provide a formal statement of the analysis provided in Section 4.3.
We consider a balanced k -class classiﬁcation problem and a given feature map f
(e.g., pretrained).
We upper bound the classiﬁcation error of the nearest-mean classiﬁer h f,S ( x ) =
arg min c ∈ [ k ] (cid:107) f
( x ) − µ f ( S c ) (cid:107) in terms of the averaged CDNV, that is, Avg i (cid:54) =
j [ V f ( P i , P j )] .
Therefore, if the averaged CDNV is small, then, we expect the nearest-mean classiﬁer to have a small classiﬁcation error.
The upper bound in Section 4.3 is obtained when choosing γ = 0 .
Proposition 5.
Let T be a balanced k -class classiﬁcation problem with distribution P along with class-conditional distributions P = { P c } kc =1 .
Let S =
(cid:83) kc =1 S c be a dataset, such that S c ∼
P n c c ( n 1 = · · · = n k = n c ∈ N ).
Let f : R d → R p be any feature map and let h f,S ( x ) =
arg min c ∈ [ k ] (cid:107) f
( x ) − µ f ( S c ) (cid:107) be the nearest-mean classiﬁer.
Then, for all γ ∈ (0 , 1) , we have
[Equation]
where s ( f, P ) = p if { f ◦ P c } lc =1 are spherically symmetric and s ( f, P )
= 1 otherwise.
Proof.
Let µ i = µ f ( P i ) , ˆ µ i = µ f ( S i ) and x i ∼ P i , u i = f
( x i ) and α ij = (cid:107) µ i − µ j (cid:107) . By union bound
[Equation]
[Equation]
Again by union bound
[Equation] Let u (cid:48)
i be the projection of u i onto the line connecting between µ i and µ j .
We notice that [Equation]
where the last inequality follows from Markov’s inequality.
We note that E [ (cid:107) u (cid:48)
i − µ i (cid:107) 2 ] ≤ Var f
( P i ) .
Next, by Markov’s inequality,
[Equation]
where the last equation follows from the fact that ˆ µ i is the average of n c i.i.d.
random variables with variance Var f ( P i ) . Substituting these inequalities in (44) implies
[Equation]
Hence, by substituting within (43) and taking expectation over i , we obtain the desired inequality
[Equation]
To obtain the second inequality, we note that E [ (cid:107) u [Equation] i − µ i
[Equation]
2 ] = Var f ( P i ) /p when the distributions f ◦ P i is spherically symmetric.