Knowledge Distillation: Bad Models Can Be Good Role Models
Abstract
Large neural networks trained in the overparameterized regime are able to ﬁt noise to zero train error.
Recent work of Nakkiran and Bansal [20] has empirically observed that such networks behave as “conditional samplers” from the noisy distribution.
That is, they replicate the noise in the train data to unseen examples.
We give a theoretical framework for studying this conditional sampling behavior in the context of learning theory.
We relate the notion of such samplers to knowledge distillation, where a student network imitates the outputs of a teacher on unlabeled data.
We show that samplers, while being bad classiﬁers, can be good teachers.
Concretely, we prove that distillation from samplers is guaranteed to produce a student which approximates the Bayes optimal classiﬁer.
Finally, we show that some common learning algorithms (e.g., Nearest-Neighbours and Kernel Machines) can often generate samplers when applied in the overparameterized regime.
Introduction
Recently, the ﬁeld of supervised learning has witnessed the success of overparameterized methods: models, such as deep neural networks, which are large enough to ﬁt their train sets but still achieve good test performance.
A core theoretical concern is to understand why such models are able to ﬁt even noisy training data without catastrophically overﬁtting [31] despite no explicit regularization.
The seminal work of Bartlett et al.
[2] proposed the theoretical framework of benign overﬁtting to capture this empirical behavior.
Brieﬂy, benign overﬁtting studies statistically consistent methods— where models approach the Bayes optimal classiﬁer, even in presence of noise.
However, recent empirical work shows that when training deep neural networks on noisy data, overﬁtting is neither catastrophic nor benign [20].
Speciﬁcally, they propose that overﬁtting leads not to a good classiﬁer, but to a good conditional sampler .
For example, suppose we train a model on a set of images sampled from some distribution D , where 20% of the images of cats are wrongly labeled as dogs.
We now train an overparameterized network to ﬁt samples from D .
Note that for the distribution D , the Bayes-optimal classiﬁer, namely f ⇤ D ( x ) := argmax y P D ( y | x ) , returns the “correct” class of every image.
We can hope that if overﬁtting is truly “benign” in the sense of [2], the overparameterized model will be close to this optimal f ⇤ D .
However, this is not what occurs in practice: as [20] point out, the trained model f reproduces noise in the training set at test time, labeling up to 20% of the cats in the test data as dogs.
In a sense, the trained model f behaves as a conditional sample: f ( x ) ⇠ P D ( y | x ) (see the leftmost confusion matrix in Figure 1).
The above example indicates that thinking about classiﬁers in the overparameterized regime as approximating the Bayes optimal predictor might be misleading.
Therefore, it is essential to develop 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
the appropriate theoretical framework for describing the behavior of samplers from the conditional distribution.
While the learning-theoretic aspects of supervised classiﬁcation are well-studied, the theory of supervised conditional sampling has, to the best of our knowledge, not been systematically explored.
In this work, we take the ﬁrst steps towards addressing this gap.
We initiate the study of conditional sampling as a learning problem, and explore its relations to other kinds of learning.
Interestingly, we relate the notion of samplers to knowledge distillation methods.
In short, knowledge distillation is the process of training a teacher network on a small labeled dataset and using its predictions to label a large unlabeled dataset, on which a student network is trained to imitate the output of the teacher.
We ﬁnd that taking an ensemble of samplers as a teacher for knowledge distillation produces a student network with minimal error with respect to the Bayes optimal classiﬁer.
Finally, our theory leads to a new algorithm for knowledge distillation, where we randomly choose a teacher from a ﬁxed pool, to label each example, which accelerates the training process in practice 1 .
We show that this new algorithm is guaranteed to ﬁnd a student with low error.
1. Our Contributions
Learning Theory of Conditional Samplers.
(Section 2)
We initiate the study of conditional sampling in the context of computational learning theory.
We formally introduce the problem of conditional-sampling, and deﬁne the sample complexity of learning a sampler.
We then present positive and negative results in this new setting.
For example, we show that there exist distributions where sampling is much easier than classiﬁcation , requiring far fewer samples.
However, if we allow polynomial blowup in sample-size and runtime, a sampler can be “boosted” to a good classiﬁer, showing that if ﬁnding a classiﬁer is computationally hard then sampling is also hard.
Theory of Knowledge Distillation.
(Section 3) One way to boost a sampler into a good classiﬁer is to run an ensemble of samplers at inference time (see middle panel of Figure 1), which is costly.
We show that by performing distillation from an ensemble of teachers, it is possible to ﬁnd a student with low error w.r.t.
the Bayes optimal.
This shows that teacher networks in ensemble-distillation need not be good classiﬁers, but just good samplers.
Finally we propose a new algorithm for distillation, where each example is labeled by a random teacher from a ﬁxed pool (see right panel of Figure 1).
We study quantitative bounds on the sample complexity of teaching and learning in our setting, for both ensemble-distillation and distillation from a random teacher.
Theory of Sampler Algorithms.
(Section 4)
We show that several classical learning algorithms provably produce good conditional samplers, and analyze their sample complexity in terms of standard problem parameters (e.g., distributional smoothness).
Speciﬁcally we show that the 1Nearest-Neighbour algorithm is a sampler, and extend our analysis to k-Nearest-Neighbour as well.
Furthermore, we show that under some distributional assumptions, Lipschitz classes such as linear methods, kernels and neural networks may also behave like samplers.
1. Related Work
Knowledge Distillation Knowledge Distillation for deep learning was proposed in [14].
Since then, a large body of work showed its practical beneﬁts for various machine learning tasks [29, 23, 30, 6, 10, 27].
Nonetheless, from a theoretical perspective, understanding why and when distillation works remains a mystery.
Hinton et al.
[14], attribute the success of distillation to the fact that the soft labels of the teacher passing additional information on the input.
In a recent paper, Menon et al.
[18], claim that when the teacher approximates the Bayes class-probabilities, distillation is possible.
Our results show that even when the teachers are far from the Bayes optimal prediction (i.e., when teachers are noisy samplers from the distribution), we can ﬁnd a student with low levels of noise through distillation.
Another work by Wei et al.
[28], shows that, assuming the data distribution has good continuity within each class, self-distillation is possible.
However, it is not clear when such assumption is actually satisﬁed.
The work of Lopez-Paz et al.
[17], relates the notion of privileged information to knowledge distillation.
A work by Frei et al.
[8], shows that self-training can boost weak learners, when the target is a linear classiﬁer over a mixture model distribution.
Other works [19, 32, 7] study distillation as a regularization method, forcing the student to learn under a “smoother” loss landscape.
Learning with Noise Learning under corrupted labels is a well-studied research area in the literature (e.g., [1, 9]).
A notable examples are the works of Blum et al., and Kearns et al.
[5, 15], showing how statistical query algorithms can be leveraged to learn under noisy labels.
A growing number of works study the effects of noise on deep learning, as well as methods to learn under label noise (see [26] for a survey).
However, most of these works do not leverage distillation for label noise robustness, as we suggest in our work.
More related to our work is [12], showing that an iterative process of training and re-labeling can combat label noise.
However, this work focuses on empirical study of distillation.
Sampling as a Learning Problem
Let X be the input space and Y = {± 1 } be the label space (for simplicity, under appropriate assumptions we can extend most of our results to multiclass).
A hypothesis class
H is some class of functions from
X to Y .
A learning algorithm
A takes a sequence of m samples S 2 ( X ⇥ Y ) m and outputs some hypothesis h :
X !
Y .
We denote by A ( S ) the hypothesis that A outputs when observing the sample S .
For some distribution
D over X ⇥ Y , we denote its X marginal as D X and denote the Bayes optimal classiﬁer by f ⇤
[Equation]
When D is a distribution where the label is not a deterministic function of the input, we think of D as a noisy version of some clean distribution D ⇤ , where each input is correctly labeled.
Naturally, we assume that the probability of seeing the right label is greater than seeing a wrong one.
In other words, D ⇤ has the same marginal distribution over X (i.e., D X = D ⇤ X ), and is labeled by the Bayes optimal classiﬁer of
D .
That is, sampling ( x , y )
⇠ D ⇤ is given by x ⇠ D X and y =
f ⇤ D ( x ) .
We denote the “noise” of the distribution D by ⌘ ( D ) :
= P ( x ,y )
⇠ D [ y 6 = f ⇤ D ( x )] .
We also consider the margin of the distribution, which deﬁnes the difference in probability between correct and wrong label for each example.
Namely, for some   0 , let   ( D ) be the supremum over  > 0 s.t.,
[Equation]
Speciﬁcally, we denote  ( D ) :
=  0 ( D ) .
We typically assume that the input distribution has a strictly positive margin  ( D ) > 0 , so for every example the probability to see the correct label is greater by  than the probability to see a wrong label.
Our objective in this setting is to approximate the Bayes optimal classiﬁer of D .
In other words, we want to minimize the 0 - 1 loss on the clean distribution D ⇤ :
[Equation]
So, the learning algorithm has access to samples from the noisy distribution D , but needs to achieve good loss on the clean distribution D ⇤ .
We deﬁne a learner in this setting to be an algorithm that minimizes (1) using a ﬁnite number of samples:
Deﬁnition 1.
For some learning algorithm A and some distribution D over X ⇥ Y , we say that A is a learner for
D if there exists a function m : (0 , 1) !
N s.t.
for every ✏ 2 (0 , 1) , taking m  m ( ✏ ) we get,
E S ⇠ D m
L D ⇤ ( A ( S ))  ✏ .
In this case, we call m ( · ) the sample complexity of A w.r.t.
D .
Additionally, for some class P of distributions over X ⇥ Y , we say that A is a learner for P if there is some m ( · ) s.t.
A is a learner with sample complexity m ( · ) for every D 2 P .
Note that this deﬁnition of learner is similar to the notion of asymptotically consistent estimator.
However, our deﬁnition explicitly accounts for the sample complexity.
To give a concrete example, let us consider agnostic learning using the ERM rule, namely: ERM H ( S ) =
arg min h 2 H L S ( h ) .
For some hypothesis class H and some margin  > 0 , let P ( H ,  ) be the class of distributions such that for every D 2 P ( H ,  ) we have  ( D )   and f ⇤ D 2 H .
Namely, P ( H ,  ) is the class of distributions with margin  for which the Bayes optimal classiﬁer comes from
H .
The following theorem states that ﬁnite VC-dimension and a non-zero margin form a sufﬁcient condition for learnability with noise.
Theorem 2.
There exists a constant C > 0 s.t.
for every hypothesis class H with VC( H ) < 1 ,
ERM H is a learner for P ( H ,  ) with sample complexity m ( ✏ ) =
C VC( H )+log(1 / ✏ ) ✏ 2 .
The proof of Theorem 2 is given in the appendix.
The main idea is the following lemma (whose proof is also in the appendix), which shows that the margin assumption connects a small relative error w.r.t.
D to a small absolute error w.r.t.
D ⇤ .
Lemma 3.
Fix some distribution D , and let h ⇤ be the Bayes optimal classiﬁer for D .
Assume that   ( D ) > 0 .
Then, for every h such that L D ( h )  L D ( h ⇤ )
+ ✏ it holds that L D ⇤ ( h )  ✏   ( D ) +  .
Combining this lemma with the Fundamental Theorem of Learning Theory (see [25]) yields Theorem 2.
From this theorem, we see that given more samples than the VC -dimension (often corresponding to the number of parameters), learning is possible.
However, complex classiﬁers with large VC -dimension such as neural networks are often trained in the overparameterized regime, when the number of parameters exceeds the number of samples.
In this regime, the bound of Theorembecomes vacuous.
That said, this does not rule out the possibility of distribution-dependent sample complexity bounds using other measures of complexity (e.g., Rademacher complexity).
2. Samplers
As previously mentioned, neural networks trained on noisy data replicate the noise to unseen samples as well, behaving like samplers from the noisy distribution.
We next deﬁne formally a sampler for some distribution D , giving a similar deﬁnition as in [20].
For some learning algorithm A , some number m 2 N and some distribution D over X ⇥ Y , deﬁne the distribution
A ( D m ) over X ⇥ Y , where ( x , y )
⇠ A ( D m ) is given by sampling S ⇠ D m , sampling x ⇠ D X and setting y =
A ( S )( x ) .
Namely, A ( D m ) is the distribution given by (re)-labeling D using a hypothesis generated by
A when observing a random sample of size m .
Using these notations, we deﬁne a sampler algorithm for the distribution D as follows:
Deﬁnition 4.
For some learning algorithm A and some distribution D over X ⇥ Y , we say that A is a sampler for
D if there exists [Equation]
for every ✏ 2 (0 , 1) , taking m  e m ( ✏ [Equation] we get,
[Equation] where TV is the Total Variation Distance.
Then, we call e m the sample complexity of A w.r.t.
D .
Additionally, for some class
P of distributions over X ⇥ Y , we say that A is a sampler for P if there exists e m [Equation] [Equation] s.t.
A is a sampler with sample complexity
[Equation]
So, a sampler for D is an algorithm that generates a distribution similar to D (the noisy input distribution) when labeling new examples.
A primary example for a sampler is the 1-NearestNeighbour algorithm: since 1-NN outputs the (possibly corrupted) label of the closest neighbour, its prediction behaves like sampling from P D ( y | x ) (see Section 4).
Figure 1 shows that neural networks behave similarly to samplers when trained on a noisy version of the CIFAR-10 dataset.
Given the above deﬁnition, it can be easily shown that, instead of approximating the Bayes optimal prediction, a sampler preserves the noise rate of the original distribution.
Lemma 5.
Let A be a sampler D with sample complexity ˜ m .
Then, for m  ˜ m ( " ) ,
[Equation]
While a sampler for D is not a good learner (in the sense of Deﬁnition 1), it does have some favorable properties.
Primarily, it can take signiﬁcantly fewer samples to get a sampler than it would take to get a learner, hence making the study of samplers more suitable for the overparameterized regime.
In fact, in the extreme case one could get a sampler using only a single example from the distribution, while getting a learner for the same distribution would require an arbitrarily large number of examples.
Indeed, ﬁx b 2 {± 1 } and  2 (0 , 1) , and let D b be the distributions concentrated on a single example x 2 X , with label y 2 {± 1 } s.t. P D b ( y = 1)
= 1+ b  2 .
To get a sampler from D b it clearly sufﬁces to take a single example ( x , y ) , and return the constant function y .
To ﬁnd the Bayes optimal for the distribution D b , on the other hand, any algorithm needs ⌦ (1 /  2 ) examples.
Using this observation, we show the folowing result:
Theorem 6.
For every M > 0 , there exists a class of distributions P M such that: a) there exists a sampler for P M with sample complexity e m ⌘ 1 , and b) any learner for P M has sample complexity satisfying m (1 / 8)  M .
Teachers
Motivated by the observed behavior of neural networks in the overparemeterized regime, we deﬁned the notion of samplers .
We showed that samplers can be much more sample efﬁcient than learners, at the cost of giving noisy predictions.
Next, we will show that while samplers are in and of themselves bad classiﬁers, they can still be good teachers .
That is, we can use samplers to label a large unlabeled dataset and train a student classiﬁer on this new dataset.
This process is often referred to as knowledge distillation , and it has been shown to work remarkably well in practice [29, 23].
The following deﬁnition captures the notion of a (good) teacher:
Deﬁnition 7.
For some learning algorithm A , some distribution D over X ⇥ Y , we say that A is a teacher for
D if there exists a function e m : (0 , 1) 2 !
N s.t.
for every ✏ , ⌧ 2 (0 , 1) , taking m  e m ( ✏ , ⌧ ) the following holds:
[Equation]
Additionally, for some class P of distributions over X ⇥ Y , we say that A is a teacher for P if there is some e m ( · , · ) s.t.
A is a teacher with sample complexity m ( · , · ) for every D 2 P .
The ﬁrst condition in the above deﬁnition means that the Bayes optimal of the original distribution and the distribution induced by the teacher are close.
The second condition means that the probability mass of low margin samples from the distribution labeled by the teacher is small.
Intuitively, an algorithm satisfying Deﬁnition 7 is a good teacher since the distribution it induces is “similar” to the original distribution.
Hence, if the student ﬁnds a good hypothesis on the teacher-induced distribution, its hypothesis is also good w.r.t.
the original distribution.
In Section 3 we formally study when and how teachers can be used for distillation, giving guarantees for getting students with small error with respect to the clean distribution D ⇤ .
Before that, let us ﬁrst show that samplers are indeed good teachers.
Theorem 8.
Let A be a sampler for D with sample complexity m ( · ) and margin  .
Then, A is a teacher for D with sample complexity
[Equation]
The main idea behind the proof of Theorem 8 is that, given an example with probability mass p , the “cost" (in terms of TV ) of ﬂipping the example’s label w.r.t.
the Bayes optimal f ⇤ D is at least p ·  .
Since our TV “budget" is limited by " we can only ﬂip a probability mass of " of the distribution.
Finally, before moving on to discuss the implication of our results, we show that learners are also good teachers.
This is almost immediate, since learners approximate the Bayes optimal classiﬁer, and hence can be used to train students to similarly imitate the Bayes classiﬁer.
Theorem 9.
Let A be a learner for D with sample complexity m ( · ) .
Then, A is a teacher for D with sample complexity
[Equation]
To conclude, Theorem 8 and Theorem 9 show that, to some extent, a teacher is an “interpolation” between a sampler and a learner.
Distillation from Teachers
We deﬁned the notion of a teacher, and showed that both samplers and learners are teachers.
Now, we show how teachers (and in particular, samplers) can be used to ﬁnd good learners.
First, we show that an ensemble of teachers can be used to get a good learner by simply outputting the majority vote of the ensemble.
Next, we show that using the ensemble to label a new set of unlabeled examples (i.e., performing knowledge distillation) guarantees ﬁnding a student with small loss, assuming the
Bayes optimal classiﬁer comes from the hypothesis class learned by the student.
Such process is favorable, since it reduces the computational cost of running the ensemble at inference time, and also allows using a different hypothesis class for the student (for example when using a student network of smaller size, e.g., [11]).
Finally, we show that distillation can also be achieved by labeling examples using a teacher that is randomly chosen from a ﬁxed set of teachers, a method that has some computational beneﬁts at training time.
To the best of our knowledge, this is a novel technique for distillation that has not been previously suggested in prior work.
3. Ensembles of Teachers
We now show how an ensemble of teachers can be used to get accurate predictions with respect to the
Bayes optimal predictor.
Given some k samples S 1 , . . .
S k
⇠ D m , each one of size m , we can use a learning algorithm
A to get k different hypothesis h 1 , . .
, h k , where h i :
= A ( S i ) .
Observe the ensemble hypothesis, which outputs the majority vote of the ensemble members:
[Equation]
We use the notation
A ens ( S 1 , . . .
, S k ) :
=
h ens to denote this ensemble hypotheses.
The following Theorem states that the ensemble hypothesis has a good loss on average, when using a large enough ensemble of teachers:
Theorem 10.
Assume that A is a teacher for some distribution D with complexity e m .
Then, for all ✏ 0 , 1) , taking m
e
[Equation]
[Equation]
⌘ and k  16log
[Equation]
we get,
[Equation]
We now give a sketch of the proof.
For some x 2 X , let y i be the prediction of the i -th teacher, and let ¯ y be the average of the predictions, namely ¯ y = 1 k
P i y i .
Observe that h ens ( x ) = sign(¯ y ) , and additionally E [¯ y ] =
E A ( D m )
[ y | x ] .
Now, since A is a teacher we get
E A
( D m )
[ y | x ] ⇡ E D [ y | x ] (with high probability over the choice of x ), and by concentration bounds this implies that w.h.p.
h ens ( x ) give the Bayes optimal prediction (i.e., h ens ( x ) =
f ⇤ D ( x ) ).
So, Theorem 10 shows that if A is a teacher for D , then A ens with k  e ⌦  1 /  ( D ) 3 is a learner for D .
More generally, if we have a teacher for some distribution D with positive margin, this implies that there exists a learner for the same distribution.
The inverse of this statement gives another interesting result—if no algorithm can learn some problem, then getting a teacher (or sampler) is also hard.
Formally, let P be a class of distributions over X ⇥ Y s.t.
for all D 2 P it holds that  ( D )   .
Then, if there is no learner for
P , there is no teacher or sampler for P .
Additionally, a similar result holds for problems which are computationally hard to learn.
That is, if there is no learner for
P that runs in polynomial time, then there is no poly-time teacher or sampler for P .
Observe that the condition that  ( D )   for all D 2 P in the previous statement is necessary.
Indeed, taking
P =
[ 1 M =1 , where P M is the distribution class guaranteed by Theorem 6, gives a class P s.t.
there is a sampler for
P with sample complexity
[Equation]
there is no learner for P .
3. Distillation from Ensembles
We showed that an ensemble of k =
˜
⌦
(   2 ) teachers gives a classiﬁer that approximates the Bayes optimal predictor.
This, however, incurs a k factor in computational cost at inference time.
To prevent this, we can instead use the ensemble to label new unlabeled data, and train a new classiﬁer to imitate the ensemble.
This moves the computational burden from inference time to training time.
Fix some class H , and deﬁne the Ensemble-Pseudo-Labeling (EPL) algorithm as follows:
For some k, m 2 N , sample S 1 , . . .
S k
⇠ D m .
Run A on S 1 , . .
, S k , and let h ens =
A ens ( S 1 , . . .
, S k ) .
Take S 0 to be a set of m 0 unlabeled examples sampled from D X , and label it using h ens .
Denote by
ERM H on the set
Intuitively, since h ens approximates the Bayes optimal classiﬁer (see Theorem 10), the labels for the new dataset S 0 are mostly correct.
In other words, the pseudo-labeled set e
S comes from a distribution that is close to the clean distribution D ⇤ .
When using pseudo-labels, it is enough to use unlabeled data, which is often abundant, so e
S can be much larger than our original labeled dataset.
In this case, we no longer need to work in the overparameterized regime, so ERM H is guaranteed to achieve good performance by standard VC bounds.
This argument is captured in Theorem 11:
Theorem 11.
For hypothesis class H with VC( H ) < 1 , and let D 2 P ( H ,  ) for some  > 0 .
Let
A be a teacher for D with distributional sample complexity e m .
Then, there exists a constant
C > 0 s.t.
for every ✏ 2 (0 , 1) , running the EPL algorithm with parameters m  e m [Equation]
C VC( [Equation] +log(1 / ✏ ) ✏and k  16log(12 / ✏ ) returns a hypothesis h satisfying E S 1 ,...,S k , e
S L D [Equation] ( h )  ✏
[Equation]
3. Distillation from Random Teachers
While the EPL algorithm moves the computation cost from inference to training, we still suffer a k factor for labeling each example.
Instead, a possible solution is to label examples by choosing a random classiﬁer from the ensemble.
Then, we only run one classiﬁer per example.
So, we deﬁne the
Random-Pseudo-Labeling (RPL) similarly to EPL , except that for each example x in the unlabeled dataset S 0 , we pick h ⇠ { h 1 , . .
h k } and label x by h ( x ) .
To understand why this method works, we can think of e
S as coming from a distribution e
D , deﬁned by sampling x ⇠ D X and sampling y s.t.
E e
D [ y | x ] =
E i ⇠ [ k ] [ h i ( x )] .
By the properties of the teacher, using concentration arguments as in Theorem 10, the distribution e
D is close to the noisy distribution
D .
However, as mentioned before, the advantage of using
[Equation] e
D is that we can use a much larger set of unlabeled data, in which case the result of Theorem 2 can be applied to show that the above algorithm ﬁnds a hypothesis with good error.
This is stated in the following result:
Theorem 12.
For hypothesis class H with VC( H ) < 1 .
let D 2 P
( H ,  ) for  > 0 .
Let
A be a teacher for D with distributional sample complexity e m .
Then, there exists a constant
C > 0 s.t.
for every ✏ 2 (0 , 1) , running the RPL algorithm with parameters m  e m
[Equation]
and k
[Equation]
returns h satisfying E S 1 ,...
[Equation]
e
[Equation]
( h [Equation]
Compare the sample complexity of the above theorem with the sample complexity achieved by the
EPL algorithm, stated in Theorem 11.
At ﬁrst glance, it seems that the gain from using the RPL algorithm is not clear, as it increases the number of unlabeled data by a factor of 1 /  ( D ) 2 (and also might increase the number of labeled examples).
This is not surprising, because a dataset labeled by a random classiﬁer can be much more noisy than a dataset labeled by the ensemble, and hence more examples are required in order to learn.
Note, that since k  ˜ ⌦ (1 /  ( D ) 2 ) , the randomized labeling takes 1 /k compute per example relative to the ensemble labeling, it will, however, need to label an order of k times more examples.
We do note that there might still be computational beneﬁts to using the random approach, as it allows the training and labeling to happen in parallel without a signiﬁcant increase in compute (e.g.,
GPU cores are enough to train in parallel without any loss of time).
On the other hand, labeling a dataset using the ensemble requires one to either increase the compute by k (applying parallelism), or otherwise wait until the full dataset is labeled, and only then start the distillation process.
Finally, our experiments show that the gain from ensemble labeling over random labeling, as captured by the ﬁnal accuracy of the trained student, is not so signiﬁcant (see Table 1).
This suggests that in some particular cases (e.g., under further assumptions on the data distribution and/or the optimization algorithm), it is preferable to use the RPL algorithm.
Samplers Exist
So far, we showed that samplers can be good teachers.
These can then be used for knowledge distillation, generating students which approximate the Bayes optimal prediction.
To complete the picture, we now show that some well-known algorithms can be used as samplers or teachers.
We start by studying the k-Nearest-Neighbour algorithm, and show that it is a teacher when the underlying distribution has some Lipschitz property.
We then show that bounded-norm ReLU networks with scalar input exhibit similar behavior.
In Appendix C.1 we further analyze Lipschitz classes of functions (e.g., linear functions, kernels and neural networks), and show that when the data is wellclustered, these algorithms can be teachers as well.
In all cases, the sample complexity needed for the samplers or teachers is lower than the complexity required for learning.
Nearest Neighbour Algorithm
For some set S = { ( x 1 , y 1 ) , . . .
, ( x m , y m ) } ✓ X ⇥ Y we denote S X
= { x 1 , . . .
, x m } .
Fix some metric d over the space
X .
In this section, we assume that the metric space ( X , d ) satisﬁes the Heine-Borel property—that is, every closed and bounded set in X is compact 4 .
For some ﬁnite set S ✓ X ⇥ Y , and some x 2 X , denote d ( x , S ) :
=
min x 0 2 S X d ( x , x 0 ) and ⇡ ( x , S ) :
= arg min ( x 0 ,y 0 ) 2 S d ( x , x 0 ) .
Additionally, we deﬁne the set k-
⇡
( x , S X ) ✓ S to be the set of k points in S that are closest to x .
That is, k- ⇡
( x , S ) is a set of size k s.t.
for any ( x 0 , y 0 ) 2 k- ⇡ ( x , S ) and (˜ x , ˜ y ) 2 S \ k-
⇡
( x , S )
it holds that d ( x , x 0 )  d ( x , ˜ x ) 5 .
For some distribution D , we say that D is  -Lipschitz if for all x , x 0 2 supp( D X ) and y 2 Y it holds that | P D [ y | x ]  P D [ y | x 0 ] |   d ( x , x 0 ) .
For some odd k  1 , deﬁne A k-NN ( S )( x ) :
= arg max ˆ y 2 Y | ( x 0 , y 0 ) 2 k- ⇡
( x , S ) , y
= ˆ y | , i.e.,
A k-NN ( S ) is the k-NN algorithm over sample S .
We start by showing that the A 1-NN is a sampler.
The distributional sample complexity of A 1-NN depends on the number of ✏ -balls that can cover a 1   mass of the distribution (Lemma 23 in the Appendix shows that this number is always ﬁnite).
Given such cover, we can guarantee that with a large enough sample, we ﬁnd a candidate example in each of the balls that have non-negligible mass.
In that case, if the distribution of labels does not change signiﬁcantly in each ball, A 1-NN is indeed a sampler:
Theorem 13.
Let D be some  -Lipschitz distribution.
Then, A 1-NN is a sampler for D .
Next, we study the k-NN algorithm for any odd k  1 .
Similarly to the analysis of the 1-NN case, we show that a large enough sample guarantees at least k candidates in each of the ✏ -ball covering the distribution.
In this case, the prediction of the k-NN algorithm is the majority vote over the k neighbours in each ball.
This will get closer to the Bayes optimal prediction as k grows, and therefore the k-NN algorithm is not a sampler for k > 1 .
However, we show that it is always a teacher:
Theorem 14.
Let D be some  -Lipschitz distribution.
Then, A k-NN is a teacher for D .
The core argument for proving Theorem 14 relies on using a variant of Condorcet’s Jury Theorem (CJT).
Roughly speaking, CJT states that the accuracy of the majority vote of a set of predictors is better than the average accuracy of the individual predictors.
In the k-NN case, each of the k candidates casts a “vote”, and using CJT we show that this improves over the 1-NN prediction, which is already a sampler (and hence a teacher).
Notice that this works for any k  1 , and we do not need to require k to be large enough, as would be required in order to get a learner.
Bounded-Norm Inﬁnite-Width ReLU Networks
We continue with investigating inﬁnite-widths neural-network with weights of bounded norm, following the setting studied in [24, 22].
We deﬁne a ReLU network of width k and depthby h ✓ ( x ) =
[Equation]
where ✓ = ( k, W (1) , W (2) , b (1) , b (2) ) , and  is the ReLU activation.
As in [24], we consider the Euclidean norm of non-biased weights:
[Equation]
Now, consider ﬁtting a sample S ✓ X ⇥ Y with a network
Experiment
[Table]
h ✓ with C ( ✓ ) acting as regularization.
Namely, observe the following objective function:
[Equation]
In the one-dimensional case, i.e. when X = R , [24] shows that R ( S ) gives the linear spline interpolation of the data points.
Using this, we show that ReLU networks in this setting are samplers:
Theorem 15.
For algorithm A that takes a sample S and returns h s.t.
h ( x ) = sign  h
ˆ ✓ ( x )  , for ˆ ✓
= R ( S ) .
Let D be some continuous  -Lipschitz distribution.
Then, A is a sampler for D .
The above shows that when we don’t limit the size of the network and use the regularization C ( ✓ ) , the resulting algorithm is a sampler.
Observe that if we do not introduce regularization, one can construct a ReLU network h ✓ that outputs a constant value (e.g., 1 ) for all x 2 R , outside of inﬁnitesimally small neighbourhoods of the points of S , where h ✓ interpolates the data (namely, h ✓ is constant with very narrow “spikes” towards the correct labels of the examples in the sample).
Thus, on new points h ✓ evaluates to 1 with high probability, so it does not behave like a sampler.
Admittedly, going beyond the one-dimensional case is more challenging, as it requires understanding the high-dimensional geometry of the function returned by R ( S ) .
While we defer this case for future work, we note that the analysis of [22] gives some technical tools for understanding the multivariate version of the above problem.
Speciﬁcally, [22] show that solving R ( S ) is equivalent to minimizing a speciﬁc norm in function-space, which controls the complexity of the learned function.
Among other things, the authors show that controlling this norm prevents a “spiking” behavior as described above.
Experiments
To this point, we saw that getting a sampler (and thus a teacher) from a noisy distribution can be more sample efﬁcient than getting a learner.
Furthermore, we showed that we can leverage multiple independent teachers to approximate the Bayes optimal classiﬁer either via ensembling at inference time or via distillation on unlabeled data.
We now complement our theoretical results with an experimental evaluation, showing the beneﬁt of using distillation when training on noisy data.
While in our theoretical setting we studied teachers that are trained on entirely disjoint training sets, in practice we ﬁnd it more effective to train the teachers on overlapping datasets, as well as training on same dataset with different random initialization.
To get the teachers, we train a ResNet-18 [13] on CIFAR-10 with 20%- ﬁxed and non-uniform label noise (see full details in D).
We see that our teachers achieve 81 .
3% test accuracy (see Table 1) and behave closely to samplers (see Figure 1) reproducing the results of [20].
We now compare the three methods considered before for using teachers to get learners: 1) Test time Ensembling; 2) Ensemble as distillation teacher and; 3) Random teacher distillation.
For distillation, we train a student network on the CIFAR-5m, a large (5-million examples) dataset that resembles the CIFARdataset [21], where the labels are provided by the previously trained teachers.
We report our results in Table 1, where the reported accuracies are on the CIFAR-10 test data.
Observe that using an ensemble for inference reduces the noise signiﬁcantly, and achieves test accuracy of 87 .
8% (versus 81 .
3% for a single teacher).
When applying distillation, both random pseudo-labeling and ensemble pseudo-labeling further increase the test accuracy to about 90% .
In addition, we study how the number of teachers affects performance (see Figure 2).
We observe that both random pseudo-labeling and ensemble majority improve in performance when the number of teachers grow.
Acknowledgements
This work supported by Simons Investigator Fellowship, NSF grants DMS-2134157 and CCF1565264, DARPA grant W911NF2010021,and DOE grant DE-SC0022199.