Original: Functional Ensemble Distillation
Revised: Functional Ensemble Distillation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Bayesian models have many desirable properties, most notable is their abilityto generalize from limited data and to properly estimate the uncertainty in theirpredictions.
Revised: Bayesian models have many desirable properties, most notable is their abilityto generalize from limited data and to properly estimate the uncertainty in theirpredictions.

Original: However, these benefits come at a steep computational cost as Bayesianinference, in most cases, is computationally intractable.
Revised: However, these benefits come at a steep computational cost as Bayesianinference, in most cases, is computationally intractable.

Original: One popular approachto alleviate this problem is using a Monte-Carlo estimation with an ensembleof models sampled from the posterior.
Revised: One popular approachto alleviate this problem is using a Monte-Carlo estimation with an ensembleof models sampled from the posterior.

Original: However, this approach still comes at asignificant computational cost, as one needs to store and run multiple models attest time.
Revised: However, this approach still comes at asignificant computational cost, as one needs to store and run multiple models attest time.

Original: In this work, we investigate how to best distill an ensemble’s predictionsusing an efficient model.
Revised: In this work, we investigate how to best distill an ensemble’s predictionsusing an efficient model.

Original: First, we argue that current approaches that simply returndistribution over predictions cannot compute important properties, such as thecovariance between predictions, which can be valuable for further processing.
Revised: First, we argue that current approaches are limited asthey are constrained to classification and the Dirichlet distribution.

Original: Second, in many limited data settings, all ensemble members achieve nearly zerotraining loss, namely, they produce near-identical predictions on the training setwhich results in sub-optimal distilled models.
Revised: Second, inmany limited data settings, all ensemble members achieve nearly zero trainingloss, namely, they produce near-identical predictions on the training set whichresults in sub-optimal distilled models.

Original: To address both problems, wepropose a novel and general distillation approach, named Functional Ensemble Distillation (FED), and we investigate how to best distill an ensemble in this setting.
Revised: To address both problems, we propose anovel and general distillation approach, named Functional Ensemble Distillation(FED), and we investigate how to best distill an ensemble in this setting.

Original: We find that learning the distilled model via a simple augmentation scheme inthe form of mixup augmentation [41] significantly boosts the performance.
Revised: We findthat learning the distilled model via a simple augmentation scheme in the formof mixup augmentation [42] significantly boosts the performance.

Original: Weevaluated our method on several tasks and showed that it achieves superior resultsin both accuracy and uncertainty estimation compared to current approaches.
Revised: We evaluatedour method on several tasks and showed that it achieves superior results in bothaccuracy and uncertainty estimation compared to current approaches.

Original:  Introduction
Revised:  Introduction

Original: In parametric Bayesian inference the output is a distribution over model parameters, known as theposterior distribution.
Revised: In parametric Bayesian inference the output is a distribution over model parameters, known as theposterior distribution.

Original: This is in contrast to standard approaches which return a single model.
Revised: This is in contrast to standard approaches which return a single model.

Original: Attest time one needs to marginalize over all models according to the posterior distribution to make aprediction.
Revised: Attest time one needs to marginalize over all models according to the posterior distribution to make aprediction.

Original: The action of averaging over multiple models allows Bayesian approaches to performwell with limited data [1, 25, 32, 40] and properly estimate uncertainties [15, 16, 32].
Revised: The action of averaging over multiple models allows Bayesian approaches to perform wellwith limited data [1, 2, 26, 33, 41] and properly estimate uncertainties [16, 17, 33].

Original: The downsideis that in most cases Bayesian inference is intractable and even methods for approximate inferencemay be costly.
Revised: The downsideis that in most cases Bayesian inference is intractable and even methods for approximate inferencemay be costly.

Original: This is especially true for Bayesian neural networks (BNNs), as the dimension of theparameter space for common neural networks is very large [3, 13].
Revised: This is especially true for Bayesian neural networks (BNNs), as the dimension of theparameter space for common neural networks is very large [4, 14].

Original: Two main approaches exist for efficient Bayesian inference.
Revised: Two main approaches exist for efficient Bayesian inference.

Original: The first is variational inference, wherewe fit a simple distribution, usually a Gaussian with diagonal covariance, to the posterior.
Revised: The first is variational inference, wherewe fit a simple distribution, usually a Gaussian with diagonal covariance, to the posterior.

Original: Whilethis can be done efficiently in some cases [18], it is more prone to over-fitting as the variationaldistribution might cover only a small part of the posterior.
Revised: Whilethis can be done efficiently in some cases [19], it is more prone to over-fitting as the variationaldistribution might cover only a small part of the posterior.

Original: A second approach is Monte-Carlo (MC)estimation where we sample M models from the posterior using an approximate sampling algorithm,e.g. Markov Chain Monte-Carlo (MCMC), and estimate the mean prediction by the average over thepredictions of the M models.
Revised: A second approach is Monte-Carlo (MC)estimation where we sample M models from the posterior using an approximate sampling algorithm,e.g. Markov Chain Monte-Carlo (MCMC), and estimate the mean prediction by the average over thepredictions of the M models.

Original: While this can better capture the diversity of the posterior, it requiresstoring and running M models instead of one.
Revised: While this can better capture the diversity of the posterior, it requiresstoring and running M models instead of one.

Original: One important question that will be the focus of thiswork is how to maintain the benefits of the Bayesian approach while reducing the computational overhead.
Revised: One important question that will be the focus of thiswork is how to maintain the benefits of the Bayesian approach while reducing the computationaloverhead.

Original: It is also important to note thatusing an ensemble of M models is commonly used outsideof Bayesian settings, e.g. bagging [2], so this line of work has wider implications.
Revised: It is also important to note that an ensemble of M models is commonly used outside of Bayesian settings, e.g. bagging [3], so this line of work has wider implications.

Original: Our focus, however,will be in the low-data regime where Bayesian methods are most impactful.
Revised: Our focus, however,will be in the low-data regime where Bayesian methods are most impactful.

Original: We propose an ensemble distillation [14, 24] method that mimics an ensemble of models using alightweight model.
Revised: We propose an ensemble distillation [15, 25] method that mimics an ensemble of models using alightweight model.

Original: Current approaches mainly consider classification where a single predictionis a probability vector, the distilled model then outputs parameters of a Dirichlet distribution thatshould fit the ensemble predictions.
Revised: Current approaches mainly consider classification where a single predictionis a probability vector, the distilled model then outputs parameters of a Dirichlet distribution thatshould fit the ensemble predictions.

Original: Our approach, which we call Functional Ensemble Distillation(FED), on the other hand tries to learn a distribution over functions by transferring the randomnessto ϵ , a noise variable that is sampled from a fixed distribution.
Revised: Our approach, which we call Functional Ensemble Distillation(FED), on the other hand tries to learn a distribution over functions by transferring the randomnessto ϵ , a noise variable that is sampled from a fixed distribution.

Original: We designed our model with tworequirements in mind.
Revised: We designed our model with tworequirements in mind.

Original: First, it should be fast to run during test time.
Revised: First, it should be fast to run during test time.

Original: This is achieved by being able torun with multiple ϵ values in parallel.
Revised: This is achieved by being able torun with multiple ϵ values in parallel.

Original: Second, the distribution of predictions from our model shouldmatch the distribution of predictions from the posterior.
Revised: Second, the distribution of predictions from our model shouldmatch the distribution of predictions from the posterior.

Original: This approach allows us more flexibility as itis not limited to classification tasks and to the Dirichlet distribution, unlike most previous approaches.
Revised: This approach allows us more flexibility as itis not limited to classification tasks and to the Dirichlet distribution, unlike most previous approaches.

Original: Furthermore, our method allows us to compute covariance between predictions which can be usefulfor further downstream tasks.
Revised: Furthermore, our method allows us to compute covariance between predictions which can be usefulfor further downstream tasks.

Original: Consider, for example, the task of object detection for autonomousvehicles.
Revised: Consider, for example, the task of object detection for autonomousvehicles.

Original: If two nearby detections are made with certain confidence then the probability that that areaof space is occupied depends heavily on the correlation between these predictions.
Revised: If two nearby detections are made with certain confidence then the probability that that areaof space is occupied depends heavily on the correlation between these predictions.

Original: Finally, we investigate how to properly distill from a model ensemble under severe overfitting.
Revised: Finally, we investigate how to properly distill from a model ensemble under severe overfitting.

Original: It iscommon for deep networks trained on small datasets such as CIFAR-10 [20] to reach almost perfectaccuracy on the training set while still generalizing well to the test set.
Revised: It iscommon for deep networks trained on small datasets such as CIFAR-10 [21] to reach almost perfectaccuracy on the training set while still generalizing well to the test set.

Original: In these cases, the predictionsof the models in the ensemble can vary considerably on the test set despite being almost identicalon the training set.
Revised: In these cases, the predictionsof the models in the ensemble can vary considerably on the test set despite being almost identicalon the training set.

Original: This means that distilling the ensemble on the training set will not capturethe diversity between predictions which is the core component of ensemble methods.
Revised: This means that distilling the ensemble on the training set will not capturethe diversity between predictions which is the core component of ensemble methods.

Original: While someprevious works tried to address this issue [24, 27], they either depended on additional unsuperviseddata that might not be readily available, e.g. medical imaging, or do not scale well.
Revised: While someprevious works tried to address this issue [25, 28], they either depended on additional unsuperviseddata that might not be readily available, e.g. medical imaging, or do not scale well.

Original: Here we proposea simple solution for this problem that works remarkably well - creating an auxiliary dataset usingthe mixup augmentation [41].
Revised: Here we proposea simple solution for this problem that works remarkably well - creating an auxiliary dataset usingthe mixup augmentation [42].

Original: The mixup procedure generates data that is on the one hand distinctenough from the training set to allow our ensemble to produce diverse predictions, while on the otherhand, being close enough to it so that models that were trained on it work well when applied to theoriginal test data.
Revised: The mixup procedure generates data that is on the one hand distinctenough from the training set to allow our ensemble to produce diverse predictions, while on the otherhand, being close enough to it so that models that were trained on it work well when applied to theoriginal test data.

Original: To conclude, we make the following novel contributions: (i)
Revised: To conclude, we make the following novel contributions: (i)

Original: We propose a general-purpose and flexiblemethod for ensemble distillation that captures well the predictive abilities and the diversity of theoriginal ensemble; (ii) we propose a simple, fast, and scalable method for obtaining a diverse auxiliarydataset for training the distilled model without any new samples using the mixup augmentation; and(iii) we evaluated our method on several image benchmarks and showed that it obtains the best resultsin terms of accuracy and calibration compared to other ensemble distillation approaches.
Revised: We propose a general-purpose and flexiblemethod for ensemble distillation that captures well the predictive abilities and the diversity of theoriginal ensemble; (ii) we propose a simple, fast, and scalable method for obtaining a diverse auxiliarydataset for training the distilled model without any new samples using the mixup augmentation; and(iii) we evaluated our method on several image benchmarks and showed that it obtains the best resultsin terms of accuracy and calibration compared to other ensemble distillation approaches.

Original:  Background
Revised:  Background

Original: We first provide a brief introduction to the main components of our model.
Revised: We first provide a brief introduction to the main components of our model.

Original: We use the followingnotations in the paper.
Revised: We use the followingnotations in the paper.

Original: We denote scalars with lower-case letters (e.g., x ), and vectors with boldlower-case letters (e.g., x ).
Revised: We denote scalars with lower-case letters (e.g., x ), and vectors with boldlower-case letters (e.g., x ).

Original: We assume we are given with a dataset D of size N ,
Revised: We assume we are given with a dataset D of size N ,

Original: D
Revised: D

Original: = { ( x i , y i ) } i =1: N ,where x
Revised: = { ( x i , y i ) } i =1: N ,where x

Original: i ∈ R
Revised: i ∈ R

Original: d are the training examples and y
Revised: d are the training examples and y

Original: i are the corresponding labels or targets.
Revised: i are the corresponding labels or targets.

Original: 2.1 Bayesian Inference
Revised: 2.1 Bayesian Inference

Original: In Bayesian inference we are mainly interested in the posterior distribution p ( θ |D )
Revised: In Bayesian inference we are mainly interested in the posterior distribution p ( θ |D )

Original: ∝ 81 (cid:81)
Revised: ∝ 80 (cid:81)

Original: i p ( y i | x i , θ ) p ( θ ) , where θ denotes the parameters of the model (e.g., a NN).
Revised: i p ( y i | x i , θ ) p ( θ ) , where θ denotes the parameters of the model (e.g., a NN).

Original: The posterior tellsus how likely θ is given the observed data.
Revised: The posterior tellsus how likely θ is given the observed data.

Original: When given a new data point x ∗ we will use the entireposterior to calculate the predictive distribution using the rule p ( y ∗ | x ∗ , D ) = (cid:82) p ( y ∗ | x ∗ , θ ) p ( [Equation]
Revised: When given a new data point x ∗ we will use the entireposterior to calculate the predictive distribution using the rule p ( y ∗ | x ∗ , D ) = (cid:82) p ( y ∗ | x ∗ , θ ) p ( θ |D ) dθ .

Original: One intuition behind this is that when one has limited data, there are many distinct models thatwork well on the training set.
Revised: One intuition behind this is that when one has limited data, there are many distinct models thatwork well on the training set.

Original: By averaging over all the possible predictors the final prediction issupported by most of the “weight" in the posterior instead of by a single model.
Revised: By averaging over all the possible predictors the final prediction issupported by most of the “weight" in the posterior instead of by a single model.

Original: This also leads tobetter uncertainty estimation, as when many different models have different predictions the resultingaverage prediction is dispersed and uncertain.
Revised: This also leads tobetter uncertainty estimation, as when many different models have different predictions the resultingaverage prediction is dispersed and uncertain.

Original: In practice, besides a few notable exceptions, computing this integral is computationally intractable.
Revised: In practice, besides a few notable exceptions, computing this integral is computationally intractable.

Original: One popular approximate inference solution is Monte-Carlo (MC) estimation.
Revised: One popular approximate inference solution is Monte-Carlo (MC) estimation.

Original: Namely, we estimatep ( y ∗ | x ∗ , D )
Revised: Namely, we estimate p ( y ∗ | x ∗ , D )

Original: ≈ 1 M (cid:80)
Revised: ≈ 1 M (cid:80)

Original: Mm =1 p ( y ∗ | x ∗ , θ m ) , where θ m are (approximate) samples from the posteriorp ( θ |D ) .
Revised: Mm =1 p ( y ∗ | x ∗ , θ m ) , where θ m are (approximate) samples from the posteriorp ( θ |D ) .

Original: Obtaining such samples can be achieved in various ways such as MCMC methods [28, 36, 42].
Revised: Obtaining such samples can be achieved in various ways such as MCMC methods [29, 37, 43].

Original: One problem with this approach is that it requires storing and running multiple models to obtain thepredictive distribution.
Revised: One problem with this approach is that it requires storing and running multiple models to obtain thepredictive distribution.

Original: This may be impractical in many situations such as when making predictionsin low resource environments or in real-time systems.
Revised: This may be impractical in many situations such as when making predictionsin low resource environments or in real-time systems.

Original: 2.2 Maximum Mean Discrepancy
Revised: 2.2 Maximum Mean Discrepancy

Original: A big question in machine learning and statistics is how to measure distances between distributions,specifically without access to the density function and with only access to samples.
Revised: A big question in machine learning and statistics is how to measure distances between distributions,specifically without access to the density function and with only access to samples.

Original: One wellknown measure is the Maximum Mean Discrepancy (MMD) [35] that was used as an objective tosuccessfully train generative models [21, 22].
Revised: One wellknown measure is the Maximum Mean Discrepancy (MMD) [36] that was used as an objective tosuccessfully train generative models [22, 23].

Original: Given a family of functions F the MMD score betweentwo distributions p, q is defined as MMD ( p, q ) = 
Revised: Given a family of functions F the MMD score betweentwo distributions p, q is defined as MMD ( p, q ) = sup

Revised: [Equation]

Original: sup f ∈F | E x ∼ p
Revised: ∈F | E x ∼ p

Original: [ f ( x )] − E
Revised: [ f ( x )] − E

Original: x ∼ q [ f ( x )] | .
Revised: x ∼ q [ f ( x )] | .

Original: A specificcase in which the MMD has a closed form formula is when F is the unit sphere in a reproducingkernel Hilbert space (RKHS) defined by a kernel function k [11].
Revised: A specificcase in which the MMD has a closed form formula is when F is the unit sphere in a reproducingkernel Hilbert space (RKHS) defined by a kernel function k [12].

Original: In that case we have
Revised: In that case we have

Original: [Equation]
Revised: [Equation]

Original:  which we estimate by Monte-Carlo means.
Revised: (1) which we estimate by Monte-Carlo means.

Original: 2.3 Mixup Training Procedure
Revised: 2.3 Mixup Training Procedure

Original: The mixup [41] training procedure is based on the Vicinal Risk Minimization principle [5].
Revised: The mixup [42] training procedure is based on the Vicinal Risk Minimization principle [6].

Original: The mainidea is to enlarge the training set by sampling points in the neighborhood of the given data.
Revised: The mainidea is to enlarge the training set by sampling points in the neighborhood of the given data.

Original: [41]proposed a simple way to achieve this goal by sampling along the line which connects two data pointsfrom the training set.
Revised: [42]proposed a simple way to achieve this goal by sampling along the line which connects two data pointsfrom the training set.

Original: Let ( x i , y i ) , ( x j , y j ) be two training examples, one can generate a new datapoint (˜ x , ˜ y ) by taking their convex combination:
Revised: Let ( x i , y i ) , ( x j , y j ) be two training examples, one can generate a new datapoint (˜ x , ˜ y ) by taking their convex combination:

Original: ˜ x
Revised: ˜ x

Original: = λ x i + (1 − λ ) x j , and ˜ y
Revised: = λ x i + (1 − λ ) x j , and ˜ y

Original: = λy
Revised: = λy

Original: i + (1 − λ )
Revised: i + (1 − λ )

Original: y j ,where λ is drawn from a Beta distribution with some fixed parameters.
Revised: y j ,where λ is drawn from a Beta distribution with some fixed parameters.

Original: Now, standard loss functions,such as the cross entropy loss, can be used on (˜ x , ˜ y ) .
Revised: Now, standard loss functions,such as the cross entropy loss, can be used on (˜ x , ˜ y ) .

Original:  Related Work
Revised:  Related Work

Original: Several previous studies proposed approaches to distill an ensemble of models into a single fastmodel.
Revised: Several previous studies proposed approaches to distill an ensemble of models into a single fastmodel.

Original: Early approaches [4, 14, 19, 30, 38] focused on capturing only the mean behaviour of theensemble but did not capture the diversity in it, which may be crucial for uncertainty estimation.
Revised: Early approaches [5, 15, 20, 31, 39] focused on capturing only the mean behaviour of theensemble but did not capture the diversity in it, which may be crucial for uncertainty estimation.

Original: Forexample, in [4] it was first shown that a single network can learn the knowledge of a large ensemble ofnetworks.
Revised: Forexample, in [5] it was first shown that a single network can learn the knowledge of a large ensemble ofnetworks.

Original: In [14] a distilled model is learned using a convex combination of the ensemble predictionsin addition to the standard cross-entropy loss.
Revised: In [15] a distilled model is learned using a convex combination of the ensemble predictionsin addition to the standard cross-entropy loss.

Original: And, in [19] a student networks is learned online froma teacher that is trained via MCMC training procedures.
Revised: And, in [20] a student network is learned online from ateacher that is trained via MCMC training procedures.

Original: More recent studies proposed alternatives for capturing the diversity as well.
Revised: More recent studies proposed alternatives for capturing the diversity as well.

Original: Ensemble Distribution
Revised: Ensemble Distribution

Original: Distillation (EnDD) [24] was the first to propose such an approach.
Revised: Distillation (EnDD) [25] was the first to propose such an approach.

Original: EnDD builds on the method in
Revised: EnDD builds on the method in[24] which tries to emulate the behaviour of an ensemble with a Dirichlet distribution parameterizedby a NN. To handle the fact that the ensemble predictions are over-confident and have low diversity

Original: Second, add Gaussian noise to intermediate features along the ResNet-18 architecture.
Revised: Second, add Gaussian noise to intermediate features along the ResNet-18 architecture.

Original: [23] which tries to emulate the behaviour of an ensemble with a Dirichlet distribution parameterizedby a NN.

Original: To handle the fact that the ensemble predictions are over-confident and have low diversityon the training set, they proposed to use an unsupervised auxiliary dataset for training the distilledmodel, and to perform temperature annealing.
Revised: on the training set, they proposed to use an unsupervised auxiliary dataset for training the distilledmodel, and to perform temperature annealing.

Original: Similarly, [8] also makes the assumption that thepredictive distribution follows a Dirichlet distribution.
Revised: Similarly, [9] also makes the assumption that thepredictive distribution follows a Dirichlet distribution.

Original: They fit the output to the teacher networkby two separate networks, one that learns the predicted class probabilities, i.e. the Dirichlet mean,and another that learns the confidence, i.e. the Dirichlet concentration.
Revised: They fit the output to the teacher networkby two separate networks, one that learns the predicted class probabilities, i.e. the Dirichlet mean,and another that learns the confidence, i.e. the Dirichlet concentration.

Original: Both [24, 8] are suitable forclassification tasks only and assume a Dirichlet distribution behaviour which may be too restrictive.
Revised: Both [25, 9] are suitable forclassification tasks only and assume a Dirichlet distribution behaviour which may be too restrictive.

Original: We argue that a more general solution for the distillation of ensembles is needed.
Revised: We argue that a more general solution for the distillation of ensembles is needed.

Original: A different line of work [34, 27] generalizes the method in [14] by distilling the ensemble membersto task heads in a one-to-one fashion.
Revised: A different line of work [35, 28] generalizes the method in [15] by distilling the ensemble membersto task heads in a one-to-one fashion.

Original: In [34] the distilled model is partitioned into two parts, a shared“body” and a member-specific “head”, one per ensemble member.
Revised: In [35] the distilled model is partitioned into two parts, a shared“body” and a member-specific “head”, one per ensemble member.

Original: Each head is trained to copy theprediction of an ensemble member using the standard cross-entropy loss function.
Revised: Each head is trained to copy theprediction of an ensemble member using the standard cross-entropy loss function.

Original: In [27] the authorspropose to use BatchEnsemble [37] for weight sharing.
Revised: In [28] the authorspropose to use BatchEnsemble [38] for weight sharing.

Original: They increase the diversity in the ensemblepredictions on the training set by randomly perturbing the training data based on three techniques:
Revised: They increase the diversity in the ensemblepredictions on the training set by randomly perturbing the training data based on three techniques:

Original: Gaussian, Output diversified sampling (ODS) [33], and ConfODS.
Revised: Gaussian, Output diversified sampling (ODS) [34], and ConfODS.

Original: The ODS perturbations are similarto adversarial perturbations and need to be computed at every iteration.
Revised: The ODS perturbations are similarto adversarial perturbations and need to be computed at every iteration.

Original: This imposes a computationaloverhead which renders a slow training process and inability to easily handle large ensembles, unlikeour proposed method.
Revised: This imposes a computationaloverhead which renders a slow training process and inability to easily handle large ensembles, unlikeour proposed method.

Original:  Our Method
Revised:  Our Method

Original: We take a similar approach to [8, 24] where we treat the ensemble distillation as a conditionalgenerative model.
Revised: We take a similar approach to [9, 25] where we treat the ensemble distillation as a conditionalgenerative model.

Original: Given an input x and a sampled model parameters θ we get a distribution p ( y | x , θ ) ,so for a given x we can push-forward the posterior distribution over weights to a distribution overdistributions over y .
Revised: Given an input x and a sampled model parameters θ we get a distribution p ( y | x , θ ) ,so for a given x we can push-forward the posterior distribution over weights to a distribution overdistributions over y .

Original: While this was done in [8, 24], there are several limitations with these approaches.
Revised: While this was done in [9, 25], there are several limitations with these approaches.

Original: First, they use a Dirichlet distribution to model this distribution over distributions, which islimitedto classification and has limited expressive power.
Revised: First, they use a Dirichlet distribution to model this distribution over distributions, which is limited toclassification.

Original: Second, given two inputs x 1 , x 2 the correlationbetween their predictions cannot be calculated by the distilled Dirichlet distributions. This can be veryimportant as one of the main motivations for Bayesian inference is its ability to measure uncertainty. In many applications, e.g. autonomous driving, the output predictions are fed into a planning modulethat needs uncertainty estimation to make the optimal plan.
Revised: Second, even in classification settings, the family of Dirichlet distributions has fixedexpressive power, and there is no evidence that the posterior is indeed a Dirichlet distribution.

Original: Our approach is to model a distribution over functions that both have low memory requirements andthat we can parallelize to get multiple predictions quickly.
Revised: Our approach is to model a distribution over functions that both have low memory requirements andthat we can parallelize to get multiple predictions quickly.

Original: We do this by training a generator neuralnetwork with parameters w and added random noise ϵ .
Revised: We do this by training a generator neuralnetwork with parameters w and added random noise ϵ .

Original: The noise can be added, for example, as anadditional channels to the input or hidden layers.
Revised: The noise can be added, for example, as anadditional channels to the input or hidden layers.

Original: Our goal is to train the generator network so that thedistribution over q ( y | x , w , ϵ ) for ϵ ∼ p ( ϵ ) will match that of p ( y | x , θ ) for θ ∼ p ( θ |D ) .
Revised: Our goal is to train the generator network so that thedistribution over q ( y | x , w , ϵ ) for ϵ ∼ p ( ϵ ) will match that of p ( y | x , θ ) for θ ∼ p ( θ |D ) .

Original: Namely, eachdraw of ϵ corresponds to a draw from the posterior.
Revised: Namely, eachdraw of ϵ corresponds to a draw from the posterior.

Original: By using this design we get a similar memorycost to a standard single network and we can parallelize easily as all the linear operations are shared.
Revised: By using this design we get a similar memorycost to a standard single network and we can parallelize easily as all the linear operations are shared.

Original: Computationally it is equivalent to running a single model on a batch instead of a single input.
Revised: Computationally it is equivalent to running a single model on a batch instead of a single input.

Original: Anillustration of our method is presented in Figure 1.
Revised: Anillustration of our method is presented in Figure 1.

Original: We further show in the appendix that for a widerange of batch sizes the runtime of our model scales approximately as 1 + (# of ϵ samples ) /
Revised: We further show in the appendix that for a widerange of batch sizes the runtime of our model scales approximately as 1 + (# of ϵ samples ) /

Original: 35resulting in a considerable speedup.
Revised: 35resulting in a considerable speedup.

Original: Since we now have a distribution over functions, we call ourmethod Functional Ensemble Distillation (FED ) .
Revised: Since we now have a distribution over functions, we call ourmethod Functional Ensemble Distillation (FED ) .

Original: Algorithm

Original: 1: input: Predictions of the ensemble on our distillation dataset: { x i , { p ji } j =1: M } i =1: N # N number of data points, M ensemble predictions 2: Initialize Generator g w 3: while not converged do 4:

Original: Sample batch of inputs and model predictions { x i , { p ji } j =1: M } i =1: B 5:

Original: Sample M latent variables - ϵ ∼ N (0 , σ 2 ) 6: q

Original: [Equation]

Original: ( x i , ϵ j ) 7: ˆ

Original: [Equation]

Original: 8: ˆ

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ∇ w L 11: end while 12: return g w

Original: We now present our training objective for the generator g w .
Revised: We now present our training objective for the generator g w .

Original: Given an ensemble member θ j anda batch of B examples, we consider the vectors of predicted probabilities (one per example) asfeatures representing the function parameterized by θ j .
Revised: Given an ensemble member θ j anda batch of B examples, we consider the vectors of predicted probabilities (one per example) asfeatures representing the function parameterized by θ j .

Original: Denote by p ji = p
Revised: Denote by p ji = p

Original: ( y i | x i , θ j ) the predictivelikelihood for the
Revised: ( y i | x i , θ j ) the predictivelikelihood for the

Original: i th example by the j th function.
Revised: i th example by the j th function.

Original: We represent the function parameterized by θ j byconcatenating these vectors to form a feature vector ˆ p j = concat ( p j 1 , ..., p jB ) . 
Revised: We represent the function parameterized by θ j byconcatenating these vectors to form a feature vector ˆ p j = concat ( p j 1 , ..., p jB ) . We repeat this process Algorithm input

Original: We repeat this processfor our generator network (only that now each ϵ i represents a function) and in a similar fashion, weobtain a representation ˆ q j = concat ( q j 1 , ..., q jB ) using the same batch.
Revised: , M ensemble predictions do 4: B 5: Sample M latent variables - ϵ ∼ N (0 , σ 2 ) 6: [Equation]

Revised: 7: ˆ

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: w − ∇ w L 11: end while 12: return g w for our generator network (only that now each ϵ i represents a function) and in a similar fashion, weobtain a representation ˆ q j = concat ( q j 1 , ..., q jB ) using the same batch.

Original: The same process is appliedto every member of the ensemble.
Revised: The same process is appliedto every member of the ensemble.

Original: Now we use the MMD objective presented in Section 2.2 toscore the difference between distribution over functions, using a simple kernel over these functions’representations.
Revised: Now we use the MMD objective presented in Section 2.2 toscore the difference between distribution over functions, using a simple kernel over these functions’representations.

Original: More formally, the MMD objective given a batch of examples is:
Revised: More formally, the MMD objective given a batch of examples is:

Original: [Equation]
Revised: [Equation]

Original: We present our method in Algorithm 1.
Revised: We present our method in Algorithm 1.

Original: In our experiments, we used standard convolutional neuralnetworks where we concatenated 3 random Gaussian channels to the input channels and we add
Revised: In our experiments, we used standard convolutional neuralnetworks where we concatenated 3 random Gaussian channels to the input channels and we add

Original: Gaussian noise to intermediate features across the network.
Revised: Gaussian noise to intermediate features across the network.

Original: We evaluated several options for insertingnoise into the generative model and noticed that to learn sufficient diversity, enough noise hasto be inserted; otherwise, the generator’s diversity suffers and collapses to a nearly deterministicmodel.
Revised: We evaluated several options for insertingnoise into the generative model and noticed that to learn sufficient diversity, enough noise hasto be inserted; otherwise, the generator’s diversity suffers and collapses to a nearly deterministicmodel.

Original: We also found it best to set the noise STD to be a learnable parameter (one parameter perinput noise layer).
Revised: We also found it best to set the noise STD to be a learnable parameter (one parameter perinput noise layer).

Original: See an illustration with ResNet18 in Figure 2.
Revised: See an illustration with ResNet18 in Figure 2.

Original: Regarding the limitations of ourapproach, we note that while the run-time is much faster then running the full ensemble, there is stilla non-negligible overhead compared to a single model.
Revised: Regarding the limitations of ourapproach, we note that while the run-time is much faster then running the full ensemble, there is stilla non-negligible overhead compared to a single model.

Original: 4.1 Auxiliary Dataset
Revised: 4.1 Auxiliary Dataset

Original: One problem with distilling ensemble models is that the predictions of the ensemble members onthe training set have a much higher agreement compared to the test set (as can be seen in Table 1).
Revised: One problem with distilling ensemble models is that the predictions of the ensemble members onthe training set have a much higher agreement compared to the test set (as can be seen in Table 1).

Original: Practically this means that distillation of the ensemble predictions on the training set can lead to adegenerate solution that returns in most cases the same predictions.
Revised: Practically this means that distillation of the ensemble predictions on the training set can lead to adegenerate solution that returns in most cases the same predictions.

Original: This leads to worse performancein both accuracy and calibration compared to the original ensemble.
Revised: This leads to worse performancein both accuracy and calibration compared to the original ensemble.

Original: In [24] the authors address this by using an auxiliary unsupervised dataset, as we can get diverseensemble predictions on this dataset and use it to train our distilled model.
Revised: In [25] the authors address this by using an auxiliary unsupervised dataset, as we can get diverseensemble predictions on this dataset and use it to train our distilled model.

Original: While this solutionperforms quite well, in several applications having access to an additional unsupervised dataset isnot trivial.
Revised: While this solutionperforms quite well, in several applications having access to an additional unsupervised dataset isnot trivial.

Original: One example is medical imaging [7], where even unsupervised data requires access toexpensive machinery and data that is not readily shared due to privacy concerns.
Revised: One example is medical imaging [8], where even unsupervised data requires access toexpensive machinery and data that is not readily shared due to privacy concerns.

Original: Another exampleis learning from ancient texts [10] where digitized data, labeled or unlabeled, is scarce.
Revised: Another exampleis learning from ancient texts [11] where digitized data, labeled or unlabeled, is scarce.

Original: We arguethat these problems in which the data is limited are exactly the problems where Bayesian methodscan make the most impact.
Revised: We arguethat these problems in which the data is limited are exactly the problems where Bayesian methodscan make the most impact.

Original: Therefore, having a method that works on these types of data is of greatimportance.
Revised: Therefore, having a method that works on these types of data is of greatimportance.

Original: This leads us to seek a solution that does not require any additional data, even if it is unsupervised.
Revised: This leads us to seek a solution that does not require any additional data, even if it is unsupervised.

Original: Weinvestigated several solutions (see Section 5.4) and found one simple solution that worked surprisinglywell, the mixup augmentation.
Revised: Weinvestigated several solutions (see Section 5.4) and found one simple solution that worked surprisinglywell, the mixup augmentation.

Original: The predictions on the mixup data are more diverse allowing us todistill from data that more closely resembles the test data.
Revised: The predictions on the mixup data are more diverse allowing us todistill from data that more closely resembles the test data.

Original: Another important property of the mixupaugmentation is that models trained on it transfer well to real images.
Revised: Another important property of the mixup augmentation is that models trained on it transfer well to real images.

Original: This is in contrast to otheralternatives that we explored, such as generating an auxiliary dataset using a generative adversarialnetwork (GAN), that did not transfer well.
Revised: This is in contrast to otheralternatives that we explored, such as generating an auxiliary dataset using a generative adversarialnetwork (GAN), that did not transfer well.

Original: To summarize we generate a synthetic, unlabeled, dataset from the original training set (that was usedto train the original ensemble) using the mixup augmentation.
Revised: To summarize we generate a synthetic, unlabeled, dataset from the original training set (that was usedto train the original ensemble) using the mixup augmentation.

Original: We then train our generator networkusing the predictions of the original ensemble on this dataset, see Algorithm 1.
Revised: We then train our generator networkusing the predictions of the original ensemble on this dataset, see Algorithm 1.

Original: In Table 3, we showthat indeed the ensemble classifier generates a more diverse set of predictions on the mixup datasetcompared to the actual training set.
Revised: In Table 3, we showthat indeed the ensemble classifier generates a more diverse set of predictions on the mixup datasetcompared to the actual training set.

Original:  Experiments
Revised:  Experiments

Original: We evaluated FED against baseline methods on different datasets and using different metrics.
Revised: We evaluated FED against baseline methods on different datasets and using different metrics.

Original: Wepresent here the results of the models using the hyperparameters with the best accuracy on thevalidation set.
Revised: Wepresent here the results of the models using the hyperparameters with the best accuracy on thevalidation set.

Original: As a result, for some models the diversity can be improved at the expense of loweraccuracy.
Revised: As a result, for some models the diversity can be improved at the expense of loweraccuracy.

Original: We will also provide our code for reproducibility.
Revised: We will also provide our code for reproducibility.

Original: Datasets.
Revised: Datasets.

Original: We evaluated all methods on CIFAR-10, CIFAR-100 [20], and STL-10 [6] datasets.
Revised: We evaluated all methods on CIFAR-10, CIFAR-100 [21], and STL-10 [7] datasets.

Original: STL-10 and CIFAR-100 are considered more challenging, the former has a relatively small amountof labeled training data ( 10 classes with 500 labeled examples per class), and the latter has morecategories.
Revised: STL-10 and CIFAR-100 are considered more challenging, the former has a relatively small amountof labeled training data ( 10 classes with 500 labeled examples per class), and the latter has morecategories.

Original: For all datasets we use train/val/test split.
Revised: For all datasets we use train/val/test split.

Original: The train/val split with a ratio 80%:20%.
Revised: The train/val split with a ratio 80%:20%.

Original: Compared methods.
Revised: Compared methods.

Original: We compare our method against the following baselines: (1) EnDD [24],
Revised: We compare our method against the following baselines: (1) EnDD [25],

Original: Ensemble Distribution Distillation framework, trained with the same train data used to capture theensemble.
Revised: Ensemble Distribution Distillation framework, trained with the same train data used to capture theensemble.

Original: (2) EnDD AUX [24], Ensemble Distribution Distillation framework, trained with the traindata and an auxiliary dataset.
Revised: (2) EnDD AUX [25], Ensemble Distribution Distillation framework, trained with the traindata and an auxiliary dataset.

Original: For CIFAR-10, CIFAR-100 is the auxiliary dataset.
Revised: For CIFAR-10, CIFAR-100 is the auxiliary dataset.

Original: For CIFAR-100,
Revised: For CIFAR-100,

Original: CIFAR-10 is the auxiliary dataset.
Revised: CIFAR-10 is the auxiliary dataset.

Original: For STL-10 we used the unlabeled data provided as part of thisdataset.
Revised: For STL-10 we used the unlabeled data provided as part of thisdataset.

Original: We stress that other methods do not have access to this data.
Revised: We stress that other methods do not have access to this data.

Original: (3) DM [27], this baselinecomes in three variations, Gaussian , ODS , and ConfODS .
Revised: (3) DM [28], this baseline comesin three variations, Gaussian , ODS , and ConfODS .

Original: (4) AMT [8], a method which also uses MMD as an objective.
Revised: (4) AMT [9], a method which also uses MMDas an objective.

Original: (5) Hydra [34], a weight sharing method based on [14].
Revised: (5) Hydra [35], a weight sharing method based on [15].

Revised: (6) Knowledge Distillation( KD ), the naive single model baseline, trained on the mean predictions on the mixup data.

Original: Ensemble.
Revised: Ensemble.

Original: We created the Bayesian ensemble (for distillation) by sampling from a posteriorobtained with the Cyclic Stochastic Gradient Hamiltonian Monte Carlo (cSGHMC) method [42].
Revised: We created the Bayesian ensemble (for distillation) by sampling from a posteriorobtained with the Cyclic Stochastic Gradient Hamiltonian Monte Carlo (cSGHMC) method [43].

Original: Overall we sampled 120 models.
Revised: Overall we sampled 120 models.

Original: In addition, we show experimented with different ensemblemethods and their implications on the distillation.
Revised: In addition, we show experimented with different ensemblemethods and their implications on the distillation.

Original: The architecture we choose for the models in theensemble is ResNet18 with Group Norm [39] and Weight Standardization [31].
Revised: The architecture we choose for the models in theensemble is ResNet18 with Group Norm [40] and Weight Standardization [32].

Original: We did not use Batchnormalization since it breaks the IID assumption  used in Bayesian Inference [15].
Revised: We did not use Batchnormalization since it breaks the IID assumption commonly used in Bayesian Inference [16].

Original: We note that sampling from the posterior with cSGHMC, instead of SGD, slightly improves theensemble diversity on the training set.
Revised: We note that sampling from the posterior with cSGHMC, instead of SGD, slightly improves theensemble diversity on the training set.

Original: If our ensemble was sampled using SGD with different seedsthe effect of mixup would be even more pronounced.
Revised: If our ensemble was sampled using SGD with different seedsthe effect of mixup would be even more pronounced.

Original: Training setup.
Revised: Training setup.

Original: For the baselines, we follow the training strategies suggested in the papers andtheir code if available, while performing an extra hyperparameter search when training on datasetswhich did not cover in the original papers, mainly CIFAR-100 and STL-10.
Revised: For the baselines, we follow the training strategies suggested in the papers andtheir code if available, while performing an extra hyperparameter search when training on datasetswhich did not cover in the original papers, mainly CIFAR-100 and STL-10.

Original: For our method, weperformed a standard training procedure with Adam optimizer [17], a learning rate scheduler withfixed milestones at epochs {35, 45, 55, 70, 80}, and a hyperparameter search done over a held-outvalidation set.
Revised: For our method, weperformed a standard training procedure with Adam optimizer [18], a learning rate scheduler withfixed milestones at epochs {35, 45, 55, 70, 80}, and a hyperparameter search done over a held-outvalidation set.

Original: Objective and Kernel choice.
Revised: Objective and Kernel choice.

Original: We trained our model using the MMD objective while consideringvarious kernels.
Revised: We trained our model using the MMD objective while consideringvarious kernels.

Original: Specifically, we examined Linear kernel, RBF kernels, and a mixture of RBFkernels.
Revised: Specifically, we examined Linear kernel, RBF kernels, and a mixture of RBFkernels.

Original: For the RBF kernel, we considered different values of length scales, as we saw that it hada considerable effect on the results.
Revised: For the RBF kernel, we considered different values of length scales, as we saw that it hada considerable effect on the results.

Original: For CIFAR-10, a mixture of RBF kernels with { 2 , 10 , 20 , 50 }length scales had the best results.
Revised: For CIFAR-10, a mixture of RBF kernels with { 2 , 10 , 20 , 50 }length scales had the best results.

Original: For CIFAR-100, the length scales are { 10 , 15 , 20 , 50 } , and for
Revised: For CIFAR-100, the length scales are { 10 , 15 , 20 , 50 } , and for

Original: STL-10 a length scale of 50 works best.
Revised: STL-10 a length scale of 50 works best.

Original: Distilled Model.
Revised: Distilled Model.

Original: We distilled using a similar ResNet18 with Group Norm and Weight Standardizationarchitecture as the ensemble models.
Revised: We distilled using a similar ResNet18 with Group Norm and Weight Standardizationarchitecture as the ensemble models.

Original: Thus, essentially trying to distill a 120 models ensemble into [Figure] a single model that has almost the same size as a member in the ensemble, i.e a ∼ 100 × memoryimprovement.
Revised: Thus, essentially trying to distill a 120 models ensemble intoa single model that has almost the same size as a member in the ensemble, i.e a ∼ 100 × memoryimprovement.

Original: Memory and Timing performance analysis can be found in the Appendix.
Revised: Memory and Timing performance analysis can be found in the Appendix.

Original: For thenoise sampled and fed into the generative model, we chose a simple Gaussian distribution, fed byconcatenating it to the input, and by inserting it in intermediate places along the model.
Revised: For thenoise sampled and fed into the generative model, we chose a simple Gaussian distribution, fed by [Figure] concatenating it to the input, and by inserting it in intermediate places along the model.

Original: Specifically,for the concatenation part, 3 channels of noise and 3 channels of the input are stacked together.
Revised: Specifically,for the concatenation part, 3 channels of noise and 3 channels of the input are stacked together.

Original: Forthe intermediate noise, the Gaussian noise was added to the features, instead of concatenation, in5 different places, one after the first convolution layer and the other four after each Block in the
Revised: Forthe intermediate noise, the Gaussian noise was added to the features, instead of concatenation, in5 different places, one after the first convolution layer and the other four after each Block in the

Original: ResNet-18 architecture (Figure 2).
Revised: ResNet-18 architecture (Figure 2).

Original: We found that setting the standard deviation of the noise to be alearnable parameter as part of the optimization, has a slight performance improvement.
Revised: We found that setting the standard deviation of the noise to be alearnable parameter as part of the optimization, has a slight performance improvement.

Original: Metrics.
Revised: Metrics.

Original: To evaluate our method, and other baselines, we define the agreement metric as theprobability that two distinct elements of our ensemble predict the same label on the same input.
Revised: To evaluate our method, and other baselines, we define the agreement metric as theprobability that two distinct elements of our ensemble predict the same label on the same input.

Original: Inaddition, we computed accuracy, and Expected Calibration Error (ECE) [12] reported in percentages.
Revised: Inaddition, we computed accuracy, and Expected Calibration Error (ECE) [13] reported in percentages.

Original: For the agreement metric, a close value to the original ensemble’s agreement is a good indication thatwe capture similar behavior.
Revised: For the agreement metric, a close value to the original ensemble’s agreement is a good indication thatwe capture similar behavior.

Original: 5.1 Ensemble Analysis
Revised: 5.1 Ensemble Analysis

Original: [Table]
Revised: [Table]

Original: We first analyze the ensemble we sampled using cSGHMC.
Revised: We first analyze the ensemble we sampled using cSGHMC.

Original: To evaluate the diversity betweenpredictions we use the agreement metric.
Revised: To evaluate the diversity betweenpredictions we use the agreement metric.

Original: We computed accuracy, agreement and Expected Calibration
Revised: We computed accuracy, agreement and Expected Calibration

Original: Error (ECE) [12, 26] over the trainset and testset.
Revised: Error (ECE) [13, 27] over the trainset and testset.

Original: We only show agreement on our suggested mixupauxiliary dataset, as it does not have discrete labels.
Revised: We only show agreement on our suggested mixupauxiliary dataset, as it does not have discrete labels.

Original: We can see that the ensemble has almost perfectaccuracy and very high agreement on the training set, with a considerable drop in both on the testset.
Revised: We can see that the ensemble has almost perfectaccuracy and very high agreement on the training set, with a considerable drop in both on the testset.

Original: In fact, for CIFAR-100 and STL-10, the ensemble has essentially no diversity over the training set,thus making the trainset inappropriate for the distillation task.
Revised: In fact, for CIFAR-100 and STL-10, the ensemble has essentially no diversity over the training set,thus making the trainset inappropriate for the distillation task.

Original: The mixup agreement closes part ofthe gap between the train and test datasets.
Revised: The mixup agreement closes part ofthe gap between the train and test datasets.

Original: 5.2 Distillation
Revised: 5.2 Distillation

Original: We now evaluate FED on learning multi-class classification tasks.
Revised: We now evaluate FED on learning multi-class classification tasks.

Original: While we are not limited toclassification tasks, In the following experiment we compare our approach to the baselines describedabove.
Revised: While we are not limited toclassification tasks, In the following experiment we compare our approach to the baselines describedabove.

Original: Results in Table 2 show that FED outperforms previous methods that do not use an auxiliarydataset by a large margin, especially on the more challenging STL-10 and CIFAR-100 datasets.
Revised: Results in Table 2 show that FED outperforms previous methods that do not use an auxiliarydataset by a large margin, especially on the more challenging STL-10 and CIFAR-100 datasets.

Original: Thisis true for both accuracy and ECE.
Revised: Thisis true for both accuracy and ECE.

Original: It is interesting to note that while EnDD AUX uses additionalunlabeled examples from the training distribution (between 50 k and 100 k ), FED is on par and oftenoutperforms this baseline across all metrics.
Revised: It is interesting to note that while EnDD AUX uses additionalunlabeled examples from the training distribution (between 50 k and 100 k ), FED is on par and oftenoutperforms this baseline across all metrics, and is the only one who outperforms the naive singlemodel distillation baseline (KD) on two out of three datasets.

Revised: We also show results for out-of-distribution (OOD) detection in Figure 3 using ROC curves (EnDDwas excluded due to poor performance), To score the examples we used the knowledge uncertainty

Original: [Table]
Revised: [Table]

Original: We also show results for out-of-distribution (OOD) detection in Figure 3 using ROC curves (EnDDwas excluded due to poor performance), To score the examples we used the knowledge uncertaintywhich is obtained by subtracting the aleatoric uncertainty from the total uncertainty [9, 24].
Revised: which is obtained by subtracting the aleatoric uncertainty from the total uncertainty [10, 25].

Original: We used
Revised: We used

Original: SVHN [29] as OOD data.
Revised: SVHN [30] as OOD data.

Original: See appendix for more details.
Revised: See appendix for more details.

Original: From the figure, FED generally outperformsall baselines here as well.
Revised: From the figure, FED generally outperformsall baselines here as well.

Original: 5.3 Mixup Augmentation as Auxiliary Dataset
Revised: 5.3 Mixup Augmentation as Auxiliary Dataset

Original: [Table]
Revised: [Table]

Original: As an ablation study, we would like to understand the separate effects of the mixup augmentationand our FED approach.
Revised: As an ablation study, we would like to understand the separate effects of the mixup augmentation andour FED approach.

Original: Thus, we show results for both FED and EnDD trained on the training setand on the mixup dataset as the auxiliary dataset.
Revised: Thus, we show results for both FED, KD and EnDD trained on the training setand on the mixup dataset as the auxiliary dataset.

Original: Results are detailed in Table 3.
Revised: Results are detailed in Table 3.

Original: We show that forour method the mixup adds a significant boost to performance, and also for EnDD  on two out ofthree datasets.
Revised: We show that forour method the mixup adds a significant boost to performance, and also for EnDD and KD on STL10.

Original: We also see that even when EnDD is trained with mixup, our approach returns betteraccuracy, ECE and also has greater diversity.
Revised: We also see that even when EnDD is trained with mixup, our approach returns better accuracy, ECEand also has greater diversity.

Original: We suspect that this is due to the limited expressivepower of the Dirichlet distribution.
Revised: We suspect that this is due to the limited expressive power of the Dirichlet distribution.

Original: Additional experiments in the Appendix show that at least partofthe lack of diversity can be attributed to the Dirichlet distribution.
Revised: Additional experiments in the Appendix show that at least part of the lack ofdiversity can be attributed to the Dirichlet distribution.

Original: 5.4 Alternatives for Learning Distilled Model Without Auxiliary Dataset
Revised: 5.4 Alternatives for Learning Distilled Model Without Auxiliary Dataset

Original: In this section, we explore an additional attempt to distill the ensemble, this time without accessto additional data.
Revised: In this section, we explore an additional attempt to distill the ensemble, this time without accessto additional data.

Original: This approach was inspired by the Out-Of-Bag (OOB) estimate in bagging.
Revised: This approach was inspired by the Out-Of-Bag (OOB) estimate in bagging.

Original: Inbagging each model is trained only on a random subset of the training set and the OOB estimateis done by averaging the predictions of all models on those data points that they were not trainedon and returning their score (which we refer to as held-out datasets).
Revised: Inbagging each model is trained only on a random subset of the training set and the OOB estimateis done by averaging the predictions of all models on those data points that they were not trainedon and returning their score (which we refer to as held-out datasets).

Original: Using this approach, we cantrain each member in the ensemble on part of the dataset, and use for distillation only the ensemblemembers’ predictions on the data points that they were not trained on.
Revised: Using this approach, we cantrain each member in the ensemble on part of the dataset, and use for distillation only the ensemblemembers’ predictions on the data points that they were not trained on.

Original: As an example, consider asingle datapoint x which is on the held-out dataset for model m 1 , but is part of the trainset for modelm 2 .
Revised: As an example, consider asingle datapoint x which is on the held-out dataset for model m 1 , but is part of the trainset for modelm 2 .

Original: To train the distillation model, the ensemble uses only the predictions of model m 1 , but does notuse those of model m 2 to get the overall output for x .
Revised: To train the distillation model, the ensemble uses only the predictions of model m 1 , but does notuse those of model m 2 to get the overall output for x .

Original: We experimented with two different partitionsfor held-out datasets: K-fold and bagging.
Revised: We experimented with two different partitionsfor held-out datasets: K-fold and bagging.

Original: In K-Fold we first split the data into K = 10 folds, D 1 , ..., D 10 .
Revised: In K-Fold we first split the data into K = 10 folds, D 1 , ..., D 10 .

Original: For each of the 10 folds we define
Revised: For each of the 10 folds we define

Original: D − j = concat ( {D i } i ∈{ 1 ,..., 10 } ,i ̸ = j ) .
Revised: D − j = concat ( {D i } i ∈{ 1 ,..., 10 } ,i ̸ = j ) .

Original: We then sample 12 models using cSGHMC on each D − j 311 dataset.
Revised: We then sample 12 models using cSGHMC on each D − j 309 dataset.

Original: As before we get 120 models and for each fold D j we can use the held-out models, i.e.models trained on D − j , for distillation.
Revised: As before we get 120 models and for each fold D j we can use the held-out models, i.e.models trained on D − j , for distillation.

Original: Note that the dataset for distillation is the entire trainset, butfor each data-point, we use only the predictions of 12 models that did not train on it.
Revised: Note that the dataset for distillation is the entire trainset, butfor each data-point, we use only the predictions of 12 models that did not train on it.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: In bagging each bag samples N data points with repetition from a dataset of size N. We explore twoalternatives, one is sampling 120 bags and sampling for each bag a single model using cSGHMC(named 120 Bagging), and the other is to sample 10 bags and from each to sample 12 models usingcSGHMC (named 10 Bagging).
Revised: In bagging each bag samples N data points with repetition from a dataset of size N. We explore twoalternatives, one is sampling 120 bags and sampling for each bag a single model using cSGHMC(named 120 Bagging), and the other is to sample 10 bags and from each to sample 12 models usingcSGHMC (named 10 Bagging).

Original: The held-out dataset for distillation is constructed in the samefashion as described in the K-Fold alternative.
Revised: The held-out dataset for distillation is constructed in the samefashion as described in the K-Fold alternative.

Original: The key difference is that for each data point, there isa different number of models that did not use it in training, due to the stochastic partitioning of bags.
Revised: The key difference is that for each data point, there isa different number of models that did not use it in training, due to the stochastic partitioning of bags.

Original: Comparing these two approaches the K-fold has an advantage since each ensemble member wastrained on exactly 90% of the training data while each ensemble member in bagging is trained on∼ 66% .
Revised: Comparing these two approaches the K-fold has an advantage since each ensemble member wastrained on exactly 90% of the training data while each ensemble member in bagging is trained on∼ 66% .

Original: Bagging (with 120 bags) however has better diversity as when we distill we see a largervariety of ensemble model combinations.
Revised: Bagging (with 120 bags) however has better diversity as when we distill we see a largervariety of ensemble model combinations.

Original: Table 4 shows results for the Bayesian ensemble used throughout this paper and the method describedin this section.
Revised: Table 4 shows results for the Bayesian ensemble used throughout this paper and the method describedin this section.

Original: As one can see the scores on the held-out datasets, is a good proxy for the test in termsof calibration, agreement, and accuracy (the lower score is due to only averaging a small number ofensemble models), but this comes in many cases at a price of reduced accuracy of the ensemble onthe test set compared to the Bayesian ensemble.
Revised: As one can see the scores on the held-out datasets, is a good proxy for the test in termsof calibration, agreement, and accuracy (the lower score is due to only averaging a small number ofensemble models), but this comes in many cases at a price of reduced accuracy of the ensemble onthe test set compared to the Bayesian ensemble.

Original: Next, we distill the new ensembles using FED.
Revised: Next, we distill the new ensembles using FED.

Original: We show the results of the distilled models using theheld-out data, and compare them to our Bayesian ensemble distilled using the mixup dataset.
Revised: We show the results of the distilled models using theheld-out data, and compare them to our Bayesian ensemble distilled using the mixup dataset.

Original: As onecan see from our results in Table 5, the Bayesian ensemble distilled using the mixup data is superior.
Revised: As onecan see from our results in Table 5, the Bayesian ensemble distilled using the mixup data is superior.

Original: We note that on all ensembles the best distillation results we achieved were by using only mixup,further proving the value of this augmentation scheme.
Revised: We note that on all ensembles the best distillation results we achieved were by using only mixup,further proving the value of this augmentation scheme.

Revised: We suspect that distillation works best on the

Revised: Bayesian ensemble as we have the most diverse examples and labels to distill from.

Original:  Conclusions
Revised:  Conclusions

Original: In this work, we present FED, a novel and general ensemble distillation approach.
Revised: In this work, we present FED, a novel and general ensemble distillation approach.

Original: We show that thisapproach can distill ensembles while keeping desirable qualities such as accuracy, calibration, anddiversity.
Revised: We show that thisapproach can distill ensembles while keeping desirable qualities such as accuracy, calibration, anddiversity.

Original: This is done by maintaining the Bayesian approach of distribution over functions whilemimicking the posterior with one that can be evaluated efficiently.
Revised: This is done by maintaining the Bayesian approach of distribution over functions whilemimicking the posterior with one that can be evaluated efficiently.

Original: Additionally, we show how theneed for an auxiliary distillation dataset can be easily solved by using mix-up augmentation.
Revised: Additionally, we show how theneed for an auxiliary distillation dataset can be easily solved by using mix-up augmentation.

Original: Finally,our experiments show that FED achieves state-of-the-art results on various benchmarks in the domainof Ensemble distribution distillation.
Revised: Finally,our experiments show that FED achieves state-of-the-art results on various benchmarks in the domainof Ensemble distribution distillation.

Original: Our work doesn’t bare negative societal impacts directly butcan be exploited for negative impact by the practitioner.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See section
Revised: [Yes] See section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See section
Revised: [Yes] See section6(d)

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] code includedas a URL.
Revised: [Yes] code included as a URL.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Full details are in the supplemental material.
Revised: [Yes] Full details are in the supplemental material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running exper-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running exper-

Original: iments multiple times)?
Revised: iments multiple times)?

Original: [Yes] Experiments comaring our method to other baselines,
Revised: [Yes] Experiments comaring our method to other baselines,

Original: done with 3 seeds, thus mean and std values presented in
Revised: done with 3 seeds, thus mean and std values presented in

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] Provided in the supplementary.
Revised: [Yes] Provided in the supplementary.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]