E NSEMBLE D ISTRIBUTION D ISTILLATION
A BSTRACT
Ensembles of models often yield improvements in system performance.
These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different forms of uncertainty.
However, ensembles come at a computational and memory cost which may be prohibitive for many applications.
There has been signiﬁcant work done on the distillation of an ensemble into a single model.
Such approaches decrease computational cost and allow a single model to achieve an accuracy comparable to that of an ensemble.
However, information about the diversity of the ensemble, which can yield estimates of different forms of uncertainty, is lost.
This work considers the novel task of Ensemble Distribution Distillation (EnD 2 ) — distilling the distribution of the predictions from an ensemble, rather than just the average prediction, into a single model.
EnD 2 enables a single model to retain both the improved classiﬁcation performance of ensemble distillation as well as information about the diversity of the ensemble, which is useful for uncertainty estimation.
A solution for EnD 2 based on Prior Networks, a class of models which allow a single neural network to explicitly model a distribution over output distributions, is proposed in this work.
The properties of EnD 2 are investigated on both an artiﬁcial dataset, and on the CIFAR-10, CIFAR-100 and TinyImageNet datasets, where it is shown that EnD 2 can approach the classiﬁcation performance of an ensemble, and outperforms both standard DNNs and Ensemble Distillation on the tasks of misclassiﬁcation and out-of-distribution input detection.
I NTRODUCTION
Neural Networks (NNs) have emerged as the state-of-the-art approach to a variety of machine learning tasks (LeCun et al., 2015) in domains such as computer vision (Girshick, 2015; Simonyan & Zisserman, 2015;
Villegas et al., 2017), natural language processing (Mikolov et al., 2013b;a; 2010), speech recognition (Hinton et al., 2012; Hannun et al., 2014) and bio-informatics (Caruana et al., 2015; Alipanahi et al., 2015).
Despite impressive supervised learning performance, NNs tend to make over-conﬁdent predictions (Lakshminarayanan et al., 2017) and, until recently, have been unable to provide measures of uncertainty in their predictions.
As NNs are increasingly being applied to safety-critical tasks such as medical diagnosis (De Fauw et al., 2018), biometric identiﬁcation (Schroff et al., 2015) and self driving cars, estimating uncertainty in model’s predictions is crucial, as it enables the safety of an AI system (Amodei et al., 2016) to be improved by acting on the predictions in an informed manner.
Ensembles of NNs are known to yield increased accuracy over a single model (Murphy, 2012), allow useful measures of uncertainty to be derived (Lakshminarayanan et al., 2017), and also provide defense against adversarial attacks (Smith & Gal, 2018).
There is both a range of Bayesian MonteCarlo approaches (Gal & Ghahramani, 2016; Welling & Teh, 2011; Garipov et al., 2018; Maddox et al., 2019), as well as non-Bayesian approaches, such as random-initialization (Lakshminarayanan et al., 2017) and bagging (Murphy, 2012;
Osband et al., 2016), to generating ensembles.
Crucially, ensemble approaches allow total uncertainty in predictions to be decomposed into knowledge uncertainty and data uncertainty .
Data uncertainty is the irreducible uncertainty in predictions which arises due to the complexity, multi-modality and noise in the data.
Knowledge uncertainty , also known as epistemic uncertainty (Gal, 2016) or distributional uncertainty (Malinin & Gales, 2018), is uncertainty due to a lack of understanding or knowledge on the part of the model regarding the current input for which the model is making a prediction.
This form of uncertainty arises when the test input x ∗ comes either from a different distribution than the one that generated the training data or from an in-domain region which is sparsely covered by the training data.
Mismatch between the test and training distributions is also known as a dataset shift (Quiñonero-Candela, 2009), and is a situation which often arises for real world problems.
Distinguishing between sources of uncertainty is important, as in certain machine learning applications it may be necessary to know not only whether the model is uncertain, but also why .
For instance, in active learning, additional training data should be collected from regions with high knowledge uncertainty , but not data uncertainty .
A fundamental limitation of ensembles is that the computational cost of training and, more importantly, inference can be many times greater than that of a single model.
One solution to speed up inference is to distill an ensemble of models into a single network to yield the mean predictions of the ensemble (Hinton et al., 2015; Korattikara Balan et al., 2015).
However, this collapses an ensemble of conditional distributions over classes into a single point-estimate conditional distribution over classes.
As a result, information about the diversity of the ensemble is lost.
This prevents measures of knowledge uncertainty , such as mutual information (Malinin & Gales, 2018; Depeweg et al., 2017a), from being estimated.
In this work, we investigate the explicit modelling of the distribution over the ensemble predictions, rather than just the mean, with a single model.
This problem — referred to as Ensemble Distribution
Distillation (EnD 2 ) — yields a method that preserves both the distributional information and improved classiﬁcation performance of an ensemble within a single neural network model.
It is important to highlight that Ensemble Distribution Distillation is a novel task which, to our knowledge, has not been previously investigated.
Here, the goal is to extract as much information as possible from an ensemble of models and retain it within a single, possibly simpler, model.
As an initial solution to this problem, this paper makes use of a recently introduced class of models, known as Prior Networks (Malinin & Gales, 2018; 2019), which explicitly model a conditional distribution over categorical distributions by parameterizing a Dirichlet distribution .
Within the context of EnD 2 this effectively allows a single model to emulate the complete ensemble.
The contributions of this work are as follows.
Firstly, we deﬁne the task of Ensemble Distribution
Distillation (EnD 2 ) as a new challenge for machine learning research.
Secondly, we propose and evaluate a solution to this problem using Prior Networks.
EnD 2 is initially investigated on artiﬁcial data, which allows the behaviour of the models to be visualized.
It is shown that distributiondistilled models are able to distinguish between data uncertainty and knowledge uncertainty .
Finally,
EnD 2 is evaluated on CIFAR-10, CIFAR-100 and TinyImageNet datasets, where it is shown that
EnD 2 yields models which approach the classiﬁcation performance of the original ensemble and outperform standard DNNs and regular Ensemble Distillation (EnD) models on the tasks of identifying misclassiﬁcations and out-of-distribution (OOD) samples.
E NSEMBLES
In this work, a Bayesian viewpoint on ensembles is adopted, as it provides a particularly elegant probabilistic framework, which allows knowledge uncertainty to be linked to Bayesian model uncertainty .
However, it is also possible to construct ensembles using a range of non-Bayesian approaches.
For example, it is possible to explicitly construct an ensemble of M models by training on the same data with different random seeds (Lakshminarayanan et al., 2017) and/or different model architectures.
Alternatively, it is possible to generate ensembles via Bootstrap methods (Murphy, 2012;
Osband et al., 2016) in which each model is trained on a re-sampled version of the training data.
The essence of Bayesian methods is to treat the model parameters θ as random variables and place a prior distribution p ( θ ) over them to compute a posterior distribution p ( θ |D ) via Bayes’ rule:
[Equation]
Here, model uncertainty is captured in the posterior distribution p ( θ |D ) .
Consider an ensemble of models { P ( y | x ∗ , θ ( m ) )
Mm =1 sampled from the posterior:
[Equation]
[Figure] where π are the parameters of a categorical distribution [ P ( y = ω 1 ) , · · · , P ( y = ω K )
] T and x ∗ is a test input .
The expected predictive distribution, or predictive posterior , for a test input x ∗ is obtained by taking the expectation with respect to the model posterior:
[Equation]
Each of the models P ( y | x ∗ , θ ( m ) ) yields a different estimate of data uncertainty .
Uncertainty in predictions due to model uncertainty is expressed as the level of spread, or ‘disagreement’, of an ensemble sampled from the posterior.
The aim is to craft a posterior p ( θ |D ) , via an appropriate choice of prior p ( θ ) , which yields an ensemble that exhibits the set of behaviours described in ﬁgure 1.
Speciﬁcally, for an in-domain test input x ∗ , the ensemble should produce a consistent set of predictions with little spread, as described in ﬁgure 1a and ﬁgure 1b.
In other words, the models should agree in their estimates of data uncertainty .
On the other hand, for inputs which are different from the training data, the models in the ensemble should ‘disagree’ and produce a diverse set of predictions, as shown in ﬁgure 1c.
Ideally, the models should yield increasingly diverse predictions as input x ∗ moves further away from the training data.
If an input is completely unlike the training data, then the level of disagreement should be signiﬁcant.
Hence, the measures of model uncertainty will capture knowledge uncertainty given an appropriate choice of prior.
Given an ensemble (cid:8) P ( y | x ∗ , θ ( m ) ) (cid:9) M m =1 which exhibits the desired set of behaviours, the entropy of the expected distribution P ( y | x ∗ , D ) can be used as a measure of total uncertainty in the prediction.
Uncertainty in predictions due to knowledge uncertainty can be assessed via measures of the spread, or ‘disagreement’, of the ensemble such as Mutual Information : [Equation] Data Uncertainty
[Equation]
This formulation of mutual information allows the total uncertainty to be decomposed into knowledge uncertainty and expected data uncertainty (Depeweg et al., 2017a;b).
The entropy of the predictive posterior, or total uncertainty , will be high whenever the model is uncertain - both in regions of severe class overlap and out-of-domain.
However, the difference of the entropy of the predictive posterior and the expected entropy of the individual models will be non-zero only if the models disagree.
For example, in regions of class overlap, each member of the ensemble will yield a high entropy distribution (ﬁgure 1b) - the entropy of the predictive posterior and the expected entropy will be similar and mutual information will be low.
In this situation total uncertainty is dominated by data uncertainty .
On the other hand, for out-of-domain inputs the ensemble yields diverse distributions over classes such that the predictive posterior is near uniform (ﬁgure 1c), while the expected entropy of each model may be much lower.
In this region of input space the models’ understanding of data is low and, therefore, knowledge uncertainty is high.
E NSEMBLE D ISTRIBUTION D ISTILLATION
Previous work (Hinton et al., 2015; Korattikara Balan et al., 2015; Wong & Gales, 2017; Wang et al., 2018; Papamakarios, 2015; Buciluˇa et al., 2006) has investigated distilling a single large network into a smaller one and an ensemble of networks into a single neural network.
In general, distillation is done by minimizing the KL-divergence between the model and the expected predictive distribution of an ensemble:
[Equation]
This approach essentially aims to train a single model that captures the mean of an ensemble, allowing the model to achieve a higher classiﬁcation performance at a far lower computational cost.
The use of such models for uncertainty estimation was investigated in (Li & Hoiem, 2019; Englesson &
Azizpour, 2019).
However, the limitation of this approach with regards to uncertainty estimation is that the information about the diversity of the ensemble is lost.
As a result, it is no longer possible to decompose total uncertainty into knowledge uncertainty and data uncertainty via mutual information as in equation 4.
In this work we propose the task of Ensemble Distribution Distillation , where the goal is to capture not only the mean of the ensemble, but also its diversity.
In this section, we outline an initial solution to this task.
An ensemble can be viewed as a set of samples from an implicit distribution of output distributions:
[Equation]
Recently, a new class of models was proposed, called Prior Networks (Malinin & Gales, 2018; 2019), which explicitly parameterize a conditional distribution over output distributions p ( π | x ∗ ; ˆ φ ) using a single neural network parameterized by a point estimate of the model parameters ˆ φ .
Thus, a
Prior Network is able to effectively emulate an ensemble, and therefore yield the same measures of uncertainty.
A Prior Network p ( π | x ∗ ; ˆ φ ) models a distribution over categorical output distributions by parameterizing the Dirichlet distribution.
[Equation]
The distribution is parameterized by its concentration parameters α , which can be obtained by placing an exponential function at the output of a Prior Network: ˆ α c =
e ˆ z c , where ˆ z are the logits predicted by the model.
While a Prior Network could, in general, parameterize arbitrary distributions over categorical distributions, the Dirichlet is chosen due to its tractable analytic properties, which allow closed form expressions for all measures of uncertainty to be obtained.
However, it is important to note that the Dirichlet distribution may be too limited to fully capture the behaviour of an ensemble and other distributions may need to be considered.
In this work we consider how an ensemble, which is a set of samples from an implicit distribution over distributions, can be distribution distilled into an explicit distribution over distributions modelled using a single Prior Network model, ie: (cid:8) P ( y | x ; θ ( m ) )
(cid:9) M m =1 → p ( π | x ; ˆ φ ) .
This is accomplished in several steps.
Firstly, a transfer dataset D ens
= { x ( i ) , π ( i, 1: M ) }
Ni =1 ∼ ^p ( x , π ) is composed of the inputs x i from the original training set D
= { x ( i ) , y ( i ) } Ni =1 and the categorical distributions { π ( i, 1: M ) }
Ni =1 derived from the ensemble for each input.
Secondly, given this transfer set, the model p ( π | x ; φ ) is trained by minimizing the negative log-likelihood of each categorical distribution π
( im ) :
[Equation]
Thus, Ensemble Distribution Distillation with Prior Networks is a straightforward application of maximum-likelihood estimation.
Given a distribution-distilled Prior Network, the predictive distribution is given by the expected categorical distribution ˆ π under the Dirichlet prior:
[Equation]
Separable measures of uncertainty can be obtained by considering the mutual information between the prediction y and the parameters of π of the categorical:
[Equation]
Similar to equation 4, this expression allows total uncertainty , given by the entropy of the expected distribution, to be decomposed into data uncertainty and knowledge uncertainty (Malinin & Gales, 2018).
If Ensemble Distribution Distillation is successful, then the measures of uncertainty derivable from a distribution-distilled model should be identical to those derived from the original ensemble [Equation]
3. T EMPERATURE A NNEALING
Minimization of the negative log-likelihood of the model on the transfer dataset D ens =
{ x ( i ) , π ( i, 1: M ) } Ni =1 is equivalent to minimization of the KL-divergence between the model and the empirical distribution ^p
( x , π ) .
On training data, this distribution is often ‘sharp’ at one of the corners of the simplex.
At the same time, the Dirichlet distribution predicted by the model has its mode near the center of the simplex with little support at the corners at initialization.
Thus, the common support between the model and the target empirical distribution is limited.
Optimization of the
KL-divergence between distributions with limited non-zero common support is particularly difﬁcult.
To alleviate this issue, and improve convergence, the proposed solution is to use temperature to ‘heat up’ both distributions and increase common support by moving the modes of both distributions closer together.
The empirical distribution is ‘heated up’ by raising the temperature T of the softmax of each model in the ensemble in the same way as in (Hinton et al., 2015).
This moves the predictions of the ensemble closer to the center of the simplex and decreases their diversity, making it better modelled by a sharp Dirichlet distribution.
The output distribution of the EnD 2 model p ( π | x ; φ ) is heated up by raising the temperature of the concentration parameters:
ˆ α c = e ˆ z c /T , making support more uniform across the simplex.
An annealing schedule is used to re-emphasize the diversity of the empirical distribution and return it to its ‘natural’ state by lowering the temperature down to 1 as training progresses.
E XPERIMENTS ON A RTIFICIAL D ATA
[Figure]
The current section investigates Ensemble Distribution Distillation (EnD 2 ) on an artiﬁcial dataset shown in ﬁgure 2a.
This dataset consists of three spiral arms extending from the center with both increasing noise and distance between the arms.
Each arm corresponds to a single class.
This dataset is chosen such that it is not linearly separable and requires a powerful model to correctly model the decision boundaries, and also such that there are deﬁnite regions of class overlap.
In the following set of experiments, an ensemble of 100 neural networks is constructed by training neural networks from 100 different random initializations.
A smaller (sub) ensemble of only 10 neural networks is also considered.
The models are trained on 3000 data-points sampled from the spiral dataset, with 1000 examples per class.
The classiﬁcation performance of EnD 2 is compared to the performance of individual neural networks, the overall ensemble and regular Ensemble Distillation (EnD).
The results are presented in table 1.
[Table]
The results show that an ensemble of 10 models has a clear performance gain compared to the mean performance of the individual models.
An ensemble of 100 models has a smaller performance gain over an ensemble of only 10 models.
Ensemble Distillation (EnD) is able to recover the classiﬁcation performance of both an ensemble of 10 and 100 models with only very minor degradation in performance.
Finally, Ensemble Distribution Distillation is also able to recover most of the performance gain of an ensemble, but with a slightly larger degradation.
This is likely due to forcing a single model to learn not only the mean, but also the distribution around it, which likely requires more capacity from the network.
The measures of uncertainty derived form an ensemble of 100 models and
[Figure]
from Ensemble Distribution Distillation are presented in ﬁgures 3a-c and ﬁgures 3d-f, respectively.
The results show that EnD 2 successfully captures data uncertainty and also correctly decomposes total uncertainty into knowledge uncertainty and data uncertainty .
However, it fails to appropriately capture knowledge uncertainty further away from the training region, as there are obvious dark holes in ﬁgure 3f, where the model yields low knowledge uncertainty far from the region of training data.
In order to overcome these issues, a thick ring of inputs far from the training data was sampled as depicted in ﬁgure 2b.
The predictions of the ensemble were obtained for these input points and used as additional auxiliary training data D AUX .
Table 2 shows how using the auxiliary training data affects the performance of the Ensemble Distillation and Ensemble Distribution Distillation.
There is a minor drop in performance of both distillation approaches.
However, the overall level of performance is not compromised and is still higher than the average performance of each individual DNN model.
The behaviour of measures of uncertainty derived from Ensemble Distribution Distillation with auxiliary training data (EnD 2+ AUX ) is shown in ﬁgures 3g-i.
These results show that successful
Ensemble Distribution Distillation of the out-of-distribution behaviour of an ensemble based purely on observations of the in-domain behaviour is challenging and may require the use of additional training data.
This is compounded by the fact that the diversity of an ensemble on training data that the model has seen is typically smaller than on a heldout test-set.
[Table]
E XPERIMENTS ON I MAGE D ATA
Having conﬁrmed the properties of EnD 2 on an artiﬁcial dataset, we now investigate Ensemble
Distribution Distillation on the CIFAR-10 (C10), CIFAR-100 (C100) and TinyImageNet (TIM) (Krizhevsky, 2009; CS231N, 2017) datasets.
Similarly to section 4, an ensemble of a 100 models is constructed by training NNs on C10/100/TIM data from different random initializations.
The transfer dataset is constructed from C10/100/TIM inputs and ensemble logits to allow for temperature annealing during training, which we found to be essential to getting EnD 2 to train well.
In addition, we also consider Ensemble Distillation and Ensemble Distribution Distillation on a transfer set that contains both the original C10/C100/TIM training data and auxiliary (AUX) data taken from the other dataset 1 , termed EnD + AUX and EnD 2+ AUX respectively.
It is important to note that the auxiliary data has been treated in the same way as main data during construction of the transfer set and distillation.
This offers an advantage over traditional Prior Network training (Malinin & Gales, 2018; 2019), where the knowledge of which examples are in-domain and out-of-distribution is required a-priori.
In this work we also make a comparison with Prior Networks trained via reverse KL-divergence (Malinin & Gales, 2019), where the Prior Networks (PN) are trained on the same datasets, both main and auxiliary, as the
EnD + AUX and EnD
+ AUX models 2 .
Note, that for Ensemble Distribution Distillation, models can be distribution-distilled using any (potentially unlabeled) auxiliary data on which ensemble predictions can be obtained.
Further note that in these experiments we explicitly chose to use the simpler VGG- (Simonyan & Zisserman, 2015) architecture rather than more modern architectures like ResNet (He et al., 2016) as the goal of this work is to analyse the properties of Ensemble Distribution Distillation in a clean and simple conﬁguration.
The datasets and training conﬁgurations for all models are detailed in appendix A.
[Table]
[Table]
Firstly, we investigate the ability of a single model to retain the ensemble’s classiﬁcation and prediction-rejection (misclassiﬁcation detection) performance after either Ensemble Distillation (EnD) or Ensemble Distribution Distillation (EnD 2 ), with results presented in table 3 in terms of error rate and prediction rejection ratio (PRR), respectively.
A higher PRR indicates that the model is able to better detect and reject incorrect predictions based on measures of uncertainty.
This metric is detailed in appendix B. Note that for prediction rejection we used the conﬁdence of the max class, which also is a measure of total uncertainty , like entropy, but is more sensitive to the prediction (Malinin & Gales, 2018).
Table 3 shows that both EnD and EnD 2 are able to retain both the improved classiﬁcation and prediction-rejection performance of the ensemble relative to individual models trained with maximum likelihood on all datasets, both with and without auxiliary training data.
Note, that on C100 and
TIM, EnD 2 yields marginally better classiﬁcation performance than EnD. Furthermore, EnD 2 also either consistently outperforms or matches EnD in terms of PRR on all datasets, both with and without auxiliary training data.
This suggests that EnD 2 is able to yield beneﬁts on top of standard
Ensemble Distillation due to retaining information about the diversity of the ensemble.
In comparison, a Prior Network, while yielding a classiﬁcation performance between that on an individual model and
EnD 2+ AUX , performs consistently worse than all models in terms of PRR and for all datasets.
This is likely because of the inappropriate choice of auxiliary training data and auxiliary loss weight (Malinin & Gales, 2019) for this task.
Secondly, it is known that ensembles yield improvements in the calibration of a model’s predictions (Lakshminarayanan et al., 2017; Ovadia et al., 2019).
Thus, it is interesting to see whether EnD and EnD 2 models retain these improvements in terms of test-set negative log-likelihood (NLL) and expected calibration error (ECE).
Note that calibration assesses uncertainty quality on a per-dataset , rather than per-prediction , level.
From table 3 we can see that both Ensemble Distillation and
Ensemble Distribution Distillation seem to give similarly minor gains in NLL over a single model.
However, EnD seems to have marginally better NLL performance, while EnD 2 tends to yield better calibration performance.
There are seemingly limited gains in ECE and NLL when using auxiliary data during distillation for EnD 2 , and sometimes even a degradation in NLL and ECE.
This may be due to the Dirichlet output distribution attempting to capture non-Dirichlet-distributed ensemble predictions (especially on auxiliary data) and over-estimating the support, and thereby failing to fully reproduce the calibration of the original ensemble.
Furthermore, metrics like ECE and NLL are evaluated on in-domain data, which would explain the lack of improvement from distilling ensemble behaviour on auxiliary data.
However, all distillation models are (almost) always calibrated as better than individual models.
At the same time, the Prior Network models yield signiﬁcantly worse calibration performance in terms of NLL and ECE than all other models.
For the CIFAR-100 and
TIM datasets this may be due to the target being a ﬂat Dirichlet distribution for the auxiliary data (CIFAR-10), which drives the model to be very under-conﬁdent.
[Table]
Thirdly, Ensemble Distribution Distillation is investigated on the task of out-of-domain (OOD) input detection (Hendrycks & Gimpel, 2016), where measures of uncertainty are used to classify inputs as either in-domain (ID) or OOD.
The ID examples are the test set of C10/100/TIM, and the test OOD examples are chosen to be the test sets of LSUN (Yu et al., 2015), C100 or TIM, such that the test
OOD data is never seen by the model during training.
Table 4 shows that the measures of uncertainty derived from the ensemble outperform those from a single neural network.
Curiously, standard ensemble distillation (EnD) clearly fails to capture those gains on C10 and TIM, both with and without auxiliary training data, but does reach a comparable level of performance to the ensemble on
C100.
Curiously, EnD + AUX performs worse than the individual models on C10.
This is in contrast to results from (Englesson & Azizpour, 2019; Li & Hoiem, 2019), but the setups, datasets and evaluation considered there are very different to ours.
On the other hand, Ensemble Distribution Distillation is generally able to reproduce the OOD detection performance of the ensemble.
When auxiliary training data is used, EnD 2 is able to perform on par with the ensemble, indicating that it has successfully learned how the distribution of the ensemble behaves on unfamiliar data.
These results suggest that not only is EnD 2 able to preserve information about the diversity of an ensemble, unlike standard
EnD, but that this information is important to achieving good OOD detection performance.
Notably, on the CIFAR-10 dataset PNs yield the best performance.
However, due to generally inappropriate choice of OOD (auxiliary) training data, PNs show inferior (with one exception) OOD detection performance on CIFAR-100 and TIM.
Curiously, the ensemble sometimes displays better OOD detection performance using measures of total uncertainty .
This is partly a property of the in-domain dataset - if it contains a small amount of data uncertainty, then OOD detection performance using total uncertainty and knowledge uncertainty should be almost the same (Malinin & Gales, 2018; Malinin, 2019).
Unlike the toy dataset considered in the previous section, where signiﬁcant data uncertainty was added, the image datasets considered here do have a low degree of data uncertainty.
It is important to note that on more challenging tasks, which naturally exhibit a higher level of data uncertainty, we would expect that the decomposition would be more beneﬁcial.
Additionally, it is also possible that if different architectures, training regimes and ensembling techniques are considered, an ensemble with a better estimate of knowledge uncertainty can be obtained.
Despite the behaviour of the ensemble, EnD 2 , especially with auxiliary training data, tends to have better OOD detection performance using measures of knowledge uncertainty .
This may be due to the use of a Dirichlet output distribution, which may not be able to fully capture the details of the ensemble’s behaviour.
Furthermore, as EnD 2 is (implicitly) trained by minimizing the forward KL-divergence, which is zero-avoiding (Murphy, 2012), it is likely that the distribution learned by the model is ‘wider’ than the empirical distribution.
This effect is explored in the next subsection and in appendix C.
5. A PPROPRIATENESS OF D IRICHLET DISTRIBUTION
Throughout this work, a Prior Network that parametrizes a Dirichlet was used for distributiondistilling ensembles of models.
However, the output distributions of an ensemble for the same input are not necessarily Dirichlet-distributed, especially in regions where the ensemble is diverse.
In the previous section we saw that EnD 2 models tend to have higher NLL than EnD models, and while
EnD 2 achieves good OOD detection performance, it doesn’t fully replicate the ensemble’s behaviour.
Thus, in this section, we investigate how well a model which parameterizes a Dirichlet distribution is able to capture the exact behaviour of an ensemble of models, both in-domain and out-of-distribution.
Figure 4 shows histograms of total uncertainty , data uncertainty and total uncertainty yielded by an ensemble, EnD 2 and EnD 2+ AUX models trained on the CIFAR-10 dataset.
The top row shows the uncertainty histogram for ID data, and the bottom for test OOD data (a concatenation of LSUN and TIM).
On in-domain data, EnD 2 is seemingly able to emulate the uncertainty metrics of the ensemble well, though does have a longer tail of high-uncertainty examples.
This is expected, as on in-domain examples the ensemble will be highly concentrated around the mean.
This behaviour can be adequately modelled by a Dirichlet.
On the other hand, there is a noticeable mismatch between the ensemble and EnD 2 in the uncertainties they yield on OOD data.
Here, EnD 2 consistently yields higher uncertainty predictions than the original ensemble.
Notably, adding auxiliary training data makes the model yield even higher estimates of total uncertainty and data uncertainty .
At the same time, by using auxiliary training data, the model’s distribution of knowledge uncertainty starts to look more like the ensembles, but shifted to the right (higher).
[Figure]
Altogether, this suggests that samples from the ensemble are diverse in a way that’s different from a Dirichlet distribution.
For instance, the distribution could be multi-modal or crescent-shaped.
Thus, as a consequence of this, combined with forward KL-divergence between the model and the empirical distribution of the ensemble being zero-avoiding , the model over-estimates the support of the empirical distribution, yielding an output which is both more diverse and higher entropy than the original ensemble.
This does not seem to adversely impact OOD detection performance as measures of uncertainty for ID and OOD data are further spread apart and the rank ordering of ID and OOD data is either maintained or improved, which is supported by results from section 5.
However, this does prevent the EnD 2 models from fully retaining the calibration quality of the ensemble.
It is possible that the ensemble could be better modelled by a different output distribution, such as a mixture of Dirichlet distributions or a Logistic-normal distribution .
C ONCLUSION
Ensemble Distillation approaches have become popular, as they allow a single model to achieve classiﬁcation performance comparable to that of an ensemble at a lower computational cost.
This work proposes the novel task Ensemble Distribution Distillation (EnD 2 ) — distilling an ensemble into a single model, such that it exhibits both the improved classiﬁcation performance of the ensemble and retains information about its diversity .
An approach to EnD 2 based on using Prior Network models is considered in this work.
Experiments described in sections 4 and 5 show that on both artiﬁcial data and image classiﬁcation tasks it is possible to distribution distill an ensemble into a single model such that it retains the classiﬁcation performance of the ensemble.
Furthermore, measures of uncertainty provided by EnD 2 models match the behaviour of an ensemble of models on artiﬁcial data, and
EnD 2 models are able to differentiate between different types of uncertainty.
However, this may require obtaining auxiliary training data on which the ensemble is more diverse in order to allow the distribution-distilled model to learn appropriate out-of-domain behaviour.
On image classiﬁcation tasks measures of uncertainty derived from EnD 2 models allow them to outperform both single NNs and EnD models on the tasks of misclassiﬁcation and out-of-distribution input detection.
These results are promising, and show that Ensemble Distribution Distillation enables a single model to capture more useful properties of an ensemble than standard Ensemble Distillation.
Future work should further investigate properties of temperature annealing, investigate ways to enhance the diversity of an ensemble, consider different sources of ensembles and model architectures, and examine more ﬂexible output distributions, such as mixtures of Dirichlets.
Furthermore, while this work considered Ensemble Distribution Distillation only for classiﬁcation problems, it can and should also be investigated for regression tasks.
Finally, it may be interesting to explore combining Ensemble
Distribution Distillation with standard Prior Network training.
A PPENDIX A D ATASETS , M ODEL ARCHITECTURE AND TRAINING
[Table]
All models considered in this work were implemented in Pytorch (Paszke et al., 2017) using a variant of the VGG16 (Simonyan & Zisserman, 2015) architecture for image classiﬁcation.
DNN and EnD models were trained using the negative log-likelihood loss of the labels and the mean ensemble predictions respectively.
EnD 2 models were trained using the negative log-likelihood of the ensemble’s output categorical distributions.
All models were trained using the Adam (Kingma & Ba, 2015) optimizer, with a 1-cycle learning rate policy and dropout regularization.
For all ensembles, models were trained using different random seed initialization
[Table]
and using different seeds for shufﬂing the data.
In addition, data augmentation was applied via random left-right ﬂips, random shifts up to ± 4 pixels and random rotations by up to ± 15 degrees.
Tables e 6 details the training conﬁgurations for all models.
Furthermore, batch normalisation was used for both Ensemble Distillation and
Ensemble Distribution Distillation, but not for Prior Networks.
[Table]
No
CIFAREnD  0.
YesEnD 2+ AUX  0.
Yes
CIFARPN 5 × 10 −  0.
No
CIFARCIFAR-
DNN 10 −  0.
EnD  0.
NoEnD + AUX  0.
No
CIFAREnD  0.
YesEnD 2+ AUX  0.
Yes
CIFARPN 5 × 10 −  0.
No
CIFARTinyImageNet
DNN 10 −  0.
EnD 5 × 10 −  0.
NoEnD + AUX 5 × 10 −  0.
No
CIFAREnD5 × 10 −  0.
YesEnD 2+ AUX 5 × 10 −  0.
Yes
CIFARPN 5 × 10 −  0.
No
CIFARTo create the transfer set D ens , ensembles were evaluated on the unaugmented CIFAR-10 and
CIFAR-100 training examples.
During distillation (both EnD and EnD 2 ), models were trained on the augmented examples with the ensemble predictions on the corresponding unaugmented inputs.
A. N UMERICAL ISSUES WITH MAXIMUM LIKELIHOOD D IRICHLET TRAINING
As shown in equation 8, ensemble distribution distillation into a prior network via likelihood maximisation is equivalent to minimising the loss function:
[Equation]
If, due to numerical precision of the implementation, one of the ensemble member sample terms π ( im ) c gets rounded to 0 .
0 , the log term ln π ( im ) c in the above equation cannot be computed.
To avoid this, we apply a small amount of central smoothing to the ensemble predictions:
[Equation]
Where the smoothing parameter parameter γ has been set to 1 × 10 − 4 for all EnD 2 experiments.
Note that after applying central smoothing, each ensemble member’s predictions still sum up to 1 . 0 .
A. T EMPERATURE A NNEALING S CHEDULE
A ﬁxed temperature of 2 . 5 was used for Ensemble Distillation as recommended in (Hinton et al., 2015), and was found to yield the best classiﬁcation performance out of { 1 , 2 . 5 , 5 , 10 } .
Temperature annealing resulted in worse classiﬁcation performance for Ensemble Distillation, and hence was not used in the experiments.
For the temperature annealing schedule, the temperature was kept ﬁxed to initial temperature T 0 for the ﬁrst half-cycle.
For the second half-cycle we linearly decayed the initial temperature T 0 down to 1 . 0 .
Then, the temperature was kept constant at 1 . 0 for the remainder of the epochs.
For Ensemble Distribution Distillation, we found that an initial temperature ofperformed best out of { 5 , 10 , 20 } .
The choice of the particular annealing schedule used has not been tested extensively, and it is possible that other schedules that lead to faster training exist.
However, we must point out that it was necessary to use temperature scaling to train models on the CIFAR-10,
CIFAR-100 and TinyImageNet datasets.
A PPENDIX B A SSESSING MISCLASSIFICATION DETECTION PERFORMANCE
In this work measures of uncertainty are used for two practical applications of uncertainty - misclassiﬁcation detection and out-of-distribution sample detection.
Both can be seen as an outlier detection task based on measures of uncertainty, where misclassiﬁcations are one form of outlier and out-of-distribution inputs are another form of outlier.
These tasks can be formulated as thresholdbased binary classiﬁcation (Hendrycks & Gimpel, 2016).
Here, a detector I T ( x ) assigns the label(uncertain prediction) if an uncertainty measure H ( x ) is above a threshold T , and label 0 (conﬁdent prediction) otherwise.
This uncertainty measure can be any of the measures discussed in sectionsand 3.
[Equation]
Given a set of true positive examples D p
= { x ( i ) p } N p
i
=1 and a set of true negative examples
D n = { x ( j ) n } N n j =1 the performance of such a detection scheme can be evaluated at a particular threshold value T using the true positive rate t p ( T ) and the false positive rate f p ( T ) :
[Equation]
The range of trade-offs between the true positive and the false positive rates can be visualized using a Receiver-Operating-Characteristic (ROC) and the quality of the possible trade-offs can be summarized using the area under the ROC curve (AUROC)
(Murphy, 2012).
If there are signiﬁcantly more negatives than positives, however, this measure will over-estimate the performance of the model and yield a high AUROC value (Murphy, 2012).
In this situation it is better to calculate the precision and recall of this detection scheme at every threshold value and plot them against each other on a
Precision-Recall (PR) curve (Murphy, 2012).
The recall R ( T ) is equal to the true positive rate t p ( T ) , while precision measures the number of true positives among all samples labelled as positive:
[Equation]
The quality of the trade-offs can again be summarized via the area under the PR curve (AUPR).
For both the ROC and the PR curves an ideal detection scheme will achieve an AUC of 100%.
A completely random detection scheme will have an AUROC of 50% and the AUPR will be the ratio of the number of positive examples to the total size of the dataset (positive and negative)
(Murphy, 2012).
Thus, the recall is given by the error rate of the classiﬁer.
This makes it difﬁcult to compare different models with different base error rates, as AUPR can increase both due to better misclassiﬁcation detection and worse error rates.
Below we provide AUPR numbers for all models and datasets in order to illustrate how it can challenging to compare models using this metric.
The difference in AUPR between models within a dataset is typically similar to the difference in classiﬁcation performance.
Thus, it is challenging to assesses whether a higher AUPR is due to better misclassiﬁcation detection or higher error rate.
Additionally, this table illustrate that conﬁdence of the prediction is a better measure of uncertainty than entropy for this task.
At the same, knowledge uncertainty yield much worse misclassiﬁcation detection performance.
These results are consistent with (Malinin & Gales, 2018;
Malinin, 2019).
In this work we consider an alternative to using AUPR to assess misclassiﬁcation detection performance proposed in (Malinin, 2019).
Consider the task of misclassiﬁcation detection - ideally we would like to detect all of the inputs which the model has misclassiﬁed based on a measure of
[Table]
uncertainty.
Then, the model can either choose to not provide any prediction for these inputs, or they can be passed over or ‘rejected’ to an oracle (ie: human) to obtain the correct prediction.
The latter process can be visualized using a rejection curve depicted in ﬁgure 5, where the predictions of the model are replaced with predictions provided by an oracle in some particular order based on estimates of uncertainty.
If the estimates of uncertainty are ‘useless’, then, in expectation, the rejection curve would be a straight line from base error rate to the lower right corner.
However, if the estimates of uncertainty are ‘perfect’ and always bigger for a misclassiﬁcation than for a correct classiﬁcation, then they would produce the ‘oracle’ rejection curve.
The ‘oracle’ curve will go down linearly to 0% classiﬁcation error at the percentage of rejected examples equal to the number of misclassiﬁcations.
A rejection curve produced by estimates of uncertainty which are not perfect, but still informative, will sit between the ‘random’ and ‘oracle’ curves.
[Figure]
The quality of the rejection curve can be assessed by considering the ratio of the area between the ‘uncertainty’ and ‘random’ curves AR uns (orange in ﬁgure 5) and the area between the ‘oracle’ and ‘random’ curves AR orc (blue in ﬁgure 5).
This yields the prediction rejection area ratio PRR :
[Equation]
A rejection area ratio of 1.0 indicates optimal rejection, a ratio of 0.0 indicates ‘random’ rejection.
A negative rejection ratio indicates that the estimates of uncertainty are ‘perverse’ - they are higher for accurate predictions than for misclassiﬁcations.
An important property of this performance metric is that it is independent of classiﬁcation performance, unlike AUPR, and thus it is possible to compare models with different base error rates.
Note, that similar approaches to assessing misclassiﬁcation detection were considered in (Lakshminarayanan et al., 2017; Malinin et al., 2017)
A PPENDIX C
A PPROPRIATENESS
OF D IRICHLET DISTRIBUTION
Section 5.1 explored the behaviour of measures of uncertainty predicted by models trained on the
CIFAR-10 dataset.
In this appendix we provide the same for models trained on CIFAR-100 and
TinyImageNet.
In general, the trends detailed in section 5.1 hold for these datasets as well - EnDconsistently yield higher uncertainties than the original ensemble, likely as a consequence of the limitations of the Dirichlet distribution.
In addition, we have also provided histograms of total uncertainty for EnD and EnD + AUX models trained on CIFAR-10, CIFAR-100 and TinyImageNet relative to the ensemble for both in-domain and OOD data.
Figure 7 shows that while all EnD models match the uncertainty of the ensemble on in-domain data for all datasets, EnD consistently under-estimates the uncertainty for OOD data.
The only exception is EnD + AUX on CIFAR-100, where it matches the ensemble’s predictions well.
This generally agrees with the results from section 5, where EnD + AUX is able to almost match the ensemble’s OOD detection performance.
[Figure]
A PPENDIX D A BLATION S TUDIES
In this paper we made several important design choices.
Firstly, we chose to use rich ensembles of 100 models, in order to generate a good estimate of the empirical distribution of the ensemble on the simplex and assess whether the Dirichlet distribution was ﬂexible enough to model it.
Secondly, we used a temperature annealing schedule with an initial temperature of 10.
In this section we conduct two ablation studies.
Firstly, we assess whether Ensemble Distribution Distillation is possible when using an ensemble of 5, 20, 50 and 100 models.
Secondly, we investigate a range of initial temperatures (1, 2, 5, 10, and 20) for temperature annealing for the models trained on the CIFARand CIFAR-100 datasets.
Here, only the EnD 2+ AUX model is considered.
Results are presented in ﬁgures 8-13.
All ﬁgures depict the mean across 3 models (for each conﬁguration) ± 1 standard deviation.
Overall, the results show two important trends.
Firstly, using using 20 models does better than using 5 models, but there are no conclusive gains for ensemble distribution distillation from using more than 20 models.
This suggests that it is sufﬁcient to use ensemble of fewer models for Ensemble
Distribution Distillation with models which parameterize the Dirichlet Distribution.
This is beneﬁcial in terms computation and memory savings.
It is possible, however, that if a more ﬂexible distribution
[Figure]
is used, such as a Mixture of Dirichlets, then it might be possible to derive further gains from a larger ensemble.
The second trend is that it is necessary to use a temperature of at least 5 in order to successfully distribution-distill the ensemble.
Using initial temperatures of 10 and 20 did not result in any signiﬁcant further increase in performance.
These results show that the temperature annealing process is important for Ensemble Distribution Distillation to work well.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]