A Reproduction of Ensemble Distribution Distillation
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility SummaryScope of Reproducibility
The authors claim that their proposed method is able to, given an ensemble of deep neural networks, capture theuncertainty estimation and decomposition capabilities of the ensemble into a single model.
The authors also claim thatthis only results in a small reduction in classiﬁcation performance compared to the ensemble.
Most of the authors’experiments on the CIFAR-10 dataset were reproduced.
Methodology
The proposed method was re-implemented in tf.keras .
The surrounding data pipelines, pre-processing, and experimentation code were also re-implemented.
As in the original paper, the models were based on VGG-16 networks withrandom initialization, but trained within the project.
Training and evaluation was done on two consumer-grade GPUs,for a total of 273 hours.
Results
Our ﬁndings support the authors’ central claims.
In terms of uncertainty estimation our EnD 2 achieved (99 ± 1) % ofthe AUC-ROC of our ensemble on the OOD-detection task.
The corresponding value in the original paper was (100 ± 1)%.
In terms of classiﬁcation our EnD 2 had (16 ± 1) % higher error than our ensemble.
The corresponding values in theoriginal paper was (11 ± 6 )%.
Other metrics showed similar agreement, but, signiﬁcantly, in the OOD-detection taskour EnD performed at least as well as our EnD 2 .
This is in stark contrast with the original paper.
We also took a novel approach to visualizing the uncertainty decomposition by plotting the resulting distributions ona simplex, offering a visual explanation to some surprising results in the original paper, while mostly supporting theauthors’ intuitive justiﬁcations for the model.
What was easy
The original paper features a thorough mathematical formulation of the method, aiding conceptual understanding.
Thedatasets used by the authors are publicly available.
The use of the simpler datasets also meant that it was computationallyfeasible for us to reproduce these results.
The base model used is well known with several implementation available,allowing us to focus on the novel aspects of the method.
What was difﬁcult
While the theoretical explanations of the method are excellent we initially found it hard to translate this into an implementation.
Our difﬁculty was likely caused by our inexperience with the subject matter.
Nonetheless, a pseudocode, such as the one we have provided would havee simpliﬁed the re-implementation. We were not able to reproduce the some of the datasets limited
Communication with original authors
Reproducibility Challenge a asked
Introduction
Uncertainty estimation can help to make deep learning safer and more usable by allowing the model to identify cases itis not suitable to handle.
There are different kinds of uncertainty, however, and it is especially interesting to separateuncertainty caused by ambiguities or contradictions in the data from the uncertainty that arises when a model faces asituation it has not been trained for.
Ensemble-based methods of uncertainty estimation are capable of making thisdistinction but suffer from computational requirements at the evaluation phase [1].
The authors of Ensemble Distribution
Distillation (EnD 2 ) [2] address this issue by using the output of an ensemble to train a so-called Prior Network (PN)
[3],distilling the ensemble down to a single model while also preserving its uncertainty decomposition abilities.
This canbe contrasted with regular ensemble distillation models [4] (EnD), which are not able to decompose uncertainty.
Thereproduced paper was accepted to ICLR2020.
Scope of reproducibility
We consider the setting of using CIFAR10 [5] as an in-distribution dataset, and LSUN [6] as an out-of-distributiondataset.
Our supplementary material also examines the setting of using a synthetic dataset in R 2 for visualization.
The claims from the original article that this reproduction is testing are as follows:
Classiﬁcation performance :
In terms of error rate, prediction rejection rate, and negative log-likelihood
EnD 2 has worse performance than the ensemble, but similar performance to EnD and PriorNet
and betterperformance than the individual model.
In terms of expected calibration error, EnD 2 has worse performance
than the ensemble, but better performance than the other methods.
On CIFAR-10 in particular, EnD 2 has the
best expected calibration error of all models.
This
claim corresponds to
Table
3 in the original paper.
Out-of-distribution detection performance :
In terms of AUC-ROC on CIFAR-10 vs. LSUN, EnD 2 without
auxiliary dataset performs worse than the ensemble and the PriorNet, similar to the individual model
and
better than EnD. With the auxiliary dataset, however, EnD 2 performs as well as the ensemble, almost as well
as PriorNet, and better than EnD. Using knowledge uncertainty
as opposed to total uncertainty on CIFAR
vs
LSUN does not yield an improved AUC -ROC.
This claim corresponds to Table 4 in the original paper.
Dependency on ensemble size : Using 20 models in the ensemble does better than using 5 models, but there
is no conclusive gain when using more than 20 models.
Dependency on temperature : It is necessary to use temperature of at least 5 to successfully distribution-distill
the ensemble.
Using higher initial temperatures do not result in conclusive improvement.5.
Uncertainty decomposition : EnD 2 trained with an auxiliary dataset is able to reconstruct the uncertaintydecomposition made possible by ensembles.
We reproduce all experiments of the main article and most of the appendix, except for the use of CIFAR100 and Tiny
Imagenet datasets. Some of these results can be found in our supplementary material. From their appendix, we do notreproduce Table 7 in appendix B. We did not recreate the OOD-detection plots when reproducing the ablation study.
Methodology
3.1 Model description
We consider the same seven models as the original authors:
• IND: A single classiﬁcation model.
• ENSM:
An ensemble of independently trained IND models.
• EnD:
A single model distilling ENSM trained according to [4].• EnD 2 : A single model distribution-distilling ENSM trained according to [2].
• EnD +AUX : Like EnD, but trained with auxiliary data. • EnD 2 +AUX : Like EnD 2 , but trained with auxiliary data.
• PN
+AUX :
A PriorNet model with auxiliary data trained according to [3]
These models are all based on almost identical VGG16 architectures [7], adapted to CIFAR-10 data as in [3] by addingdropout, batch normalization and reducing the size of the fully connected layers.
The only exception being that batchnormalization is not used for PN.
[Table]
Link
[Table] [Table]
The training set of CIFAR-10 was used as the primary training dataset.
The training set of CIFAR-100 was usedas an auxiliary dataset.
For evaluating the classiﬁcation task the test set of CIFAR-10 was used.
For evaluating theout-of-distribution detection task the CIFAR-10 test set was used as in-domain dataset, while the LSUN test set wasused as the out-of-domain dataset.
Information about the datasets is listed in Table 1.
Each image x was normalized according to x (cid:48)
=
x/ 127 .
5 − 1 where the operations are elementwise, causing all valuesto lie in the range (-1, 1).
The LSUN images were also scaled down to 32x32.
Furthermore, dataset augmentation wasused for all models, consisting of rotations with 15 ◦ range, horizontal ﬂips, width and height shifts of up to 4 pixels ineach direction, and using nearest-neighbour interpolation.
3.3 Hyperparameters
The models were trained with the hyperparameters listed in Table 2.
3.4 Experimental setup and code
Using these models and dataset we ran a number of experiments, as detailed below.
The full code is availableon https://anonymous.4open.science/r/4ee2c9ef-295f-44e2-8214-f0818b932817/.
Our implementation was made in
TensorFlow Keras, as opposed to the original implementation which was made in PyTorch.
Classiﬁcation: The classiﬁcation task was evaluated on the test set of CIFAR-10.
We use the same four metrics as inthe original paper, ERR, PRR, ECE, and NLL.
ERR is the mean classiﬁcation error.
PRR is the prediction rejectionarea ratio introduced in Appendix B of [2].
ECE is the expected calibration error 1 .
Finally, NLL is the negativelog-likelihood.
This experiment tests Claim 1.
Out-of-distribution detection: The OOD-detection task was evaluated with the CIFAR-10 test set as the in-domainset, and the LSUN test set as the out-of-domain set.
The AUC-ROC was computed both when total uncertainty andwhen only knowledge uncertainty is used to make rejection decisions.
This experiment tests Claim 2.
Ensemble size ablation study: Our examination of the effect of ensemble size goes slightly beyond the originalauthors.
We extend the error analysis to also consider the sensitivity of EnD 2 to variations in the underlying ensemble.
We began by training a set of 400 VGG16 models on CIFAR-10.
Next, we sampled randomly from this set to create 4different sets, each consisting of 100 models.
For each N ∈ { 1 , 2 , 3 , 4 , 6 , 8 , 10 , 13 , 16 , 20 , 25 , 30 , 45 , 60 , 75 , 100 } wetrained four EnD 2 models on an ensemble consisting of the ﬁrst N models in the ﬁrst of the four sets, corresponding towhat was done in the original study.
We also trained one model on an ensemble consisting of the ﬁrst N models foreach of the three remaining sets, capturing the sensitivity of EnD 2 to changes in the underlying ensemble.
All ensembleand EnD 2 models were then evaluated on the classiﬁcation task.
This experiment tests Claim 3.
[Table]
[Table]
ablation
[Table]
We reproduce the temperature ablation study by training EnD 2 models for various initialtemperatures.
For each T ∈ { 1 , 2 , 3 , 4 , 5 , 7 . 5 , 10 , 15 , 20 } we trained three EnD 2 models with initial temperature T onan ensemble consisting of 100 VGG16 models.
The EnD 2 models were then evaluated on the classiﬁcation task.
In thisexperiment, we have chosen to use a slightly ﬁner spacing between the temperatures than what the original authorsused.
This experiment tests Claim 4.
Simplex visualization: A key motivation for EnD 2 is the idea that an ensemble can distinguish between knowledgeuncertainty and data uncertainty, and that this distinction is retained by the EnD 2 model.
This is communicated using aschematic ﬁgure showing ensemble predictions on a simplex.
A similar schematic ﬁgure can be found in [3], depictinga Dirichlet PDF of a PriorNet on a simplex.
We recreated these ﬁgures using experimental data in order to examine
Claim 5 from a novel perspective.
A new training set was created, consisting of all images from the CIFAR10 train setwith one of three labels chosen for their similarity: ’deer’, ’horse’, and ’dog’.
The remaining images were reserved asout-of-distribution dataset for testing.
CIFAR-100 was used as auxiliary data.
An ensemble and EnD 2 was then trainedon this data using the same architecture and processed as before.
We then selected various images from the test set andvisualized the ensemble predictions as well as the PDF of the EnD 2 model.
The simplex visualization was created usingopen source code 2 .
3.5 Computational requirements
Training and evaluation were performed on two mid-range consumer GPUs (RTX 2070, GTX 1660s) locally.
Regarding
VRAM, at least 4711 MiB is required for the models.
The total number of GPU time required for the ﬁnal results is11.4 GPU days on an RTX 2070.
The accumulated GPU days during the reproduction is 3-5 times this amount.
Weprovide detailed numbers in the supplementary materials.
Results
4.1 Classiﬁcation performance
The classiﬁcation results are shown in Table 3.
Overall, the ensemble seems to perform best, and when it does not, it isstill within error bounds.
Curiously, EnD 2 +AUX seems to perform worse than the individual model in regards to ERR.
4.2 Out-of-distribution detection performance
The OOD-detection results are shown in Table 4.
The results suggest plain EnD 2 performs worse than ENSM, butthat the addition of an auxiliary dataset brings the performance up to at least the level of ENSM.
More surprising,perhaps, is that EnD 2 seems to perform worse than EnD.
In both metrics PN +AUX has a signiﬁcant lead.
Using knowledgeuncertainty instead of total uncertainty decreases the effectiveness of all tested models.
The supplemental materialcontains histograms showing the distribution of estimated total and knowledge uncertainty over the images.
4.3 Ensemble size ablation study
Figure 1 shows the results of the ensemble size ablation study.
The lines ’ENSM Paper’ and ’EnD 2 Paper’ show theresults of the original paper.
The bands indicate two standard deviations.
Two bands surround the ’EnD 2 +AUX ’ line,representing the two types of variation we have examined.
The purple band represents the variation of four EnD 2models each trained on a different ensemble.
The orange band represents the variation of four EnD 2 models all trainedon the same ensemble.
The band surrounding the ’EnD 2 +AUX Paper’ line corresponds to the latter type of variation.
There appears to be a trend of small improvement when the number of models is increased, but the high level ofuncertainty makes it difﬁcult to draw conclusions from the remaining points.
Nonetheless, the results seem to generallyindicate that EnD 2 is not particularly sensitive to ensemble size.
4.4 Temperature ablation study
[Figure] The results of our temperature ablation study are shown in Figure 2, along with the results of the original paper.
Forinitial temperature equal to 1 and 2 our models fail to converge, resulting in poor classiﬁcation performance.
Raising theinitial temperature to 3 allows the model to converge.
Increasing the initial temperature further has no signiﬁcant effect.
It is worth noting the negative PRR values for T = 2 .
The original authors mention this possibility when they proposethe metric, and offer the interpretation that this means that the model is increasing the classiﬁcation error by rejectingsamples, performing worse than simply rejecting at random.
4.5 Simplex visualization
Predictions for four images are visualized in Figure 3.
These four images were selected from the CIFAR10 dataset forrespectively having the lowest total uncertainty, highest data uncertainty, highest knowledge uncertainty, and highesttotal uncertainty, as measured by the ensemble.
The third row shows the Dirichlet PDF of EnD 2 .
There is a strongtendency towards extremely sharp distributions, even when the ensemble has high spread, making comparison difﬁcult.
For this reason the fourth row plots the PDF after being transformed by the transformation log ( x +1) .
It is now possibleto see that the PDF is adapting to the distribution of the ensembles.
We also plot randomly selected images from the in, out, and auxiliary datasets respectively.
The PDF has again beentransformed using log ( x + 1) .
Figure 4 shows images from the in-domain dataset, and Figure 5 shows images from theout-of-domain dataset.
The PDF appears to follow the ensemble fairly well, but it is noteworthy that the ensemblesshow such a low degree of spread despite encountering samples on which they have not been trained.
Discussion
5.1 Comparison with original paper
We now revisit the six claims which we speciﬁed in Section 2.1.
Classiﬁcation performance: When compared to the original table we see overall worse performance.
This islikely rooted in the fact that we were unable to achieve as high accuracy on our base VGG16 as in the originalarticle.
We therefore instead consider the relative performance between the models.
Our supplementarymaterial contains a table allowing for easy comparison with the original results.
For example, we ﬁnd that
our
EnD 2 has 112.5% of the classiﬁcation error of the ensemble, while in the original paper this ﬁgure is 117.7%.
The absolute difference is the same in both papers, 1.1 percentage units.
Our results generally agree well, withthose of the authors.
There are some discrepancies in expected calibration error, but our extremely high ECEfor the individual model suggests that there might be an issue in our computations of this metric.
Overall ourﬁndings support Claim 1.2.
Out-of-distribution detection performance : For the most part, our results agree with Claim 2.
For instance,we found that using total uncertainty EnD 2 without auxiliary data had 98.1% of the AUC-ROC of the ensemble,while the corresponding ﬁgure with auxiliary data was 100.0%.
In the original paper, these ﬁgures were96.8% and 99.8% respectively.
There is one very signiﬁcant discrepancy, however.
With auxiliary dataset, our
EnD 2 had 99.6% of the AUC-ROC of our EnD, while in the original paper this ﬁgure is 106.5%.
A similarrelationship exists without the auxiliary dataset.
It is worth noting that in the original paper EnD performsworse than even the individual model, and the authors themselves note that this is odd.
Since EnD 2 is designedto overcome certain shortcomings of EnD in terms of uncertainty estimation we believe that this warrantsfurther investigation.
results conﬁrm the
relative performance between ensembles and EnD 2 +AUX , with increased resolution. For expected calibration
error, the relative performance is conﬁrmed for a large number of models, but for a small number of models, weget contradictory results.
Their results seem to suggest that smaller ensembles have worse calibration, which isnot expected, as per [1].
Our results conﬁrm this expectation.
In their paper, they state this expectation, but wesee no comment for this discrepancy.
For prediction rejection rate, we conﬁrm the relative performance, andalso show that it starts to drop rapidly below their tested range.4.
Dependency on temperature annealing : Our results diverge heavily from the results in the paper fortemperatures 1 and 2.
While the original authors are able to train working but sub-par models with thesetemperatures, we are unable to get the models to converge at all.
We re-did the experiments with a newensemble, and experimented with the smoothing factor and auxiliary data, but were unable to ﬁnd anyexplanation for this difference.
Nevertheless, these ﬁndings support the claim that temperature annealing isessential for successful use of the EnD 2 method.
The authors suggested temperature 5 as a minimum valuebeyond which larger values make no difference.
Our ﬁndings support this as well, although our increasedresolution reveals that the minimum value for the CIFAR-10 dataset is closer to 3 than 5.5.
Uncertainty decomposition :
Based on the description in [3] an image with a high knowledge uncertaintyshould produce a Dirichlet PDF with a close to uniform spread.
Our simplex visualizations on the 3-class+AUXdataset shows that this is not the case.
This is not too surprising, given that high knowledge uncertaintycorrelates with small alphas, and this in turn produces convex as opposed to ﬂat probability density surfaces.
Overall, these plots suggest that EnD 2 can capture the uncertainty decomposition of the ensemble.
The plots also show an interesting behaviour in the ensemble.
The ensembles agree to a surprising extent onthe out-of-domain samples.
In fact, when they do disagree it normally takes the form of data uncertainty asopposed to knowledge uncertainty.
This could perhaps shed some light on the observation that knowledgeuncertainty does not seem to be useful for OOD-detection on CIFAR-10.
The original authors explain thisas essentially being a property of the dataset.
We feel, based on the visualizations, that another possibilitymight be that the ensemble models simply are not diverse enough to provide a useful measure of knowledgeuncertainty.
5.2 What was difﬁcult
Although the general idea of the paper is well formulated in mathematical terms, the original paper does not providemany hints regarding how to implement the method.
In our case, this imposed a signiﬁcant barrier to immediatelyreproducing the work, since our inexperience meant that we’re unable to immediately see how it could be implementedin a modern deep learning framework.
There is some code available in a public repository hosted by one of the authorsbut this is not mentioned in the paper, and so we could not treat it as an ofﬁcial implementation.
We have provided apseudocode in our supplemental material, in order to hopefully assist future reproducers.
There are also some missing details regarding the models used.
Most importantly, the authors mention that they haveused a modiﬁed VGG model, but do not specify what these modiﬁcations are.
The authors also do not specify the minand max value of the cyclic LR.
These details may explain the consistently worse performance of our models despitethe attempt of replication.
5.3 What was easy
The synthetic dataset was fairly easy to reconstruct, and the other datasets are well known and publicly available.
Thedata augmentation was straightforward and easy to incorporate into a training pipeline.
The base model (VGG16) usedin most of the experiments is well known and was computationally feasible to train.
Similarly, the datasets are notexcessively demanding in terms of computation, although in our case training time did become a limiting factor due tothe amount of time we spent on implementation and experimentation.
The mathematical formulation of the model isvery good, helping the conceptual understanding.
5.4 Communication with original authors
We did not communicate with the authors while reproducing their work, although we did refer to some resources whichone of the authors has made publicly available, including an repository 3 made for [3] containing an implementation of
EnD 2 .
At the same time as submitting this report, we also sent a copy to the authors and asked for their comments