Paper:
Ensembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different forms of uncertainty. However, ensembles come at a computational and memory cost which may be prohibitive for many applications. There has been significant work done on the distillation of an ensemble into a single model. Such approaches decrease computational cost and allow a single model to achieve an accuracy comparable to that of an ensemble. However, information about the diversity of the ensemble, which can yield estimates of different forms of uncertainty, is lost. This work considers the novel task of Ensemble Distribution Distillation (EnD 2 ) -distilling the distribution of the predictions from an ensemble, rather than just the average prediction, into a single model. EnD 2 enables a single model to retain both the improved classification performance of ensemble distillation as well as information about the diversity of the ensemble, which is useful for uncertainty estimation. A solution for EnD 2 based on Prior Networks, a class of models which allow a single neural network to explicitly model a distribution over output distributions, is proposed in this work. The properties of EnD 2 are investigated on both an artificial dataset, and on the CIFAR-10, CIFAR-100 and TinyImageNet datasets, where it is shown that EnD 2 can approach the classification performance of an ensemble, and outperforms both standard DNNs and Ensemble Distillation on the tasks of misclassification and out-of-distribution input detection.

Review:
========= Post Rebuttal ========= 
I appreciate the authors' effort in addressing the raised issues. I think the revised paper has higher quality in that the additional ablation studies are very useful to understand the effectiveness of the method and the importance of the hyperparameters. It now also has higher clarity in that the presentation of the work is considerably more detailed in the last revision. The outstanding issue is that the final results are not consistent across datasets, baselines, and metrics which is concerning. But, I think, overall, the paper is pushing on an interesting line of research which is relevant and educational for ICLR audience. Th concerns on consistency and conclusiveness of the results can hopefully be addressed in a journal version of the work as suggested by the authors.
========= Summary ========= 
 
Distilling an ensemble of deep networks into a single student network is a common approach to reduce the inference-time computational complexity. In those cases, the paper poses the question of whether it is useful for the student to capture the diversity of ensemble members’ predictions on top of the mean distribution (as in the standard distillation). 
The main motivation is that this captured diversity will enable the student network to decompose the total (predictive) uncertainty into data (aleatoric/irreducible) and knowledge (epistemic/model) uncertainty components. 
In order to distill the diversity, it proposes to use “prior networks [Malinin&Gales 2018]” to output parameters of a  Dirichlet distribution over the simplex of possible predictive categorical distributions.
It uses one toy dataset as well as 3 small image classification datasets to compare the performance of the proposed method (EnD^2) with a) standard ensemble distillation (EnD), b) ensemble of deep networks, as well as c) an individual deep network. The performance is measured in terms of classification accuracy, and the quality of the predictive uncertainty for in-distribution (ID) and out-of-distribution (OOD) samples.
It shows that for OOD sample detection, the proposed EnD^2 outperforms the standard EnD and bridges the gap to the full ensemble. The classification accuracy and ID uncertainty quality is similar to the standard EnD.
 
 
========= Strengths and Weaknesses ========= 
 
+ the paper studies the interesting problem of decoupling data/knowledge uncertainty in the commonly-used framework of knowledge distillation. It proposes a simple solution to the problem. So, it is highly relevant for the field.
+ Results in Table 3 suggest significant improvements over the standard distillation (EnD). 
+ Figure 3 is pedagogical and intuitive for the data vs knowledge uncertainty decomposition and the effectiveness of auxiliary dataset for this toy problem.
 
(I) General concerns:
- The technical novelty is limited to the combination of knowledge distillation and prior networks.
- Two closely-related works are not cited:
[Li&Hoiem, “Reducing Over-confident Errors outside the Known Distribution” 2019] uses auxiliary dataset for ensemble distillation.
[Englesson&Azizpour, “Efficient Evaluation-Time Uncertainty Estimation by Improved Distillation”, 2019] addresses the problem of efficient uncertainty estimation using knowledge distillation.
 
(II) Concerns regarding the experiments
- As also found curious by the authors, it is concerning that the EnD_Aux results are so low for OOD detection. Both [Li&Hoiem 2019] and [Englesson&Azizpour 2019] suggest that the standard distillation can perform similarly to (or even outperform) the individual model and the ensemble using an auxiliary dataset [Li&Hoiem 2019] or teacher label for augmented samples [Englesson&Azizpour 2019]. It should be mentioned that the OOD dataset setup is different across these works.
- Why only 100-ensembles are used for the real-world experiments? Does a “successful” training of the prior network require a large number of samples? This would be important since prior ensemble works (e.g. [Lakshminarayanan et al. 2017] ) hardly improve beyond 5-15 networks. Training 100 networks is very costly which would be an important limitation of this work in case it’s necessary. An ablation study on the number of networks in the ensemble is required for investigating this trade-off.
- Hyperparameter optimization: Appendix A provides the final values for the hyperparameters of each method. However, it is not clear how these were optimized? Was grid search used? What was the range for all the hyperparameters? Is there a validation set or cross validation is used and in each case how is the split done? Is it exactly the same hyper-parameter optimization that is done for EnD and EnD^2? 
 
(III) Missing from experiments
- The main goal/motivation of the paper is to be able to decompose the total uncertainty when distilling an ensemble into a single model. In that respect, richer and more experiments are required to evaluate this ability. Currently, the experiments are focused on showing that EnD^2 outperforms EnD. The toy dataset, qualitatively, evaluates the decomposition quality with interesting results but is not enough to make conclusions for real-world datasets. For instance, as an additional experiment, plots/statistics can be given on data vs knowledge uncertainty for ID vs OOD samples.
- Prior network [Malinin&Gales 2018] in its standard form is an important single-network baseline that should be included for both ID and OOD experiments. 
- Temperature scaling is mentioned to be a vital part of the model. As such, it requires a thorough ablation study to see the results with or without it as well as when changing the temperature and the annealing factor. It should be studied from two aspects: accuracy and convergence failure (as mentioned by the paper)
- P.7: “Prediction Rejection Ratio” (PRR) is a measure proposed in this work but it’s only defined in the appendix. That is the only metric that measures the quality of uncertainty for ID samples (Table 3). As such, I believe it’s important to define PRR in the main paper and also further include more standard metrics such as NLL, AUROC, AUPRcurve in table 3 so that some context is given to the newly-proposed PRR.
- Along the same line, it seems NLL is consistently and “more significantly” worse for EnD^2 compared to EnD.
 
(IV) Missing training and implementation details
- More details should be provided for the training of the student prior network including it’s loss function given a dataset and Dirichlet distribution. This can be obtained from [Malinin&Gales 2018] but it’s important for this work to be self-contained.
- the loss function in equation 8 has log(p(\pi|x;\theta)); is there any numerical issue regarding the Dirichlet distribution and/or the log? If so, how significant and what measures are accordingly taken? Could it be that the temperature scaling is more a way of alleviating these issues as opposed to the shared support of distributions for KL divergence?  
- the details of the annealing algorithm is entirely missing.
 
 
========= Final Decision ========= 
 
The paper addresses a highly relevant problem in a simple (and potentially effective) way. This is great. However, there are several concerns as listed above which altogether makes me lean towards an initial “weak reject” rating. (II) and (III) are more central to this initial rating. I will carefully read the authors rebuttal as well as other reviewers’ comments before I finalize my rating.
 
 
========= Minor points ========= 
 
general:
- P.2: “[...] limitation of ensembles is that the computational cost of training and inference [...] One solution is to distill [...]“ -> this only remedies the computational complexity of inference and in fact increases the training time.
- P.3: [Malinin&Gales 2018] should be cited for equation 4 and the discussion around it.
- P.5: “Ensemble Distillation (EnD) is able to recover the classification performance [...] with only very minor degradation in performance”. Table 1 does not show any degradation for EnD. It shows some degradation for EnD^2 when 100-ensemble is used.
- P.7: “Note, that on C100 and TIM, EnD2 yields better classification performance than EnD”: almost all of the classification improvements are well within one-std. In the specific case of C100, it’s a stretch to call it a “better classification performance”.
- P.7: how many runs are used to obtain the mean and std reported in table 3 and 4.
- P.7: is the PRR in table 3 calculated using the total or knowledge uncertainty for ensemble and EnD2?
Typo:
- P.2: “. Consider an ensemble of models {P[…]” --> a closing parenthesis is missing.
- P.3: “Each of the models P([...]” --> a closing parenthesis is missing.
- P.3: “ the entropy of the expected and“ -> “the entropy of the expectation”
- P.4: “=\hat{p}(x,\pi)” → “\sim\hat{p}(x,\pi)”
- P3-4.: hat and star seem to have been arbitrarily used for input x, parameters theta, pi and p with most of them undefined.
- P.5: script MI is used for mutual information while in P.4 script I is used.
- P.6: “may require the use additional training” -> use of additional
- P.6: Results of EnD in table 2 does not match table 1.