Paper:
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet.

Review:
The paper proposes a simple non-Bayesian baseline method for estimating predictive uncertainty. This is achieved by using ensembles of NNs, where M distinct NNs, from different random initializations, are trained independently through optimizing a scoring rule. The authors simply used a conventional loss function (i.e. softmax cross entropy) as the score rule for classification task, while for regression problems the authors replaced a regular loss function (i.e. MSE) by negative log-likelihood (NLL), which should capture predictive uncertainty estimates. In addition to this, an adversarial training (AT) schema is proposed to promote smoothness over the predictive distribution. To demonstrate the capacity of ensemble and AT to estimate predictive uncertainty, the authors evaluated the proposed method on a large number of datasets across different tasks, both for regression and classification (vision) tasks. Their method outperforms MC dropout for classification tasks while being competitive on regression datasets in term of NLL. Furthermore evaluations demonstrate the ability of the proposed approach to provide uniform predictive distributions for out-of-distribution (unknown) samples, for an object classification task (ImageNet). 
The paper proposed the use of NNs ensemble, which trained independently on ordinary training samples and their corresponding adversary, in order to estimate the predictive uncertainty. Although the proposed method is not significantly novel, nor very sophisticated, the paper is contributing to the domain by expliciting an idea that have not been very much exploited so far. However, the results obtained from the experimentations are not able to support well the proposal.
Training NNs by adversaries is one of the contributions of the paper, but the relationship between smoothness and obtaining a well-calibrated predictive uncertainty estimates is a bit vague. How smoothness obtained using AT can help to achieve a well-calibrated predictive uncertainty estimates? In other words, the effectiveness of AT on the predictive uncertainty estimates is not clearly justified, and the experiments are not supporting this well. For examples, the authors showed AT has no considerable effect for regression problems (Table 2 in Appendix), while it has a significant effect for vision classification problems. Why? Moreover, if we consider AT as data augmentation, can we demonstrate the positive effects of AT on the predictive uncertainty over other simple data augmentation approach, such as random crop? I think that having another experiment with simple data augmentation for training NNs and ConvNets can highlight whether the smoothness has a definite effect on the predictive uncertainty or not, and whether AT is truly key to achieve this.
Recognizing erroneous samples, which are confidently misclassified by NNs, is also another concern for AI safety. The high confidence of misclassified samples prohibit a human intervention then some major deficiencies (accidents) can occur. So, the question can be asked that whether this ensemble can recognize the misclassified samples by providing uniform distributions? The experiments should support that.
Estimating the predictive uncertainty is a key for recognizing out-of-distribution and misclassified samples.  However, a few non-Bayesian papers are missed from the literature review of the paper. For example, Hendrycks and Gimpel (2016) propose a baseline method by using softmax statistics to estimate the probability of error and probability of out-of-distribution samples. Also, ensemble of DNNs have been evaluated by Abbasi and Gagné (2017) for computing the uncertainty of the predictions for adversaries cases, a hard case of out-of-distribution examples.
References: 
Hendrycks, Dan, and Kevin Gimpel (2016). "A baseline for detecting misclassified and out-of-distribution examples in neural networks." arXiv preprint arXiv:1610.02136.
Abbasi, Mahdieh, and Christian Gagné (2017). "Robustness to Adversarial Examples through an Ensemble of Specialists." arXiv preprint arXiv:1702.06856.
** Update following reviewers discussions **
I likely underappreciated the paper, re-reading it in the light of other reviews and discussions with other reviewers, I increased my score to 6. I still maintain my comments on AT, which I think is the weakest part of the paper.