Paper:
Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance. At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation. In this work, we focus on in-domain uncertainty for image classification. We explore the standards for its quantification and point out pitfalls of existing metrics. Avoiding these pitfalls, we perform a broad study of different ensembling techniques. To provide more insight in the broad comparison, we introduce the deep ensemble equivalent (DEE) and show that many sophisticated ensembling techniques are equivalent to an ensemble of very few independently trained networks in terms of the test log-likelihood.

Review:
The paper provide an extensive review of current advances in uncertainty estimation in neural networks with the analysis of drawbacks of currently used uncertainty metrics and comparison on scale the recent method to estimate uncertainty. The paper covers a lot of uncertainty metrics and a wide range of methods. The paper focuses on in-domain uncertainty estimation complementing the recent similar review on out-of-domain uncertainty estimation.
It seems that the paper provides the analysis missing in the current literature. Whereas as mentioned Yukun Ding in a public comment there is a related work on identifying issues with popular uncertainty metrics, the mentioned paper is missing the through comparison of the methods for estimating uncertainty. 
Such kind of thorough analysis (especially performed on scale on large datasets) and comparison is of obvious interest to the community as well as objective comparison of the current state-of-the-art. 
The paper is clearly written and easy to follow.
Based on this, I believe this is a strong technical paper and it should be accepted. However, the analysis in the paper is not overwhelmingly exhaustive. Some of the arguments on that are listed below.
Below is the list of comments/thoughts for potential improvement of the paper:
1.	“In this case, a model is expected to provide correct probability estimates:” – may be not the best choice of words, because for out of domain uncertainty estimation we still expect a model to provide correct probability estimates
2.	The first paragraph on page 3 seems to better fit in Section 2, for example, on the very beginning of Section 2.
3.	“Comparison of the log-likelihood should only be performed at the optimal temperature.” and others alike – personally, I do not support this kind of formatting for a scientific paper
4.	“can produce an arbitrary ranking of different methods. <…> Empirically,” – in the current form it seems that the first statement is somehow theoretically justified and then additionally it is confirmed empirically in this paper. I believe that the authors use empirical observation itself as the justification of the first statement, if that the case it should be reworded here. For example, “can produce an arbitrary ranking of different methods as we show below/ as we show empirically. We demonstrate that the overall …” If my belief is incorrect and there are other grounds that justify the first statement that it is required a reference after this statement.
5.	Italic and non-italic LL usage is unclear
6.	Having “Brier score” emphasised as a paragraph, it seems that there should be a paragraph log-likelihood as well
7.	“In that case, both objects and targets of induced binary classification problems remain fixed for all models” – do the authors consider in this case all out-of-domain objects as having a positive class and all in-domain objects as having a negative class? Because the models are still going to make individual misclassification mistakes
8.	Figure 2 – legend occupies too much space of the plot occluding almost a third part of the plot. Maybe taking the legend out of the plot to the right and squeezing the plot to make a room for the legend would be a better solution
9.	In eq. (4) and (5) subscript DE is not defined
10.	“SSE and cSGLD outperform all other techniques except deep ensembles” –cSGLD was not applied on ImageNet, therefore this statement is a bit misleading
11.	Colour of SWAG in Figure 3 is not very clear. Only excluding other colours I can determine which line is SWAG. Similar to cSGLD, is seems that SWAG was not applied on ImageNet. Why is that if that is the case? And it should be clearly stated at least in experimental setup in Supplementary  
For colours in general, lines in legends are very thin and it is difficult to assess their colour. I appreciate the authors compare a lot of methods and therefore have to use a lot of colours, but it is quite difficult to assess them even on screen not to mention if the paper is printed out. Could the authors please use thicker lines in legends at least?
12.	“Being more “local” methods” – without any context in the main paper this referral to “local” methods is unclear. Also it is good to add a reference to Appendix review of the considered methods in the main text.
13.	Missing details of what kind of augmentation is used in Section 4.3. Is it the same as training augmentation specified in Supplementary? It would require a reference to Supplementary
14.	“(Figure 1, Table REF)” – missing number for Table
15.	“Our experiments demonstrate that ensembles may be severely miscalibrated by default while still providing superior predictive performance after calibration.” – unclear which experiments demonstrate this and superior in comparison to what
16.	The issues of uncalibrated log-likelihood and TACE are clearly shown in the paper, whereas the issues with misclassification detection are only verbally discussed. An illustrative example, at least a toy thought example, could really improve the paper here
17.	The chosen main performance metric is not very convincingly motivated. It is clear why it is based on calibrated log-likelihood, but it is not very convincing why one cannot just used calibrated log-likelihood as a performance metric, why one should base the metric on deep ensembles instead. Also from the long-term perspective, if the community comes up with methods clearly outperforming deep ensembles, the metric would need to be based on one of these new methods 
18.	There is an indirect uncertainty metric that is not mentioned in the paper – uncertainty used in active learning (see, e.g., Hernández-Lobato and Adams, 2015. Probabilistic backpropagation for scalable learning of Bayesian neural networks)
19.	Figures 4 and 5 are too small
20.	“the original PyTorch implementations of SWA” – SWA is not considered in the paper
21.	“hidden inside an optimizer … The actual underlying optimization problem” – it seems that the ICLR audience should be familiar with “actual optimization problems” rather than using blindly the optimizer. It is always good to explicitly write down an equation that is used in a paper, but this wording seems a bit off for ICLR 
22.	“\hat{p}(y^∗_i = j | x_i, w) denotes the probability that a neural network with parameters w assigns to class j when evaluated on object x_i” – it should be \hat{p}(y_i = j | x_i, w), y^*_i is observed
23.	
a.	Why was dropout applied only for limited number of architectures and not applied on ImageNet at all?
b.	Why wasn’t cSGLD applied on ImageNet
24.	“On CIFAR-10/100 parameters from the original paper are reused” – it is better to repeat the reference here
Minor:
1.	Font size in eq. (10) should be the same as the rest of the paper
2.	“Or models achived top-1 error of”: “Or” - ?, “achived” -> achieved 
3.	“for a 45 epoch form a per-trained model”: “form” -> “from”?