Paper:
Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. We propose new measures for calibration, the Static Calibration Error (SCE) and Adaptive Calibration Error (ACE). These measures taks into account every prediction made by a model, in contrast to the popular Expected Calibration Error.

Review:
Paper summary: This paper proposes two simple extensions to Expected Calibration Error (ECE):  1) SCE which accounts for multiclass classification settings by averaging over all the errors due to all classes (as opposed to error in the top one class only in ECE) and 2) ACE which attempts as distributing the predictions equally across the bins (as opposed to have too few highly populated bins in the interval). Authors evaluated their approach using ResNet-110 on CIFAR100 and ResNet-50 on ImageNet against most common post-training calibration methods. 
Pros:
(+): The paper is well-motivated.
(+): The problem is important and has direct real world applications.
(+): The idea is simple and viable to improve ECE.
Cons that significantly affected my score and resulted in rejecting the paper are as follows:
1 - Experimental setting and evaluations:
The biggest drawback in this paper is the experimental setting which is not rigorous enough to show the effectiveness of the proposed metrics due to the following reasons:
(a) The first proposed extension (SCE) is too incremental. However, while I am not against simple and effective methods (I have even listed this as a pro above), I think it should be backed up with more thorough experiments and discussions. SCE is supposed to be more effective in case of having more number of classes however authors do not shed light into this properly by taking advantage of comparing their CIFAR100 and ImageNet 1K experiments. They keep emphasizing on the fact that the numbers provided in Table 1 and 2 are not comparable so how is the reader supposed to understand their differences? They also show their metrics give far less error to models than ECE in Table 1 and 2 and yet it is not clear why that is. Not to mention that authors need to state these results are obtained on how many runs and to report stds because the numbers appear too close and are hence not conclusive. 
(b) Datasets and architectures: Authors have used MNIST and FASHION-MNIST when discussing the shortcomings of ECE. I was wondering what architecture they used for these experiments? The reason I am asking is that it is a known fact (see Guo et al., ICML17) that the simpler the architecture is, the more calibrated its predictions tend to be and vice versa. For instance LeNet 5 appears to be much more well-calibrated than more modern neural networks such as ResNet variants despite being less accurate (Guo et al, 2017). Therefore, it will probably be in their favor to show these issues on a less calibrated model.
(c) Can authors please explain why they have used ResNet-110 on CIFAR100 and ResNet-50 on ImageNet? I assume it might be because they intended to compare to (Guo et al, 2017) but there are more experiments there for comparison. Have they also tried any other architecture? It has been shown before that there can be a noticeable difference across different architectures used on a fixed dataset. Authors may want to add the arch effect to their evaluation. 
2) The second metric (ACE) while it is well-motivated, it is defined such that it leaves the impression that applying it will involve lots of heuristics as there is no systematic procedure is given. In section 6.3 (where I think could be a section to address the implementation details for this) no effective information is given either expect that authors recommend using 50+ bins for CIFAR and ImageNet without any quantitative support. It is also not clear why authors used 15 bins in their experiments for these datasets in Table 1 and 2 (maybe comparing to the baseline? but they could show both specially if 50+ bins is better for ACE). Moreover, in section 5.2 where it says “the overall calibration error the metric should focus on the regions where the predictions are made (and focus less on regions with few predictions).“, I was wondering if authors have any suggestions on how to identify the predictions with low confidence as coming from in or out-distribution?
2- Structure of the paper, writing, and visualizations:
(a) Writing: The paper, in its current form, needs to be thoroughly proofread and reorganized. The text does not read well and is vague in most parts. Authors have spent too much time explaining the drawbacks of the prior work in 4 pages (entire sections 2,3, and 4) only to propose their ideas in section 5.1 and 5.2 in almost half a page. Unfortunately the analysis section (6) acts very poorly in providing a thorough exploration into their method.
(b) Dividing each section to too many subsections has hurt the flow as each section is too short and does not provide deep evidence to support its title/subtitle. 
<<< Note that the followings are less major and are given only to help, and not necessarily part of my decision assessment >>>
(c) Results shown in Table 3 and 4 are reporting “TACE” which is technically explained in the Appendix. Apart from the fact that authors need to make sure the Table comes before the references (not in the middle) but more importantly, as a paper submitted to ICLR I would expect it to be self-contained and be able to provide all the details needed. Authors should either move this Table to the appendix or move section (A.1) to the main text. 
(d) In section 3.4. Authors provide support for their claim on issues with binning scheme with a screenshot of their code in the Appendix. I think it is important to show this effect but in perhaps a table with quantitative results within the main text. 
(e) Title for Section 5 as the main contribution of this paper should be changed to reflect that.
(f) Abstract in its current wording, does not provide absolute no detail into the proposed metrics. Whereas it can, because 1) it encourages the reader to keep reading 2) the method is simple enough to be summarized here.
(g) The figures do not meet the conventional scientific standards and have to be significantly improved. Standard deviations are missing.
(h) There are also grammar errors and typos, (for example on page 1 paragraph 5, the word "may" should be "many"), for which I have found passing my writing through the free version of Grammarly very helpful in getting rid of most such errors. 
As a final note, I think this paper can be contributive to the field as it provides a novel and simple extension to a widely used calibration metric. However it needs to be written in a more effective way and be supported by a more rigorous experimental setting. Therefore, I will be willing to change my score if the presented issues will be addressed.