Paper: Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing (ICLR-2024)

Reviewer comment (Novelty):
The overall contributions of the paper seem a bit limited as it boils down to applying (Gaussian) kernel to the original distribution for obtaining a smoothed estimate of it (which is already done in the prior work) and a technique to determine the kernel bandwidth.

Author response:
Note that our contribution is twofold: (1) deriving a theoretically-principled choice of Gaussian bandwidth, which was an unsolved complication in practice, and (2) proving that, with this bandwidth choice, the induced calibration measure satisfies the strong theoretical guarantees of a "consistent calibration measure".
We acknowledge that the concept of Gaussian smoothing for reliability diagrams is not new (as we cite on page 3), but the details matter significantly, and it was not previously known that this method could satisfy meaningful theoretical guarantees.

Reviewer comment (Experiments):
The empirical results are missing experiments on more image datasets, especially that cover out-of-distribution scenarios, and different types of networks.

Author response:
See above; we consider this important but out-of-scope for our paper.

Reviewer comment (Presentation):
It is a bit difficult to understand that, how the smooth reliability is visually more interpretable than the binned reliability.

Author response:
We have included a new section (Appendix E,  FIgure 3) in the updated pdf, which demonstrates how SmoothECE can behave *much* better than binned ECE in certain settings.

Reviewer comment (Presentation):
Page 3 bottom (initial proposal for smECE): The actual smECE defined in eq. (4) seems to make more sense than \widetilde{smECE} de facto, so the \widetilde{smECE} is slightly distracting. The reviewer questions the reason for not just introducing the actual smECE earlier.

Author response:
Regarding distinction between $\widetilde{SmECE}$ and $SmECE$: we agree that indeed $SmECE$ is a slightly more natural notion as a calibration measure. Those two notions are close to each other, both in spirit, as well as formally (see for instance Lemma 10 in appendix A). On the other hand, the $\widetilde{SmECE}$ is more directly reflected in a visually interpretable reliability diagram, which we argue for constructing. As such, we chose to define both those notions and highlight the close relations between those two, while arguing for SmECE to be the more fundamental construction, which should be eventually used to report calibration error itself. A slightly more detailed discussion about this is present in Appendix A.

Reviewer comment (Writing):
The discussion of Błasiok et al.'s work should be more detailed for someone unfamiliar with it. After Definition 5, comments should be provided on why consistent calibration measures are a useful notion.

Author response:
Thank you for pointing out that the importance of consistent calibration measures was not discussed enough, that is a great point. We have included additional discussion in the current write-up (Appendix E), which we will consider moving into the main body in the full version (where we will have more space).

Reviewer comment (Theory):
Theorem 6: Is this a good, bad, or ok "rate" (perhaps in terms of the \alpha_1 and \alpha_2)? What does it mean in practice? Does the rate show up in experiments?

Author response:
The more precise upper and lower bounds between smECE and distance to the nearest perfectly calibrated predictor are given at the end of Section 3.3 (giving $\alpha_1 = 1, \alpha_2=1/2$). This provides a decent continuity guarantee for the smECE measure. We provided some synthetic examples in Figure 3, where this continuity is highlighted — binning ECE and the associated binning diagram behaves extremely differently on two very close datasets of (prediction, outcome), while smECE provides a stable picture of mis-calibration.

Reviewer comment (Evaluation):
The experimental analysis focuses on producing some reliability diagrams, which look nice, but the reviewer asks if there is a way to show the benefits of the highlighted consistency property.

Author response:
Indeed, that is a great point. We have added a set of synthetic reliability plots highlighting how commonly used binning variants of ECE behave extremely unreasonably (on a carefully crafted data) due to discontinuity issues, and how our method provides much more sensible diagrams in this exact scenario. These are included in the new Appendix E.

Reviewer comment (Experiments):
Can further evidence be shown that the smECE is close to the true ECE in finite samples (on synthetic data)? Fig 1d presents one simple setup.

Author response:
The current experimental section shows some examples on which SmoothECE happens to behave closely to binning ECE on actual data. In contrast, and more importantly, we have now added synthetic examples for which SmoothECE behaves drastically differently than binning ECE (Appendix E). We argue that in those scenarios it is the binning ECE that behaves in a very undesirable way, and SmoothECE is fixing inherent problems with this definition, while being accompanied by informative reliability diagram.

Reviewer comment (Evaluation):
Page 3 top: The reviewer agrees with the statement “Thus we have a situation ... consistent calibration measures.” and further adds that reliability diagrams are nice because they are interpretable by humans. Widmann et al.'s kernel-ECE is useful for comparison but its adoption has been limited due to the interpretability problem (lack of reliability diagram).

Author response:
Thank you. Yes, that was exactly the point why we have introduced the SmECE: ECE by itself is mathematically poorly behaved, and better methods for stand-alone measuring calibration error has been proposed (as well as abstracting away the desirable properties of such measures, which ECE fails to posses). Unfortunately, those metrics were not widely adapted, in part because binning ECE can be accompanied with a useful reliability diagram. We proposed a method which has desirable mathematical properties of those better calibration measures, while being associated with a visually interpretable calibration diagram.

Reviewer comment (Writing):
Page 3 bottom: \tilde{K} is used before being defined.

Author response:
Thank you, fixed.

Reviewer comment (Novelty):
Page 5 (center): The methods would in fact apply to any of the binary-reduction-based calibration notions discussed in the paper "Top-label calibration and multiclass-to-binary reductions".

Author response:
Indeed, thank you for pointing us to this work. We have added this citation in the revised pdf.

Reviewer comment (Experiments):
The experiment section is weak. If the proposed method alleviates the discontinuity problem of BinnedECE (to be confirmed in Questions), some experiments showing how it can be beneficial (e.g. optimizing a loss involving the calibration metric) should be included.

Author response:
* Re. discontinuity problem: Thank you for your suggestion – it will surely improve our paper to provide experimental data highlighting the issues with discontinuity of BinnedECE and associated reliability diagrams, as well as showing how those problems disappear when using our method instead. We have included such a diagram now, in Appendix E (Figure 3) of the updated pdf.

Reviewer comment (Reproducibility):
The code is not provided. Perhaps the author(s) can use https://anonymous.4open.science to share it anonymously.

Author response:
* Re. code: We have now uploaded the anonymous code here: https://anonymous.4open.science/r/smoothece-anon/ (also in the attached .zip file), as we mention in our common response.

Reviewer comment (Presentation):
In figure 2, what are the shaded areas for the smooth reliability diagrams? I thought they are kernel density estimates of the predictions but I don't understand why for (d), the red line is surrounded by that area.

Author response:
* Thank you for pointing out the inconsistency in Figure 2, plot (d). The shaded areas (and the thickness of the red line) indeed were meant to indicate the kernel density estimates. The plot (d) was produced by using our library with a different option enabled, showing the confidence intervals for the red line. We will change this back to the standard format, so that it is consistent with other plots. (We chose not to focus on confidence intervals in this paper in order to provide cleaner plots, but they are an option supported by our library).

Reviewer comment (Experiments):
The major contribution of the paper is the new smooth ECE estimator over the commonly used Binned ECE. The paper discussed several disadvantages or flaws of the binned ECE in the introduction, but I think these flaws are not well demonstrated in the experiments, e.g. "changing the predictor by an infinitesimally small amount may change its ECE drastically", "overly sensitive to the choice of bin widths." I think it is beneficial to include some synthetic experiments to demonstrate these problems of the Binned ECE estimator and show how the proposed estimator overcomes them.

Author response:
Thank you for this suggestion. We have now updated the pdf with a new section in Appendix E, including a new Figure (3), to help illustrate the problems with Binned ECE, and the resolution of Smooth ECE. We hope this addresses your concern.