From: ICLR.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Reject
comment: The reviewers all generally find the paper both well-motivated in addressing an important challenge as well as well-written. However, there's quite a bit of hesitation around whether the proposed metric is convincing enough as an approach to measure local calibration.

Reviewer 76PS and 784d's concerns around the choice of feature map and associated hyperparameters remain unaddressed, and I agree with their concern. There is no clear understanding of what constitutes a "good" feature map, which makes the metric quite difficult to use whether as a benchmark of ML methods or for general application.  I recommend the authors use the reviewers' feedback to enhance their preprint should they aim to submit to a later venue.

From: ICLR.cc/2022/Conference/Paper1895/Reviewer_784d
title: Thanks for the follow-up
comment: Thanks for the informative comment and I appreciate the effort to improve the experiments section.

*"When all protected groups are explicitly specified, existing methods can ensure fairness, but because of the sheer number of possible minority groups, practitioners may not be able to enumerate all of them. On the other hand, we show that LoRe achieves good fairness amongst explicit groups without needing to specify them ahead of time."*

This still requires the knowledge of a feature map that can (to some degree) recognize the minority groups, right? Otherwise, the definition of 'local' can be messed up for minority groups and as a consequence, the group-wise fairness for such groups might not be achieved.

From: ICLR.cc/2022/Conference/Paper1895/Reviewer_76PS
title: Thanks for additional results and clarifications, however arguments are still not convincing
comment: I would like to thank the authors for their response and additional experiments to address the points raised. 

I believe the crux of my concern is that this paper proposes a metric, and there are certain aspects that would be expected of a robust metric, which the proposed metric does not seem to meet.

While it is encouraging to see stable performance across different feature maps in the additional experiments, the dependence on "how good" a feature map is still remains as a concern: e.g. if you were to deploy this method and metric on a non-benchmark dataset where there is no standard pre-trained model, should you simply use your own model's feature maps for both LoRe and the LCE metrics? and what are the possible pitfalls for overfitting in this case? Or should you train another classifier to obtain an independent feature map? With no argument on what constitutes a "good enough" feature map to calculate the metric with, I can easily see how the metric can be exploited with degenerate or manipulated feature maps.

The argument for choosing an appropriate $\gamma$ is still very vague....does it really just come down to visual inspection and roughly choosing a midpoint between two limiting behaviors? A metric should be principled, and I'm not sure how principled this would be.
It does not seem to be explicitly stated, but I assumed that a fixed $\gamma$ was used for the method LoRe, even while different $\gamma$ values were used in calculating MLCE in the MLCE vs. $\gamma$ plots....this is the case, right? What happens when you choose a different $\gamma$ in applying LoRe?

Further, the authors have not adequately addressed the question, "why should I necessarily care more about LCE than classwise-ECE?" Since the notion of local calibration still seems conceptually very similar to classwise-ECE, I believe it is important to report classwise-ECE in the experiments, and show that the signals provided via evaluation with classwise-ECE is lacking and thus convince the reader that inspecting LCE is necessary (especially in the case when we are not necessarily concerned with the fairness setting). Simply stating that "classwise-ECE doesn't work well with many classes" is not convincing.

From: ICLR.cc/2022/Conference/Paper1895/Reviewer_Es9F
title: Regarding the added definition in Section 3.1
comment: Thanks for the detailed responses.

On the definition of (Perfectly Local Calibrated), I have some additional comments.

Denote $X$ as the feature random variable, $Y$ the target RV and $S=f(X)$ the model output RV. The definition of a Bayes optimal model is solely based on the joint distribution of $P(X, Y)$. The definition of calibration is solely based on the joint distribution of $P(S, Y)$. Both these two definitions don't require any additional evaluation metrics. However, the current definition of (Perfectly Local Calibrated) is actually based on the LCE measure as well as an unknown kernel function.  So it still feels less solid compared to the Bayes-opt and conventional calibration definitions.

From: ICLR.cc/2022/Conference/Paper1895/Authors
title: Response to Reviewers
comment: We thank all reviewers for their time and thoughtful feedback on our work. We have uploaded an updated revision which we believe incorporates the reviewer suggestions with additional experiments and clarifications (new changes are in blue). 

In summary, our work presents the following contributions: 
We introduce the Local Calibration Error (LCE), a novel calibration metric that spans the gap between fully global and fully individual calibration. Our definition considers locality based on the feature space, and thus provides more flexibility in defining groups (users can choose a feature space according to their downstream applications and don’t need to explicitly pre-define groupings) 
We propose a local recalibration method, LoRe, that transforms a model’s confidence predictions to improve the local calibration. 
We demonstrate empirically that LoRe achieves better local calibration and leads to better downstream performance on some fairness and decision-making tasks.

Some key changes and clarifications are as follows:
- [R1, R2, R4, R5] Additional experiments to demonstrate LoRe’s efficacy over a wide range of parameter choices, including various additional feature spaces, dimensionality reduction methods, and additional datasets
- [R1, R2, R4] Clarification of our procedure for choosing gamma
- [R3] Clarification of the definition of perfect location calibration. We note that better local calibration does imply that the model is closer to the Bayes-optimal model
- [R4] Clarification of differences from existing fairness metrics
- [R4] Additional experiments to compare with group-wise recalibration

From: ICLR.cc/2022/Conference/Paper1895/Authors
title: Thank you for the followup.
comment: Thank you for following up.

On the definition of local calibration: I think the confusion here may be over the LCE itself - the LCE itself is not an analog to the ECE, but the average LCE is. Similarly, the MLCE is an analog to the MCE.

With global calibration metrics such as the ECE, the points are binned by confidence, and we consider the difference between the confidence and the accuracy /within each bin/. Then, we average over all of the bins. We do something similar for the LCE: we consider the points that are within the same confidence bin, and then we take a summary statistic such as the mean over all of them to get e.g. the average LCE. Thus, the average LCE is the natural extension of the ECE and the MLCE is the natural extension of the MCE.

In our paper, we mostly used the MLCE for reliability, since we want our predictions to be calibrated for every confidence level, but using the average LCE would be completely fine too.

On the CDFs in Fig. 2: We note that it tells us that the LCE is nontrivial - i.e. it’s different from the ECE. These plots show that different local neighborhoods have different calibration errors, which may have implications for fairness or reliability. This also means that using only the global average is insufficient, and we should inspect these neighborhoods in a more fine-grained way.

From: ICLR.cc/2022/Conference/Paper1895/Authors
title: Response to Reviewer fisU
comment: We thank the reviewer for their constructive comments. We clarified the rationale behind choosing a binning scheme over model confidences, added experiments demonstrating that LoRe achieves good results over a wide range of hyperparameter choices, clarified the procedure used to choose gamma, and addressed concerns about the theoretical soundness of the approach.

_"1. Why choose the binning scheme over predicted model confidences? [...] 2. For a unique sample, the neighborhood the authors defined maybe change when the size of its bin is changed."_

Global calibration measures like the ECE compute the expected gap between the model’s confidence and accuracy, conditioned on the model’s confidence. Conditioning on the model’s confidence (often approximated in practice by binning) is crucial to evaluating how closely that confidence reflects the accuracy one can expect. In our proof of Lemma 1 (see Appendix C), we show that using this binning scheme causes the maximum LCE to converge to the global MCE as the kernel bandwidth gamma goes to infinity. One can also show a similar convergence result for the average LCE and ECE.

One could use other schemes to define the LCE (e.g. soft kernel similarity in confidence space, or a strategy similar to the reviewer’s ref [1]), but they would not converge to the MCE/ECE as the kernel bandwidth goes to infinity. However, these are interesting directions which we plan to investigate in future work.

_"3. Is the promising experimental result due to the superiority of the method, or is it an accidental result from a specific kernel function and feature map?"_

We have run additional experiments with several different feature maps including AlexNet, DenseNet121, and ResNet101 features (Figures 10-13), and using both t-SNE and PCA (Tables 1-3 & updates to Figures 5-7). The results remain consistent with before. LoRe is still the most effective method, ​​showing that it has some universality. In our initial experiments, we also tried using a Gaussian kernel, and it yielded very similar results as well. The only kernel hyperparameter is gamma, and we selected it by taking the midpoint between the two limiting regimes of the LCE (see additional explanation below). 

_"4. Since performance is sensitive to many choices, [...] the authors set \gamma as 0.2 and show the good performance, but how can I choose the best one?"_

We would first like to clarify that because the MLCE depends heavily on $\gamma$, we plot graphs showing the MLCE achieved by various recalibration methods over a wide range of $\gamma$. On the other hand, our method LoRe requires us to choose a single value of $\gamma$. We choose this value based on the following reasoning: there are 2 limiting regimes for the value of LCE, a worst-case individual calibration error as $\gamma$ approaches 0 and the global MCE as $\gamma$ approaches infinity. Thus, we visualized the graph of MLCE vs. $\gamma$ and chose the midpoint between these limiting behaviors (e.g. see Figure 1), which was $\gamma=0.2$ for t-SNE features and $\gamma=0.4$ for PCA features. The appropriate bandwidth depends on the specific feature space, but we present a clear empirical method to select it.

_"5. The difference between the confidence and the accuracy for one sample is not reasonable when using it in Eq. 4. Since the accuracy is not a good metric for only one sample, although the authors propose using neighbors (what about the case the prediction of the current sample is wrong?)"_

We formally show in the proof of Theorem 1 (see Appendix D) that our definition of LCE(x) is reasonable because it can be estimated sample-efficiently, unlike the accuracy for a single sample (as you rightly point out). This is specifically because we introduce a kernel similarity term which compares a sample to its neighbors. Informally, LCE(x) computes a weighted average of the difference between the model’s confidence and accuracy in a neighborhood of x. This weighted average is high if the model’s classification of x is confident and incorrect, and it is low if the model’s classification of x is confident and correct. The intuition underlying this choice is that trained classifiers should make similar mistakes on similar inputs.

_6. Using kernel to introduce locality is not new._

We respectfully disagree. To our knowledge, our work is the first to introduce any notion of feature-space locality in the context of calibration.

_"1. The style of Table 3 is different from other tables"_

Because this table is information-dense, we use a slightly different formatting to improve readability.

_"2. In Fig. 3 and Fig. 4, there are no error bars, while the other same type figures (Fig. 5, Fig. 6 and Fig. 7) have reported that."_

Because training multiple ImageNet classifier models is expensive, we used pre-trained models for ImageNet. Thus, we could not calculate the confidence intervals over multiple base classifiers, as we did with the smaller datasets.

From: ICLR.cc/2022/Conference/Paper1895/Reviewer_hPxn
title: Thanks for the Clarifications
comment: Thanks for the clarifications. Some quick responses:

On the definition of local calibration: let me clarify and see if I'm misunderstanding. Global calibration metrics, such as ECE, first condition on the difference between confidence and accuracy within a confidence bin, and then average over the whole dataset, including *all* confidence bins. However, local calibration error only considers a single confidence bin. Wouldn't the natural analogue to ECE be averaging over all bins, but only considering datapoints near x in the feature space? I'm not sure why we wouldn't take this definition, instead of the single-bin definition proposed in the paper.

On the CDFs in Fig. 2: I understand them a bit better now, thank you. It's still not clear to me that they communicate much about clustering of LCE beyond random chance: we would never really expect in practice to see the delta distribution you describe. It's not necessary you come up with such a statistical test in the paper, but it's not really clear what to take away from the figure.

From: ICLR.cc/2022/Conference/Paper1895/Authors
title: Response to Reviewer 76PS minor comments
comment: _What is $d$ in the denominator of the Laplacian kernel definition (Section 3.2)?_

$d$ is the dimension of the feature space.

_In Figure 2, is the vertical dashed line simply ECE? It is referred to as "average calibration error" in text in Section 3.3._

The vertical dashed line is the ECE within the specific bin being examined.

_Are the experiments with ImageNet run with just 1 seed?_

Because training multiple ImageNet classifier models is expensive, we used pre-trained models for ImageNet. Thus, we could not calculate the intervals over multiple base classifiers trained with different seeds, as we did with the smaller datasets.

From: ICLR.cc/2022/Conference/Paper1895/Authors
title: Response to Reviewer 784d
comment: We thank the reviewer for their constructive comments. We have clarified the differences between the LCE and existing fairness metrics, clarified our procedure for selecting a $\gamma$ value, and added additional experiments to demonstrate the effectiveness of LoRe in several additional settings.

*“The authors indeed mentioned about fairness and causal inference as two potential interesting applications, and conducted fairness related tasks in the experimental section. But the connection is not clear to me. Furthermore, the authors should provide some comments on the pros and cons of LCE over existing fairness related group-wise calibration measures (ref i).”*

The goal of LCE is similar to existing measures like equalized odds as mentioned in [i]: both measures (and associated algorithms) aim to ensure fairness for multiple groups. The main difference is that the LCE defines soft groups based on locality in a semantic feature space (e.g. visual similarity for image data), whereas methods like [i] consider explicitly defined groups. When all protected groups are explicitly specified, existing methods can ensure fairness, but because of the sheer number of possible minority groups, practitioners may not be able to enumerate all of them. On the other hand, we show that LoRe achieves good fairness amongst explicit groups without needing to specify them ahead of time.

*“The dimensional reduction procedure. There should be some detailed discussion on how to choose the kernel bandwidth (based on the empirical observation of the transition behavior?).”*

Thank you for the comment. We have run additional experiments using PCA as well (see Tables 1-3 and updates to Figures 5-7 in Appendix B), and we found that it also works well. This indicates that our method is robust to the method of dimensionality reduction. The dimension reduction step serves the role of reducing high-dimensional concentration effects.

We chose $\gamma$ for LoRe based on the following reasoning: there are 2 limiting regimes for the value of LCE, a worst-case individual calibration error as $\gamma \to 0$ and the global MCE as $\gamma \to \infty$. Thus, we visualized the graph of MLCE vs. $\gamma$ and selected the midpoint between these limiting behaviors (e.g. see Figure 1), which was $\gamma=0.2$ for t-SNE features and $\gamma=0.4$ for PCA features. 

*“Experiments. While the provided experiments are highly useful, I think both 5.2 and 5.3 need improvement. Specifically, in 5.2, include some popular baseline datasets such as CIFAR 10/100 ; in 5.3, the authors might want to compare against existing group-wise calibration measures & recalibration methods.”*

Thank you for the comment. We have added additional experimental results for CIFAR-100 and CIFAR-10 (see Figures 8-9 in Appendix B). For Section 5.3, we have added results using group temperature scaling and group histogram binning, where we apply temperature scaling or histogram binning within each group. Interestingly, LoRe actually achieves better maximum group MCE than either of these methods (Table 1), though the mean group ECE is similar (LoRe achieves 0.079, 0.042, and 0.048 in settings 1, 2, and 3, respectively; group temperature scaling achieves 0.089, 0.039, and 0.050; group histogram binning achieves 0.088, 0.043, 0.042). We speculate that LoRe’s use of feature space locality allows it to better improve the worst-case group calibration. 

*“Some discussions and/or comparisons about highly related and recent work are missing, such as [i] for fairness and calibration and [ii, iii] for recently proposed kernel-based calibration measures.”*

Thank you for the suggestion; we have added these references into our paper. We discuss comparisons to [i] above. We note that similar to [1], [ii] and [iii] only consider kernel similarity in the confidence space (though they do so in more sophisticated ways), while LCE considers kernel similarity in the feature space. Analogues to [ii] and [iii] which use more sophisticated methods to compute feature space similarity are interesting directions for future work.

[1] Kumar et al. Trainable calibration measures for neural networks from kernel mean embeddings. NIPS 2018.

From: ICLR.cc/2022/Conference/Paper1895/Authors
title: Response to Reviewer hPxn
comment: We would like to thank the reviewer for their detailed comments about our work. We have clarified several aspects of our paper (why binning matters in local calibration, how to read the CDFs of the local calibration plots in Figure 2, additional details about how we applied t-SNE, and additional details about how we calculated the correlation in Table 2). We have also added several experiments to demonstrate that LoRe achieves good results over a wide range of choices of feature representations and dimensionality reduction methods, and that using the average LCE rather than the MLCE yields similar results as well.

_“it's not clear to me why we are filtering for prediction bin as well as weighting along input space”_

Thank you for the question. Global calibration measures like the ECE compute the expected gap between the model’s confidence and accuracy, conditioned on the model’s confidence. Conditioning on the model’s confidence (often approximated in practice by binning) is crucial to evaluating how closely that confidence reflects the accuracy one can expect. The key contribution of local calibration is to add a dependence on the input space to the existing calibration framework, which necessarily depends on the output space. In our proof of Lemma 1 (see Appendix C), we show that using this binning scheme causes the maximum LCE to converge to the global MCE as the kernel bandwidth $\gamma \to \infty$. One can also show a similar convergence result for the average LCE and ECE.

_“the choices around representations which are operated on  [especially t-SNE] seem under-discussed.”_

We have run additional experiments with a different dimensionality reduction method (PCA), as well as different feature maps (InceptionNet, AlexNet, DenseNet121, and ResNet101). Our new results are shown in Appendix B, in updates to supplemental figures 5-7, and in the new supplemental figures 10-15. The results are qualitatively identical.

_“how is a test point placed in a t-SNE representation space which was fit on a validation set?”_

We computed the t-SNE representation of the recalibration and test set together, in batch. However, in initial experiments we found similar results using neural t-SNE, so we used t-SNE for simplicity. We also perform new experiments using PCA which also yield similar results (see updates in Table 1-2 and updates to supplemental figures 5-7). For both neural t-SNE and PCA, we trained the embeddings on only the recalibration data.

_“2-d pictures in Fig 2 are very nice but I find the right side confusing ... Also, you state that the regions of over/under-confidence are clustered in input space, but it's not clear how we would tell if this is more than we'd expect given random variation or not”_

Thank you for the question. Figure 2 demonstrates that there is value in computing LCE(x) for a particular input x. If the local calibration error gave us no new information over global metrics, we would expect to see $LCE(x) = e := E[ 1[f(x’) = y’] | \hat{p}(x’) \in Bin(\hat{p}(x)) ]$ (the “bin calibration error”) identically. This would correspond to a CDF which is flat everywhere but increases sharply at e. Instead, both CDF’s have a near-linear rise, which indicates that LCE(x) is distributed almost uniformly in a fairly large range. In other words, LCE(x) captures local miscalibrations that global metrics miss. To better understand whether the input-space clustering of LCE(x) is random, one could manually inspect examples from these clusters to see if there are identifiable trends associated with systematic under/over-confidence. 

_“I'm also curious about something like average LCE”_

Thank you for the comment. We have added results for the average LCE, and we find similar trends to the maximum LCE. See figures 16-19 for plots of average LCE vs. gamma on ImageNet, Setting 1, Setting 2, and Setting 3.

_“Table 2: how is this correlation calculated?”_

For each setting, we train multiple base classifiers and then compute the relevant metrics for all baseline methods. This gives us 6 data points per base classifier (no recalibration, temperature scaling, histogram binning, isotonic regression, MMCE optimization, and LoRe). We then train k base classifiers with different random seeds to get 6*k data points of form, (ECE, MCE, NLL, Brier, MLCE, max group MCE). We compute the correlations within this set of points.

_“Table 4: I'd like to see standard deviations”_

Because training multiple ImageNet classifier models is expensive, we used pre-trained models for ImageNet. Thus, we could not calculate the intervals over multiple base classifiers, as we did with the smaller datasets.

_“Fig 3: tell me what these legend abbreviations mean”_

Thank you for pointing this out. Original means no recalibration; TS means temperature scaling; HB means histogram binning; IR means isotonic regression; MMCE means optimizing the MMCE; and LoRe is our local recalibration method. We have clarified this in the paper.

From: ICLR.cc/2022/Conference/Paper1895/Authors
title: Response to Reviewer Es9F
comment: *“One main concern with this paper is the ambiguity of the proposed definition of Local Calibration. On the one hand, we have an existing definition of a perfectly calibrated model, which is solely defined in terms of the probabilistic output space. On the other hand, we also have the definition of a Bayes-optimal model, which is defined only on the feature space. For both cases, we can define a classifier that is optimal for either measure.”*

Thank you for the comment. We have added a paragraph to Section 3.1 which defines perfect local calibration, and discusses its relationship to both perfect calibration and Bayes-optimal models. In particular, being a Bayes-optimal model does satisfy perfect local calibration.

*“Also, while the authors used NLL and BS for their experiments, they did not discuss much the framework of Proper Scoring Rules. [...] I would therefore encourage the authors to consider potential links to PSRs further so that the local calibration concept can be clearly linked to calibrated / Bayes-optimal classifiers.”*

Thank you for the suggestion. Decomposing a proper scoring rule into local calibration and refinement components is an interesting direction that we plan to investigate in future work. The goal of this work is to introduce the novel concept of local calibration which future work can expand upon.

*“A minor suggestion is also to consider links to existing kernel-based probabilistic classifiers (relevant vector machines / Gaussian Processes), where the probability output is directly optimised with a given kernel space.”*

Thank you for the suggestion. We note that we consider MMCE [1] as a representative baseline for kernel-based methods. 

[1] Kumar et al. Trainable calibration measures for neural networks from kernel mean embeddings. NIPS 2018.

From: ICLR.cc/2022/Conference/Paper1895/Authors
title: Response to Reviewer 76PS
comment: We thank the reviewer for their thoughtful comments. Many of these concerns seem to revolve around LoRe and LCE having multiple tunable hyperparameters. We have performed several additional experiments which demonstrate good results across a wide range of hyperparameter choices. Ultimately, these hyperparameters are used to capture some notion of locality based on a semantic feature space. For instance, in our experiments, there is locality based on visual similarity (note that deep neural networks have been shown to capture visual similarity remarkably well [1]).

[1] Zhang, et. al. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.

*“What happens empirically when different feature maps and kernels are used for applying LoRe and evaluating with LCE, MLCE?”*

We have run additional experiments using different feature maps for recalibration (i.e. applying LoRe) and evaluation. (Specifically, we use Inception features for evaluation and AlexNet features for applying LoRe, and we use DenseNet121 features for evaluation and AlexNet features for applying LoRe.) The results are shown in Figures 14 and 15 in Appendix B; they are qualitatively identical to the original results.

*“It seems as though computing the proposed metric (LCE, MLCE) depends on 1) a specific feature map, 2) a specific method for further reducing dimensions, 3) a specific kernel function, and 4) specific settings of the hyperparameters for the kernel.”*

We have run additional experiments with several different feature maps including AlexNet, DenseNet121, and ResNet101 features (Figures 10-13 in Appendix B), and using both t-SNE and PCA (Tables 1-3 & updates to Figures 5-7 in Appendix B). The results remain consistent with before. LoRe is still the most effective method, ​​showing that it has some universality. In our initial experiments, we also used a Gaussian kernel, and it yielded similar results as well. Our only kernel hyperparameter is $\gamma$, and we selected it by taking the midpoint between the two limiting regimes of the LCE. 

*“Continuing from the above point, is there a reason why the authors chose a Laplacian kernel?[...] How about different feature maps?”*

We ran initial experiments with both a Gaussian kernel and a Laplacian kernel, and the results looked qualitatively identical; thus, we arbitrarily chose a Laplacian kernel for the full experiments for consistency. There is nothing special about the Laplacian kernel; the main requirement is that the kernel obeys the form described in Paragraph 2 of Section 3.1. We also added additional experiments with different feature maps in Figures 10-15. 

*“How should $\gamma$ be chosen?”*

There are 2 limiting regimes for the value of LCE, a worst-case individual calibration error as $\gamma \to 0$ and the global MCE as $\gamma \to \infty$. Thus, we visualized the graph of MLCE vs. $\gamma$ and selected the midpoint between these limiting behaviors (see Figure 1), which was $\gamma=0.2$ for t-SNE features and $\gamma=0.4$ for PCA features. The appropriate bandwidth depends on the specific feature space, but we present a clear empirical method to select it.

*“How much does the LCE metric diverge from classwise-ECE? [...] I believe it would have been more meaningful to 1) compare against methods that target classwise-ECE, and 2) report classwise-ECE values in the experiments.”*

You are correct that LCE reduces to classwise-ECE with a specific choice of (improper) kernel. However, classwise-ECE does not work well for datasets with a large number of classes. For example, in Table 3 of [2] the classwise-ECE recalibration algorithm always achieves better classwise-ECE than baseline algorithms for CIFAR-10, but only sometimes does better for CIFAR-100. [3] further shows that classwise-ECE recalibration does not work well for Imagenet-1000.  We conjecture that many similar classes (such as Husky and Poodle) get mapped to similar representations, so our method treats them as similar examples, but classwise-ECE treats them as distinct examples. Finally, in many applications of groupwise calibration that are concerned with fairness, we are interested in groupings that are explicitly distinct from the classes (Section 5.3).

[2] Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
[3] Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration

*“Is LoRe an efficient algorithm?”*

The computational complexity of applying LoRe to any data point is O(Nd), where N is the size of the recalibration set and d is the dimension of the feature map. When d is low, LoRe’s runtime is negligible, taking $\sim$150ms on the UCI Communities and Crime dataset (500 points), $\sim$7.5s on CelebA ($\sim$20k points), and $\sim$8.5s on ImageNet ($\sim$25k points). Note that these runtimes are for recalibrating the full dataset, not just a single data point! This is comparable to the time taken by temperature scaling.

From: ICLR.cc/2022/Conference/Paper1895/Reviewer_fisU
summary_of_the_paper: This paper focuses on proposing the local calibration error (LCE) to span the gap between average and individual reliability, which measures the average reliability of a set of similar predictions, proposing a metric to measure calibration locally around a given prediction. Experiments are conducted and the results seem good, beating most baselines in the experiments chosen.
main_review: Strengths:
    1. The motivation of this work is very clear.
    2. The article is very clear and easy to understand.
    3. A large number of experiments were carried out to prove authors' claim.
Weakness:
    1. Why choose the binning scheme over predicted model confidences? How about taking all samples into account, or other schemes? 
    2. For a unique sample, the neighborhood the authors defined maybe change when the size of its bin is changed, does it make sense? I think you can refer to [1].
    3. Is the promising experimental result due to the superiority of the method, or is it an accidental result from a specific kernel function and feature map?
    4. Since performance is sensitive to many choices, as discussed in Section 3.2 ``the MLCE is very sensitive to \gamma", the authors set \gamma as 0.2 and show the good performance, but how can I choose the best one in real-word applications? So clear and strong ablation study is necessary.
   5.  The difference between the confidence and the accuracy for one sample is not reasonable when using it in Eq. 4. Since the accuracy is not a good metric for only one sample, although the authors propose using neighbors (what about the case the prediction of the current sample is wrong?)
   6. Using kernel to introduce locality is not new.

Other concerns:
    1. The style of Table 3 is different from other tables, is there anything special the authors want to emphasize? Although the article is clear on the whole, there will still be misunderstandings on this point.
    2. In Fig. 3 and Fig. 4, there are no error bars, while the other same type figures (Fig. 5, Fig. 6 and Fig. 7) have reported that. 

[1] Karandikar, A. , et al. "Soft Calibration Objectives for Neural Networks." (2021).
summary_of_the_review: The paper proposes a new metric for calibration, and propose a post-calibration algorithm for this metric. Overall, the novelty is a little weak and the improvements are not quite clear.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

From: ICLR.cc/2022/Conference/Paper1895/Reviewer_76PS
summary_of_the_paper: This work proposes a new metric for calibration in classification where calibration is measured over localities in the input space, and the localities are determined with a kernel over the feature space. A recalibration algorithm (LoRe) is additionally proposed, which aims to recalibrate the class predictions w.r.t. local calibration error (LCE). The experiments show that 1) LoRe achieves better local calibration (measured by MLCE, i.e. maximum LCE) than baseline recalibration methods on ImageNet, 2) LoRe achieves better group-wise calibration (measured by worst group-wise maximum calibration error) across 3 datasets with identifiable groups, and 3) achieves competitive performance in a simple decision making task where confident incorrect predictions incur high costs. 
main_review: Strengths:
- This paper is generally well written and clear. It is easy to follow and the experiments support the argument made in this paper.
- The notion of local calibration seems novel to me. Although I have not scoured papers in this area, conditioning calibration error not just on confidence, but also feature similarity in the classification setting seems new to me.

Weaknesses:
- While the experimental results look positive, I believe they are not surprising because, as far as I can tell, LoRe uses the same feature map and kernel as the core evaluation metrics (LCE, MLCE). What happens empirically when different feature maps and kernels are used for applying LoRe and evaluating with LCE, MLCE?
- Further, it seems as though computing the proposed metric (LCE, MLCE) depends on 1) a specific feature map, 2) a specific method for further reducing dimensions, 3) a specific kernel function, and 4) specific settings of the hyperparameters for the kernel. It is hard to be convinced that this is a robust metric when calculation depends on so many choices, i.e. I can imagine different rankings for the methods if the evaluation metric was computed with different choices of the above. 
- Continuing from the above point, is there a reason why the authors chose a Laplacian kernel? Have the authors experimented with other kernels? How about different feature maps?
- Choosing a suitable $\gamma$ seems quite arbitrary. Last line of Section 3.2, the authors simply state, "0.2 is a good intermediate point" without any justification. In practice, how should $\gamma$ be chosen?
- How much does the LCE metric diverge from classwise-ECE? If I understand correctly, LCE is meant to measure ECE in localities over the input feature space. Classwise-ECE, in a certain way, provides a proxy of this, because ECE is measured for each true class label, and inputs features should be well clustered within each class label (assuming meaningful features). In fact, if the features were separable by class, I believe LCE should be identical to classwise-ECE with an appropriate kernel and hyperparameter.
- Continuing from the point above, I believe it would have been more meaningful to 1) compare against methods that target classwise-ECE (i.e. perform recalibration by class), and 2) to additionally report classwise-ECE values in the experiments.
- Is LoRe an efficient algorithm during test-time? It seems as though Equation 6 needs to be computed for each test point x, where x_i indexes all points in the recalibration dataset $\mathcal{D}$. This would be in contrast with other post-hoc methods which, e.g. learn a lookup table or tune a scalar during the recalibration phase, and can predict during test-time with a single forward pass of the classifier.

Minor comments:
- What is $d$ in the denominator of the Laplacian kernel definition (Section 3.2)?
- In Figure 2, is the vertical dashed line simply ECE? It is referred to as "average calibration error" in text in Section 3.3.
- Are the experiments with ImageNet run with just 1 seed?
summary_of_the_review: While the concepts introduced and methods proposed are interesting, I do not believe the proposed metric is necessarily robust, and am not convinced of its advantages over an established metric (classwise-ECE, which also has not been reported in this work).
I believe elaborating (theoretically or experimentally) on these points could make a much stronger case for this work.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

From: ICLR.cc/2022/Conference/Paper1895/Reviewer_Es9F
summary_of_the_paper: This paper proposes a new measure of calibration called Local Calibration.
While conventional calibration measures are only defined with probabilistic outputs, the proposed local calibration measure further incorporates the feature space by considering the neighbouring region with a kernel.
The authors also propose a calibration method according to the definition and experimentally demonstrate the advantages.
main_review: In general, the idea of evaluating a model on both the feature space and output space is quite interesting. It provides a potential direction to extend the current probability calibration framework.

On the positive side, while the proposed LCE straightforwardly applies the kernel method, the authors put some reasonable effort into investigating the proposed measure's theoretical properties. While the proof of Theorem 1 relies on some existing work on kernel methods, it might provide some valuable references to other researchers interested in similar problems.

On the negative side, one main concern with this paper is the ambiguity of the proposed definition of Local Calibration. On the one hand, we have an existing definition of a perfectly calibrated model, which is solely defined in terms of the probabilistic output space. On the other hand, we also have the definition of a Bayes-optimal model, which is defined only on the feature space. For both cases, we can define a classifier that is optimal for either measure. Furthermore, we know that being a Bayes-optimal model is a sufficient but not necessary condition for being a perfectly calibrated model.
However, while the proposed local calibration tries to incorporate both output space and feature space, it is unclear what is an optimal classifier under the definition due to the introduced kernel space. The author did not clearly show the requirements for a classifier to be perfectly local calibrated and its relationship to the Bayes-optimal model / perfectly calibrated models. It hence could be problematic when we want to treat local calibration as a measure to optimise.  

Also, while the authors used NLL and BS for their experiments, they did not discuss much the framework of Proper Scoring Rules. PSRs can be used to evaluate the overall quality of a probabilistic forecaster in terms of both refinement and calibration. A classifier with minimal refinement loss and calibration loss is Bayes-optimal and optimal both on the feature and probabilistic output spaces. It is therefore closely related to the local calibration concept proposed in this paper. I would therefore encourage the authors to consider potential links to PSRs further so that the local calibration concept can be clearly linked to calibrated / Bayes-optimal classifiers.

A minor suggestion is also to consider links to existing kernel-based probabilistic classifiers (relevant vector machines / Gaussian Processes), where the probability output is directly optimised with a given kernel space.
summary_of_the_review: This paper provides an interesting method to improve probabilistic outputs with the proposed definition of local calibration. However, the local calibration concept is not linked with the existing definition of calibrated / Bayes-optimal classifiers, so it might not stand firm as a measure to optimise. Some further links to exciting kernel-based classifiers will also bring better insights to the paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

From: ICLR.cc/2022/Conference/Paper1895/Reviewer_784d
summary_of_the_paper: The paper proposes a novel calibration metric, the local calibration error (LCE) that measures the average miscalibration degree of local predictions, which complements the existing global ECE metric. In addition, the authors also propose a novel local recalibration method called LoRe to minimize LCE.
main_review: Strength: the paper is well written and the proposed LoRE method seems simple yet effective. The proposed local calibration concept is novel. 

Weakness:

1. Lack of motivation for the proposed local calibration concept. The authors indeed mentioned about fairness and causal inference as two potential interesting applications, and conducted fairness related tasks in the experimental section. But the connection is not clear to me. Furthermore, the authors should  provide some comments on the pros and cons of LCE over existing fariness related group-wise calibration measures (ref i).  

2. The dimensional reduction procedure. There should be some detailed discussion on how to choose the kernel bandwidth (based on the empirical observation of the transition behavior?). 

3. Experiments. While the provided experiments are highly useful, I think both 5.2 and 5.3 need improvement. Specifically, in 5.2, include some popular baseline datasets such as CIFAR 10/100 ; in 5.3, the authors might want to compare against existing group-wise calibration measures & recalibration methods.

4. Some discussions and/or comparisons about highly related and recent work are missing, such as [i] for fairness and calibration and [ii, iii] for recently proposed kernel-based calibration measures.

[i].Pleiss, Geoff, et al. On Fairness and Calibration. NeurIPS, 2017.
[ii]. Zhang et al. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. ICML, 2020. 
[iii] Gupta et al. Calibration of neural networks using splines. ICLR, 2021. 
summary_of_the_review: As discussed above, my primary concerns for this paper is on the motivation of local calibration, as well as some possible improvement over the experimental section (such as comparison with existing approaches). I hope that those concerns can be addressed in the rebuttal process.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

From: ICLR.cc/2022/Conference/Paper1895/Reviewer_hPxn
summary_of_the_paper: This paper introduces a new metric for measuring model reliability called local calibration, which allows for some insight into which regions of input space a model may be less well-calibrated in. They provide a post-hoc adjustment method for improving local calibration and empirically show that it is effective, without hurting overall measures too much.

main_review: Strengths:

- local calibration is a sensible idea and it's good to see it worked out like this
- local/individualized notions of calibration are an important problem
- same thing for the proposed method: it's simple and seems like the thing you would try in this situation
- the experiments show fairly consistently that this method provides gains in improving local calibration metrics

Weaknesses:

- it's not clear to me why we are filtering for prediction bin as well as weighting along input space - my intuition around what "local" calibration should mean would involve reliability among all similar inputs, rather than just those that also have similar outputs. The analogy to global calibration metrics is not immediately obvious to me and merits more discussion
- the choices around representations which are operated on seem under-discussed. For instance, my guess from reading this paper is that t-SNE is quite important to the practical results of this method - would be good to hear if that's true or not, or how other approaches compare results-wise. If t-SNE is essential to the application of the method, it indicate the method may not be that robust
- Additionally, I'm curious about the application of t-SNE here - how is a test point placed in a t-SNE representation space which was fit on a validation set? As far as I know, t-SNE representations are usually not amortized
- 2-d pictures in Fig 2 are very nice but I find the right side confusing - I need some clarification on how to read these CDFs. Also, you state that the regions of over/under-confidence are clustered in input space, but it's not clear how we would tell if this is more than we'd expect given random variation or not
- I'm skeptical that MLCE is the only metric worth looking at here - I'm also curious about something like average LCE
- Table 2: how is this correlation calculated - over what sample of models?
- Table 4: I'd like to see standard deviations/confidence intervals here. This type of metric is similar to an average LCE, which is good, but the wins seem uneven (which is fine). Also curious to here why some metrics show different results than others


Other thoughts:

- would be interesting to think about customizable kernels to inject notions of similarity we care about (e.g. the improper group-wise kernel mentioned in section 4)
- Fig 3: tell me what these legend abbreviations mean
- as bandwidth goes to \infty, can you discuss what other methods it becomes similar to?
- I would be interested to see exploration of which regions are uncalibrated for models

summary_of_the_review: This is a sensible idea (metric and method) applied to an important problem. The exposition is fairly clear although some design choices are under-discussed. The experiments show that the method works as intended, although they could benefit from some deeper analysis.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.