Paper: A Benchmark Study on Calibration (ICLR-2024)

Reviewer comment (Novelty):
Other questions are mostly expansion of existing studies. This seems to undermine the overall contributions of the paper to some extent.

Author response:
**Re Expansion of existing studies:** We agree studies are the expansion of current research, however, they provide some different results from previous study. For example,
- Previous works use AuC on OoD datasets  as a measurement of calibration[1][2]. However, in Section 4.2 we found that there is no correlation between the AuC and ECE, which indicate AuC on OoD may not be a reliable calibration measurement.
- In section 4.4, we observe that a well calibrated model may not achieve better post-hoc calibration performance than a poor calibrated model. This observation supports the “calibratable” objective proposed in [1][2][3] in a way that research should focus more on obtaining a model with better post hoc calibration performance. Since the post-hoc calibration methods like temperature scaling are computationally cheap and effective, an overall calibration performance worth more attention. See more details in section "Re significance of Section 4.4".
- Section 4.6 study the relationship between accuracy and calibration and section 4.7 study the architecture design for better calibration performance,  which are neither studied before from our understanding.

Reviewer comment (Experiments):
Only post-hoc temperature scaling is used as a calibration technique to evaluate pre- and post calibration performance of a large chunk of models. There have been several new calibration methods, especially in train-time calibration paradigm, such as [A], [B] and [C].

Author response:
**Re other calibration methods:** Following the suggestion, we incorporated Focal Loss[2] and MMCE Loss[11] as additional calibration methods. We trained six human-designed CNN models—ResNet18, ResNet34, ResNet50, ResNet110, Wide-ResNet-26-10, and Densenet121—on CIFAR-10 and CIFAR-100 using Focal Loss and MMCE Loss, both recognized as classic train-time calibration methods. Our observations, consistent with those reported in Appendix B, are detailed in Figure 16. As depicted in Figure 16 and similar to Figure 2, models trained using different train-time calibration methods all exhibit little correlation between results on CIFAR-10 and CIFAR-100. This suggests that the calibration properties of a specific architecture may not generalize effectively across different datasets. In Figure 17, ECE evaluated across different bin sizes shows little correlation between pre and post temperature scaling, indicating that well-calibrated models do not necessarily show improved calibration performance after post-hoc calibration techniques. This trend is particularly pronounced on CIFAR-100, where post-hoc calibration performance becomes negatively correlated with pre-calibration performance. Additionally, we observe that bin size can significantly impact post-hoc calibration performance, aligning with the observations in Section 4.4. In terms of the reliability of calibration metrics, we conducted an analysis of the correlation between all calibration metrics, as presented in Figure 14 in Appendix A and Figure 18 and Figure 19 in Appendix B. Notably, equal-mass classwise ECE exhibits a different pattern compared to other metrics, especially on CIFAR-100, reinforcing the observations outlined in Section 4.3.

Reviewer comment (Evaluation):
In 4.3: 1) What are the possible reasons of ECE showing consistent results with other metrics, although some other metrics are theoretically different? 2) Also, it is not clear that which calibration metric should be preferred over others in the scope of studies?

Author response:
**Re consistency between metrics and preferred calibration metrics:**
Although some other metrics are theoretically different from ECE, they share the same objective which aims to measure the alignment between the true likelihood and the predicted confidence. Thus, ECE shows consistent results with other metrics. However, this consistency is significantly influenced by the way how to approximate this alignment, such as the binning scheme. We mainly attribute the gap between class wise ECE and other metrics to the scenarios where the equal mass binning strategy aims to distribute samples uniformly across bins in a multi-class setting. This approach often leads to a scenario where the negative class is predominant. In practical applications of multiclass prediction, it's common for the model to assign very low confidence scores to a majority of the classes, often more than 95 out of 100. These low scores, hovering close to zero, indicate an absence of uncertainty. Equal mass class wise ECE will assign almost all bins to these near-zero confidence samples, except for one. Thus, equal mass binning experiences a significant decrease in estimation accuracy, making it unreliable. Thus, although all these metrics share the same objective, some of them cannot provide a precise estimation of this alignment in some scenarios. Considering this fact, we conduct the evaluation of the reliability of calibration metrics as in section 4.3, which we observe the unreliability of equal mass class wise ECE. On the other hand, the consistency between different calibration metrics with different theories such as ECE and MMCE support the reliability of metrics against each other. In the current stage, we recommend the use of calibration metrics that are more consistent with each other, such as ECE and MMCE.

Reviewer comment (Writing):
The abstract mentions ‘data pre-processing’ methods for improving calibration but such methods are discussed nowhere in the paper, including the related work.

Author response:
**Re ‘data pre-processing’ methods:** By “data pre-processing”, we mean data augmentation methods such as mixup[6] and Augmix[7], we apologize for the confusion using “data pre-processing”.

Reviewer comment (Novelty):
How the proposed study and dataset would be helpful toward the development of new calibration methods for classification?

Author response:
**Re help to develop to new calibration method:** 
- This paper can benefit the calibration research in different ways, for example:
As an illustration, delving deeper into the architecture design reveals insights into achieving better calibration. For instance, as depicted in Figure 10, a discernible trend emerges, indicating that better-calibrated models exhibit a preference for Conv3*3 in edge 1 over Conv1*1 and favor a residual connection in edge 4 within the NATS-Bench scope. This implies that a better-calibrated model may lean towards incorporating larger kernel sizes in the early layers of a CNN block.
- One can assess the validity of newly proposed metrics by evaluating their consistency with the set of metrics we have examined. Specifically, using the provided checkpoints and the 11 different metrics we assessed, a newly proposed metric can gauge its consistency by evaluating the checkpoints and calculating the Kendall ranking coefficient in comparison to the existing metrics. If the results exhibit little correlation or are negatively correlated with the established metrics, it suggests that the new metric warrants more careful analysis and scrutiny.

Reviewer comment (Experiments):
Would the finding that the bin size has a more substantial impact on Post-ECE be relevant for any train-time calibration technique too?

Author response:
**Re bin-size impact on train-time calibration:** We conduct a toy experiments on bin-size involved train time calibration algorithm DECE[12] and observe that different bin size can bring larger impact on the post-hoc ECE, where small bin size works better for DECE, as shown in the table
| Bin Size |  Accuracy%|Pre ECE%|Post ECE%|
|--|--|--|--|
| 2 | 95.85|3.994 |1.278|
| 5 | 95.04| 3.893|1.107|
| 10 | 95.07| 3.908|1.372|
| 15 | 95.23 |  3.566| 1.361|
| 50 | 94.96 |  4.031| 1.415|
| 100 |95.02  |  3.943|1.523|
| 200 |95.00  |  3.723|1.523|
|  500|  94.97|  4.004|1.561|

Reviewer comment (Novelty):
The motivation about why NAS + Calibration is important is missing in the paper. Unfortunately, the paper lacks a clear justification for studying NAS + Calibration. It is not clear intuitively why this is a good direction to explore. It is not clear why a wholistic approach is not worth exploring over NAS + Calibration. Unfortunately, the paper makes the reader believe that the analysis was done just because it has not done before. I think the paper really needs to justify why NAS + Calibration is a good angle to study.

Author response:
We appreciate the valuable comments from Reviewer Xzoe


**Re NAS + Calibration:** To avoid potential misunderstanding regarding the term "NAS + Calibration", please let us clarify that this study does not involve conducting neural architecture search (NAS) specifically for calibration purposes. The primary focus of this research is the examination of calibration properties. To address calibration-related research questions, such as the reliability of calibration metrics, one approach is to assess the consistency of different metrics based on a substantial number of well-trained models. However, collecting such a substantial dataset is often challenging due to the associated training costs. Fortunately, NATS-Bench [1] provides access to 117.9K well-trained models with various architectural designs, enabling us to conduct a comprehensive and generalisable study.

**Re datasets:** 
Modern large-scale datasets, such as LAION, primarily serve image-text alignment tasks, notably in training stable diffusion models. However, it's worth noting that calibration tasks typically revolve around classification and regression. In recent years, standard benchmarks for calibration studies have included well-known datasets like CIFAR-10, CIFAR-100, and ImageNet [2] [3] [4] [5]. While the prospect of applying calibration tasks to LAION is intriguing, it falls outside the scope of the present work.

---
[1] Dong, Xuanyi, et al. "Nats-bench: Benchmarking nas algorithms for architecture topology and size." IEEE transactions on pattern analysis and machine intelligence 44.7 (2021): 3634-3646.

[2] Tao, Linwei, Minjing Dong, and Chang Xu. "Dual Focal Loss for Calibration." arXiv preprint arXiv:2305.13665 (2023).

[3] Ghosh, Arindam, Thomas Schaaf, and Matthew Gormley. "AdaFocal: Calibration-aware Adaptive Focal Loss." Advances in Neural Information Processing Systems 35 (2022): 1583-1595.

[4] Minderer, Matthias, et al. "Revisiting the calibration of modern neural networks." Advances in Neural Information Processing Systems 34 (2021): 15682-15694.

[5] Gawlikowski, Jakob, et al. "A survey of uncertainty in deep neural networks." Artificial Intelligence Review 56.Suppl 1 (2023): 1513-1589.

Reviewer comment (Experiments):
The study uses small- and medium- scale datasets for analyzing the relationship between NAS + Calibration. In particular, the study uses CIFAR-10, CIFAR-100, and ImageNet datasets for the analysis. While I understand that small datasets are easier to handle given the NAS component of the study, it is questionable the conclusions one can get from these small datasets. While ImageNet is larger, compared to modern large-scale datasets, such as, LAION, its use is also questionable. Modern image classification methods are trained using foundation models that use really large-scale datasets (e.g., LAION) and those are the worth studying in my opinion since they are the ones adopted in industry and are making an impact. I think the paper needs to justify the use of these small- and medium-scale datasets. Otherwise, the conclusions drawn from the studies are not solid.

Author response:
We appreciate the valuable comments from Reviewer Xzoe


**Re NAS + Calibration:** To avoid potential misunderstanding regarding the term "NAS + Calibration", please let us clarify that this study does not involve conducting neural architecture search (NAS) specifically for calibration purposes. The primary focus of this research is the examination of calibration properties. To address calibration-related research questions, such as the reliability of calibration metrics, one approach is to assess the consistency of different metrics based on a substantial number of well-trained models. However, collecting such a substantial dataset is often challenging due to the associated training costs. Fortunately, NATS-Bench [1] provides access to 117.9K well-trained models with various architectural designs, enabling us to conduct a comprehensive and generalisable study.

**Re datasets:** 
Modern large-scale datasets, such as LAION, primarily serve image-text alignment tasks, notably in training stable diffusion models. However, it's worth noting that calibration tasks typically revolve around classification and regression. In recent years, standard benchmarks for calibration studies have included well-known datasets like CIFAR-10, CIFAR-100, and ImageNet [2] [3] [4] [5]. While the prospect of applying calibration tasks to LAION is intriguing, it falls outside the scope of the present work.

---
[1] Dong, Xuanyi, et al. "Nats-bench: Benchmarking nas algorithms for architecture topology and size." IEEE transactions on pattern analysis and machine intelligence 44.7 (2021): 3634-3646.

[2] Tao, Linwei, Minjing Dong, and Chang Xu. "Dual Focal Loss for Calibration." arXiv preprint arXiv:2305.13665 (2023).

[3] Ghosh, Arindam, Thomas Schaaf, and Matthew Gormley. "AdaFocal: Calibration-aware Adaptive Focal Loss." Advances in Neural Information Processing Systems 35 (2022): 1583-1595.

[4] Minderer, Matthias, et al. "Revisiting the calibration of modern neural networks." Advances in Neural Information Processing Systems 34 (2021): 15682-15694.

[5] Gawlikowski, Jakob, et al. "A survey of uncertainty in deep neural networks." Artificial Intelligence Review 56.Suppl 1 (2023): 1513-1589.

Reviewer comment (Experiments):
Most architectures and networks are generated from the similar search space, which might have similar effects and are limited for the conclusions. Varying the search spaces and using human designed architectures are necessary.

Author response:
**Re NAS search space:** In an effort to extend the applicability of our observations beyond a limited search space, we conducted experiments on human-designed Transformers and CNNs. The outcomes of these experiments align with the findings in the main text, as detailed in Appendices A and B.

Reviewer comment (Experiments):
Different architectures on different benchmarks/dataset might require different hyper-parameters. Though it’s hard to search optimal/sub-optimal parameters for different models, it will affect the experimental results.

Author response:
**Re hyper-parameters fine tune:** While we acknowledge that hyperparameter tuning for each model could potentially yield more precise results, the vast number of models involved in our study (117,702 unique neural networks) makes this approach computationally intensive. As a pragmatic tradeoff, we opted to establish a fixed and fair set of hyperparameters for all models to conduct our experiments.

Reviewer comment (Novelty):
This paper explores the calibration properties in neural networks, but I’m much concerned about how those findings impact the future research or guide the designing process for both accuracy and calibration performance.

Author response:
**Re border impact:** Our several observations can help improve the research in calibration and accuracy, for example,
- In section 4.1, we suggest testing model calibration performance on downstream tasks specifically, since the fact that a well calibrated model on source tasks does not necessarily perform well on downstream tasks.
- In section 4.2 and 4.3, we suggest avoiding equal mass class wise ECE or AuC on OoD datasets for calibration evaluation, which helps the calibration evaluation.
- In section 4.4, we observe that a well calibrated model may not achieve better post-hoc calibration performance than a poor calibrated model. This observation supports the “calibratable” objective proposed in [1][2][3] in a way that research should focus more on obtaining a model with better post hoc calibration performance.
- In section 4.5, we observe the tradeoff between accuracy and ECE, which can facilitate the research in prediction accuracy.
- In section 4.6, the impact of bin-size might support the study of those train-time calibration methods which involve the hyperparameter bin size, such as SoftECE[4] and DECE[5].
- In section 4.7, the analysis on architecture design of better calibration models can support the research on calibration from the view of architecture design.