Paper: Calibration-then-Calculation: A Variance Reduced Metric Framework (ICLR-2024)

Reviewer comment (Evaluation):
I don't understand the first line of Table 7 shows the log loss acc is only 5.6% for resnet101 over resnet18. Could you please explain more?

Author response:
A1: We discussed the reason at the bottom of page 8. It indicates that Log Loss and Classification Accuracy will give different results when comparing the performance of pipeline A and pipeline B (e.g. pipeline A performs better if using metric Log Loss while pipeline B performs better if using metric Classification Accuracy). This metric inconsistency can be mitigated by Calibrated Log Loss. This phenomenon indicates that some models may have bad calibration (and hence bad Log Loss) while their performances are actually good if measured by accuracy.

Reviewer comment (Evaluation):
Log loss can sometime indicate the classification accuracy of a model, but they are not identical. How you define model A is better than model B? If I want to focus on accuracy, not log loss, how to apply your method?

Author response:
A2: Thanks for the great question. The definition of “better” is at page 2, equestion 3. If only the accuracy metric is of interest, then the proposed method is not applicable as calibration won’t change accuracy.

Reviewer comment (Writing):
The notations are unnecessarily redundant. For example, we can use \( e, f \) to represent the metrics, and \( A, B \) to represent the pipelines. \( h \) represents the model name. It is redundant since it is a subset of the pipeline. Please reduce the usage of necessary superscripts and subscripts.

Author response:
Q1: concern regarding notations

A1: Thanks for the suggestion. We will fix it in the updated version of this paper.

Reviewer comment (Evaluation):
The paper focuses on the comparisons between two pipelines, which was not necessary. The paper should pay attention to the variance reduction in the performance evaluation of pipelines, i.e., the equation at the bottom of Page 2. For example, the histograms in Figure 1 are good visualization for the variance reduction. Please add similar figures for other experiments.

Author response:
Q2: concern regarding “comparisons between two pipelines”

A2: We believe “comparisons between two pipelines” is necessary and important. Simply doing variance reduction is not useful. As an extreme case, assigning 0 as the metric number will have zero variance, but it’s not useful at all. The goal of this paper is to introduce a new metric to compare pipelines more accurately. Consider a potential application of this new metric — AutoML. For each AutoML search, we need to determine which configuration is better. Due to metric randomness, there might be some noises in the AutoML search process. Using the new metric could improve the accuracy of search, and ideally the final model would be better. We haven’t conducted the AutoML experiments yet, and this is a planned follow-up work.

This also answers the next question about “bias-variance tradeoff”: we evaluate the metric based on its accuracy, defined in Equation 5.

Reviewer comment (Theory):
How do we handle the bias-variance tradeoff in the equation at the bottom of Page 2?

Author response:
Q2: concern regarding “comparisons between two pipelines”

A2: We believe “comparisons between two pipelines” is necessary and important. Simply doing variance reduction is not useful. As an extreme case, assigning 0 as the metric number will have zero variance, but it’s not useful at all. The goal of this paper is to introduce a new metric to compare pipelines more accurately. Consider a potential application of this new metric — AutoML. For each AutoML search, we need to determine which configuration is better. Due to metric randomness, there might be some noises in the AutoML search process. Using the new metric could improve the accuracy of search, and ideally the final model would be better. We haven’t conducted the AutoML experiments yet, and this is a planned follow-up work.

This also answers the next question about “bias-variance tradeoff”: we evaluate the metric based on its accuracy, defined in Equation 5.

Reviewer comment (Novelty):
The Calibrated Loss for multiclass classification is the existing method of "Temperature Scaling". Why is it a proposed method? What is the contribution on that in this paper?

Author response:
Q3: concern regarding “Calibrated Loss for multiclass classification”

A3: We demonstrated that the new metric has a higher accuracy compared with its vanilla counterpart. This demonstrates that our framework “first calibrate, then calculate metric number” is a general method that can be applied to different applications.

Reviewer comment (Novelty):
Calibration of the evaluation results is a general topic. The paper lacks discussions and comparisons of the related work.

Author response:
Q4: concern regarding “Calibration of the evaluation results”

A4: We are aware of the existing work on model calibrations, but to our knowledge we are the first to tackle the deep learning pipeline evaluation problem from the calibration perspective. Our main contribution is to propose that the strategy “first calibrate, then calculate metric number” improves the metric accuracy.

Reviewer comment (Evaluation):
In Section 4, there is less randomness with a larger test dataset. The original metric \( e \) is good enough for evaluation. The difference between \( e_1 \) and \( e \) will become smaller with a larger \( n \). However, for the \( e_1 \), we may need to split the test dataset into two disjoint subsets, one for calibration and one for true evaluation, which will decrease the effective dataset size \( n \). In that case, can we claim that \( e_1 \) is better than \( e \)? If \( n=2 \), \( e \) can use the average of these two samples to obtain the final result, while \( e_1 \) should use one example for calibration and use the other one as the final result. In this case, which case is better? What if \( n=10k \) and even infinity?

Author response:
Q5: concern regarding “less randomness with a larger test dataset”

A5: From the pure statistical perspective, it is correct that large n will create less randomness. As you mentioned, there is a gap between theory and experiments. In the experiments with NN, randomness from SGD plays a more important role, so the theory may not directly apply there. In the SGD case, batch size may serve the role of n. Our theory serves as supporting evidence to our framework that under some simple circumstances (e.g. linear regression), our framework can reduce the variance in theory. We support our framework under more complex settings via experiments.

Reviewer comment (Experiments):
Since the log loss accuracy reported in Tables 1 - 7 does not use the validation test, whereas the calibrated log loss accuracy uses this validation set, it is no surprise that the calibrated log loss accuracy is always higher than the log loss accuracy. This is similar to comparing a model whose hyperparameters are tuned via cross-validation with one that only sees the training set and does not undergo hyperparameter tuning via cross-validation, with the only difference being that the additional hyperparameter introduced in this method is separately introduced via a connection to calibration. The variance reduction property should also not be considered a surprise if the baseline is overfit on the training set.

Author response:
There is a misunderstanding here. When calculating the vanilla log loss, we apply D_test = D_val-test + D_remaining-test. We intend to revise our paper to emphasize this crucial setting, and we're grateful for your attention to this detail.

Reviewer comment (Presentation):
Is the first pipeline in Table 2 labeled "dense" the same as the first pipeline B in Table 1 labeled "remove dense"? If not, the reason for removing dense in Table 1's first row and keeping dense in Table 2's first row is not clear.

Author response:
Thanks for the great suggestion and spotting the typo. They are the same, and we should’ve used the same label “remove dense”. We will fix it in the updated version.

Reviewer comment (Writing):
My understanding is that the proposed framework applied a posthoc calibration on the trained model using a calibration set to adjust model prediction, then used the same loss function (cross-entropy loss for classification and $L_2$ loss for regression) to the adjusted model output on a separate test set. Frame it as a new loss is a bit confusing.

Author response:
Thanks for the great suggestions. We agree that the term “loss” is a bit confusing. We will change the framing to “a new metric”.

Reviewer comment (Evaluation):
The $\text{Acc}(\bar{e})$ defined in equation 5 relies on a ground truth metric, which makes the framework a bit hard to interpret. For example, in the CIFAR10 experiment, accuracy and log loss give different comparison results, I think this can happen because the larger model overfits in terms of cross-entropy loss, but accuracy doesn't drop (e.g. Figure 3 in [1]). The larger model is more accurate but not well-calibrated [1]. In the paper, it says "this metric inconsistency can be mitigated by Calibrated Log Loss", but I think a better way to explain the result is that after applying the posthoc calibration method, the larger model can have better calibration (in terms of cross-entropy loss).

Author response:
Thanks for the great question. In real world applications, models are built to achieve some business needs. In different contexts, the best model may not be the same. For example, if the models are only used to classify images, then accuracy should be the main metric. If the models need to output the confidence, then calibration should be taken into account. That’s the main reason why we need a ground truth metric. This ground truth metric represents the business goal of the models.

Reviewer comment (Evaluation):
Accuracy is used as the ground truth metric in CIFAR10 experiments, but a model with higher accuracy or lower loss is better. This can cause some problems in definition 3.

Author response:
Thanks for the great suggestion. In this case, we can use minus Accuracy as “risk”. We will make it clear in the paper.

Reviewer comment (Theory):
On top of page 4, it says the bias-adjusted predictions $q_i$ are well-calibrated. But calibration usually means $E(Y|f(X)) = f(X)$ for all $f(X)$; the property only says the calibration error is 0 if evaluated using only 1 bin. So I think the claim is not appropriate.

Author response:
Thanks for the great suggestion. We will fix it in our next version of the paper.

Reviewer comment (Evaluation):
I am not fully convinced that the proposed algorithm is realistic. For instance, to obtain the calibrated metric one must have a separate validation set, which is not always the case. One could split the test set into test-val and test-test splits like described in the submission, but that means using a different set of data to test the model, which in itself would be another problem.

Author response:
We would like to express our gratitude to the reviewer for the feedback. We have carefully examined the comments, and it seems there is a **misunderstanding** about this paper. We present our responses to the reviewer's concerns below:

Q1: Concern regarding the proposed algorithm

A1: We don’t need a separate validation set for the algorithm to work. We can always split test data into test-val and test-test splits as you mentioned. This test data split is deterministic, so every model will be evaluated on the same test-val data and test-test data. 

And could you elaborate on “the calibrated metric seems to be model and data specific”? Every metric is model specific and data specific — every metric needs to use model predictions on some data as input to calculate a number. At a high level, the metric is just a function of the model and data, and the function form could be general, and may not admit the expectation form, as we discussed in detail in Section 2 ” PRELIMINARIES AND PROBLEM SETTING”.

Reviewer comment (Evaluation):
The calibrated metric seems to be model and data specific as it uses a specific model's predictions and validation data to calibrate the bias term. This would lead to evaluating multiple models which all use individually different metrics. I do not particularly think that would be a step towards robust and fair comparison of deep learning models.

Author response:
Thank you for the response.

1. Indeed. Given we are proposing a new metric, the new metric result is not directly comparable with previous benchmark results. However, let's consider a potential application of this new metric — AutoML. For each AutoML search, we need to determine which configuration is better. Due to metric randomness, there might be some noises in the AutoML search process. Using the new metric could improve the accuracy of search, and ideally the final model would be better. We haven’t conducted the AutoML experiments yet, and this is a planned follow-up work.

2. We can think of each metric as a function of the test data (not the test-remaining set), and the model. The function associated with our new proposed metric is deterministic on the test data. The function is indeed not deterministic on the test-remaining set for different model, but that's not a problem. That's exactly the reason why the new proposed metric has the desired variance reduced property.

Reviewer comment (Evaluation):
As mentioned in the weakness, wouldn't the proposed algorithm lead to different metrics being used for different combinations of models and validation data?

Author response:
Thank you for the response.

1. Indeed. Given we are proposing a new metric, the new metric result is not directly comparable with previous benchmark results. However, let's consider a potential application of this new metric — AutoML. For each AutoML search, we need to determine which configuration is better. Due to metric randomness, there might be some noises in the AutoML search process. Using the new metric could improve the accuracy of search, and ideally the final model would be better. We haven’t conducted the AutoML experiments yet, and this is a planned follow-up work.

2. We can think of each metric as a function of the test data (not the test-remaining set), and the model. The function associated with our new proposed metric is deterministic on the test data. The function is indeed not deterministic on the test-remaining set for different model, but that's not a problem. That's exactly the reason why the new proposed metric has the desired variance reduced property.