Paper: On the Limitations of Temperature Scaling for Distributions with Overlaps (ICLR-2024)

Reviewer comment (Experiments):
It would be better to have more real data experiments. In the "IMAGE CLASSIFICATION BENCHMARKS", the overlap is introduced rather artificially.

Author response:
**Real data experiments.** We also agree that having a more canonical "overlap" setting would be useful, however we felt that label noise was a natural way to introduce this overlap that also is readily encountered in practice (since samples are often mislabeled). Our results in Sections 5.2 and B.3 also include the 0\% label noise setting, which involves no artificial modification.

Reviewer comment (Evaluation):
"We also trained d-Mixup models on the same data, but we found that the confidence regularization effect for d > 2 led to underconfidence on these datasets, so we report just the results for Mixup.": do we know why this may happen?

Author response:
**Explaining regularization effect of $d$-Mixup.** When mixing several points with a uniform mixing distribution, mixtures in which one point gets a label very close to its original label (i.e. a mixing weight very close to 1) become less likely, and as a result we get even less spiky predictions (much closer to uniform probabilities).

Reviewer comment (Evaluation):
In the image experiments, for CIFAR-100, why having label noise makes NLL worse but ACE / ECE better? This makes the experiments less convincing if we don't have a solid explanations.

Author response:
**NLL gets worse but ECE/ACE get better.** It is possible to have very poor NLL while still having good (or even perfect) ECE, so worse NLL does not necessarily imply worse ECE performance. For example, a model that predicts uniformly at random can achieve zero ECE in the balanced class regime despite having poor NLL. Additionally, the ECE results in Table 4 for Mixup on CIFAR-100 are roughly within the 1 standard deviation error bounds of each other (so this does not represent a marked improvement), which we have also now pointed out in the surrounding discussion.

Reviewer comment (Evaluation):
The mixing training introduces an extra degree of freedom (i.e. d-mixing). Based on table 2, we see that mixing is actually negative for NLL when classes are relatively separate, but only shows performance improvement as the overlap increases. I also believe that the model performance would not be strictly better as we increase the degree of mixing, not to mention the additional computational complexity.

Author response:
1. **Amount of mixing in $d$-Mixup.** We definitely agree that the optimal mixing choice depends on the underlying data distribution being considered, and it is certainly true that mixing more points is not necessarily better (as we mention, this can lead to over-regularization of the predicted probabilities). However, it is tricky to analyze the optimal mixing choice in practice outside of treating it as an additional hyperparemeter. This is due to the fact that we do not know the underlying ground truth distribution - if we did have some way to quantify the amount of overlap in the ground truth, then it is certainly conceivable that we could develop theoretical guidelines for the optimal mixing number (although this would be separate from the theory in our work, which mixes $d + 1$ points since we do not make assumptions on the model class). Additionally, even when only two classes are overlapping there can still be significant benefit to mixing more than two points, which is demonstrated by the high overlap case in our experiments in Section 5.1 (4-Mixup performs the best in terms of NLL and ECE). The intuition here is again that the number of mixed points corresponds to the amount of regularization, and in the cases of high overlap we want to make sure the predicted probabilities are far away from being spiky (i.e. close to point masses).

Reviewer comment (Experiments):
Intuitively, the optimal d should have to do with the structure of overlapping in the dataset. I think it would be beneficial for the authors to have a more in-depth discussion on the choice of mixing in practice.

Author response:
1. **Amount of mixing in $d$-Mixup.** We definitely agree that the optimal mixing choice depends on the underlying data distribution being considered, and it is certainly true that mixing more points is not necessarily better (as we mention, this can lead to over-regularization of the predicted probabilities). However, it is tricky to analyze the optimal mixing choice in practice outside of treating it as an additional hyperparemeter. This is due to the fact that we do not know the underlying ground truth distribution - if we did have some way to quantify the amount of overlap in the ground truth, then it is certainly conceivable that we could develop theoretical guidelines for the optimal mixing number (although this would be separate from the theory in our work, which mixes $d + 1$ points since we do not make assumptions on the model class). Additionally, even when only two classes are overlapping there can still be significant benefit to mixing more than two points, which is demonstrated by the high overlap case in our experiments in Section 5.1 (4-Mixup performs the best in terms of NLL and ECE). The intuition here is again that the number of mixed points corresponds to the amount of regularization, and in the cases of high overlap we want to make sure the predicted probabilities are far away from being spiky (i.e. close to point masses).

Reviewer comment (Experiments):
Does it make sense to generate additional classes for the overlapping case (y=1, y=2, y=1&2)? In that case would temperature scaling still work and what is the tradeoff here?

Author response:
2. **Generating additional class labels.** Generating synthetic labels as a means of regularization is definitely a useful idea, and in fact this is already done by methods such as label smoothing and Mixup (which drives the theory behind Mixup in our paper). The main difficulties with what you are suggesting (constructing a new class entirely) is the lack of knowledge of the ground truth distribution required to do so (that is, the input data does not specify which samples are in the overlapping part between class 1 \& 2). We need to specify some strategy for determining which samples get split off into the newly constructed class, and any such strategy will necessarily come with some baked-in biases about how we expect the ground truth distribution to behave. Naive strategies such as randomly assigning samples to the new class probably fall within the realm of existing label augmentation techniques, such as variants of label smoothing and Mixup.

Reviewer comment (Theory):
Is ERM interpolator the best model class to capture the datasets with overlaps? The properties of the interpolator seem to be naturally mismatched.

Author response:
3. **ERM interpolator drawbacks.** It is true that the properties of the ERM interpolators that we study are not ideal for calibration, which also drives the failure of temperature scaling in this case. However, we focus on this class of interpolating models as they are incredibly common in practice, and for these models we observe empirically the kind of assumptions we make in our theory (spiky probabilities that are well-separated). It is definitely possible that training ERM models with early stopping can provide sufficient regularization for better calibration (especially when combined with temperature scaling), which would be an interesting avenue to explore.

Reviewer comment (Theory):
It is commonly believed that temperature scaling is very effective for post-hoc calibration scaling, although it is found that this technique cannot be used for all the cases. Can you explain what causes this gap between your theoretic results and the commonly observed empirical success?

Author response:
1. **Reconciling theory with common temperature scaling wisdom.** As you correctly point out, temperature scaling has been demonstrated repeatedly since [1] to be useful for calibration performance. Our theory does not contradict the *relative benefit* of temperature scaling (i.e. ERM + TS will almost certainly be better than ERM alone), but rather says that in *absolute* terms that there are classes of data distributions for which the improvement due to temperature scaling is simply not enough to achieve good calibration. This is made clearer by a new set of comparisons (Tables 6-8) that we have added as part of Appendix Section B.4, which show the relative gain from temperature scaling for both ERM and Mixup. As can be seen in this table, temperature scaling improves ERM calibration performance, but nowhere near enough to be even close to competitive with Mixup alone. Also worth pointing out is that in our setting, we do not train our ERM models with additional regularization (data augmentation, weight decay, etc.) as was done in prior work (so as to stay within the setting of our theory), and these almost certainly play a non-trivial role in how much temperature scaling can improve the base model (since the aforementioned modifications can lead to less spiky predictions and also better test accuracy).

Reviewer comment (Experiments):
The main experimental results in tables only show the comparison between ERM+TS vs Mixup. I think that the results of ERM baseline and Mixup+TS should be presented at least. Moreover, is the results influenced by the training schemes used for training models (such as learning epochs, learning rate and regularization)?

Author response:
2. **Non-temperature-scaling comparisons and effect of hyperparameters.** Thank you for the useful point; as mentioned above, we have added ERM vs ERM + TS and Mixup vs Mixup + TS comparisons to the Appendix as part of Section B.4. The main obversation from these comparisons is that while TS can significantly improve ERM performance (at least with respect to negative log-likelihood), it simply does not make a substantial difference with respect to the gap in performance between ERM and Mixup (exactly as suggested by our theory). With regards to the question about hyperparameters - calibration performance will certainly be affected heavily by hyperparameter tuning, with early stopping (among the modifications you mention) in particular having a significant effect (since this can prevent predicted probabilities from becoming too spiky for ERM). Our results are constrained to the common interpolation regime in which models are trained to have very low training loss, and in this regime we found that minor tweaks to hyperparameters such as learning rate did not have a significant effect (since we are training for a long enough time horizon). Additionally, as mentioned above, in this regime it is also harder to improve calibration performance since model predictions are even closer to point masses (which drives the failure of temperature scaling in our theoretical results).