Paper: A Data-Driven Measure of Relative Uncertainty for Misclassification Detection (ICLR-2024)

Reviewer comment (Experiments):
In order to assess the paper contribution, the authors need to compare their methods with stronger techniques to assess predictive uncertainty, such as conformal prediction. For instance, the authors could compare their method with a conformal prediction algorithm that, instead of using data to obtain the uncertainty metric s_d, uses such data as calibration data to obtain prediction intervals. Such comparison is also important in that the assessment of prediction uncertainty in terms of prediction sets seems to be more useful in practice than that in terms of the metric s_d(x).

Author response:
We thank the reviewer for bringing this to our attention. In order to address this point, we have added a dedicated section on conformal prediction in the related work section, including a short explanation in section A.3.7 in the appendix, and most importantly a new set of experiments in section A.6 (Performance of conformal prediction). We considered 2 modesl, 2 datasets and 3 possible training techniques for a total of 12 additional numerical results. The content related to these changes is reported in blue in the revised version of the paper.

Reviewer comment (Writing):
Can you articulate the meaning of the metric in (3) in more precise terms? Referring to the distance d as an observer is a bit strange; what does the data-based distance d(y, y') describe?

Author response:
According to [Rao, 1982], D must be chosen to reflect some intrinsic dissimilarity between instances relevant to a particular task at hand. In practice, in our paper, the positive definite matrix D implicitly defines an inner product, which in general measures the similarity of vectors. We use data to optimize this inner product for the task of detecting misclassifications. 

**References**

[Rao, 1982] Rao, C. R.
Diversity and dissimilarity coefficients: a unified approach
*Theoretical population biology, Elsevier*, 1982, 21, 24-43

Reviewer comment (Theory):
It is not correct to refer to p_x,y as “probability density function” since Y has a finite range of values.

Author response:
We agree with the suggested terminology for the symbol  $p_{x,y}$. We updated the manuscript accordingly, referring to it as a probability distribution.

Reviewer comment (Presentation):
The paper is overall not easy to follow. For example, the related work section comes at a random place. Also, I encourage authors to conduct wider and more structured literature reviews, to cover related topics such as deep model calibration and classification with rejection, and present in a more structured way.

Author response:
We thank the reviewer for the observations. We moved the related works section up and placed it after the introduction. We also structured it better, by highlighting important related domains, adding additional comments and references pointed out in the review process.

Reviewer comment (Novelty):
Please also give review on the measure of diversity investigated in Rao (1982), and how the proposed method is inspired. This accounts for the core contribution for the paper.

Author response:
The measure investigated in Rao (1982) is introduced in Eq. (3). We improved the phrasing, to highlight this fact.

Reviewer comment (Writing):
Terminology is not unified. For example, "soft-prediction", "the posterior distribution", "the predicted distribution" is used, which seem to mean one same thing.

Author response:
We invested significant efforts in standardizing our terminology. We refer the reviewer to the updated manuscript.

Reviewer comment (Writing):
Some words are used without clear definition or in a confusing way.

Author response:
We thank the reviewer for pointing out these concerns. We made the necessary changes, updating the terminology and refer the reviewer to the current version of the manuscript.

Reviewer comment (Writing):
"feature" is used in cases where it seems to mean a data instance, such as an image.

Author response:
We thank the reviewer for pointing out these concerns. We made the necessary changes, updating the terminology and refer the reviewer to the current version of the manuscript.

Reviewer comment (Writing):
Lambda parameter for the proposed method is mentioned in experiments, but there is no definition.

Author response:
The variable $\lambda$ is a real number in $[0,1]$. It is introduced in Definition 1, and the related eq. 4.

Reviewer comment (Writing):
At first, the presentation of "positive" and "negative" is confusing. I was even thinking about binary classification for a second.

Author response:
We thank the reviewer for their suggestions. We added an example of positive and negative samples right after these terms are introduced and rephrased some text to eliminate any possible ambiguity between an iterative learning algorithm and the optimization of the objective function with Lagrangian multipliers.

Reviewer comment (Reproducibility):
How to prepare positive and negative data for calculating the distance matrix at the first place?

Author response:
We refer the reviewer to the newly introduced Algorithm 1 in the Appendix A.2 where we detailed how to prepare negative and positive data for the computation of the $D^*$ matrix.

Reviewer comment (Experiments):
What does match and mismatch mean in experiments?

Author response:
We investigate two sources of mismatch:

   1. Datasets with different label domains, which translates to experiments of CIFAR-10 vs CIFAR-100 and vice versa or CIFAR-10 vs SVHN for instance. \
   2. Perturbation of the feature space domain generated using popular distortion filters, which translates to running experiments on CIFAR-C datasets [A].

Reviewer comment (Evaluation):
Why use AUC instead of the metric mentioned in Granese et al. 2021?

Author response:
We thank the reviewer for pointing out this typo. Actually, we meant AUROC, not just  AUC. The evaluation metrics are indeed the same as in Granese et al. 2021.

Reviewer comment (Experiments):
A more detailed discussion / description of the baselines, such as Doctor, would be beneficial. The same applies to different objectives like Mixup, RegMixUp etc.

Author response:
We added discussion on the baselines and objective functions. Please see Section A.3 in the appendix.

Reviewer comment (Novelty):
It should be mentioned that the paper is actually not the first one introducing a data-driven measure of uncertainty. There is a whole batch of works (see for example [1, 2, 3, 4]) that use data to derive estimates of epistemic or/and aleatoric uncertainties. This could help readers to correctly place the current paper in the right position in the broader research landscape.

Author response:
We thank the reviewer for pointing out the references [1-4]. They were added and properly cited in the discussion on Related Works.

Reviewer comment (Novelty):
A discussion on how "relative uncertainty" compares to well-known epistemic and aleatoric uncertainties would be insightful.

Author response:
We employ relative uncertainty to measure the overall uncertainty of a model, encompassing a mixture of both aleatoric and epistemic uncertainty components. On a principled basis, our method, like any approach solely utilizing the model output and validation samples, does not anticipate capturing specifically either aleatoric or epistemic uncertainty. This limitation arises from inherent constraints within the model architecture. We have added a clarification in the revised version of the paper.

Reviewer comment (Experiments):
Choice of lambda: can we choose it based on the data? Like $\lambda \sim \frac{1}{\text{Npos.examples}}$

Author response:
While Rel-U does not appear sensitive to changes in λ as shown in Figure 3, a reasonable choice would be $\lambda = N_+ / (N_+ + N_-)$ in order to balance the effect of positive and negative examples. We added a discussion in Appendix A.5.

Reviewer comment (Theory):
In page 9: "no class labels required", but it is not really true. You need class labels to make positive/negative pairs. This means that the approach cannot be used in a semi-supervised scenario, when objects without labels are available, as we still need to know labels.

Author response:
We thank the reviewer for bringing this useful comment to our attention. We meant that strictly speaking, eq. 4 only needs positive and negative instances, which are task-specific. Of course in our case, the labels are implicitly needed to produce negative and positive instances. Thus, we removed the sentence from the paper.

Reviewer comment (Evaluation):
The confusion matrix in Figure 1, as understood, was computed using the validation data. Shouldn't it be more fair to build it on the test set?

Author response:
In order to obtain the matrix D we utilize the validation set. Figure 1 illustrates that D can approximate relatively well the confusion matrix for the same samples that were used for computing it.

Reviewer comment (Experiments):
In scenarios where there's a very good classifier and a high-quality dataset (like MNIST or CIFAR10, with low aleatoric uncertainty), there might be a significant imbalance between positive and negative samples. This could lead to the situation that the trained uncertainty measure is unreliable. How should one address this situation?

Author response:
This might be addressed in part by tuning the $\lambda$ parameters as mentioned above. However, if the imbalance is too severe (e.g., near perfect classification), it is questionable whether misclassification detection is feasible (or even suitable) in such a setting. This would certainly require further analysis and could be a possible direction for future research.

Reviewer comment (Presentation):
I think the paper can improve on its presentation. Most importantly, the authors should provide a clear description of how the learn their metric function $s_d$ because that's the crux of their paper.

Author response:
We added an overview of our strategy in Algorithm 1 in the Appendix Section A.2 and improved the discussion on how the metric function is learned.

Reviewer comment (Reproducibility):
I think the paper would benefit from having an algorithm block just to clarify the exact learning procedure. For instance, information like the detector is tuned on a held-out set which the model wasn't trained out should be highlighted much clearly in the paper.

Author response:
We added an overview and algorithm block in the appendix Section A.2 and expanded discussion on assumptions and requirements.

Reviewer comment (Novelty):
A high-level question here is: how is this different from some logit/feature transformation on held-out set? There has been some work in the literature (although around OOD detection) that looks at using latent representations from deep neural network backbone and training models that use them (see for e.g. [1]). I see that you have taken a slightly different approach here that looks at using closed form solution, but the idea seems very similar. Can you describe how your paper differentiates from this high-level approach of taking features from a deep net backbone and then applying some transformation to it to generate an uncertainty metric?

Author response:
Our metric doesn't directly incorporate uncertainty statistics obtained from the model, as relying on such statistics may create a misleading sense of the model's genuine, as opposed to self-assured, confidence in detecting its classification errors. Unlike metrics that depend on transformations applied to latent code, our approach restricts observation to black-box access of the model, where only the final layer statistics can be observed (with $\epsilon$ set to 0). Instead of adopting a strategy that involves learning a metric for uncertainty through an additional model (as seen in [1] for OOD), which might raise concerns about the uncertainty of this secondary model, our metric focuses on learning to recognize the signature of uncertainty for each pair of classes using the soft-prediction of the existent model. In essence, our observer '$d$', given a sufficient number of observed positive and negative samples, can effectively weigh the uncertainty metric based on the model's soft-probabilities in a manner specific to the class pair in question.

Reviewer comment (Experiments):
Building upon the prev point, would it make a lot of difference if you were to use a simple MLP architecture rather than the closed form solution you have right now to transform the soft outputs into an uncertainty measure that's being fine-tuned on held-out tuning set? The MLP can also be trained on held-out tuning set.

Author response:
Experiments with a MLP trained on the output logits of the validation set to predict mistakes were performed. Please see Table 3 in the Appendix for details. We added additional details on the hyper-parameters and the exact architecture used in the new Section A.3 in the Appendix.

Reviewer comment (Novelty):
How do you think your approach compares with Bayesian neural networks which also introduce other forms of uncertainty such as model uncertainty? (see [2] for e.g.)

Author response:
We compared to MCDropout [A] and Deep Ensembles [B], which are estimates of Bayesian Neural Networks, in Table 3 in the Appendix. We added further details on these baselines in Section A.3 in the appendix.

   In contrast to [2], we are given a pre-trained model and we want to gauge the uncertainty w.r.t. pairs of labels. This is slightly different from measuring the uncertainty on a class of models or on a given label for a family of models. It is worth mentioning that our focus is to develop a methodology that can be applied to any classifier that outputs soft-predictions, which includes, but is not limited to (Bayesian) neural networks.