Paper: R-EDL: Relaxing Nonessential Settings of Evidential Deep Learning (ICLR-2024)

Reviewer comment (Theory):
Could you present the detailed derivation of Formulas 7, 9 and 10? These formulas are the essential steps, but I may not catch up with your idea.

Author response:
Thanks for the comment. Presented below are the step-by-step derivations of Formulas 7, 9, and 10, respectively. For the sake of brevity, some explanation of notations, which have been provided in the main text, have been omitted here.

Formula 7 is a generalized form of the projected probability $P_X$, where the prior weight $W$ is not fixed to the class number $C$. Its derivation is presented below:
- In Definition 2, the projected probability $P_X$ of the opinion $\omega_X$ is defined by $P_X(x) = b_X(x) + a_X(x) u_X$.
- Without fixing the prior weight $W$ to $C$, Formula 5 will be transformed into  $b_X(x) = \frac{e_X(x)}{\sum_{x^\prime\in \mathbb X}e_X(x^\prime) + W},\quad u_X= \frac{W}{\sum_{x\in \mathbb X}e_X(x) + W}$.
- By setting the base rate $a_X(x)$ as a uniform distribution, i.e., $a_X(x)=1/C$ and plugging the above $b_X(x)$ and $u_X$ into the definition of $P_X$, we come to formula 7: $P_X(x)=\frac{e_X(x) + \frac{W}{C}}{\sum_{x^\prime\in \mathbb X}e_X(x^\prime) + W}$.

Formula 9 is a generalized form of the concentration parameter $\alpha_X$ of the constructed Dirichlet PDF, whose derivation is presented below:
- Formula 4 shows $\alpha_X(x) = \frac{b_X(x) W}{u_X} + a_X(x)W$.
- By setting the base rate $a_X(x)$ as a uniform distribution, i.e., $a_X(x)=1/|\mathbb{X}|=1/C$, and plugging the above $b_X(x)$ and $u_X$ into Formula 4, we have $\alpha_X(x) = e_X(x) + \frac{W}{C}=e_X(x)+\lambda$, where $\lambda=W/C$.

Formula 10 is equivalent to Formula 7. We reiterated it in Section 3.3 to facilitate subsequent discussion about the optimization objective. Employing Formula 9, its derivation can be more easily achieved:

- Formula 9 shows $\alpha_X(x)=e_X(x)+\lambda$.
- As defined below Formula 6，$S_X$ denotes the sum of $\alpha_X(x)$, thus we have $S_X = \sum_{x^\prime\in \mathbb X}e_X(x^\prime) + C\lambda$.
- Then we come to Formula 10: $P_X(x) = \frac{\alpha_X(x)}{S_X}=\frac{e_X(x) + \lambda}{\sum_{x^\prime\in \mathbb X}e_X(x^\prime) + C\lambda}$.

We hope that the above derivation process will aid in the comprehension of our work. Should any aspects of our paper remain ambiguous, we warmly welcome further discussion.

Reviewer comment (Presentation):
The authors are recommended to make necessary explanations on different shapes of I-EDL with EDL and R-EDL in Figure 2 and Figure 5. Additionally, uncertainty representations for ID (MNIST) and OOD (FMNIST) are better to show the differences among EDL, I-EDL, and R-EDL, as presented in I-EDL (figure 4).

Author response:
Thanks for the constructive comment! **The necessary explanations and the uncertainty density plots (Figure 7-10) for ID (MNIST) and OOD (FMNIST) have been added to the appendix of the revised version.**

The explanations on different shapes of I-EDL with EDL and R-EDL are as follows: The I-EDL method utilizes the Fisher information matrix to measure the amount of information that the categorical probabilities carry about the concentration parameters of the corresponding Dirichlet distribution, thus allowing a certain class label with higher evidence to have a larger variance. Consequently, the predictions made by the I-EDL approach are typically less extreme, resulting in a bimodal distribution on the uncertainty density plot where the two peaks are closer to the center of the density axis compared to the EDL and R-EDL methods. Additionally, we deduce that the similarity in the shapes of the uncertainty density plots between EDL and R-EDL may stem from the fact that R-EDL's modifications to EDL only consist of relaxations of non-essential settings, without introducing any additional mechanisms.

Reviewer comment (Writing):
What role did W play in the process of uncertainty estimation or evidence collection?

Author response:
Thanks for the comment.

Theoretically, the existence of $W$ serves as a necessary condition for the bijective relationship between subjective opinions and Dirichlet PDFs in Theorem 1. As shown in Formula 4, a single subjective opinion could correspond to multiple Dirichlet PDFs in the absence of $W$.

Empirically, as elaborated in Section 3.2, the value of $W$ determines the extent to which the projected probability $P_X(x)$ is influenced by the magnitude and proportion of evidence respectively. As one of our key contributions, relaxing the rigid setting of fixing $W$ to the number of classes has been shown to enhance the quality of uncertainty estimation.

**In the end, thanks for all your time and consideration in reviewing our paper!**

Reviewer comment (Writing):
More intuition about some of the key concepts of the paper can be provided such as subjective opinion, belief mass, uncertainty mass, and Dirichlet distribution.

Author response:
Thanks for the valuable comment. Your comment serves as a crucial reminder that some concepts in the paper may be perplexing to readers. Here we provide some useful intuition about these concepts:

- **Subjective opinions** are based on **belief mass** distributions over a domain $\mathbb{X}$. **Belief mass** assigned to a singleton value $x\in\mathbb{X}$ expresses support for $x$ being TRUE. 
- **Belief mass** distributions are subadditive, meaning that the sum of **belief massess** can be less than one. The sub-additivity of the **belief mass** distribution is complemented by **uncertainty mass**. 
- The **uncertainty mass** represents vacuity of evidence, i.e., the lack of support for the variable $X$ to have any specific value. It can also be interpreted as **belief mass** assigned to the entire domain.
- The sub-additivity of the **belief mass** distribution and the complement property of the **uncertainty mass** are expressed by the additivity requirement in Definition 1.
- An important aspect of **subjective opinions** is that they are equivalent to Beta or **Dirichlet PDFs** (probability density functions) under a specific mapping, which has been elaborated on by Theorem 1.

**A more explicit explanation of belief mass and uncertainty mass has been added to the revised version (below Definition 1).** Besides, **we have also incorporated necessary guidance in the appendix**, directing readers to references [1,2] for a more comprehensive understanding. Thanks again for the reminder!

[1] Jøsang, Audun. A logic for uncertain probabilities. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 2001.

[2] Jøsang, Audun. Subjective logic. Vol. 3. Cham: Springer, 2016.

Reviewer comment (Evaluation):
In the experiments, max probability and UM are used as confidence scores. However, it is not clear how to interpret them and what is a good score and a bad one.

Author response:
Thank you for the comment, but there appears to be some misunderstanding.

**The scores listed beneath "MP" and "UM" are all AUPR scores**. MP and UM are two choices of confidence scores, and then they are used to compute the AUPR, which evaluates the model's ability of discriminating classified and misclassified samples, as well as ID and OOD samples.
  
Besides, regarding the question which is the better option for confidence scores between "MP" and "UM", as indicated by Table 1, there is no absolute consensus. Instead, the preferable choice depends on the specific methods, datasets, and tasks involved.

Reviewer comment (Evaluation):
Table 1 caption mentioned AUPR score, but it does not seem to be actually reported.

Author response:
Thank you for the comment, but there appears to be some misunderstanding.

**The scores listed beneath "MP" and "UM" are all AUPR scores**. MP and UM are two choices of confidence scores, and then they are used to compute the AUPR, which evaluates the model's ability of discriminating classified and misclassified samples, as well as ID and OOD samples.
  
Besides, regarding the question which is the better option for confidence scores between "MP" and "UM", as indicated by Table 1, there is no absolute consensus. Instead, the preferable choice depends on the specific methods, datasets, and tasks involved.

Reviewer comment (Experiments):
Considering the prior weight as a hyperparameter also means that the algorithm requires hyperparameter tuning.

Author response:
Thanks for the insightful comment. Indeed, our algorithm necessitates the tuning of hyperparameters. Nonetheless, as Figure 1(b) demonstrably illustrates, the algorithm consistently exhibits satisfactory performance, not merely at an isolated hyperparameter value, but across a continuous range. This indicates that the process of hyperparameter tuning should not pose significant challenges.

Reviewer comment (Writing):
The sentence "Despite its great success" in the abstract is difficult to parse. Consider breaking it into a few sentences.

Author response:
Thanks for the suggestion. We guess what you refer to is the "In this work, our analysis" sentence in the abstract. We have modified its expression as suggested in the revised version.

Reviewer comment (Novelty):
The paper below addresses the over/under regularization problem of EDL using an alternative approach, strictly speaking via Bayesian inference: **“Kandemir et al., Evidential Turing Processes, ICLR, 2022”**. It could be worthwhile to clarify the advantage of the proposed approach with respect to this alternative.

Author response:
We extend our gratitude for directing our attention to this important work.  A brief discussion of the advantage and the disadvantage of this alternative has been added to the Related Work section in the revised version as follows: 

"Kandemir et al. (2022) combines EDL, neural processes, and neural Turing machines to propose the Evidential Tuning Process, which shows stronger performances than EDL but requires a rather complex memory mechanism."

Reviewer comment (Writing):
The phrase “multiple mutually disjoint values” in Definition 1 is a little confusion and potentially imprecise. A better alternative could be **“Let X be a categorical random variable on the domain $\mathbb{X}$”**

Author response:
Thanks for the valuable comment. We have modified the expression as suggested in the revised version.

Reviewer comment (Theory):
While the paper justifies all the steps up till Equation 10 rigorously, the choice of the loss in Equation 11 looks a bit arbitrary. Why should one choose the MSE between the class probabilities and one-hot labels instead of the simple and straightforward cross-entropy loss? How would it work numerically? If it works better or worse, what is the interpretation of the outcome?

Author response:
Thanks for the insightful comment!

When choosing which loss function to optimize the projected probability, we followed the previous EDL-related works [1],[2] to use the Mean Squared Error (MSE) loss without extensive additional exploration. Empirically, we find that MSE results in better performance than Cross-Entropy (CE) loss, which is also mentioned by [1], particularly when employing a non-exponential evidence activation function. In our study, as detailed in the appendix, , we use Softplus as the evidence activation function.

In contrast, when $\exp$ is used as the evidence activation function, as seen in EDL-related works [3] and [4], the CE loss is commonly preferred over the MSE loss. However, we noted that the use of the exponential function as the evidence activation leads to significant fluctuations in the value of the evidence, complicating the tuning of our introduced parameter. Therefore, based on these observations and considerations, we selected the Softplus function as our evidence activation function and the MSE as the loss function form for our model.

[1] Sensoy, Murat, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. NeurIPS, 2018.

[2] Deng, Danruo, et al. Uncertainty estimation by fisher information-based evidential deep learning. ICML, 2023.

[3] Bao, Wentao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. ICCV, 2021.

[4] Chen, Mengyuan, et al. Dual-evidential learning for weakly-supervised temporal action localization. ECCV, 2022.

Reviewer comment (Evaluation):
Why were the MP and UM scores chosen as the performance metrics to measure in-distribution performance instead of the well-established Expected Calibration Error (ECE)?

Author response:
Thanks for the insightful comment! 

Strictly speaking, we are using the area under the precision-recall curve (AUPR) of discriminating correctly classified and misclassified samples to measure in-distribution confidence estimation performance. MP and UM are two choices of the confidence scores for calculating the AUPR score. To elucidate our rationale for adopting the AUPR as the evaluation metric, as suggested in reference **"Deng, Danruo, et al. Uncertainty estimation by fisher information-based evidential deep learning. ICML, 2023"**, over the Brier Score or ECE, we present our justification below.

As delineated in Section 5.2, our stance is that a reliable uncertainty estimator should **(1)** assign higher uncertainties to misclassified samples than to correctly classified ones, and **(2)** assign higher uncertainties to out-of-distribution (OOD) samples than to in-distribution (ID) ones. In other words, **our primary criterion for assessing confidence estimation is the model's ability in differentiating between correctly classified and misclassified samples, as well as between ID and OOD samples based on the predicted confidence**. This is the fundamental reason for our selection of AUPR as the evaluation metric. The term "over-confidence," as used in our context, specifically pertains to the tendency of neural networks to assign excessively high confidence to misclassified or OOD samples, a perspective that deviates from some existing literature. 

We recognize that ECE is frequently employed to assess the degree of correspondence between the model's confidence and the true correctness likelihood, which is important for classification models in many applications. However, **a confidence distribution accompanied by a low ECE does not inherently ensure a clear distinction between correct and incorrect predictions**. For instance, in a balanced two-class dataset scenario, if a binary classifier categorizes all samples into a single class with a consistent confidence output of 50%, the ECE would be zero, yet this result lacks practical significance. Therefore, we posit that for the objectives of our study, AUPR provides a more pertinent evaluation. Furthermore, whereas ECE is focused on the absolute values of confidence, AUPR concentrates on the ranking order of confidence. We consider both metrics as direct evaluators of the quality of predicted confidence, each with its unique perspective and utility.

| Method | ECE (15 bins)$\downarrow$ | Brier score$\downarrow$ | AUPR (confidence) | AUPR (OOD against SVHN)
| :---: |:---: |:---: |:---:|:---:|
| EDL | 20.08$\pm$1.77 | 40.68$\pm$2.39 |97.86$\pm$0.17|78.87$\pm$3.50|
|I-EDL| 39.96$\pm$0.37 | 55.32$\pm$0.50 |98.72$\pm$0.12|83.26$\pm$2.44|
|R-EDL| **3.48**$\pm$0.30 | **18.14**$\pm$0.51 |**98.98**$\pm$0.05|**85.00**$\pm$1.22|

As shown in the above table, which summarizes results from five runs on CIFAR10, our proposed method outperforms EDL and I-EDL with a large margin when evaluated by the ECE and Brier score. It is mainly due to that we relax the rigid setting of fixing the prior weight parameter to the number of classes. With the original setting of $W$ (or $\lambda$), as illustrated in Section 3.2, it is difficult for previous EDL-related works to make predictions of high confidence, despite the fact that they actually output correct evidence distributions, thus resulting in excessively high ECE and Brier scores. With our modification, model's predicted confidence becomes much more aligned with the expected classification accuracy. It is noteworthy that, although previous EDL-related works paid little attention to the correspondence of confidence and expected accuracy, they are still effective in detecting misclassified and OOD samples, which also aligns with our primary objective.

In summary, our stance is that **employing the AUPR scores for evaluation purposes aligns more closely with our objectives than using ECE or Brier score**. Additionally, for the sake of comprehensive analysis and transparency, **the results evaluated using ECE and Brier score have been included in the appendix of the revised version**, serving as a reference point.

Reviewer comment (Experiments):
How does the method compare against post-hoc calibration algorithms such as temperature scaling?

Author response:
We appreciate the constructive comment, and **the comparison between our method and the temperature scaling method has been added to the appendix of our revised version**.

The results of temperature scaling, derived from five runs on the CIFAR10 dataset, is reported above. We utilized the [official code](https://github.com/gpleiss/temperature_scaling) of temperature scaling to implement this method, which indeed exhibits excellent performances when measured by ECE. However, there still exists a performance gap in OOD detection when compared to our method. Besides, as stated above, employing the AUPR scores for evaluation aligns more closely with our objectives than using ECE.

Reviewer comment (Writing):
Typo: The first paragraph on p7 KL-PN is mentioned twice.

Author response:
> Comment 2: Typo: The first paragraph on p7 KL-PN is mentioned twice.

Thanks for the reminder! The second "KL-PN" should be "RKL-PN". It has been corrected.

**In the end, we extend our sincerest thanks for all your time and consideration! Thanks again for your constructive suggestions and postive rating!**

Reviewer comment (Evaluation):
Can the authors report additional calibration results? E.g., by reporting the expected calibration error (Guo et al., 2017)?

Author response:
Thanks for your constructive comments and positive rating. We have meticulously revised the manuscript as suggested.  In the following, your comments are first stated and then followed by our point-by-point responses.

> Comment 1: The experiments lack an in-distribution calibration evaluation. Can the authors report additional calibration results? E.g., by reporting the expected calibration error (Guo et al., 2017)?

Thanks for the constructive comment! **The additional results have now been reported in the appendix of the revised version.**

Instead of performing an in-distribution calibration evaluation, we have selected AUPR scores to evaluate whether the model can assign higher confidence to correctly classified samples than misclassified ones. We would like to provide more explanation for our choice when reporting the results evaluated by ECE and Brier score. 

| Method | ECE (15 bins)$\downarrow$ | Brier score$\downarrow$ | AUPR (confidence) | AUPR (OOD against SVHN)
| :---: |:---: |:---: |:---:|:---:|
| EDL | 20.08$\pm$1.77 | 40.68$\pm$2.39 |97.86$\pm$0.17|78.87$\pm$3.50|
|I-EDL| 39.96$\pm$0.37 | 55.32$\pm$0.50 |98.72$\pm$0.12|83.26$\pm$2.44|
|R-EDL| **3.48**$\pm$0.30 | **18.14**$\pm$0.51 |**98.98**$\pm$0.05|**85.00**$\pm$1.22|

As shown in the above table, which summarizes results from five runs on CIFAR10, our proposed method outperforms EDL and I-EDL with a large margin when evaluated by the ECE and Brier score. It is mainly due to that we relax the rigid setting of fixing the prior weight parameter to the number of classes. With the original setting of $W$ (or $\lambda$), as illustrated in Section 3.2, it is difficult for previous EDL-related works to make predictions of high confidence, despite the fact that they actually output correct evidence distributions, thus resulting in excessively high ECE and Brier scores. With our modification, model's predicted confidence becomes much more aligned with the expected classification accuracy. It is noteworthy that, although previous EDL-related works paid little attention to the correspondence of confidence and expected accuracy, they are still effective in detecting misclassified and OOD samples, which also aligns with our primary goal.

As delineated in Section 5.2, our stance is that a reliable uncertainty estimator should (1) assign higher uncertainties to misclassified samples than to correctly classified ones, and (2) assign higher uncertainties to OOD samples than to ID ones. In other words, our primary criterion for assessing confidence estimation is the model's ability in differentiating between correctly classified and misclassified samples, as well as between ID and OOD samples based on the predicted confidence. This is the fundamental reason for our selection of AUPR as the evaluation metric. The term "over-confidence," as used in our context, specifically pertains to the tendency of neural networks to assign excessively high confidence to misclassified or OOD samples, a perspective that deviates from some existing literature.

We recognize that ECE is frequently employed to assess the degree of correspondence between the model's confidence and the true correctness likelihood, which is important for classification models in many applications. However, a confidence distribution accompanied by a low ECE does not inherently ensure a clear distinction between correct and incorrect predictions.

Reviewer comment (Experiments):
It would have felt more natural to me to include the Natural Posterior Network [1] instead of / besides the Posterior Network as a baseline, since it boasts better results than its predecessor.

Author response:
We extend our gratitude for directing our attention to the important work "Natural Posterior Network" (NatPN) [1]. The primary reason for the initial omission of [1] from our analysis was our reliance on the baseline selection in reference [4] from ICML 2023. Given that [1] did not conduct experiments using the CIFAR10 dataset with a VGG16 backbone, we reproduced their method and generated the following results, which were assessed using the same metrics as those presented in Table 1:

| Method | Classification Accuracy | Confidence Estimation | OOD Detection (SVHN) | OOD Detection (CIFAR100)|
| :---: | :---: | :---: | :---: | :---: |
| PostN | 84.85$\pm$0.0 | 97.76$\pm$0.0 | 80.21$\pm$0.2 | 81.96$\pm$0.8 |
| NatPN | 84.72$\pm$0.0 | 97.81$\pm$0.0 | 80.14$\pm$0.3 | 81.80$\pm$1.0 |
| R-EDL | **90.09**$\pm$0.30 | **98.98**$\pm$0.05 | **85.00**$\pm$1.22 | **87.72**$\pm$0.31 |

It is noteworthy that the performance of NatPN does NOT surpass that of its predecessor, which aligns with the assertions made by the authors in [1]. They state that a single NatPN achieves accuracy and calibration performance **comparable to** PostNet. The principal advantage of NatPN lies in its efficiency, as it utilizes a single normalizing flow, thereby offering a significant speed advantage over PostNet, whose computational requirements scale linearly with the number of classes due to its evaluation of a separate normalizing flow for each class.

[4] Deng, Danruo, et al. "Uncertainty estimation by fisher information-based evidential deep learning." ICML, 2023.

Reviewer comment (Experiments):
Regarding the creation of a new hyperparameter, I do appreciate the authors' ablation study in 5.5. However, the investigation, given that this step is one of the key contributions of the paper, feels a bit shallow. I would be interested to understand better how to interpret the best values found in the context of the authors' original argument and to distill any trends. I found Figure 1b) particularly puzzling, since lambda seems to have major effect on classification performance, but less so in terms of actual OOD detection (~2 points AUPR). A similar graph about misclassification detection, which seems to align better with the authors' original arguments, is not given.

Author response:
We appreciate your insightful comment. 

To interpret the best values, in our discussion in Section 3.2, we elaborate on the significance of the parameter $W$ (or $\lambda$) in influencing the projected probability $P_X(x)$, specifically in terms of the magnitude and proportion of evidence. Theoretically, for any specific case, there should be an optimal value of $W$ that effectively balances the trade-off between utilizing the proportion and magnitude of evidence. This optimal value is what we refer to as the "best value." As evidenced in Figure 1(b), this best value is not a singular point but rather spans across a suitable continuous range.
 
The trend analysis follows a similar logic. At the point where $W$ (or $\lambda$) is reduced to zero, the model's predictive probability is entirely dependent on the proportion of evidence. Conversely, as $W$ (or $\lambda$) increases substantially, the model progressively prioritizes the magnitude of evidence over its proportion. This trend is visually represented in Figure 1(b), where it is clear that an over-reliance on either magnitude or proportion leads to suboptimal performance.

We acknowledge that our current interpretation does not comprehensively elucidate the intricate mechanisms that determine the optimal value of this parameter. Indeed, deciphering this aspect poses a significant challenge, as the output of modern neural networks is shaped by a plethora of complex factors, including dataset characteristics, architectural choices of the backbone, etc., rather than being solely dependent on the number of classes or any isolated factor. As we have posited in the Conclusion (Section 6) of our paper, *the underlying mechanism dictating its optimal value is a topic worth further investigation*.

We are grateful for the valuable comment.

In our view, it is reasonable that the value of $\lambda$ impacts the accuracy of classification. As the value of $\lambda$ increases, the neural network is required to generate evidence for the target class with greater absolute values to attribute a high level of confidence to its predictions, which can escalate the propensity for overfitting. We conjecture that this phenomenon may contribute to the comparatively lower classification accuracy (83.55%) observed for EDL as presented in Table 1.

We appreciate your suggestion. **Now we have included this graph in the appendix of the revised version.** As shown in Figure 2(b) of the revised version, this graph displays a trend similar to the existing two parameter analysis graphs, characterized by an initial increase followed by a subsequent decline.

Reviewer comment (Evaluation):
Although the second contribution – effectively deprecating the variance term in the loss function – is supposed to help overconfidence, confidence is never directly evaluated. This might be due to the common conflation of the terms uncertainty and confidence, with the latter often referring to the actual probability assigned to predicted class – however works like the aforementioned Charpentier et al. [1] evaluate both the quality of uncertainty estimates obtained through different EDL metrics as well as the direct confidence of their model through the Brier score. The authors do try to assess confidence through measuring the AUPR for their misclassification task (including the MP value, i.e. the maximum probability across all classes), which I would argue is appropriate, but assesses confidence only in its ability to discriminate between correctly and incorrectly classified samples, not in its degree to correspond to the (in expectation) chance of classifying correctly, like in the instance of the expected calibration error.

Author response:
Thanks for the insightful comment. To address this concern, in the revised version, **we have enhanced the clarity and included results evaluated using the Brier score and Expected Calibration Error (ECE) in the Appendix as Table 8**. Furthermore, to elucidate our rationale for adopting the Area Under the Precision-Recall Curve (AUPR) as the evaluation metric, as suggested in reference [4], over the Brier score or ECE, we present our justification below:

As delineated in Section 5.2, our stance is that a reliable uncertainty estimator should **(1)** assign higher uncertainties to misclassified samples than to correctly classified ones, and **(2)** assign higher uncertainties to out-of-distribution (OOD) samples than to in-distribution (ID) ones. In other words, **our primary criterion for assessing confidence estimation is the model's ability in differentiating between correctly classified and misclassified samples, as well as between ID and OOD samples based on the predicted confidence**. This is the fundamental reason for our selection of AUPR as the evaluation metric. The term "over-confidence," as used in our context, specifically pertains to the tendency of neural networks to assign excessively high confidence to misclassified or OOD samples, a perspective that deviates from some existing literature.

We recognize that ECE is frequently employed to assess the degree of correspondence between the model's confidence and the true correctness likelihood, which is important for classification models in many applications. However, **a confidence distribution accompanied by a low ECE does not inherently ensure a clear distinction between correct and incorrect predictions**. For instance, in a balanced two-class dataset scenario, if a binary classifier categorizes all samples into a single class with a consistent confidence output of 50%, the ECE would be zero, yet this result lacks practical significance. Therefore, we posit that for the objectives of our study, AUPR provides a more pertinent evaluation. Furthermore, whereas ECE is focused on the absolute values of confidence, AUPR concentrates on the ranking order of confidence. We consider both metrics as direct evaluators of the quality of predicted confidence, each with its unique perspective and utility.

| Method | ECE (15 bins)$\downarrow$ | Brier score$\downarrow$ | AUPR (confidence) | AUPR (OOD against SVHN)
| :---: |:---: |:---: |:---:|:---:|
| EDL | 20.08$\pm$1.77 | 40.68$\pm$2.39 |97.86$\pm$0.17|78.87$\pm$3.50|
|I-EDL| 39.96$\pm$0.37 | 55.32$\pm$0.50 |98.72$\pm$0.12|83.26$\pm$2.44|
|R-EDL| **3.48**$\pm$0.30 | **18.14**$\pm$0.51 |**98.98**$\pm$0.05|**85.00**$\pm$1.22|

As shown in the above table, which summarizes results from five runs on CIFAR10, our proposed method outperforms EDL and I-EDL with a large margin when evaluated by the ECE and Brier score. It is mainly due to that we relax the rigid setting of fixing the prior weight parameter to the number of classes. With the original setting of $W$ (or $\lambda$), as illustrated in Section 3.2, it is difficult for previous EDL-related works to make predictions of high confidence, despite the fact that they actually output correct evidence distributions, thus resulting in excessively high ECE and Brier scores. With our modification, model's predicted confidence becomes much more aligned with the expected classification accuracy. It is noteworthy that, although previous EDL-related works paid little attention to the correspondence of confidence and expected accuracy, they are still effective in detecting misclassified and OOD samples, which also aligns with our primary objective.

In summary, our stance is that **employing the AUPR scores for evaluation purposes aligns more closely with our objectives than using ECE or Brier score**. Additionally, for the sake of comprehensive analysis and transparency, the results evaluated using ECE and Brier score have been included in the Appendix as Table 8, serving as a reference point.

[4] Deng, Danruo, et al. "Uncertainty estimation by fisher information-based evidential deep learning." ICML, 2023.