Paper: Leveraging Uncertainty Estimates To Improve Classifier Performance (ICLR-2024)

Reviewer comment (Writing):
This paper is not well written. I have the impression that the authors lack a thorough understanding of the literature. They use unconventional notions, and they don't discuss the problem in a formal way.

Author response:
**Response:**  (a) We acknowledge that we used some non-standard terms such as train/test positivity and model score estimation bias. We will revise the draft to ensure these are defined precisely the very first time they are used rather than in footnotes or further down in the writeup or appendix. (b) We do present a formal definition of the decision-making problem in Sec 4. Could the reviewer kindly clarify their expectations on the problem definition?

Reviewer comment (Writing):
In the introduction, the authors write "benefits of combining model score with estimates of aleatoric and epistemic uncertainty". This is a very weird reasoning, because the model score corresponds to the estimate of aleatoric uncertainty. It is better to use conditional class probabilities instead of model score, because model score is a very generic term.

Author response:
**Response:**    
(a) Yes, aleatoric uncertainty is a deterministic function of model score but that is not true of epstemic uncertainty and will rephrase the sentence to read better.   
(b) We are using the phrase “model score“ instead of ”conditional probability of the positive class as estimated by model“ only for brevity and have clarified this in Footnote 1[Pg 1]

Reviewer comment (Writing):
In the introduction, RQ1 states: "Does model score estimation bias (deviation from positivity) depend on uncertainty?" This is a really weird sentence that is difficult to understand. The meaning only became clear after reading Section 3. The authors first have to explain what they mean by model score estimation bias—is it a bias in the sense of the traditional bias-variance decomposition or something else? Similarly, "positivity" is a term never used in the literature; it appears they mean the probability of the positive class on the population level. After reading Section 3, it became clear that "uncertainty" refers to the differential entropy of the second-order distribution. This may be a way to measure epistemic uncertainty with posterior networks, but the authors must be more precise in their description.

Author response:
**Response:**  (a) We do mention that the “model score estimation bias“ is the “difference between test positivity rate and the model score varies with uncertainty” [Pg 2 Para 2] but will move this up to where RQ1 is mentioned. (b) Apologies for not having defining “positivity rate” upfront. It is just the “probability of the positive class on population level”. “Train positivity” and “Test positivity” corresponds to the training and test distribution respectively. We will make it a point to clarify this as well upfront as we revise the draft. (c) Just to clarify in case of Posterior networks and Theorem 3.1(b), “uncertainty” refers to the differential entropy of the second-order distribution in Posterior networks. However for Sections 4 and 5, we are using the term “uncertainty” in a broader sense and have also included results using other methods for estimating uncertainty (e.g. MCDropout) [Sec 6.2, Pg 8, RQ2 and Appendix C.5, Table 5].

Reviewer comment (Writing):
In the introduction, the term "test positivity rate" is used. This term is unfamiliar and needs to be defined.

Author response:
**Response:**  (a) We do mention that the “model score estimation bias“ is the “difference between test positivity rate and the model score varies with uncertainty” [Pg 2 Para 2] but will move this up to where RQ1 is mentioned. (b) Apologies for not having defining “positivity rate” upfront. It is just the “probability of the positive class on population level”. “Train positivity” and “Test positivity” corresponds to the training and test distribution respectively. We will make it a point to clarify this as well upfront as we revise the draft. (c) Just to clarify in case of Posterior networks and Theorem 3.1(b), “uncertainty” refers to the differential entropy of the second-order distribution in Posterior networks. However for Sections 4 and 5, we are using the term “uncertainty” in a broader sense and have also included results using other methods for estimating uncertainty (e.g. MCDropout) [Sec 6.2, Pg 8, RQ2 and Appendix C.5, Table 5].

Reviewer comment (Writing):
In Related work, the overview is described in a very vague way. What is called "Uncertainty modelling" refers to methods that intend to quantify aleatoric and epistemic uncertainty, such as Bayesian methods, ensemble methods, and evidential deep learning methods.

Author response:
**Response:**  We will expand the related work section to have a more detailed discussion of various uncertainty modeling methods including those focusing on aleatoric uncertainty as well. We focused on methods that also capture epistemic uncertainty because given the model score $p(y=1|x)$, epistemic uncertainty provides additional discriminating ability to determine the optimal decision boundary unlike aleatoric uncertainty that has a deterministic relationship with model score and no additional information.

Reviewer comment (Theory):
In Section 3 on Posterior networks, the authors should motivate why this group of methods is their first choice to improve classification performance by leveraging epistemic uncertainty scores. Recent papers have criticized this group for not quantifying epistemic uncertainty correctly, for example, Bengs et al., "Pitfalls of epistemic uncertainty quantification through loss minimization", NeurIPS 2022. Thus, it is questionable to consider this group as the first choice.

Author response:
**Response:**  First, we wish to clarify that our work does not qualify any uncertainty quantifying/estimating method (eg: Posterior / MC-dropout) as being superior over the other. We are aware of the work by Bengs et al. which discusses why optimizing the model based on uncertain-cross entropy loss function does not provide faithful estimates of epistemic uncertainty and will include it in the revised version of related work, but the gaps in the optimization approach are not central to our work.  
  
Our choice of Posterior Networks for our experiments was based on two considerations. First, we already had an existing implementation for one of our real-world applications and experiments using 2D-decision boundary algorithms yielded promising results. Second, for certain controlled scenarios (Fig 2), we could derive a closed analytic form for the model score estimation bias in terms of the model score, ratios of priors and observed evidence ($\gamma(x)$ ), which is positively correlated with the notion of “uncertainty” (entropy of the epistemic distribution) in Posterior networks, allowing us to compare empirical observations with expected behavior.   
  
Note that it is quite possible that other methods of quantifying epistemic uncertainty yield superior results in terms of optimizing decision boundaries and that remains an area of further exploration. We do provide results on MC-Dropout method as well [Sec 6.2, Pg 8, RQ2 and Appendix C.5, Table 5] and observe similar benefits in combining  uncertainty estimates with conditional class probabilities to oprimize recall at a precision bound. We will attempt to include results from other recent approaches in a revised version of the Appendix.

Reviewer comment (Writing):
In Section 3.1, the uncertainty score \( u(x) \) is a measure of epistemic uncertainty. The authors should be more precise.

Author response:
**Response:**  The uncertainty score $u(x)$ needs to definitely capture epistemic uncertainty since that is the element that provides additional information beyond the model estimated $p(Y=1|x)$, but depending on the modeling method, it could also have a dependence on the aleatoric uncertainty. We will clarify this in the revision.

Reviewer comment (Writing):
In Section 3.2, notions such as "true positivity" and "empirical positivity in the train and test set" are very weird terms. It is assumed that these refer to true and estimated conditional class probabilities for the positive class, but the authors should be more precise.

Author response:
**Response:**  We are using these terms only for brevity and will make it a point to define these first.

Reviewer comment (Theory):
In Section 3.2, the authors state that the true positivity is sampled from a Beta distribution. It is unclear whether this is to generate synthetic data, or for another purpose. Since Theorem 3.1 is a theoretical result, it is assumed there is a different reason for considering a Beta distribution.

Author response:
**Response:** (a) We should have explained our choice of data generation setting better. Before going to Sec 4-5, we wanted to first establish that there is a systemic deviation between the conditional probability $p(y=1|x)$ from the model and the observed distribution on the test set and that the deviation has a dependence on “uncertainty”. Since the general case was not as tractable, we restricted our analysis to a common representative scenario where we have Beta prior and differential sampling across classes for train and test set. Being able to verify the theoretical analysis with synthetic data was a secondary motivation. (b) We will update Figure 2 to include the exact distributions mentioned in Sec 3.2 and also Appendix E.1. We, in fact, had a more detailed figure but chose the simpler one for lack of space.

Reviewer comment (Theory):
In Theorem 3.1, the phrase "When data is generated according to Figure 2" is unclear. The figure is not detailed enough, and the data generation process must be described in a formal way.

Author response:
**Response:** (a) We should have explained our choice of data generation setting better. Before going to Sec 4-5, we wanted to first establish that there is a systemic deviation between the conditional probability $p(y=1|x)$ from the model and the observed distribution on the test set and that the deviation has a dependence on “uncertainty”. Since the general case was not as tractable, we restricted our analysis to a common representative scenario where we have Beta prior and differential sampling across classes for train and test set. Being able to verify the theoretical analysis with synthetic data was a secondary motivation. (b) We will update Figure 2 to include the exact distributions mentioned in Sec 3.2 and also Appendix E.1. We, in fact, had a more detailed figure but chose the simpler one for lack of space.

Reviewer comment (Writing):
Because of the very imprecise write-up, it is difficult to follow what the novelty of this paper is. The reviewer was completely lost after Theorem 3.1, even though they are very familiar with posterior networks.

Author response:
**Response:** Thanks again for your valuable feedback and taking the time to read through our submission. We hope we have answered some of your questions. Based on the clarifications, if possible, please do take a look at Section 4-6 to share any further feedback or revise your score.

Reviewer comment (Novelty):
The paper ignores a large part of the literature on decision-making under uncertainty. There already exist many works dedicated to this topic, with the purpose of taking uncertainty into account to improve the decision-making process or abstaining to make decisions when uncertainty is too high. Some of these works are rooted in formalisms alternative to probabilities, while others make use of the classical probabilistic framework. Examples include:  
[1] Mathias C. M. Troffaes. Decision making under uncertainty using imprecise probabilities. International Journal of Approximate Reasoning, Volume 45, Issue 1, May 2007, Pages 17-29.  
[2] Thierry Denoeux. Decision-making with belief functions: A review. International Journal of Approximate Reasoning, Volume 109, June 2019, Pages 87-110.  
[3] Thomas Mortier, Marek Wydmuch, Krzysztof Dembczyński, Eyke Hüllermeier, Willem Waegeman. Efficient set-valued prediction in multi-class classification. Data Mining and Knowledge Discovery, Volume 35, Issue 4, Jul 2021, Pages 1435-1469.

Author response:
**Response:**  Thanks for the pointers. We acknowledge this gap and will definitely expand the related work to include a discussion of the suggested papers . While the papers do deal with decision making under uncertainty, the problem setting and contributions are slightly different as we discuss below.   
  

1. Decision-Making with Belief Functions (Deneoux et al. , Troffaes et al. ) : These works deal with decision making where an input state or policy (possibly multi-dimensional) maps to a consequence which results in a real-valued payoff function with uncertainty modeled via (possibly Bayesian) belief functions. While there are parallels to our decision boundary estimation, in these works, the decision making over competing options is based on a single risk-adjusted (or uncertainty adjusted) utility determined by various axiomatic criterion, e.g., minmax, Hurwicz without any data-based calibration. Second, most of this work is applicable for utility functions that are additive over elements of the input decision space, e.g., overall utility of a stock portfolio can be expressed as the sum of utilities over the individual stocks.. Lastly, there is no discussion around bias in estimating model scores or the effects of differential sampling. 
2. Set-value prediction: Mortier et al. present an efficient approach for set-value predictions in a multi-class setting based on maximizing the expected utility. This work differs from ours in three key aspects. First, the focus of Mortier et al. is on efficient optimization over the power set of classes with the gains being primarily relevant for multi-class settings with a large number of classes where the set of possible prediction sets, (i.e., any non-empty element of the power set) is extremely large. For the binary classification scenario, there are only 3 prediction sets { {0}, {0,1}, {1} }. Secondly, they do not consider the question of the model estimated conditional probabilities being biased which is an important focus of our work. Lastly, while Mortier et al., discuss unconstrained optimization of multiple different utility functions, such as aggregated precision, recall and f-measure, they do not explicitly consider optimizing recall for a precision bound or vice-versa which is more common in real-world applications and the objective function that we focus on.

Reviewer comment (Theory):
It is difficult to determine to what extent the analyses reported in the proposal can be generalized outside of the framework considered, either regarding the analysis of estimation bias (Section 3.2) or the proposed decision-making strategy (Sections 4 and 5). The Bayesian framework may not hold for all datasets, and both the analysis and the proposed decision-making strategy seem hard to generalize to more than two classes, particularly for computational reasons. These limitations should have been discussed in the paper.

Author response:
**Response:**  The decision making strategy in Sec 4-5 is independent of the Bayesian framework and is applicable for datasets following any distribution as well as any choice of uncertainty modeling method.    
  
The purpose of Sec 3 was primarily to motivate the 2D decision-making problem by establishing that that there exist some common scenarios where the conditional probability $p(y=1|x)$ of the positive class in the training dataset as well as the model estimate deviates from that of the underlying distribution and the test set in a systemic way with dependence on uncertainty. In Sec 4-5, we do NOT make any assumption about the nature of the dependence on the uncertainty.   
  
(b) The analysis in Sec 3 [both Theorem 3.1 (a) and (b)] does in fact generalize to multi-class setting with more than two classes when we consider a global Dirichlet prior instead of a Beta prior. We can express the expectation of true and test conditional class probabilities conditioned on the train conditional class probabilities. In case of Posterior networks, we can do the same while conditioning on model’s conditional class probabilities.   
  
On the other hand, generalization of the decision-making strategy to more than two classes is somewhat non-trivial since it first requires characterizing the objective function that the decision boundary(ies) need to be optimized for . We are exploring this setting by considering monotone operators over the conditional probability distributions that also take into account other uncertainty-based discriminating signals. Having said that, there are a large number of real-world applications that require decision making over binary classes which ensures broad applicability. We also consider differential sampling which is a common to multiple large scale applications.

Reviewer comment (Writing):
The writing should be improved. There are typos, and the general structure of the paper might be enhanced (e.g., Sections 4 "2-D Decision Boundary Problem" and 5 "2-D Decision boundary algorithms" could be aggregated). Some parts should be further developed for clarity (e.g., the "Dynamic Programming (DP) Algorithm" part in Section 5.2 is verbose but lacks formalization). The subsection numbering format changes inconsistently, from bold lowercase letters in Section 3 to capitalized in Section 5. Figures are very small and hard to read; Equation 2 is awkwardly displayed; Algorithm 1 is densely packed and consequently hard to read.

Author response:
**Response:**  Thanks so much for these suggestions. We will address all of these and do a more careful proof reading as we revise our draft.

Reviewer comment (Theory):
Why choose the differential entropy of distribution \( H(q(\mathbf{x})) \) as a measure of uncertainty? While this choice is intuitively reasonable, it has also been criticized by some authors (see e.g., the works of Hüllermeier).

Author response:
**Response:**  We would like to clarify that the decision-making strategy in Sec 4-5 does not assume anything about the nature of uncertainty modeling. In Theorem 3.1(b) and for our experimental section, we focus on Posterior networks and choose the differential entropy of the epistemic distribution as a measure of uncertainty. The reason for this is that it is correlated to density of observations and thus, the ratio of prior to evidence (\lambda(x), which impacts the expected test positivity rate conditioned on the train positivity rate.   
  
We are aware of the work by Bengs, Hullermeier et al . which discusses why optimizing the model based on uncertain-cross entropy loss function does not provide faithful estimates of epistemic uncertainty but the gaps in the optimization approach are not central to our work.    
  
We are only focused on optimizing the decision boundary and do not qualify any uncertainty quantifying/estimating method as being superior over the other in terms of estimation of “epistemic uncertainty” or better calibrated estimates. Do note that the lack of calibration of uncertainty estimates does not also impact the decision boundary estimation.   
  
Our choice of Posterior Networks for our experiments was based on two considerations. First, we already had an existing implementation for one of our real-world applications and experiments using 2D-decision boundary algorithms yielded promising results. Second, for certain controlled scenarios (Fig 2), we could derive a closed analytic form for the model score estimation bias in terms of the model score, ratios of priors and observed evidence ($\gamma(x)$), which is positively correlated with the notion of “uncertainty” (entropy of the epistemic distribution) in Posterior networks, allowing us to compare empirical observations with expected behavior. We also report some results on MC-Dropout methods. (Sec 6.2, Pg 8, RQ2 and Appendix C.5, Table 5).

Reviewer comment (Theory):
The interpretation of \( \gamma(\mathbf{x}) \) in Section 3.2 is not really an interpretation. Given that the correlation between \( u(\mathbf{x}) \) and \( \gamma(\mathbf{x}) \) is central to the score estimation bias depending on score and uncertainty, this part should be further developed and discussed.

Author response:
**Response:**  Thanks for the suggestion. We mentioned the definition of $u(x) = H(q(x))$ in footnote 3 (Pg 2) but agree that further explanation would be helpful. We will expand on this item with a small plot of $u(x)$ and $\gamma(x)$ as well as explanation of the positive correlation either in the main paper or the appendix in the revised draft.

Reviewer comment (Evaluation):
Can insights be provided on generalizing the analysis of estimation bias to other settings (as compared to Fig. 2)?

Author response:
**Response:**  The generality of Figure 2 is primarily limited by the choices of (i) the global prior (Beta distribution) and (ii) estimation of model score and uncertainty using Posterior networks.    
  
Note that Thm 3.1(a) on the expectation of test/true conditional class probabilities conditioned on the train positive class probability holds true independent of assumption (ii). For any alternative method of estimating model score and uncertainty, we can derive the relationship between the model score (i.e., model estimated $p(y=1|x)$ ) and test positivity rate based on the dependency between the train positivity and the model score. The term ( $\lambda(x)$ ), which is related to the density of observations will continue to incorporate the effect of epsitemic uncertainty.   
  
Figuring out an equivalent closed form result for Thm 3.1(a) is non-trivial when assumption (i) does not hold. However, for a global unimodal prior (not necessarily a Beta distribution), the expectation of test/true conditional class probabilities conditioned on the train positive class probability is shifted away from the train distribution in the direction of the global prior since the posterior distribution of the true positivity conditioned o the train positivity combines the effects of the evidence (train positivity) and the global prior.

Reviewer comment (Theory):
The decision-making approach seems debatable. In Section 4: is it reasonable to leave \( b \) unconstrained? Should a monotonicity assumption be imposed? Is it reasonable to always keep the decision-making process precise? Would it be preferable to abstain under high (epistemic) uncertainty? Aleatoric and epistemic uncertainties are not distinguished, which raises the question of whether it is reasonable to make decisions if \( u(\mathbf{x}) \) is maximal. What is the effect of discretizing the Score \(\times\) Uncertainty space, and should independent splitting on both dimensions be avoided?

Author response:
**Response:** Please see the comments below  
  
**Comment 7(a):** Q4.1 In Section 4: is it reasonable to leave $b$ unconstrained ? Shouldn't a monotonicity assumption be imposed ?   
  
**Response:** Since for each uncertainty level $i$, there is a single threshold $b_i$, we do assume monotonicity of the true conditional probability of the positive class with respect to the model score for a fixed uncertainty but not with respect to uncertainty for a fixed model score. This is because monotonicity with respect to uncertainty that does not hold true across the entire score range. Figure 3(a) shows the behavior (flipped trends for different score ranges) for the case of Beta priors with uncertainty estimated using Posterior Networks.    
  
In fact, we did experiment with a variant (EW_DPMT-MONO) of the DP algorithm where we constrained the decision boundary such that the true conditional probability is monotonically decreasing in uncertainty (i.e., $b_i \geq b_j$ for $i > j$, the boundary function to be look like a staircase). This variant has higher complexity $O(K^2L^3)$ than EW_DPMT where $K$ and $L$ are the number of uncertainty and score bins. The Table below shows a comparison of the performance of the two algorithms for the same bin configuration. We observe that for high precision bound (as in Criteo), the monotonicity assumption does hold, but that is not true as the precision bound is lowered and the unconstrained version results in better performance.   
  
Table 1: Performance with different unconstrained and monotonic variants of EW-DPMT with uncertainty bins = 100 and score bins = 3. Note that here we used a smaller number of uncertainty bins instead of 500 (in the paper) to reduce computational effort.   

|   |   |   |
|---|---|---|
|Dataset|Criteo (Tau=3) [Recall@90% Precision]|Avazu (Tau=5) [Recall@70% Precision]|
||Score_bins= 100, Unc_bins=3|Score_bins= 100, Unc_bins=3|
|ST|2.2±0.2%|1.9±0.6%|
|EW_DPMT|2.6±0.3%|2.3±0.6%|
|EW_DPMT - MONO|2.6±0.3%|1.9±0.6%|

**Comment 7(b):** Q4.2 Is it reasonable to always keep the decision-making process precise ? Wouldn't it be preferable to abstain in presence of a high (epistemic) uncertainty ?   
  
**Response:** In the setting we are considering, we are attempting to optimize recall of detecting something as positive (e.g., fraudulent transaction or malignant tumor) or not while maintaining a certain precision level. Here, we are not distinguishing between abstaining and a negative decision since we are focusing on the precision and recall of positive detection.  We could reformulate the problem to find two decision boundaries that separate out three regions: positives, abstain, and negatives (e.g., malignant tumor, needs further testing, no tumor) but need to redefine the objective function appropriately.   
  
  
**Comment 7(c):** Q4.3 Aleatoric and epistemic uncertainties are here not distinguished from each other, which seems to be an issue: what if u(x) is maximal ? Is it reasonable to make decisions in this case ?   
  
**Response:**  Making a decision about positive class prediction, abstaining or negative class prediction based only on uncertainty level of a bin is likely to be suboptimal. Instead, it is preferable to consider the model output score, uncertainty level as well as the actual positive class probability in the calibration set for the corresponding bin. If the actual positive class probability  is high, then the bin will get picked to be on the positive side of the decision boundary. (Similarly with negative class and abstaining if neither is high).  
  
  
Since aleatoric uncertainty is a deterministic function of the model estimated conditional probability (model score), the additional predictive power of the uncertainty signal primarily comes from the ‘epistemic uncertainty’. In case of the posterior networks, the differential entropy of the ‘epistemic distribution’ q(x) includes contributions from both aleatoric and epistemic uncertainty. However, this is not an issue since the calibration approach adjusts for it and utilises the non-redundant information given the model score.  
  
**Comment 7(d):** Q4.4 What is the effect of discretizing the Score-Uncertainty space ? To this extent, shouldn't the independent splitting on both dimensions avoided ?  
  
**Response:** Fig 5(a-b) show the dependence of performance on varying bin counts. In general, very fine bins lead to high computation and poor generalization, while coarse bins lead to over smoothing even if the compute costs are lower. There is a sweet spot that is dataset dependent.   
  
We did not understand the comment on avoiding the independent splitting. In Sec 6.2, Table 1, we do report results on the equi-span setting with independent splitting. In most cases, the resulting recall @precision bound from nested splitting is comparable or better.

Reviewer comment (Theory):
The "proof" that the partitioning problem of the Score \(\times\) Uncertainty space is NP-hard (mentioned in the introduction) seems to directly follow from the bin-packing problem behind it.

Author response:
**Response:**  Please note that partitioning of the score-uncertainty space does not readily map to any of the standard bin-packing or knapsack problems because while the profit (recall) is additive, the cost (1-precision) is not additive and there is a dependence on the bins selected till that point. The NP-hardness result follows from a mapping to the subset-sum problem through a careful constructed reduction (Appendix F). To the best of our knowledge, this is not a previously published finding and there are potentially other applications for this result.

Reviewer comment (Evaluation):
In the equi-weight binning case, "it is possible that a bin with lower positivity rate might be preferable to one with higher positivity rate due to different number of samples": can more details be provided on this situation?

Author response:
**Response:**  Thanks for the question. We will rewrite the line to explain it better. Let us say we wish to maximize recall for a precision bound of $\sigma$. Let the current selected bins have a net total of N samples with P of them positive. Given two bins A and B with total and positive samples and $(n_A, p_A)$ and $(n_B, p_B)$. Even if $p_A/n_A > p_B/n_B$, it might be preferable to choose bin B over bin A since it is possible that $(P+p_B)/(N +n_B) > \sigma > (P+p_A)/(N +n_A)$ when the bin sizes $(n_A, n_B)$ are different. For example, consider the case where P=10, N=20, p_A=5, n_A = 15, p_B = 1, and n_B =4 . Here $p_A/n_A = 5/15 > p_B/n_B = 1/4$, but $( P+p_B)/(N +n_B) = 11/24 > (P+p_A)/(N +n_A) = 15/35$.

Reviewer comment (Experiments):
Regarding the binning strategy, for any given score level, the uncertainty bin indices are monotonic with respect to actual values; however, this does not correspond to the same level of model score. Does this have an impact on the results? More generally, what is the impact of the binning strategy on the overall procedure? The procedure may be sensitive to the bins computed.

Author response:
**Response:**  We explore three binning strategies (a) independent binning or equ—span (b) splitting first on uncertainty followed by score [Unc-Score] (c) splitting first on score followed by uncertainty [Score-Unc]. In Sec 6.1, Table 1, we present results on (a) and (b). We skipped including results on (c) for the lack of space and to avoid confusion. The table below summarizes the results for binning strategy (c) which yields similar results. There is also a dependence on the number of bins chosen (see Figures 5(a) and 5(b)) In future, it might able be preferable to explore perform binning and decision boundary detection jointly.  
  
Table 2: Performance with different equi-weight binning strategies. Score-Unc involves splitting on Score quantiles followed by that Uncertainty while it is the opposite for Unc-Score. Same number of score and uncertainty bins is used in both.  

| Dataset                | Criteo (Tau=3) [Recall@90% precision] |                | Avazu (Tau=5) [Recall@70% precision] |                |
|------------------------|-------|----------------|-------|----------------|
| Binning nest Order         | Score-Unc   | Unc-Score   | Score-Unc   | Unc-Score   |
| #bins                  | (500, 3)    | (3, 500)    | (500, 3)    | (3, 500)    |
| ST                     | 2.2±0.2%  | 2.2±0.2%  | 1.9±0.6%  | 1.9±0.6%  |
| MIST                   | 2.6±0.2%  | 2.6±0.3%  | 2.9±0.2%  | 2.6±0.3%  |
| GMT                    | 2.6±0.4%  | 2.7±0.3%  | 2.9±0.2%  | 2.7±0.3%  |
| EW_DPMT                | 2.7±0.2%  | 2.7±0.3%  | 2.9±0.2%  | 2.7±0.3%  |

Reviewer comment (Experiments):
The proposed approach is motivated by the applications such as medical diagnosis and fraud detection. However, only advertising and e-commerce datasets are considered in the empirical experiment section. It would strengthen the work, if other additional imbalanced datasets from different application domains could have been experimented, as well.

Author response:
Due to the space limitation we could only present results on a few datasets, but we do wish to clarify that the E-Com dataset was derived from a proprietary fraud/abuse detection application. Certain transformations were applied to avoid disclosing sensitive information while preserving the core integrity of the data for research purposes.

Reviewer comment (Experiments):
Why choosing the particular datasets? (additional data from some other application domains could support the work and findings further)

Author response:
Due to the space limitation we could only present results on a few datasets, but we do wish to clarify that the E-Com dataset was derived from a proprietary fraud/abuse detection application. Certain transformations were applied to avoid disclosing sensitive information while preserving the core integrity of the data for research purposes.