Paper: Class-Conditional Conformal Prediction for Imbalanced Data via Top-$k$ Classes (ICLR-2024)

Reviewer comment (Novelty):
The value of the k-CCP contribution over CCP is unclear. It seems relatively modest to restrict to the top-k classes.

Author response:
The $k$-CCP algorithm adapts the double calibration mechanism (calibration on both non-conformity score and ranks in Equation (9), instead of a single calibration in standard CCP) to take a good trade-off of coverage and efficiency. The key advantage of the additional calibration over rank is to exclude those data where their true labels are ranked beyond top-$\widehat k(y)$ positions (i.e., poorly predicted data) in the calibration process, which in turn helps remove labels beyond top-$\widehat k(y)$ positions in the testing process.

Reviewer comment (Experiments):
A concrete real-world example would be better. Most of the experiments are on vision, but the only application mentioned in the abstract and introduction are general references to medical diagnoses.

Author response:
It is necessary to set UCR to the same value (also close to $0$) for all methods, because it is a measure of the violation of class-conditional coverage. Our paper studies the efficiency of CP methods that guarantees the class-conditional coverage. If UCR is not close to $0$ (like MCP), we cannot verify the class-conditional coverage is achieved, so that it is consequently not meaningful to compare prediction set sizes as the measure of efficiency.

Reviewer comment (Theory):
A concrete example instead of or combined with Proposition 1 would help. Intuitively, it seems like conformal prediction with imbalanced classes would suffer the same problems as accuracy optimization—the classes that have few samples in training/calibration data are unlikely to be predicted. It seems like a more intuitive example could be provided.

Author response:
We would like Proposition 1 to be a general and easy-to-verify theoretical result. This is why we did not connect it to the learning observation (recall that a key advantage of CP is model-agnostic, i.e., invariant to the learning algorithm). Instead, we can easily interpret from Proposition 1 and experiments to verify the conditions. The first columns of Figure 1 and 4 show the ease of under-coverage of MCP over each class, which in fact provides a concrete illustration for Proposition 1.

Reviewer comment (Writing):
The notation of the paper is slightly confusing. For example, you use $r_f(X, Y)$ at the beginning of Section 4.2, where $X, Y$ are datasets. However, earlier, you use $r_f(X_{n+1}, y)$ to correspond to a specific datapoint $X_{n+1}, y$. Which is it? Moreover, one uses $\epsilon_y$ to denote the top-k error but $\epsilon_{n_y}$ to denote a general constant. Using $\epsilon$ for a general constant definition is not a great practice, in my opinion, and greatly complicates the reading.

Author response:
Q2: **The notation of the paper is slightly confusing. Using $r_f(X,Y)$, where $X,Y$ are datasets. However, earlier, using $r_f(X_{n+1},y)$ correspond to a specific datapoint $X_{n+1},y$. Which is it?**  

A2: We summarize our main notations in Section 2. Specifically, “$(X, Y)$ is a data sample”, and $\mathcal D$ denotes the dataset, e.g., the training set $\mathcal D_{\text{tr}}$ and the calibration set $\mathcal D_{\text{cal}}$. Moreover, $r_f(X, Y)$ denotes the rank of the ground-truth label $Y$ predicted by the classifier $f$ on input $X$.  

In Equation (3) of Section 3, we introduce “a new testing input $X_{n+1}$”. Accordingly, in Section 4.1 and 4.2, $r_f(X_{n+1}, y)$ denotes the rank of $y$ (this $y$ may be any label rather than the ground-truth one) predicted by classifier $f$ on the testing data $X_{n+1}$, where $y$ is a fixed label. Both CCP and $k$-CCP require iterating over all possible $y$ from $\mathcal Y$ (in our multiclass case, $\mathcal Y = \{1, \cdots,C\}$ for a total of $C$ classes).

Reviewer comment (Experiments):
The computation of $\hat{k}(y)$ that achieves sufficiently small $\epsilon_{y}$ seems very difficult. The equation provided to calculate this calibrated class value seems very computationally expensive to find. Moreover, it seems that the quality of the generated sets is highly dependent on $g$. An ablation on how different values of $g$ affect the generated length of the sets should be included. If the prediction sets' quality is only strong for different $g$, which vary greatly between different hyperparameters, then doing a hyperparameter search for different $g$ for every individual dataset is very expensive. Comparing a method for which the hyperparameters have been extensively tuned to another algorithm for which this is not the case is not a fair empirical comparison. Additionally, a detailed section on the computability of the calibrated class is warranted. It seems that you need to do a linear search over the calibrated dataset many times, which is far more expensive than traditional Split Conformal Prediction. A breakdown here would be useful.

Author response:
Q3: **The computation of $\hat k(y)$ that achieves sufficiently small ϵy seems very difficult. Indeed, the equation provided to calculate this calibrated class value seems very computationally expensive to find.**  

A3: Verifying if Equation 10 is satisfied in practice does not require significant computation/data overhead. The complexity for verification is $O(n C)$ (or improved to $O(n \log(C))$ with binary search over classes from top-$1$ to top-$C$ for a large value $C$) with $n$ calibration samples (or $n$ testing samples if verification on testing data) and $C$ classes. Under the standard i.i.d. condition, the empirical estimates approximate population values in $O(1/\sqrt{n})$.  

Q4: **Comparing a method for which the hyperparameters have been extensively tuned to another algorithm for which this is not the case is not a fair empirical comparison.**  

A4: Our $k$-CCP does not require more hyper-parameters to be tuned compared with baselines CCP and cluster-CP [R13] (an added baseline in the updated paper, Table 1, 15, 16, and 17, Figure 1, 2, 4, 5, and 24 in our updated paper). The only hyper-parameter $g$ has to be tuned for each CP method to achieve class-conditional coverage (their UCR close to $0$), under which we can compare their prediction set sizes as an efficiency measure. As a result, we believe our experiments provide a fair comparison.  

Q5: **Moreover, a detailed section on the computability of the calibrated class is in order.**  

A5: See A3 above Q3.  

Q12: **An ablation study on how different values of $g$ affect the generated length of the sets.**  

A12: We add the ablation study to verify how hyper-parameter $g$ affects the performance of CCP, cluster-CP, and $k$-CCP. We use the under coverage ratio (UCR) and average prediction set size (APSS) as metrics (see Figure 2, 5, and 23 in the updated paper). According to the results, we observe that UCR and APSS of $k$-CCP are much smaller than CCP and cluster-CP with the same $g$ value. It verifies that $k$-CCP achieves a better trade-off between coverage and prediction set size.

Reviewer comment (Experiments):
There is only one baseline for CCP. A broader range of methods for CCP should be compared. This could include the methods from ("Class-Conditional Conformal Prediction With Many Classes", "Classification with Valid and Adaptive Coverage", "Improving Conditional Coverage via Orthogonal Quantile Regression", "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification"). There is a vast array of CCP works, and comparing to the most basic method is insufficient.

Author response:
Q11: **Only compare the method to the simplest baseline of CCP.**  

A11: We added the following experimental results as requested by the reviewers.  

(1) Comparison with Cluster-CP [R13], an accepted NeurIPS-2023 paper, using both APS and RAPS scoring functions  

(2) Comparison experiments for MCP [R14], CCP [R16], [R17], Cluster-CP, and $k$-CCP with both APS and RAPS scoring functions  

(3) Comparison experiments with conformal training to show the synergistic benefits of CP methods including $k$-CCP using a better classifier  

(4) Ablation study for hyper-parameter $g$ with CCP, Cluster-CP, and $k$-CCP  

(5) Verification of $\sigma_y$ on calibration data for different datasets  

Input space conditional CP methods pointed out by reviewers are not applicable for output space conditional CP (i.e., class-conditional CP). So we could not add such comparison experiments. We would like to point out that we included all the baselines used by the Cluster-CP paper which is accepted for publication at NeurIPS-2023.

Reviewer comment (Evaluation):
If the nominated coverage $\tilde{\alpha}$ is larger than $\alpha$, then why are the generated sets shorter? This is not immediately clear. Is it because this allows for the use of only the top-k classes, which shortens the intervals?

Author response:
Q8: **If the nominated coverage $\tilde \alpha$  is larger than $\alpha$, then why are the generated sets shorter?  Is it because this allows for the use of only the top-$k$ classes, which shortens the intervals?**  

A8: Yes, the additional calibration for the top-$k$ class improves efficiency (produced smaller prediction sets). The $k$-CCP algorithm adopts the double calibration mechanism to achieve improved trade-off between coverage and prediction set size. The key advantage of the additional calibration over rank is to exclude those data where their true labels are ranked beyond top-$\widehat k(y)$ positions (i.e., poorly predicted data) in the calibration process, which in turn helps remove labels beyond top-$\widehat k(y)$ positions in the testing process.

Reviewer comment (Experiments):
In the experiments, how were the hyperparameters chosen? It is unclear where values such as $g$ were chosen. Were the hyperparameters for the baselines chosen in a similar manner to that of the proposed method?

Author response:
Q9: **In the experiments, how were the hyperparameters chosen? Were the hyperparameters for the baselines chosen in a similar manner to that of the proposed method?**  

A9: See A4 above Q4.

Reviewer comment (Novelty):
Do existing works guarantee that their method of CCP outperforms the naive CCP method in terms of the size of the prediction set, or is this unique to this work?

Author response:
Q10: **Do existing works guarantee that their method of CCP outperforms the naive CCP method in terms of the size of the prediction set, or is this unique to this work?**  

A10: We add a baseline cluster-CP [R13] that achieves approximate class-conditional coverage (see Table1, 15, 16, and 17 in the updated paper) and find our $k$-CCP out performs it in efficiency. In [R13], the authors empirically showed improved efficiency, but did not provide theoretical analysis or guarantee. In contrast, our paper theoretically analyzes the factors ($\sigma_y$ in Theorem 2) that reduce prediction set sizes of our $k$-CCP and also verifies in experiments in the last column of Figure 1 (we further verify these condition numbers on calibration data in Figure 24 of the updated paper).

Reviewer comment (Writing):
The notations are sometimes a bit convoluted, e.g. in Proposition 1. Moreover, the terminology is sometimes somewhat non-standard. CCP from the paper is often called Mondrian conformal prediction in the literature, and CCP itself is also used as an abbreviation for cross-conformal prediction.

Author response:
Thank you for your suggestion, but our $k$-CCP is not a Mondrian conformal predictor. In fact, Mondrian conformal prediction (MCP) is a general procedure that encompasses many kinds of conditional conformal prediction [R11] [R12], including class conditional conformal prediction and group (attribute) conditional conformal prediction, etc. Our paper focuses on only class-conditional conformal prediction, instead of the general conditional coverage. Therefore, we use the terminology “class-conditional conformal prediction” to represent the relevant algorithms.

Reviewer comment (Theory):
Proposition 1 feels a bit overly complex. Although technically sound, the content should be straightforward. E.g. if the coverage of two classes should average to 0.9 and one is larger than that, the other is going to be lower. This has also been discussed in a recent paper by Ding et al. "Class-conditional conformal prediction with many classes", Neurips 2023.

Author response:
Proposition 1 is a general result to show the ease of under-coverage for Marginal CP. Technically, we only use very mild assumptions in Equation (7) as the condition for the violation of class-conditional coverage. Although other literature may discuss the same observation, our Proposition 1 theoretically quantifies the gap of class-conditional coverage and importantly makes this quantity easily verifiable in experiments (See the first column of Figure 1 and 4).

Reviewer comment (Theory):
I cannot see what Theorem 2 contributes. The theorem more or less assumes that the new method is better (the sigma factors), so obviously it will be better on average.

Author response:
The contribution of Theorem 2 is to theoretically identify the source of reduced prediction set sizes by $k$-CCP, so that it enables verification in experimental results shown in Figure 1 (last column). We additionally verify it on calibration data in Figure 24 of the updated paper. We believe that Theorem 2 reveals this insight/knowledge and wraps up the key factors into the condition numbers $\sigma_y$.

Reviewer comment (Theory):
The purposed method heavily relies on the ranking of candidate class labels by the classifier, which could be a limitation if the classifier's ranking is not reliable.

Author response:
No, $k$-CCP does not make assumptions on the top-$k$ accuracy to configure (see Theorem 1 and Remark 2). 

$k$-CCP degenerates to CCP in the worst-case when the label rank threshold for each class is set to the total number of classes. Intuitively, the reduction in prediction set sizes from $k$-CCP over CCP depends on how small the label rank thresholds are which is classifier-dependent. If the top-$k$ accuracy of the classifier is good, then we will achieve significantly small prediction sets using $k$-CCP (Theorem 2 and condition numbers $\sigma_y$). Our experimental results on four diverse datasets demonstrate that $k$-CCP significantly reduces prediction set size over CCP and Cluster-CP (NeurIPS-2023 paper).

Reviewer comment (Novelty):
Chapter 3 appears somewhat redundant. Using an overall score for each class in conformal prediction (MCP) would lead to marginal coverage rather than class-specific coverage seems evident. Previous works by (Lei, 2014; Sadinle et al., 2019) studied overall coverage and class-specific coverage separately.

Author response:
We do not use the fact that MCP does not guarantee class-conditional coverage as the only key motivation. Instead, we use its failure analysis to highlight the ease of under-coverage on imbalanced data (see the first column of Figure 1 for significant deviation of class-wise quantiles even for relatively balanced data where $\rho = 0.5$). On the other hand, the existing CCP method guarantees the class-conditional coverage but produces large prediction sets. These two observations together motivate the need for efficient CP algorithms which provide class-conditional coverage with small prediction sets.

Reviewer comment (Theory):
In Theorem 2, could assumption (10) be too strong to achieve? It suggests that kCCP algorithm should lead to a coverage that is less than or equal to the CCP and achieve a shorter length as a trade-off. Also, could you elaborate on how this assumption was validated?

Author response:
The assumption of condition numbers $\sigma_y$ in Equation (12) is commonly satisfied and supported by empirical results (see Section 5.2, Figure 1, the last column with the standard training process). In addition, in Figure 24 of the updated paper, we also verify this assumption firmly holds on the calibration set.

To verify the assumption in Equation (10), we directly estimate the empirical values of $\sigma_y$ for each class label $y$ on testing data (Section 5.2, the last column of Figure 1). For fixed $y$, we go over all testing samples $X_{n+1}$ and calculate the frequencies of two events:
(1) the joint event “the coverage of non-conformity scores and calibrated rank”, i.e., $V(X_{n+1}, y) \leq \widehat Q_{1-\tilde \alpha}^{\text{class}}$ with $\tilde \alpha \leq \alpha$” and meanwhile the rank of $y$ stays in top-$\widehat k(y)$, i.e., $r_f(X_{n+1}, y) \leq \widehat k(y)$

(2) the event “the coverage of non-conformity scores”, i.e., $V(X_{n+1}, y) \leq \widehat Q_{1-\alpha}^{\text{class}}$ with nominated mis-coverage $\alpha$”.

After that, we compute the fraction of these two frequencies, to estimate $\sigma_y$.

Reviewer comment (Experiments):
I might have overlooked this detail, but how can we choose $k$ by hyper-parameter $g$ in general?

Author response:
In Appendix C.2, we explained that we selected the value of $g$ from the interval $[0.1, 1]$ in increments of $0.05$ to find the minimal $g$ that $k$-CCP and CCP achieved the target class conditional coverage. We will move this explanation to the main text in the revised paper.

Reviewer comment (Evaluation):
The Average Prediction Set Size (APSS) you presented takes the average prediction length evenly for each class. I wonder if you can present the comparison of the overall average prediction lengths. I'm curious if we will have a larger set-valued prediction for the ''majority'' classes.

Author response:
We report the overall average prediction lengths on the balanced testing set. 
Because our goal is class-conditional coverage, the macro average prediction set size is more suitable for fair comparison. Additionally, we use a balanced calibration dataset, even though the training dataset itself is imbalanced. It means that the macro average prediction set size equals the micro average prediction set size, which is also adapted by another class conditional conformal prediction method [R9].

We added new experiments to compare the average prediction set size of majority, medium, and minority groups in MCP, CCP, cluster_CP [R9], and  $k$-CCP (See Figure 21 with APS scoring function and Figure 22 with RAPS [R10] scoring function in Appendix H of the revised paper).  We select the top $1/4$ classes of the largest number of data to the majority group. Similarly, we assign the bottom $1/4$ classes of smallest number of data to the minority group and the remaining $1/2$ classes to the medium group. In conclusion, for the same CP method, we find that the group-wise average prediction set sizes do not differ too much, while the sizes generated by different CP methods can be large, e.g., $k$-CCP outperforms baselines significantly.

[R9] Ding, T., Angelopoulos, A. N., Bates, S., Jordan, M. I., & Tibshirani, R. J. (2023). Class-Conditional Conformal Prediction With Many Classes. arXiv preprint arXiv:2306.09335.

[R10] Angelopoulos, A., Bates, S., Malik, J., & Jordan, M. I. (2020). Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193.

Reviewer comment (Theory):
One of the advantages of conformal guarantees is that they do not depend on the classifier used to compute the prediction sets. Assumptions on the classifier make the method no longer applicable as a post-hoc procedure working with a black-box ML model. In addition, in realistic scenarios it may be hard to make valid assumptions on the top-k accuracy of the classifier. Moreover, if a classifier is trained using an imbalanced dataset, one should expect the accuracy of the classifier to be higher for majority classes, which does not necessarily guarantee good performance on the top-$k$ accuracy.

Author response:
No, $k$-CCP does not make assumptions on the top-$k$ accuracy to configure (see Theorem 1 and Remark 2). 

$k$-CCP degenerates to CCP in the worst-case when the label rank threshold for each class is set to the total number of classes. Intuitively, the reduction in prediction set sizes from $k$-CCP over CCP depends on how small the label rank thresholds are which is classifier-dependent. If the top-$k$ accuracy of the classifier is good, then we will achieve significantly small prediction sets using $k$-CCP (Theorem 2 and condition numbers $\sigma_y$). Our experimental results on four diverse datasets demonstrate that $k$-CCP significantly reduces prediction set size over CCP and Cluster-CP (NeurIPS-2023 paper).

Reviewer comment (Theory):
The computation of $\hat{k}$ depends on the hyper parameter $g$. One would expect that tuning such a hyper parameter would add a data and computation overhead. The authors do not elaborate on this overhead, neither on the process of selecting the hyper parameter nor on the implications that this process can have. Furthermore, it is not clear if the proposed $\hat{k}$ will never be such that it violates the target coverage $1-\alpha$. A definition of $\hat{k}$ in theorem 1 as well as guarantees that the error terms in theorem 1 are small or can be small with high probability for that $\hat{k}$ could strengthen the contribution.

Author response:
Tuning the hyper-parameter $g$, similar to the model selection process for the other hyper-parameters like learning rate, only requires minor computational overhead. We select the $g$ from the interval [0.1, 1] to find the minimal $g$ that UCR of $k$-CCP and CCP equal to $0$. The detailed tuning approach is introduced in Remark 2 and Appendix C.2. Moreover, it is necessary for every CP method (including compared baselines) to tune the effective $1-\alpha$ to ensure their UCR is smaller than a universal threshold close to $0$ in practical experiments (refer to our response to Q12 for why UCR has to be set near $0$ for all class-conditional CP methods uniformly). In other words, tuning this hyperparameter $g$ for $k$-CCP does not cost more overhead compared to baselines.

Reviewer comment (Theory):
The results of theorem 2 strongly depend on the assumption (10). This assumption seems to depend on the probability distributions that may be hard to safely evaluate and verify in practice without a significant data overhead. It would be very useful to provide insights, or a data efficient method with which one could check if (10) is satisfied.

Author response:
Verifying if Equation 10 is satisfied in practice does not require significant computation/data overhead. The complexity for verification is $O(n C)$ (or improved to $O(n \log(C))$ with binary search over classes from top-$1$ to top-$C$ for a large value $C$) with $n$ calibration samples (or $n$ testing samples if verification on testing data) and $C$ classes. Under the standard i.i.d. condition, the empirical estimates approximate population values in $O(1/\sqrt{n})$.

In our verification experiments, it reduces to verifying if condition numbers $\sigma_y \leq 1$. If calibration data and testing data are i.i.d., then this verification experiment can also be done on the calibration set (See Section I, Figure 24 in our updated paper).

Reviewer comment (Novelty):
Given that there are already other algorithms to address class conditional coverage, showing that MCP does not necessarily provide class-conditional guarantees does not seem as a strong motivation for the current work. Existing prior work [2] has already provided evidence that MCP does not satisfy class conditional guarantees. Besides, MCP was never designed or claimed to achieve class conditional coverage.

Author response:
We do not use the fact that MCP does not guarantee class-conditional coverage as the only key motivation. Instead, we use its failure analysis to highlight the ease of under-coverage on imbalanced data (see the first column of Figure 1 for significant deviation of class-wise quantiles even for relatively balanced data where $\rho = 0.5$). On the other hand, the existing CCP method guarantees the class-conditional coverage but produces large prediction sets. These two observations together motivate the need for efficient CP algorithms which provide class-conditional coverage with small prediction sets.

Reviewer comment (Experiments):
In the experimental results k-CPP and CPP seem to perform very similarly in terms of conditional coverage, suggesting that the main advantage of k-CPP is the reduced set size. However, there is no comparison with CPP using the RAPS method [1] that improves over APS. It would be important to investigate (theoretically and/or experimentally) how the proposed method compares to other approaches with conditional guarantees such as [3], or other approaches to improve the size of the prediction sets [4]. Providing results on how the proposed approach advances over such works would make the contribution much stronger.

Author response:
We added the following experimental results as requested by the reviewers.

(1) Comparison with Cluster-CP, an accepted NeurIPS-2023 paper, using both APS and RAPS scoring functions

(2) Comparison experiments for MCP, CCP, Cluster-CP, and $k$-CCP with both APS and RAPS scoring functions

(3) Comparison experiments with conformal training to show the synergistic benefits of CP methods including $k$-CCP using a better classifier

(4) Ablation study for hyper-parameter $g$ with CCP, Cluster-CP, and $k$-CCP

(5) Verification of $\sigma_y$ on calibration data for different datasets

Input space conditional CP methods pointed out by reviewers are not applicable for output space conditional CP (i.e., class-conditional CP). So we could not add such comparison experiments. We would like to point out that we included all the baselines used by the Cluster-CP paper which is accepted for publication at NeurIPS-2023.

Reviewer comment (Writing):
The entire manuscript gives the impression that it was written in a rush, as typos, grammatical errors and poor sentence structure appear very frequently. These make the work cumbersome to read. Moreover, there is considerable room for improvement in terms of clarity, flow, and cohesion.

Author response:
Thank you for your careful corrections and suggestions. We will fix typos, grammatical errors, unclear presentation and complicated notations in text/tables/captions in the revised paper.

Reviewer comment (Writing):
On page 5, in the definition of the top-k error, the random variable $Z$ is not defined. In the same paragraph, it is not clear what is the distinction between $\hat{k}(y)$ and $k(y)$. Also, $\hat{k}(y)$ is not formally defined the first time it is introduced, which is confusing.

Author response:
We highlight that  $\widehat k(y)$ is formally defined in Equation (9) and the paragraph before Theorem 1. Later we introduce how to determine its value in Remark 2. However, we do not use $k(y)$ in our paper. 
For the notation $Z$, we define it as $Z = (X, Y)$ as the features-label pair. We will include a clear definition of the random variable $Z$ in revision.

Reviewer comment (Theory):
It is not clear why $\sigma_{y}$ is necessary in (10), as the denominator cancels out. One could directly state the condition in (10) using just the numerator of $\sigma_{y}$.

Author response:
We use $\sigma_y$ for a simpler notion of the condition number of the learned classifier. Our ablation study verifies the validity of Theorem 2 (see Section 5.2, the last column of Figure 1, $\sigma_y$ is exactly the X-axis).

Reviewer comment (Theory):
The under coverage indicator in the Under Coverage Ratio assumes a fixed calibration set. However, conformal coverage guarantees on $1 - \alpha$ are in expectation over the calibration set and the test sample. As a result, the UCR definition appears incorrect.

Author response:
We disagree that the UCR definition appears incorrect as a measure of under-coverage. Although CCP algorithms guarantee class-conditional coverage in expectation, we do not have access to the underlying distribution, so we can only measure the coverage empirically. As a result, we define UCR as the measure of under-coverage on a realized testing set, i.e., it is the percentage of classes on which under-coverage occurs. To better avoid the bias of the realized testing set, we also “repeat experiments over 10 different random calibration-testing splits and report the average performance with standard deviation”. Therefore, we believe UCR is a reasonable metric to measure the class-conditional coverage and provides a fair condition for efficiency (prediction set size) comparison over CP methods.

Reviewer comment (Writing):
The manuscript contains multiple typos, grammatical issues, and unclear phrasing throughout, such as:
- Abstract, 1st line “Classification tasks … arises” —> “Classification tasks…. arise”
- Abstract, 8th line from the end, “…threshold”—> “…thresholds…”
- Introduction, 2nd line of 1st paragraph, “with long tail distribution” —> “with long tail distributions"
- “…minority classes are typically very important”; considered an over-generalization. Suggested rephrase: “…minority classes can be very important…”
- Introduction, first paragraph, before end “imbalance data”—> “imbalanced data”
- Introduction, in paragraph before contributions: “calibrated rank threshold for each class c.”—> “a calibrated rank threshold for each class c.“
- Introduction, contributions “Novel CP algorithm…” —> “A novel CP…” and clarification of “rank threshold pair”
- Multiple further examples as listed in the review including table captions not following ICRL instructions.

Author response:
Thank you for your careful corrections and suggestions. We will fix typos, grammatical errors, unclear presentation and complicated notations in text/tables/captions in the revised paper.