Paper: Conformal Prediction for Deep Classifier via Label Ranking (ICLR-2024)

Reviewer comment (Experiments):
In the experiments, choosing $\lambda$ for SAPS uses a subset of the calibration set. Do all baselines use the (same) remaining calibration set? That is, is SAPS calibrated on a smaller set due to $\lambda$?

Author response:
Thank you for the recognition. We address specific concerns below.


**1. Choice of $\lambda$.  [Q1]**

For SAPS and RAPS, we utilize a validation set to tune hyperparameters and the remaining dataset to calibrate the $\tau$. For APS, we calibrate the threshold $\tau$ on the whole calibration set.

Reviewer comment (Evaluation):
I have a reservation about the claimed contribution of higher adaption, i.e., the adaption is not that convincing: For the example of Figure 3(b), now that both SAPS and RAPS achieve the same coverage, why should we require a larger prediction set for difficult observations? In general, the smaller the better. RAPS gives more efficient predictions on those observations with higher difficulty, while the proposed SAPS only gives efficient predictions on a shorter interval of difficulty.

Author response:
In the literature of conformal prediction, the size of prediction sets is expected to represent the inherent uncertainty of the classifier's predictions. With a comparable set size, methods with high adaption can reflect instance-wise uncertainty precisely [R1]. Specifically, prediction sets should be larger for hard examples than for easy ones. Adaption is particularly significant in high-stakes scenarios where assessing the model's reliability is necessary. In Figure 3b, we show SAPS preserves more distinguishable information than RAPS, while achieving a smaller average set size and desired coverage rate. This advantage further strengthens the value of the proposed method.

Reviewer comment (Evaluation):
Even though the proposed method has a promising performance compared to several methods, how is the proposed method far away from the ground truth?

Author response:
We guess that the "ground truth" is the sets produced by APS with the oracle model (correct us if we are mistaken). In Proposition 2, we provide a theoretical result, suggesting that SAPS is capable of producing smaller sets than APS, with the oracle model.

Reviewer comment (Reproducibility):
To make it clearly catch the whole scope, it would be better to explicitly outline the calibration and prediction under the frame of a pseudo-algorithm as the one in RAPS. For example, I believe "We choose the hyper-parameter that achieves the smallest set size on a validation set" fails to disclose the entire picture because the smallest set size on a validation set cannot secure the desired coverage. How did the authors handle this issue?

Author response:
Thank you for the great suggestion. We have included pseudo-code algorithms in Appendix H of the revised manuscript. 

For the validation set, we would like to clarify that various score functions caused by different values of $\lambda$ always satisfy the desired coverage. In other words, the value of $\lambda$ does not affect the coverage guarantee, which is theoretically proved by Proposition 1. This is also supported by the empirical results in Sec 4.2.

Reviewer comment (Writing):
The proofs are not friendly reading (see the section on questions).

Author response:
We thanks for pointing out the typos. We have fixed them in the updated version. 
- $\mathcal{C}\left(\boldsymbol{x}\_{i},y_i,u_i\right)$ is a typo. In Section 4.1, we revised $\mathcal{C}\left(\boldsymbol{x}\_{i},y_i,u_i\right)$ to $\mathcal{C}\left(\boldsymbol{x}\_{i}\right)$ representing the  prediction set for $\boldsymbol{x}\_{i}$.  
- In the proof of Lemma 1, we assume that $p_{(k)}>\frac{1}{k}$ for any $k\geq 2$. Thus,  for any $\tilde{k}< k$, it holds that $p_{(\tilde{k})}>\frac{1}{k}$.
- In Proposition 1, the subscript of symbol $\mathcal{C}\_{1-\alpha}(\boldsymbol{x}\_{n+1},u\_{n+1})$  hase been completed. 
- In proof of Proposition 2, the inequality is the $\frac{1}{\lambda} \leq \frac{1}{1-\frac{1}{K}}$.

Reviewer comment (Writing):
Do you mean $\mathcal{C}(\boldsymbol x_i, y_i, u_i)$ for the definition of coverage rate?

Author response:
We thanks for pointing out the typos. We have fixed them in the updated version. 
- $\mathcal{C}\left(\boldsymbol{x}\_{i},y_i,u_i\right)$ is a typo. In Section 4.1, we revised $\mathcal{C}\left(\boldsymbol{x}\_{i},y_i,u_i\right)$ to $\mathcal{C}\left(\boldsymbol{x}\_{i}\right)$ representing the  prediction set for $\boldsymbol{x}\_{i}$.  
- In the proof of Lemma 1, we assume that $p_{(k)}>\frac{1}{k}$ for any $k\geq 2$. Thus,  for any $\tilde{k}< k$, it holds that $p_{(\tilde{k})}>\frac{1}{k}$.
- In Proposition 1, the subscript of symbol $\mathcal{C}\_{1-\alpha}(\boldsymbol{x}\_{n+1},u\_{n+1})$  hase been completed. 
- In proof of Proposition 2, the inequality is the $\frac{1}{\lambda} \leq \frac{1}{1-\frac{1}{K}}$.

Reviewer comment (Theory):
In the proof of Lemma 1, did you intend to assume $p_{(k)}\geq \frac{1}{k}$? Where will be $\tilde{k}$ used in the proof?

Author response:
We thanks for pointing out the typos. We have fixed them in the updated version. 
- $\mathcal{C}\left(\boldsymbol{x}\_{i},y_i,u_i\right)$ is a typo. In Section 4.1, we revised $\mathcal{C}\left(\boldsymbol{x}\_{i},y_i,u_i\right)$ to $\mathcal{C}\left(\boldsymbol{x}\_{i}\right)$ representing the  prediction set for $\boldsymbol{x}\_{i}$.  
- In the proof of Lemma 1, we assume that $p_{(k)}>\frac{1}{k}$ for any $k\geq 2$. Thus,  for any $\tilde{k}< k$, it holds that $p_{(\tilde{k})}>\frac{1}{k}$.
- In Proposition 1, the subscript of symbol $\mathcal{C}\_{1-\alpha}(\boldsymbol{x}\_{n+1},u\_{n+1})$  hase been completed. 
- In proof of Proposition 2, the inequality is the $\frac{1}{\lambda} \leq \frac{1}{1-\frac{1}{K}}$.

Reviewer comment (Theory):
Is (2) generally correct? In other words, are the prediction results always nested? Particularly in Theorem 2, Since there is a random variable $u$ introduced, why $\mathcal{C}_{1-\alpha}(\boldsymbol{x}, u)$ have the nesting property?

Author response:
The nesting property defined by Eq.2 is a common property holding on all prediction sets for any conformal predictor [R2]. Specifically, if a lower error rate is expected, the set will have a larger size for higher coverage. In this work, since the calibrated threshold $\tau$ is the $1-\alpha$ quantile of scores, the nesting property of $\alpha$ is equivalent to a nesting property for $\tau$, i.e., $$\alpha_1>\alpha_2 \rightarrow \tau_1\leq\tau_2 \rightarrow \lbrace y\in\mathcal{Y}: S(\boldsymbol{x},y,u;\hat{\pi})\leq \tau_1 \rbrace \subseteq \lbrace y\in\mathcal{Y}: S(\boldsymbol{x},y,u;\hat{\pi})\leq \tau_2\rbrace,$$ for a test point $\boldsymbol{x}$ and a random variable $u$. Therefore, we get the nesting property of $\mathcal{C}_{1-\alpha}(\boldsymbol{x},u)$.

Reviewer comment (Writing):
Proposition 1: How is $\mathcal{C}_{1-\alpha}(\boldsymbol x, u)$ defined as in Eq. 3? They have totally different notations.

Author response:
Thank you for pointing out the ambiguous notation. The prediction set in Proposition 1 is defined as  $$\mathcal{C}_{1-\alpha}(\boldsymbol{x},u) =\lbrace y\in\mathcal{Y} : S(\boldsymbol{x},y,u;\hat{\pi})\leq \tau \rbrace.$$ Thus, it is mathematically equivalent to Eq. 3. To mitigate this confusion, we added a detailed description in Proposition 1.

Reviewer comment (Theory):
I didn’t get the point of the proof for Proposition 1. What is the difference between your proof of proposition 1 and Theorem 2? The conclusion of coverage is for the popped $\mathcal{C}(\boldsymbol x_{n+1},u_{n+1})$ but there is no $\mathcal{C}(\boldsymbol x_{n+1},u_{n+1})$ during your proof. I think the authors need to well-articulate the proof.

Author response:
Proposition 1 is a corollary of Theorem 2. Specifically, Theorem 2 gives a coverage guarantee for CP methods whose prediction set has a general formulation $\mathcal{C}(\boldsymbol{x},u,\tau)$. The prediction set of SAPS defined as $\lbrace y\in\mathcal{Y}:S(\boldsymbol{x},y,u;\hat{\pi})\leq \tau \rbrace$ is a specific instance of $\mathcal{C}(\boldsymbol{x},u,\tau)$. Therefore, Proposition 1 offers a coverage guarantee for SAPS. We apologize for this misunderstanding and have revised  Appendix B to make this clearer.

Reviewer comment (Theory):
Why $\frac{1}{\lambda}\leq1-\frac{1}{K}$ but previously you require $\lambda>1-\frac{1}{K}$?

Author response:
We thanks for pointing out the typos. We have fixed them in the updated version. 
- $\mathcal{C}\left(\boldsymbol{x}\_{i},y_i,u_i\right)$ is a typo. In Section 4.1, we revised $\mathcal{C}\left(\boldsymbol{x}\_{i},y_i,u_i\right)$ to $\mathcal{C}\left(\boldsymbol{x}\_{i}\right)$ representing the  prediction set for $\boldsymbol{x}\_{i}$.  
- In the proof of Lemma 1, we assume that $p_{(k)}>\frac{1}{k}$ for any $k\geq 2$. Thus,  for any $\tilde{k}< k$, it holds that $p_{(\tilde{k})}>\frac{1}{k}$.
- In Proposition 1, the subscript of symbol $\mathcal{C}\_{1-\alpha}(\boldsymbol{x}\_{n+1},u\_{n+1})$  hase been completed. 
- In proof of Proposition 2, the inequality is the $\frac{1}{\lambda} \leq \frac{1}{1-\frac{1}{K}}$.

Reviewer comment (Theory):
The theoretical contribution of this paper seems limited. Proposition 1 represents a common property of any non-conformity measure. Moreover, the condition in Proposition 2, $\hat{\pi} = \pi$, is challenging to satisfy in practice. As for another condition  $\lambda \geq 1 - \frac{1}{K}$, note that $\lambda$ used in experiments is searched in the range of 0.001 to 0.5, which conflicts with this condition. Figure 4a shows that when $\lambda$ exceeds 0.2, the set size increases with $\lambda$. It is important to address these concerns.

Author response:
**1. Proposition 1 represents a common property [W1]**

Yes, many existing works have shown their methods possess the property [R1, R2] while some works do not provide theoretical proofs [R3, R4]. In this work, we present the proposition to show that the proposed method also satisfies the finite-sample coverage guarantee from a theoretical perspective. Without the proposition, it becomes ambiguous if our method can obtain the theoretical property. In other words, Proposition 1 enhances the completeness of our work.


**2. Calibrated condition of Proposition 2**

In the literature of conformal prediction, it is a common assumption that the given model is well-calibrated in theoretical analysis [R5, R6, R7]. Based on the assumption, we can then provide the subsequent analysis to show the inherent advantages of SAPS compared to APS. On the contrary, it would be non-trivial to give an in-depth understanding of the proposed method, without the condition. Thus, the theoretical explanation is still valuable to the community. In addition, the empirical results in Section 4 demonstrate the effectiveness of our method with imperfect models.


**3. The range of $\lambda$ in Proposition 2**

We would like to clarify that the range of 0.001 to 0.5 is for the hyperparameter of RAPS, instead of our method. We tune the $\lambda$ of SAPS in the range of 0.02 to 0.6. Moreover, the threshold of $\lambda$ in Proposition 2 is a sufficient condition to show the advantage of SAPS, but it does not reflect that the optimal value of $\lambda$ must exist in the interval $[1-\frac{1}{K},\infty)$ in the experiments.

Here, we conduct an experiment to validate the effectiveness of SAPS with a $\lambda$ in the interval $[1-\frac{1}{K},\infty)$. In particular, we set $\lambda = 1$. The results on three datasets are shown in the table below. We show that SAPS with $\lambda=1$ still outperforms APS, which validates the proposition.

|  Method   | APS  |  SAPS($\lambda=1$) |
|  :----:  | :----:  | :----:  |
| ImageNet  | 20.95 | 3.82     |
| CIFAR-100  | 7.88 | 3.35    |
| CIFAR-10  | 1.97 | 1.80   |

Finally, we would like to clarify that the set size of SAPS would not maintain the upward trend with the increase of $\lambda$. Instead, the set size will converge to a specific value, and SAPS is equivalent to the APS without probability value when $\lambda \to \infty$. To demonstrate this, we update Figure 4a in the revised version by extending the value of $\lambda$ and the updated figure validates the above statement.

Reviewer comment (Theory):
In Equation (8), it is unclear why the authors use 2 instead of another constant in $o(y,\hat{\pi}(x)) - 2 + u$. The authors should provide a motivation and explanation for this choice.

Author response:
**4. Clarification of proposed score function [W2]**

The constant $2$ in Eq. (8) is not a user-guided value. The term $(o(y,\hat{\pi}(\boldsymbol{x}))-2+u)$ contains ranking scores from rank $2$ to rank $(o(y,\hat{\pi}(\boldsymbol{x}))-1)$, and a random ranking score at the rank $o(y,\hat{\pi}(\boldsymbol{x}))$. As the ranking score is defined as a constant $\lambda$, the sum of ranking scores from rank $2$ to rank $(o(y,\hat{\pi}(\boldsymbol{x}))-1)$ is $(o(y,\hat{\pi}(\boldsymbol{x}))-2) \cdot \lambda$. Thus, the formulation of the proposed score function is defined as $\hat{\pi}_{max} (\boldsymbol{x}) + (o(y,\hat{\pi}(\boldsymbol{x}))-2) \cdot \lambda$+ $u\cdot \lambda$.

Reviewer comment (Theory):
The authors only provide theoretical analysis comparing APS and SAPS, and empirical comparisons between RAPS and SAPS. It is necessary to include a detailed comparison with RAPS theoretically, since it is also a modified version of APS.

Author response:
**5. Theoretical comparison with RAPS [W3]**

In this work, we aim to provide an in-depth understanding of SAPS through the comparison of APS and RAPS. While there might be some new insights in the comparison between SAPS and RAPS, it is challenging to analyze these two methods in a unified framework, as RAPS introduces two hyper-parameters. Instead, we empirically show the superiority of SAPS over RAPS in set size, conditional coverage violation, and adaptation. Furthermore, we compare SAPS with RAPS($k_r=1$) in Sec 5 to further demonstrate the negative effect of probability values. We believe the thorough analysis of RAPS is sufficient to support our conclusion.

Reviewer comment (Writing):
On page 3, just above (4), it stated "In the APS method (Romano et al., 2019)". However, are you sure it's not Romano et al., 2020? The 2019 paper was Conformalized quantile regression.

Author response:
Thank you for the correction. We have fixed the citation in the revised version.