Paper: Provably Robust Conformal Prediction with Improved Efficiency (ICLR-2024)

Reviewer comment (Evaluation):
It seems like RCT is not making too much difference, and it requires retraining the model. While it is an interesting idea, I wouldn't say the experiment supports that this helps, and is kind of distracting from the main idea of the paper.

Author response:
**Weakness #1:** *It seems like RCT is not making too much difference, and it requires retraining the model. While it is an interesting idea, I wouldn't say the experiment supports that this helps, and is kind of distracting from the main idea of the paper.*

**Ans:** Thank you for the comments! We partially agree with your comments that the RCT alone is more expensive and makes less improvement compared to PTT. However, as evident in Table 1 on p.9 of our draft, if we combine RCT with PTT, in some cases we can further improve the result of PTT, which shows the value of RCT. 

We also want to point out that: RCT does not need to train the model from scratch. As discussed in Appendix I.1, in RCT we froze the backbone and only finetuned the last linear layer. This approach could be much cheaper, compared with training from scratch. Therefore, we believe RCT will be useful for users who want to further boost performance and are less sensitive to computational cost.

&nbsp;

Reviewer comment (Evaluation):
Missing discussion of potential failure mode (see Q1)

Author response:
**Ans:** As we discussed in the response to Q1(b), our transformed score is equivalent to the original score in vanilla conformal prediction, hence there would not be a failure mode under this setting. For robust conformal prediction, despite theoretical possibility, our empirical study does not observe a case where PTT inflates more than the original score. We will leave it as  future works to show theoretical possibility or impossibility results for such cases and explore the failure mode of the PTT method. 

Dear Reviewer y2Uk,

Thank you for the suggestion! Despite that we did not find failure cases empirically, we agree that exploring potential failure mode will be theoretically interesting. 

Hence, we have revisited our 1D synthetic data example in Appendix D of our original draft, where the theoretical analysis is more feasible. Following your suggestion, we found that indeed under certain conditions, it is possible to create an example where our PTT method could generate a larger set than the original score. The key idea of the construction is to build a “slightly wrong” base score and carefully choose $\alpha$ and $\sigma$ to make PTT amplify this error.

Our analysis is detailed and added to the Appendix N.6 of the revised draft and our result is in the Table R2 below. It can be seen that if we choose $\sigma = 0.3$, then the average size of PTT is slightly larger than the average size of the baseline (the original score without transformation). 

| $\sigma$ | Avg. Size(Baseline) | Avg. Size(PTT) |
|--------|-----------------|----------------|
| 0.2    |           0.529 |          0.511 |
| 0.3    |           0.525 |          0.529 |
Tab. R2: Average size for original score (Baseline) and PTT in our constructed failure case. We use $\alpha = 0.5$ and $\frac{\epsilon}{\sigma} = 0.01$. 


However, despite being theoretically possible, we also want to point out that this failure mode requires (1) a deliberately designed base score and (2) a very specific choice of $1-\alpha$ and $\sigma$ (Note that in this case, we choose $1- \alpha=0.5$, which is usually much lower than the setting in practice where high coverage is usually desired).  Hence, the chance that this kind of failure happens in practice is very rare – We didn’t observe this failure mode in our experiments, and we found that PTT works very well with improved efficiency across standard image classification benchmarks including cifar 10, cifar 100 and imagenet. 

Nevertheless, we think it is important to explore the potential failure modes of PTT and we sincerely thank the reviewer for this valuable suggestion and insights for us to consolidate our result and improve the draft! Accordingly, we have also added a sentence at the end of Sec 4.1 PTT to point out this theoretical possibility. 

We hope that this example has addressed your concern on the potential failure modes of the PTT method in **Q1(b)**. Please let us know if you still have any reservations, and we would be happy to discuss further!

Reviewer comment (Theory):
While the coverage guarantee always holds, are there cases where the PTT transform hurts the efficiency? For example, it seems like the optimality proof hinges on the uniform distribution, and if the noise is adversarial to this sigmoid transformation can PTT actually inflate more? It's just a bit difficult to imagine a transformation is dominating the original score.

Author response:
**Q1(b):** *While the coverage guarantee always holds, are there cases where the PTT transform hurts the efficiency? For example, it seems like the optimality proof hinges on the unfiorm distribution, and if the noise is adversarial to this sigmoid transformation can PTT actually inflate more? It's just a bit difficult to imagine a transformation is dominating the original score.*
	
**A1(b):** We would like to clarify that our transformation is not designed to “dominate” the original score , but to enhance its robustness to threshold inflation which is required in RSCP/RSCP+. In Appendix L (p.34 of our draft), we discussed this and showed that by our design, the transformed score will be **equivalent** to the original score under vanilla conformal prediction. For RSCP/RSCP+, however, the transformed score would be better against adversarial perturbation due to enhanced robustness. 

&nbsp; 

Dear Reviewer y2Uk,

Thank you for the suggestion! Despite that we did not find failure cases empirically, we agree that exploring potential failure mode will be theoretically interesting. 

Hence, we have revisited our 1D synthetic data example in Appendix D of our original draft, where the theoretical analysis is more feasible. Following your suggestion, we found that indeed under certain conditions, it is possible to create an example where our PTT method could generate a larger set than the original score. The key idea of the construction is to build a “slightly wrong” base score and carefully choose $\alpha$ and $\sigma$ to make PTT amplify this error.

Our analysis is detailed and added to the Appendix N.6 of the revised draft and our result is in the Table R2 below. It can be seen that if we choose $\sigma = 0.3$, then the average size of PTT is slightly larger than the average size of the baseline (the original score without transformation). 

| $\sigma$ | Avg. Size(Baseline) | Avg. Size(PTT) |
|--------|-----------------|----------------|
| 0.2    |           0.529 |          0.511 |
| 0.3    |           0.525 |          0.529 |
Tab. R2: Average size for original score (Baseline) and PTT in our constructed failure case. We use $\alpha = 0.5$ and $\frac{\epsilon}{\sigma} = 0.01$. 


However, despite being theoretically possible, we also want to point out that this failure mode requires (1) a deliberately designed base score and (2) a very specific choice of $1-\alpha$ and $\sigma$ (Note that in this case, we choose $1- \alpha=0.5$, which is usually much lower than the setting in practice where high coverage is usually desired).  Hence, the chance that this kind of failure happens in practice is very rare – We didn’t observe this failure mode in our experiments, and we found that PTT works very well with improved efficiency across standard image classification benchmarks including cifar 10, cifar 100 and imagenet. 

Nevertheless, we think it is important to explore the potential failure modes of PTT and we sincerely thank the reviewer for this valuable suggestion and insights for us to consolidate our result and improve the draft! Accordingly, we have also added a sentence at the end of Sec 4.1 PTT to point out this theoretical possibility. 

We hope that this example has addressed your concern on the potential failure modes of the PTT method in **Q1(b)**. Please let us know if you still have any reservations, and we would be happy to discuss further!

Reviewer comment (Theory):
How do we choose $T$ in practice? Obviously we have to fix it ex-ante to avoid breaking the i.i.d. condition? In fact, given the proof/theorem, what prevents us from using $T=\infty$?

Author response:
**Q2:** *How do we choose T in practice? Obviously we have to fixed it ex-ante to avoid breaking the i.i.d. condition? In fact, given the proof/theorem, what prevents us from using $T = \infty$?*

**A2:** In Appendix F, we conduct a theoretical analysis of PTT and suggest we should choose T high enough. We realized there was a typo in the sentence of original Appendix F after Eq (F.3) and we have fixed it: we change it to “Hence with T large enough, we could achieve our goal…”. We are sorry for any confusion on this. 

We also add an empirical study on the impact of T on CIFAR10 in below Table R1 and summarize it in Appendix N.4 in the revised draft. The results suggest that our PTT method is not sensitive to the choice of T, as long as it’s sufficiently large. Hence, we simply choose $T=400$. We hypothesize that the reason the performance saturates after T becomes sufficiently large is that the linear approximation we performed in Sec 4.1 becomes less accurate: the linear term may not be dominant when the slope is small.

| $T$       | 25   | 50    | 100   | 200   | 400   | 800   | 1600  |
|-----------|------|-------|-------|-------|-------|-------|-------|
| Avg. Size | 2.99 | 2.344 | 2.240 | 2.224 | 2.224 | 2.226 | 2.227 |
Table R1: Impact of $T$ to prediction sets size on CIFAR10.

&nbsp;

Reviewer comment (Presentation):
The difference between RSCP+ and RSCP should be discussed earlier. As the most important contribution of this paper, the algorithmic design of RSCP+ appears until Page 4. It would be better to summarize the main idea or unique part of RSCP+ in the Introduction.

Author response:
Thank you for the suggestions. Following your suggestion, we have added a summary of the main idea of RSCP+ in the revised draft.

Reviewer comment (Experiments):
The empirical comparison to [1] and [2] should be added to the experiments.

Author response:
Thank you for your advice! Following your suggestions, we studied [1][2] and discussed their differences with our work. We found that the empirical results of [1][2] may not be directly comparable to ours: as we discussed in Weakness #1, both works chose a different setting than ours. As far as we know, our RSCP+ framework is the first to provide a valid adversarial robustness guarantee for conformal prediction. 

[1] Ghosh et al. "Probabilistically robust conformal prediction." UAI 2023.

[2] Bastani et al. "Practical adversarial multivalid conformal prediction." NeurIPS 2022.

Reviewer comment (Novelty):
Is the ranking-transformation technique proposed in your paper or already proposed in previous work? If the latter, the citation should be added.

Author response:
Thank you for the questions! We have discussed in Appendix M (Theorem M.13) that calculating the rank is an elementary operation in the statistics community and is well-studied, where we have cited the work [Kuchibhotla (2020)]. We will add a sentence in Sec 4.1 to make it more clear. As far as we know, within the scope of robust conformal prediction, no previous work uses this transformation to improve efficiency. 

[Kuchibhotla (2020)] Arun Kumar Kuchibhotla. Exchangeability, conformal prediction, and rank tests, 2020.

Reviewer comment (Theory):
Why does the Sigmoid transformation require uniformly distributed scores?

Author response:
We believe that there may be a slight misunderstanding here. In PTT, the Sigmoid transformation does not require uniformly distributed scores. As we discussed in Sec. 4.1, the idea is to turn the score distribution into a known distribution, so that we don’t need to craft a specific transformation for every new dataset. We choose a uniform distribution for two reasons:
It’s easy to generate with our ranking transformation.
It’s simple and has good theoretical properties, facilitating our analysis.

Reviewer comment (Theory):
For the ranking transformation, the new score \( Q_{rank} \circ S_i \) is uniformly distributed when we consider the randomness from both \( S_i \) and the hold-out set. In this vein, the new calibration scores \( Q_{rank} \circ S_{i}: i=1,...,n \) and the test score \( Q_{rank} \circ S_{n+1} \) are no longer independent. How can we guarantee the validity of RSCP+?

Author response:
This is a very good question, thank you for asking! We try to formalize your question as: $Q_{rank} \circ S_i$ could be viewed as a function of $S_i$ and the holdout set $D_{holdout}$: $Q_{rank} \circ S_i = f(S_i, D_{holdout})$. Hence, the scores are no longer independent because they depend on the same holdout set. 

However, conformal prediction only requires scores to be exchangeable [Kuchibhotla (2020)] (i.e. Any permutation on these variables does not change the joint distribution), which is satisfied by our transformed scores (because the scores are symmetric). We present a formal proof in Appendix N.1 in the revised draft.

[Kuchibhotla (2020)] Arun Kumar Kuchibhotla. Exchangeability, conformal prediction, and rank tests, 2020.

Reviewer comment (Presentation):
The paper, to its credit, tackles three different techniques, which has unfortunately degraded the reading experience. The proposed techniques have little to do with each other, and jumping between them was a bit hard to grasp in the first few reads. Having to fit all this in the page limit certainly does not help the authors. There is also an over reliance on the Appendix, which made the reading experience very choppy. I am not sure what is the best way to tackle this frankly.

Author response:
**Weakness #1:** *The paper, to its credit, tackles three different techniques, which has unfortunately degraded the reading experience. The proposed techniques have little to do with each other, and jumping between them was a bit hard to grasp in the first few reads. Having to fit all this in the page limit certainly does not help the authors. There is also an over reliance on the Appendix, which made the reading experience very choppy. I am not sure what is the best way to tackle this frankly.*

**Ans:** Thank you for sharing your thoughts! Please let us clarify below. The key goal of our paper is to build a prediction set with a certified robustness guarantee. For this goal, we first proposed RSCP+ in Sec 3 to amend the theoretical flaw in previous works, and then proposed PTT/RCT in Sec 4 to address additional challenges to improve the efficiency of robust conformal prediction. We will make it more clear in the motivation and introduction section to show that these three parts are related and necessary for our goal of developing robust conformal prediction with guarantee:
RSCP+ (obtain certified robustness guarantee) 
PTT and RCT (improve the efficiency of RSCP+). 

We plan to organize the Appendix into three main parts: supplementary contents related to RSCP+, supplementary contents related to PTT, and supplementary contents related to RCT. Note that this is not revised yet in the current revised draft, as we are referring to many different appendix in the original draft, so we think it will be less confusing to keep the original appendix for now. But we plan to organize the Appendix as above in the camera-ready version, and we hope that this will make the supplementary materials more clear and organized to improve the readability. Thank you for helping us improve the draft!

Reviewer comment (Experiments):
The PTT method requires another set of held-out, independent of the calibration set. In order for a fair comparison with vanilla RSCP+, would it make sense to have the combined holdout and calibration sets as the baseline calibration set for vanilla RSCP+? This would eliminate any notion of PTT improvement coming from having access to more data, rather than the transformation.

Author response:
**Q1:** *In order for a fair comparison with vanilla RSCP+, would it make sense to have the combined holdout and calibration sets as the baseline calibration set for vanilla RSCP+?*

**A1:** Thank you for the suggestions! Following your suggestion, we conducted additional experiments for RSCP+ using the combination of the holdout set and calibration set as the baseline calibration set. We find that the results are the same, as shown in Table N.2: they still produce trivial results, where the average prediction set size is the same as the number of classes.

Reviewer comment (Reproducibility):
How were the adversarial examples constructed? is it vanilla PGD? or similar to the SmoothAdv [Salman et al., 2019] paper?

Author response:
**Q2:** *How were the adversarial examples constructed? is it vanilla PGD? or similar to the SmoothAdv [Salman et al., 2019] paper?*

**A2:** Thanks for the question! We discussed in the Appendix I.2 (p.28) in the main draft that we follow the approach of RSCP in [1, Sec 5.1, Attack algorithm], which uses SmoothAdv [Salman et al., 2019] to generate adversarial examples. We optimize objective Eq. (S) in Sec 2.2 in [Salman et al., 2019] with PGD and the gradient approximation in [Salman et al., 2019]. 

[1] Gendler etal. Adversarially robust conformal prediction. ICLR 2022.
[Salman et al., 2019] Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers. NeurIPS 2019

Reviewer comment (Evaluation):
RSCP+ generates relatively large prediction sets on dataset such as ImageNet even with PTT, limiting its application.

Author response:
**Weakness #2:** *RSCP+ generates relatively large prediction sets on dataset such as ImageNet even with PTT, limiting its application.*

**Ans:**
As far as we know, our PTT method provides **the first non-trivial result on ImageNet** as reported in Table 2 in the main draft (p.9). It can be seen in Table 2 that the baselines [1] gives trivial results (prediction set size = full class number = 1000) on ImageNet, while with our method, we can already reduce it by 10-16.9 $\times$ to a size of 59.12. 

In fact, as we discussed in the Q3 and Appendix N.3, following your suggestion, our results can be further improved by ~5$\times$ if we increase the number of monte-carlo samples $N_{MC}$, reaching a prediction set size equal to **10.34** for ImageNet. This is equal to an overall 96.7$\times$ improvement compared to the baseline, which is significant.   

This suggests that our proposed RSCP+ and PTT are scalable and can be used in real-world applications (e.g. safety-critical tasks) where guaranteed robustness is of significant importance.

Reviewer comment (Experiments):
The impact of number of Monte Carlo on RSCP+ remains ambiguous, since the experiment is only conducted on Cifar-10.

Author response:
**Weakness #3:** *The impact of number of Monte Carlo on RSCP+ remains ambiguous, since the experiment is only conducted on Cifar-10.*

**Ans:** Following your suggestion, we have conducted additional experiments on ImageNet and discussed in the above response for **Q3 (see Table R1)** and Appendix N.3 in the revised draft. We found that the size of the prediction set could be reduced further by 3-5.7 $\times$ if we increase the number of Monte Carlo examples.

Reviewer comment (Theory):
The Theorem 1 in [R1] provides a guarantee for RSCP prediction set; moreover, I didn't find noticeable coverage violation in empirical results. Thus, I'm confused why you refine the non-conformity score to bound the estimation error?

Author response:
**Q1:** *The Theorem 1 in [R1] provides a guarantee for RSCP prediction set; moreover, I didn't find noticeable coverage violation in empirical results. Thus, I'm confused why you refine the non-conformity score to bound the estimation error?*

**A1:** We didn’t see the reference provided for the [R1] you are referring to, but we believe you are referring to the RSCP paper by Gendler et al [1].  

The Theorem 1 in [1] is valid *theoretically* but becomes invalid in practice because it assumes the smoothed non-conformity score $\tilde{S}$ is used to construct the prediction set $C_{\delta}$ [1, their Eq (8) at p.4] (or see Eq (9) in Sec 2 in our main draft at p.3), while this is approximated by the Monte Carlo estimator $\hat S_{RS}$ (see Eq (12) in the main draft) in practice. This gap between theory and practice is usually filled by giving an estimation bound in randomized smoothing literature, while this bound is missing in [1] as we point out in Sec 3.
Without giving an estimation error bound of $\hat S_{RS}$, the computed prediction set is invalid in [1], and hence the guarantee in Theorem 1 in [1] will not hold in experiments and in practice. This is why we said at the beginning of Sec 3 that there is a missing step (the step 2: estimation error bound in our main draft) in [1] and missing this step makes the RSCP’s guarantee invalid, despite working empirically. 

To fix this issue, we propose to incorporate the Monte Carlo estimator $\hat S_{RS}$ directly as the non-conformity score, as discussed in the first 3 paragraphs in Sec 3 in our draft (p.4). We show in Fig 2 and prove in our Theorem 1 and Corollary 2 to incorporate the estimation error for $\hat S_{RS}$ such that the prediction set $C_{\epsilon}^+$ satisfies robust coverage guarantees in Eq (2) with Monte Carlo estimation parameters (e.g. $b_{hoef}$).  


[1] Gendler et al. Adversarially robust conformal prediction. ICLR 2022.

Reviewer comment (Experiments):
Can you share the result of PTT+RCT of RSCP+ on ImageNet?

Author response:
**Q2:** *Can you share the result of PTT+RCT of RSCP+ on ImageNet?*

**A2:** Existing conformal training methods, e.g. [2][3] are not scalable to the ImageNet and focus on smaller datasets like CIFAR10 and CIFAR100. As our proposed robust conform training (RCT) builds upon existing conformal training methods (e.g. [2]) with additional modification to incorporate robustness (which is our novelty in sec 4.2), our RCT currently also suffers from the same issues as existing conformal training methods [2][3]. 

Nevertheless, we would like to highlight that RCT is not the only contribution of our work. In fact, we have proposed another novel training-free method called PTT (see sec 4.1 in our draft) that is scalable to ImageNet. Indeed, for practitioners working on datasets at ImageNet scale, we recommend using our PTT method which is scalable and computationally efficient. 

In particular, we have reported in Table 2 of our main draft that the prediction set size could be reduced to **~59** for ImageNet, which gives meaningful and non-trivial predictions unlike baselines. Furthermore, the performance of PTT could be boosted (i.e. the average prediction size can be further reduced to a size around **10** for ImageNet, which is significant) by increasing the number of Monte-Carlo examples, as we discussed immediately in the response below (Q3 and Table R1). This suggests that PTT is very effective and scalable. The result is summarized in the revised draft in Appendix N.3.  

[2] Stutz et al. "Learning Optimal Conformal Classifiers." ICLR 2021.

[3] Einbinder et al. "Training uncertainty-aware classifiers with conformalized deep learning." NeurIPS 2022.

Reviewer comment (Experiments):
Can you conduct additional experiments on the impact of number of Monte Carlo on RSCP+?

Author response:
**Q3:** *Can you conduct additional experiments on the impact of number of Monte Carlo on RSCP+?*

**A3:** 
Thank you for the suggestion! Following your suggestion, we have conducted additional experiments and reported the results below and the revised draft in Appendix N.3 (Table N.1). As shown in Appendix N.3,  the size of the prediction set could be further reduced by increasing the number of Monte Carlo examples: e.g. for ImageNet, we can further reduce the prediction size by 3-5.7 $\times$, which is a significant improvement.   

| $N_{MC}$   | 256    | 512   | 1024      |
|----------|--------|-------|-----------|
| CIFAR100 |  26.06 | 13.60 | **12.43** |
| ImageNet | 59.12  | 19.68 | **10.34** |

Table R1: Impact of $N_{MC}$ to prediction sets size on CIFAR100 and ImageNet.

Reviewer comment (Theory):
Although PTT and RCT successfully improve efficiency (as shown by empirical results), I don't understand your statement that the coverage gap $\alpha_{gap}$ reflects the conservativeness of RSCP. Can you provide more theoretical or empirical explanation?

Author response:
**Q5:** *Although PTT and RCT successfully improve efficiency (as shown by empirical results), I don't understand your statement that the coverage gap reflects the conservativeness of RSCP. Can you provide more theoretical or empirical explanations?*

**A5:** Recall that the coverage gap is defined as the change in coverage after threshold inflation:  $\alpha_{\text{gap}} = \alpha - \alpha_{\text{adj}}$ (Eq. 19 in our draft), and $\textbf{conservativeness}$ is defined as the change in the prediction sets size after threshold inflation: $\textbf{conservativeness} =  \mathbb E (|C_\epsilon(x)| - |C(x)|)$(Eq. K.1 in our draft). Intuitively, given a non-conformity score function, higher coverage implies a larger prediction set. 

When the coverage gap goes to 0, the predictor converges to the vanilla conformal prediction and then the conservativeness is also 0.
In Appendix D (p.19-22) we theoretically analyze a 1-D example and also show that in Table D.1 (p.20), $\alpha_{\text{gap}}$ is positively correlated with conservativeness, supporting our intuition.

Reviewer comment (Evaluation):
In Section 4, the paper points out that RSCP and RSCP+ is conservative: RSCP gives a larger prediction set on both clean and perturbed data; this conservativeness then passes on to RSCP+, as supported by Table 1 and Table 2. However, the complementary results in Appendix I don't strongly indicates RSCP is conservative: for example, on Cifar-10, RSCP generates prediction sets of average size 2.751 based on APS. Is it improper to say that RSCP+ is conservative simply because it builds upon RSCP?

Author response:
**Q6:** *However, the complementary results in Appendix I don't strongly indicates RSCP is conservative: for example, on Cifar-10, RSCP generates prediction sets of average size 2.751 based on APS. Is it improper to say that RSCP+ is conservative simply because it builds upon RSCP?*
	
**A6:** In this paper, conservativeness is measured by the set inflation compared with *vanilla conformal prediction*. As shown in Figure I.1 (b) (p.29), the RSCP has a much larger average size (almost 2-2.5$\times$ larger) compared to vanilla conformal prediction, thus we say it’s conservative. 

In Sec. 4, we said RSCP+ inherits the conservativeness of RSCP to emphasize that RSCP+ is also conservative because it uses the threshold inflation similar to RSCP, which is a root cause of conservativeness. To reduce this conservativeness, we propose both PTT and RCT in sec 4.1 and sec 4.2 and show that we can effectively reduce the set size by 11.57%-31.12% in the Table I.3 that you are referring to.