Paper: The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks (ICLR-2024)

Reviewer comment (Evaluation):
It looks like the improvements on the CIFAR datasets are smaller than the improvements on the Caltech256 and the CUB200 datasets. Is this specifically related to the number of classes in those datasets? If so, would it be better to normalize the improvements by the number of classes?

Author response:
Yes, we agree on this point. Thus, in the updated paper, we normalize the PSS by the number of classees and show the result in Table 3.

Reviewer comment (Presentation):
Section 4 and Section 5 are relatively short compared to other sections. Maybe you can merge those sections into one section (that summarizes the discovery regarding CP against adversarial examples) with two subsections.

Author response:
We agree on this point. We have revised the paper accordingly, see Section 4 with the name NECESSITATE AT FOR ROBUST AND EFFICIENT COVERAGE.

Reviewer comment (Evaluation):
The fact that a model trained without adversarial training does not produce small conformal prediction sets on adversarial examples is not surprising. Since such a model has poor performance (often random or worse), this trend is expected; the prediction set size depends on the quality of the underlying model [Vovk et al. (2005), Shafer and Vovk (2008)].

Author response:
Note that we never claim this effect is surprising in our paper. We show this effect to make the point that adversarial training is necessary for conformal prediction to work in an adversarial environment.

Reviewer comment (Theory):
The correlation between the prediction set size with the prediction entropy and the predicted rank of the ground-truth class is not surprising. It is a consequence of the non-conformity function (or the set function) used; the correlation is apparent from the function definition. Instead, for example, if one were to use the 0-1 loss that outputs 0 if the top-1 prediction is correct and 1 otherwise, the prediction set size would correlate with the top-1 accuracy (which this paper argues is not true).

Author response:
Our paper is an empirical study into the prediction set size of adversarially trained models and how to make the conformal prediction more efficient in such robustly trained models. We are glad to hear that our proposed method makes sense intuitively, as the reviewer admits, while our paper empirically proves that this intuitive design works quite well across several datasets and AT methods. 
However, it is unclear to us what is the difference between the mentioned 0-1 loss function and top-1 accuracy and how to compute the non-conformity score based on the mentioned 0-1 loss function. It is appreciated if the reviewer could elaborate on this point to make it a valid weakness.

Reviewer comment (Experiments):
Gendler et al. (2021) and Ghosh et al. (2023) are highly related to this paper; they propose conformal algorithms to do well on adversarial examples. While this paper highlights the comparisons with Gendler et al. (2021) in Section 1, the more recent work of Ghosh et al. (2023) is not. It is unclear if the authors compared against their proposed method.

Author response:
The reason why we did not compare with Ghosh et al. (2023) is that Ghosh et al. (2023) only considers the l_2 norm instead of l_inf norm and it does not consider the training stage, same as Gendler et al. (2021). Thus, similar to what the reviewer claims in Q1, we believe that the coverage of  Ghosh et al. (2023) would be unsurprisingly bad at adversarially robust coverage.

Reviewer comment (Writing):
The subsection "Adversarial Robustness" (Section 2) contains an incorrect citation. It may have been meant to be Gendler et al. (2021) instead of Salman et al. (2020).

Author response:
This is a typo, we have fixed it in the updated paper.

Reviewer comment (Theory):
In the proof for Theorem 1, when $A$ and $B$ are combined, the bound $c \leq K^{- \alpha}$ is used. This is not satisfied when setting $c = \max \\{ K^{- \alpha} , \cdot \\}$.

Author response:
We have explained how to get the $A \leq cK^c$ in the updated version, page 17. Other theoretical issues like using general Gamma function for real-value $\alpha$ and $\beta$ will be fixed in our future work.

Reviewer comment (Theory):
In the proof for Theorem 1, it is unclear how the bound $A \leq c K^{- c}$ is obtained.

Author response:
We have explained how to get the $A \leq cK^c$ in the updated version, page 17. Other theoretical issues like using general Gamma function for real-value $\alpha$ and $\beta$ will be fixed in our future work.

Reviewer comment (Evaluation):
Theorem 1 is not empirically supported (cf. Table 3).

Author response:
Note that Theorem 1 does not guarantee that the top-1 test accuracy will be improved. Theorem 1 indicates that the generalization error of the beta-weighting method is comparable to that of ERM, so we can train a classifier with the beta-weighting in practice.

Reviewer comment (Evaluation):
It is unclear what evidence exists to show that the majority of data lies in the promising region (stated in Section 6.2).

Author response:
Fig. 5 shows that most training samples are in the promising region. We compute the ratio of test samples in the promising region (Fig. 6) to all test samples on CIFAR100. The ratio is 44.36%.

Reviewer comment (Reproducibility):
It is unclear what is done to reduce the importance weight when $r_{i} = 1$ or $\hat{r}_{i} = 1 / K$ (since this is not part of the promising region).

Author response:
We use small importance weights for those samples. See Fig. 4 and Equation 6 for the importance weight for samples outside the promising region.

Reviewer comment (Writing):
The paper should explicitly mention how the predicted ranks are normalized, i.e., define $\hat{r} ( x , y ) = r ( x , y ) / K$.

Author response:
We will make it explicit in the next version.

Reviewer comment (Presentation):
In Fig. 5, the histograms look indistinguishable. The entire x-axis could be included to highlight any differences.

Author response:
Including the whole x-axis reduces the visual difference. The quantitative difference is shown in the bottom of Fig. 4.

Reviewer comment (Presentation):
In Fig. 4, it is unclear how the promising region is determined in this illustration.

Author response:
According to the Beta-distribution shown in Fig. 4, we choose top 25% as the promising region for visualization purpose.

Reviewer comment (Presentation):
In Fig. 4, it is unclear if the reported ratio is not the fraction of points in that region. The caption and the supporting text for this figure are confusing.

Author response:
It is the ratio of p(r) instead of the ratio of samples, which is explained in the paragraph following Equation (7). We will make it clear in the next version.

Reviewer comment (Writing):
APS is claimed to be more stable than RAPS in Section 7, but it is unclear what this stability refers to.

Author response:
As the prediction set is a quantification of prediction uncertainty, we use the stable APS instead of RAPS to have a low-variance uncertainty quantification to have a better contrast between different AT methods.

Reviewer comment (Evaluation):
The generalizations for which version of the proposed method to use in Section 7.2 are done for datasets, but they seem dependent on the adversarial training method used.

Author response:
We provide our observations in Section 7.2. It makes sense that the performance depends on the AT method, as different AT has different prediction set sizes as Table 1 shows.

Reviewer comment (Theory):
The paper includes empirical risk minimization but does not discuss its assumptions. The generalization error is bound under the i.i.d. assumption.

Author response:
As in every machine learning paper uses empirical risk minimization, we assume the data are i.i.d.. We cite the corresponding papers (adversarial examples, PGD attack and adversarial training) in our paper as most adversarial robustness papers do, as we all assume a basic understanding into the research topic. We have explicitly defined the adversarial training in Equation (2).

Reviewer comment (Writing):
The paper does not explain what adversarial examples are, what the PGD attack is, or what adversarial training is. These details should be provided.

Author response:
As in every machine learning paper uses empirical risk minimization, we assume the data are i.i.d.. We cite the corresponding papers (adversarial examples, PGD attack and adversarial training) in our paper as most adversarial robustness papers do, as we all assume a basic understanding into the research topic. We have explicitly defined the adversarial training in Equation (2).

Reviewer comment (Theory):
The conformal prediction framework is not explained well. Additionally, the paper does not discuss its assumptions and statistical guarantees in detail.

Author response:
We pointed the readers to Romano et al. 2020 for more details about CP in our paper.

Reviewer comment (Evaluation):
Section 4 does not show that adversarial training is indispensable (as mentioned in the last paragraph). It shows that standard non-adversarial training methods lead to large conformal prediction sets. Additionally, saying that the non-adversarially trained models are "completely broken" is incorrect. The paper might want to emphasize that the prediction sets are large, making them less informative.

Author response:
When the prediction set size is almost as large as the class numbers, the conformal prediction provides no uncertainty information, thus we think it is broken. We will revise the paper to make it clear.

Reviewer comment (Evaluation):
It is worth mentioning in the main paper that the top-1 accuracy decreases when using the proposed method.

Author response:
We have revised the paper to make it clear.

Reviewer comment (Reproducibility):
It is worth including details of the experimental setup in Sections 4-6 or pointing to where they are in the paper.

Author response:
We have revised the paper to make it clear.

Reviewer comment (Experiments):
Is the gain related to the entropy minimization loss tied with the APS framework, which, due to the additional randomization, favors prediction with low entropy? Do you still observe the improvement of using the entropy minimization loss using other conformal prediction constructions? For example, using Sadinle et all with average-coverage/per-class coverage.

Author response:
When we use the RAPS as the conformal prediction method, the entropy minimization loss is also better than the AT baseline on Caltech256, see the following table. We think that the mentioned method Sadinle et all is not comparable in our context since their target coverage is not for adversarial examples.

| Caltech256 |    Coverage    |         PSS        |
|:----------:|:--------------:|:------------------:|
|     AT     | 90.533 (1.483) |   32.738 (6.618)   |
|    AT-EM   | 90.322 (1.369) | **30.303 (5.133)** |

Reviewer comment (Experiments):
How do different attack budgets influence the results?

Author response:
In addition to the $\epsilon$=8.0, we test different attack budgets $\epsilon$=4.0, 12.0, 16.0 and add the result on Caltech256 below. It shows that across the attack budgets, our method is consistently better than the AT baseline. 

| Caltech256,$\epsilon$=4.0 |    Coverage    |         PSS        |
|:----------:|:--------------:|:------------------:|
|     AT     |  92.728 (0.814) | 21.913 (1.593)   |
|    AT-EM   |  92.928 (0.672) | 21.639 (1.895)  |
| AT-Beta    |  91.851 (1.062) | **17.172 (2.049)**  |
| AT-EM-Beta | 91.826 (0.761) | 17.589 (1.213)  |

| Caltech256,$\epsilon$=12.0 |    Coverage    |         PSS        |
|:----------:|:--------------:|:------------------:|
|     AT     |  89.684 (0.893) | 78.800 (5.430)  |
|    AT-EM   |  89.483 (0.754) | 76.386 (4.210) |
| AT-Beta    |  89.779 (1.217) | 79.789 (7.094)  |
| AT-EM-Beta | 89.968 (0.922) | **74.383 (4.507)**  |

| Caltech256,$\epsilon$=16.0 |    Coverage    |         PSS        |
|:----------:|:--------------:|:------------------:|
|     AT     |   90.223 (1.096) | 136.863 (7.198) |
|    AT-EM   |  89.968 (0.666) | 132.146 (4.077) |
| AT-Beta    |  90.125 (0.872) | 142.175 (6.892)  |
| AT-EM-Beta |  90.053 (0.980) | **131.636 (5.591)** |

Reviewer comment (Theory):
The assumption on how pk distributed in Theorem 1 seems very stringent. In addition, even if I accept the assumption (which the authors certainly need to justify), the proof also needs to be discussed in more detail and I am not completely convinced currently. For example, the first term in Lemma 1 seems to be dropped in Theorem 1's proof, but isn't it the case that the first term will change as you change the weights and will be higher for the beta-weighted problem?

Author response:
Assumption on $p_k$ does not necessarily require that the polynomial distribution exactly hold. Instead, in practice, it can be generalized by setting a worse scenario as an upper bound.

R6: In generalization analysis, the dominating term is the second error term $O(\sqrt{ d_2(\mathcal P || \mathcal P / \omega) / m })$, rather than the first one $O(1/m)$, since $1/m$ is in higher order compred with $1/\sqrt{m}$. As a result, it is common to make effort to improve the second term. See the variance-based robust learning [R1] and [R2] as examples.

[R1] Namkoong, Hongseok, and John C. Duchi. "Variance-based regularization with convex objectives." Advances in neural information processing systems 30 (2017).

[R2] Maurer, Andreas, and Massimiliano Pontil. "Empirical bernstein bounds and sample variance penalization." arXiv preprint arXiv:0907.3740 (2009).

Reviewer comment (Writing):
What is d(x,y) in the proof of Theorem 1?

Author response:
$d(x,y)$ means the differential of the variable in the calculus. Since we couple the input $x$ and output $y$ together as a joint random variable, we use $d(x,y)$ as the differential of the joint variable.

Reviewer comment (Theory):
The summation in d2(P||P/w) should be r=k instead of r=1.

Author response:
Thanks for pointing out this typo. We will revise it.

Reviewer comment (Presentation):
I need to go to later sections in order to understand Table 1 in the preliminary results; some brief explanations about the evaluation metrics will be helpful.

Author response:
We will revise the paper and add some brief introduction to the evaluation metrics.