Original: A DVERSARIALLY R OBUST C ONFORMAL P REDICTION
Revised: A DVERSARIALLY R OBUST C ONFORMAL P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Conformal prediction is a model-agnostic tool for constructing prediction sets that are valid under the common i.i.d.
Revised: Conformal prediction is a model-agnostic tool for constructing prediction sets that are valid under the common i.i.d.

Original: assumption, which has been applied to quantify the prediction uncertainty of deep net classiﬁers.
Revised: assumption, which has been applied to quantify the prediction uncertainty of deep net classiﬁers.

Original: In this paper, we generalize this framework to the case where adversaries exist during inference time, under which the i.i.d.
Revised: In this paper, we generalize this framework to the case where adversaries exist during inference time, under which the i.i.d.

Original: assumption is grossly violated.
Revised: assumption is grossly violated.

Original: By combining conformal prediction with randomized smoothing, our proposed method forms a prediction set with ﬁnite-sample coverage guarantee that holds for any data distribution with (cid:96) 2norm bounded adversarial noise, generated by any adversarial attack algorithm.
Revised: By combining conformal prediction with randomized smoothing, our proposed method forms a prediction set with ﬁnite-sample coverage guarantee that holds for any data distribution with (cid:96) 2norm bounded adversarial noise, generated by any adversarial attack algorithm.

Original: The core idea is to bound the Lipschitz constant of the non-conformity score by smoothing it with Gaussian noise and leverage this knowledge to account for the effect of the unknown adversarial perturbation.
Revised: The core idea is to bound the Lipschitz constant of the non-conformity score by smoothing it with Gaussian noise and leverage this knowledge to account for the effect of the unknown adversarial perturbation.

Original: We demonstrate the necessity of our method in the adversarial setting and the validity of our theoretical guarantee on three widely used benchmark data sets: CIFAR10, CIFAR100, and ImageNet.
Revised: We demonstrate the necessity of our method in the adversarial setting and the validity of our theoretical guarantee on three widely used benchmark data sets: CIFAR10, CIFAR100, and ImageNet.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural net classiﬁers have achieved tremendous accomplishments over the last several years.
Revised: Deep neural net classiﬁers have achieved tremendous accomplishments over the last several years.

Original: Nevertheless, the increased deployment of these algorithms in real-world applications, especially ones that can be life-threatening like autonomous driving, raises major concerns about their reliability (Heaven, 2019).
Revised: Nevertheless, the increased deployment of these algorithms in real-world applications, especially ones that can be life-threatening like autonomous driving, raises major concerns about their reliability (Heaven, 2019).

Original: To alleviate these issues, it is important to develop techniques that allow the users to assess the uncertainty in predictions obtained by complex classiﬁers, revealing their limitations.
Revised: To alleviate these issues, it is important to develop techniques that allow the users to assess the uncertainty in predictions obtained by complex classiﬁers, revealing their limitations.

Original: Conformal prediction (Vovk et al., 2005) is a simple yet powerful tool for generating prediction sets whose size reﬂects the prediction uncertainty.
Revised: Conformal prediction (Vovk et al., 2005) is a simple yet powerful tool for generating prediction sets whose size reﬂects the prediction uncertainty.

Original: Speciﬁcally, suppose we are given n training examples { ( X i , Y i ) } ni =1 with feature vector X i ∈ R d , discrete and unordered class label Y i ∈ { 1 , 2 , . . .
Revised: Speciﬁcally, suppose we are given n training examples { ( X i , Y i ) } ni =1 with feature vector X i ∈ R d , discrete and unordered class label Y i ∈ { 1 , 2 , . . .

Original: , L } =
Revised: , L } =

Original: Y , and any learning algorithm that aims at predicting the unknown Y n +1 of a given test point X n +1 .
Revised: Y , and any learning algorithm that aims at predicting the unknown Y n +1 of a given test point X n +1 .

Original: Under the assumption that the training and test examples are sampled i.i.d. from P XY , conformal prediction algorithms construct a distribution-free prediction set C ( X n +1 ) ⊆
Revised: Under the assumption that the training and test examples are sampled exchangeably—e.g., they may be drawn i.i.d.—from an unknown distribution P XY , conformal prediction algorithms construct a distribution-free prediction set C ( X n +1 ) ⊆

Original: Y guaranteed to contain the test label Y n +1 at any desired coverage probability 1 − α ∈ (0 , 1) :
Revised: Y guaranteed to contain the test label Y n +1 at any desired coverage probability 1 − α ∈ (0 , 1) :

Original: [Equation]
Revised: [Equation]

Original: For example, it is common to set the desired coverage level 1 − α to be 90% or 95%.
Revised: For example, it is common to set the desired coverage level 1 − α to be 90% or 95%.

Original: Note that the coverage probability P [ Y n +1 ∈ C ( X n +1 )] is marginal because it is taken over all the training examples { ( X i , Y i ) } ni =1 and the test point X n +1 .
Revised: Note that the coverage probability P [ Y n +1 ∈ C ( X n +1 )] is marginal because it is taken over all the training examples { ( X i , Y i ) } ni =1 and the test point X n +1 .

Original: The key idea of conformal prediction is to ﬁt a classiﬁer on the training set and use this model to assign non-conformity scores for held-out data points.
Revised: The key idea of conformal prediction is to ﬁt a classiﬁer on the training set and use this model to assign non-conformity scores for held-out data points.

Original: These scores reﬂect the prediction error of the underlying classiﬁer, where, loosely speaking, a smaller prediction error would lead to the construction of smaller and more informative sets.
Revised: These scores reﬂect the prediction error of the underlying classiﬁer, where, loosely speaking, a smaller prediction error would lead to the construction of smaller and more informative sets.

Original: However, the sets constructed by the vanilla conformal method may not have the right coverage when the training and test points violate the i.i.d.
Revised: However, the sets constructed by the vanilla conformal method may not have the right coverage when the training and test points violate the exchangeability assumption (Cauchois et al., 2020;

Original: assumption (Cauchois et al., 2020; Gibbs & Cand`es, 2021; Tibshirani et al., 2019; Podkopaev & Ramdas, 2021; Guan & Tibshirani, 2019), which is hardly satisﬁed by real data in practice as distribution shift happens frequently (Koh et al., 2021).
Revised: Gibbs & Cand ` es, 2021; Tibshirani et al., 2019; Podkopaev & Ramdas, 2021; Guan & Tibshirani, 2019), which is hardly satisﬁed by real data in practice as distribution shift happens frequently (Koh et al., 2021).

Original: In particular, we consider the potential threat of adversarial attacks (Goodfellow et al., 2015; Szegedy et al., 2014; Carlini & Wagner, 2017)—carefully crafted human-imperceptible noise perturbations that drives the ﬁtted model to err at test (inference) time.
Revised: In particular, we consider the potential threat of adversarial attacks (Goodfellow et al., 2015; Szegedy et al., 2014; Carlini & Wagner, 2017)—carefully crafted human-imperceptible noise perturbations that drives the ﬁtted model to err at test (inference) time.

Original: Such noise perturbations can introduce a large and arbitrary distribution shift that is extremely hard to estimate.
Revised: Such noise perturbations can introduce a large and arbitrary distribution shift that is extremely hard to estimate.

Original: In this setting, the prediction sets constructed by the vanilla conformal approach are often invalid, i.e., do not satisfy(1), as illustrated in Figure 1.
Revised: In this setting, the prediction sets constructed by the vanilla conformal approach are often invalid, i.e., do not satisfy

Original: Following that ﬁgure, while this method achieves the desired 90% coverage

Original: V a n ill a C P
Revised: V a n ill a C P

Original: C l e a n T e s t
Revised: C l e a n T e s t

Original: V a n ill a C P
Revised: V a n ill a C P

Original: A d v e r s a r i a l
Revised: A d v e r s a r i a l

Original: T e s t
Revised: T e s t

Original: O u r M e t h o d ( R S C P )
Revised: O u r M e t h o d ( R S C P )

Original: A d v e r s a r i a l
Revised: A d v e r s a r i a l

Original: T e s t 0.
Revised: T e s t 0.

Original: M a r g i n a l C o v e r a g e
Revised: M a r g i n a l C o v e r a g e

Original: Nominal Level
Revised: Nominal Level

Original: V a n ill a C P
Revised: V a n ill a C P

Original: C l e a n T e s t V a n ill a C P
Revised: C l e a n T e s t V a n ill a C P

Original: A d v e r s a r i a l
Revised: A d v e r s a r i a l

Original: T e s t
Revised: T e s t

Original: O u r M e t h o d ( R S C P )
Revised: O u r M e t h o d ( R S C P )

Original: A d v e r s a r i a l
Revised: A d v e r s a r i a l

Original: T e s t 0.
Revised: T e s t 0.

Original: A v e r a g e S e t S i z e  when applied to clean test data, the empirical coverage obtained when applying the same method on adversarial test examples falls dramatically below 90%, providing a coverage level of about 30%.
Revised: A v e r a g e S e t S i z e (1), as illustrated in Figure 1. Following that ﬁgure, this method achieves the desired 90% coverage when applied to clean test data, the empirical coverage obtained when applying the same method on adversarial test examples falls dramatically below 90%, providing a coverage level of about 30%.

Original: Therefore, it is the main motivation of this work to construct prediction sets that are robust to adversarial attacks.
Revised: Therefore, it is the main motivation of this work to construct prediction sets that are robust to adversarial attacks.

Original: We formalize this requirement as follows:
Revised: We formalize this requirement as follows:

Original: [Equation]
Revised: [Equation]

Original: where ˜ X n +1 =
Revised: where ˜ X n +1 =

Original: X n +1 + (cid:15) is the test adversarial example, and (cid:107) (cid:15) (cid:107) 2 ≤ δ is a norm-bounded adversarial perturbation.
Revised: X n +1 + (cid:15) is the test adversarial example, and (cid:107) (cid:15) (cid:107) 2 ≤ δ is a norm-bounded adversarial perturbation.

Original: Note that we use the notation C δ to distinguish between the new setting from the i.i.d. case for which δ = 0 .
Revised: Note that we use the notation C δ to distinguish between the new setting from the exchangeability case for which δ = 0 .

Original: Crucially, we require (2) to hold in ﬁnite samples, for any P XY , any adversarial perturbation of magnitude bounded by δ , and regardless of the choice or accuracy of the underlying classiﬁer.
Revised: Crucially, we require (2) to hold in ﬁnite samples, for any P XY , any adversarial perturbation of magnitude bounded by δ , and regardless of the choice or accuracy of the underlying classiﬁer.

Original: At the same time, we wish C δ to be as small as possible.
Revised: At the same time, we wish C δ to be as small as possible.

Original: To realize (2) with the desired coverage, we propose to combine randomized smoothing (Duchi et al., 2012;
Revised: To realize (2) with the desired coverage, we propose to combine randomized smoothing (Duchi et al., 2012;

Original: Cohen et al., 2019;
Revised: Cohen et al., 2019;

Original: Salman et al., 2019) with the vanilla conformal prediction procedure, and hence we name our technique R andomly S moothed C onformal P rediction ( RSCP ).
Revised: Salman et al., 2019) with the vanilla conformal prediction procedure, and hence we name our technique R andomly S moothed C onformal P rediction ( RSCP ).

Original: Randomized smoothing allows us to bound the Lipschitz constant of any non-conformity score function by convolving it with the Gaussian distribution function.
Revised: Randomized smoothing allows us to bound the Lipschitz constant of any non-conformity score function by convolving it with the Gaussian distribution function.

Original: Leveraging this bound, we show how to modify conformal prediction and rigorously construct prediction sets that account for the adversarial perturbation.
Revised: Leveraging this bound, we show how to modify conformal prediction and rigorously construct prediction sets that account for the adversarial perturbation.

Original: Figure 1 illustrates that our proposed RSCP approach successfully attains the desired coverage level whereas the vanilla conformal method fails.
Revised: Figure 1 illustrates that our proposed RSCP approach successfully attains the desired coverage level whereas the vanilla conformal method fails.

Original: Observe that RSCP constructs slightly larger prediction sets, reﬂecting the increased uncertainty induced by the adversarial noise.
Revised: Observe that RSCP constructs slightly larger prediction sets, reﬂecting the increased uncertainty induced by the adversarial noise.

Original: The contributions of this paper are two-fold: (i)
Revised: The contributions of this paper are two-fold: (i)

Original: We propose, for the ﬁrst time, a new conformal prediction method that can account for the potential adversarial threats during inference time.
Revised: We propose, for the ﬁrst time, a new conformal prediction method that can account for the potential adversarial threats during inference time.

Original: Our
Revised: Our

Original: RSCP method, described in Section 3, is model-agnostic in that it can work with any classiﬁer and non-conformity score, scalable since the smoothing can be done by Monte Carlo integration with many i.i.d.
Revised: RSCP method, described in Section 3, is model-agnostic in that it can work with any classiﬁer and non-conformity score, scalable since the smoothing can be done by Monte Carlo integration with many i.i.d.

Original: Gaussian noise realizations, and robust against any (cid:96) 2 -norm bounded adversarial attack.
Revised: Gaussian noise realizations, and robust against any (cid:96) 2 -norm bounded adversarial attack.

Original: (ii)
Revised: (ii)

Original: We prove that the prediction sets constructed by RSCP are valid in the sense of (2), and, in
Revised: We prove that the prediction sets constructed by RSCP are valid in the sense of (2), and, in

Original: Section 5, support this theoretical result with numerical experiments on CIFAR10, CIFAR100, and
Revised: Section 5, support this theoretical result with numerical experiments on CIFAR10, CIFAR100, and

Original: ImageNet data sets.
Revised: ImageNet data sets.

Original: A code that implements our methods is included in the Supplementary Material.
Revised: A code that implements our methods is included in the Supplementary Material.

Original:  C ONFORMAL PREDICTION
Revised:  C ONFORMAL PREDICTION

Original: Since the focus of this paper is on how to adapt the vanilla conformal prediction method to the adversarial setting, in this section, we give background on conformal prediction.
Revised: Since the focus of this paper is on how to adapt the vanilla conformal prediction method to the adversarial setting, in this section, we give background on conformal prediction.

Revised: While we focus here on classiﬁcation problems, the method of conformal prediction can also be applied to regression tasks.

Original: The vanilla conformal prediction can be divided into two categories: the split conformal prediction and the full conformal prediction.
Revised: The vanilla conformal prediction can be divided into two categories: the split conformal prediction and the full conformal prediction.

Original: The ﬁrst one involves data splitting while the second one constructs valid sets without splitting the data but at the cost of a signiﬁcant increase in computational complexity at test time (Papadopoulos et al., 2002; Papadopoulos, 2008; Vovk et al., 2005).
Revised: The ﬁrst one involves data splitting while the second one constructs valid sets without splitting the data but at the cost of a signiﬁcant increase in computational complexity at test time (Papadopoulos et al., 2002; Papadopoulos, 2008; Vovk et al., 2005).

Original: To avoid prohibitive computation complexity, in this paper we focus only on split conformal prediction.
Revised: To avoid prohibitive computation complexity, in this paper we focus only on split conformal prediction.

Original: This method starts by splitting the training data into two disjoint subsets: a proper training set
Revised: This method starts by splitting the training data into two disjoint subsets: a proper training set

Original: I tr ⊆ { 1 , . . . , n } and a calibration set I cal = { 1 , . . . , n } \ I tr .
Revised: I tr ⊆ { 1 , . . . , n } and a calibration set I cal = { 1 , . . . , n } \ I tr .

Original: Then, a classiﬁer ˆ f ( x ) ∈ [0 , 1] L is ﬁt to the proper training set, estimating the conditional class probabilities P
Revised: Then, a classiﬁer ˆ f ( x ) ∈ [0 , 1] L is ﬁt to the proper training set, estimating the conditional class probabilities P

Original: [ Y = y
Revised: [ Y = y

Original: | X = x ] for all y ∈ Y .
Revised: | X = x ] for all y ∈ Y .

Original: In the case of deep net classiﬁers, which is the focus of this work, this may be the output of the softmax layer.
Revised: In the case of deep net classiﬁers, which is the focus of this work, this may be the output of the softmax layer.

Original: Next, we compute a non-conformity score S i =
Revised: Next, we compute a non-conformity score S i =

Original: S ( X i , Y i ) ∈
Revised: S ( X i , Y i ) ∈

Original: R for each calibration point { ( X i , Y i ) } i ∈I cal .
Revised: R for each calibration point { ( X i , Y i ) } i ∈I cal .

Original: This score expresses how well the model prediction
Revised: This score expresses how well the model prediction

Original: ˆ f
Revised: ˆ f

Original: ( X ) is aligned with the true label Y , where a lower score implies better alignment.
Revised: ( X ) is aligned with the true label Y , where a lower score implies better alignment.

Original: For example, the score from Vovk et al.
Revised: For example, the score from Vovk et al.

Original: Lei et al.
Revised: Lei et al.

Original: (2013) is given by
Revised: (2013) is given by

Original: [Equation]
Revised: [Equation]

Original: where ˆ f ( x ) y ∈ [0 , 1] is the y th entry in the vector ˆ f ( x ) .
Revised: where ˆ f ( x ) y ∈ [0 , 1] is the y th entry in the vector ˆ f ( x ) .

Original: Another example is the score proposed by Romano et al.
Revised: Another example is the score proposed by Romano et al.

Original: (2020b), which can be expressed as
Revised: (2020b), which can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where I is the indicator function and u is a random variable distributed uniformly over the segment [0 , 1] .
Revised: where I is the indicator function and u is a random variable distributed uniformly over the segment [0 , 1] .

Original: We refer to the score from (3) as HPS as it was shown to construct homogeneous prediction sets .
Revised: We refer to the score from (3) as HPS as it was shown to construct homogeneous prediction sets .

Original: Analogously, we refer to (4) as APS since it tends to yield adaptive prediction sets that reﬂect better the underlying uncertainty across sub-populations; see Romano et al.
Revised: Analogously, we refer to (4) as APS since it tends to yield adaptive prediction sets that reﬂect better the underlying uncertainty across sub-populations; see Romano et al.

Original: (2020b)
Revised: (2020b)

Original: for more details.
Revised: for more details.

Original: Given the desired coverage level 1 − α , the prediction set for a new test point X n +1 is formulated as
Revised: Given the desired coverage level 1 − α , the prediction set for a new test point X n +1 is formulated as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: th empirical quantileof { S
Revised: quantile

Original: [Equation]
Revised: [Equation]

Original: is the score positioned (cid:100) ( n + 1) (1 − α ) (cid:101) in the sorted array of calibration scores S i , i ∈
Revised: is the score positioned (cid:100) ( n + 1) (1 − α ) (cid:101) in the sorted array of calibration scores S i , i ∈

Original: I cal .
Revised: I cal .

Original: In plain words, in (5) we sweep over all possible labels y ∈ Y and include in C ( X n +1 ) the ‘guessed’ labels y whose scores S ( X n +1 , y ) are smaller than most of the calibration scores S ( X i , Y i ) .
Revised: In plain words, in (5) we sweep over all possible labels y ∈ Y and include in C ( X n +1 ) the ‘guessed’ labels y whose scores S ( X n +1 , y ) are smaller than most of the calibration scores S ( X i , Y i ) .

Original: Since the calibration and test points are drawn i.i.d. from P XY and ˆ f is ﬁxed, the score S ( X n +1 , y ) for the guess y
Revised: Since the calibration and test points are drawn exchangeably from P XY and ˆ f is ﬁxed, the score S ( X n +1 , y ) for the guess y

Original: = Y n +1 can fall anywhere in the sorted array of calibration scores with equal probability.
Revised: = Y n +1 can fall anywhere in the sorted array of calibration scores with equal probability.

Original: This property guarantees that the prediction set (5) satisﬁes (1); see Vovk et al.
Revised: This property guarantees that the prediction set (5) satisﬁes (1); see Vovk et al.

Original:  R ANDOMLY SMOOTHED CONFORMAL PREDICTION
Revised:  R ANDOMLY SMOOTHED CONFORMAL PREDICTION

Original: In this section, we introduce our proposed RSCP framework for constructing prediction sets that are valid in the adversarial regime.
Revised: In this section, we introduce our proposed RSCP framework for constructing prediction sets that are valid in the adversarial regime.

Original: Recall that an adversarial attack can lead to a signiﬁcant distributional shift between the clean calibration points and the corrupted test example ˜ X n +1 =
Revised: Recall that an adversarial attack can lead to a signiﬁcant distributional shift between the clean calibration points and the corrupted test example ˜ X n +1 =

Original: X n +1 + (cid:15) , thus violating the fundamental i.i.d. assumption of the split conformal procedure.
Revised: X n +1 + (cid:15) , thus violating the fundamental exchangeability assumption of the split conformal procedure.

Original: Focusing on the guess y
Revised: Focusing on the guess y

Original: = Y n +1 , an effective attack would result in a larger non-conformity score for the corrupted test point S ( ˜ X n +1 , y ) compared to that of the clean input S ( X n +1 , y ) .
Revised: = Y n +1 , an effective attack would result in a larger non-conformity score for the corrupted test point S ( ˜ X n +1 , y ) compared to that of the clean input S ( X n +1 , y ) .

Original: Therefore, a naive comparison of S ( ˜ X n +1 , y ) to the same threshold Q 1 − α from (6), which neglects the increased uncertainty caused by the adversarial perturbation, will result in a prediction set that may not achieve the desired coverage, as already illustrated in Figure 1.
Revised: Therefore, a naive comparison of S ( ˜ X n +1 , y ) to the same threshold Q 1 − α from (6), which neglects the increased uncertainty caused by the adversarial perturbation, will result in a prediction set that may not achieve the desired coverage, as already illustrated in Figure 1.

Original: To address this, we should compare the test score to an inﬂated threshold, larger than Q 1 − α , which rigorously accounts for the effect of the adversarial noise.
Revised: To address this, we should compare the test score to an inﬂated threshold, larger than Q 1 − α , which rigorously accounts for the effect of the adversarial noise.

Original: This is the core idea behind our proposal described in detail below; seealso Figure 2.
Revised: This is the core idea behind our proposal described in detail below; see Figure 2.

Original: 3. A DVERSARIALLY ROBUST CALIBRATION
Revised: 3. A DVERSARIALLY ROBUST CALIBRATION

Original: Suppose we are given a non-conformity score function ˜ S for which we can bound by how much its value could be increased due to the adversarial noise (cid:107) (cid:15) (cid:107) 2 ≤ δ added to X n +1 .
Revised: Suppose we are given a non-conformity score function ˜ S for which we can bound by how much its value could be increased due to the adversarial noise (cid:107) (cid:15) (cid:107) 2 ≤ δ added to X n +1 .

Original: Formally, we require the score ˜ S to satisfy the following relation:
Revised: Formally, we require the score ˜ S to satisfy the following relation:

Original: [Equation]
Revised: [Equation]

Original: where M δ ≥ 0 is a constant that is a function of δ , such that M δ 1 ≥ M δ 2 for δ 1 ≥ δ 2 and M δ =for δ = 0 .
Revised: where M δ ≥ 0 is a constant that is a function of δ , such that M δ 1 ≥ M δ 2 for δ 1 ≥ δ 2 and M δ =for δ = 0 .

Original: In essence, we would like (7) to hold for the smallest possible M δ .
Revised: In essence, we would like (7) to hold for the smallest possible M δ .

Original: We denote this score function by ˜ S to emphasize that it must satisfy (7), distinguishing it from existing non-conformity scores S , e.g., (3)–(4); see Section 3.2 for a concrete and very general framework for designing ˜ S for which the constant M δ can be easily derived.
Revised: We denote this score function by ˜ S to emphasize that it must satisfy (7), distinguishing it from existing non-conformity scores S , e.g., (3)–(4); see Section 3.2 for a concrete and very general framework for designing ˜ S for which the constant M δ can be easily derived.

Original: Importantly, M δ serves as a bridge between the observed score ˜ S ( ˜ X n +1 , y ) and the unobserved one ˜ S ( X n +1 , y ) for a ﬁxed y ∈ Y .
Revised: Importantly, M δ serves as a bridge between the observed score ˜ S ( ˜ X n +1 , y ) and the unobserved one ˜ S ( X n +1 , y ) for a ﬁxed y ∈ Y .

Original: Leveraging this property, we propose to construct a prediction set robust to a norm-bounded adversarial attack by applying the following decision rule:
Revised: Leveraging this property, we propose to construct a prediction set robust to a norm-bounded adversarial attack by applying the following decision rule:

Original: [Equation]
Revised: [Equation]

Original:  where ˜ S i =
Revised: 8) where ˜ S i =

Original: ˜ S
Revised: ˜ S

Original: ( X i , Y i ) .
Revised: ( X i , Y i ) .

Original: In contrast to the vanilla split conformal approach (5), the prediction set deﬁned above is generated by comparing the test score to an inﬂated threshold Q 1 − α + M δ , as illustrated in Figure 2.
Revised: In contrast to the vanilla split conformal approach (5), the prediction set deﬁned above is generated by comparing the test score to an inﬂated threshold Q 1 − α + M δ , as illustrated in Figure 2.

Original: Notice that the level of inﬂation is affected by the magnitude of the adversarial perturbation as well as the robustness of ˜ S , i.e., the value of M δ .
Revised: Notice that the level of inﬂation is affected by the magnitude of the adversarial perturbation as well as the robustness of ˜ S , i.e., the value of M δ .

Original: A larger perturbation implies larger inﬂation, and a more robust score ˜ S implies smaller inﬂation.
Revised: A larger perturbation implies larger inﬂation, and a more robust score ˜ S implies smaller inﬂation.

Original: The theorem below states that the constructed prediction set (8) is guaranteed to contain the unknown target label Y n +1 with a probability of at least 1 − α , for any distribution P XY , sample size n , score function ˜ S that satisﬁes (7), and adversarial perturbation of magnitude δ generated by any attack algorithm.
Revised: The theorem below states that the constructed prediction set (8) is guaranteed to contain the unknown target label Y n +1 with a probability of at least 1 − α , for any distribution P XY , sample size n , score function ˜ S that satisﬁes (7), and adversarial perturbation of magnitude δ generated by any attack algorithm.

Original: The proof of this and all other results can be found in Section S1 of the Supplementary Material.
Revised: The proof of this and all other results can be found in Section S1 of the Supplementary Material.

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume that the samples { ( X i , Y i ) } n +1 i =1 are drawn i.i.d. from some unknown distribution P XY .
Revised: Assume that the samples { ( X i , Y i ) } n +1 i =1 are drawn exchangeably from some unknown distribution P XY .

Original: Let ˜ X n +1 =
Revised: Let ˜ X n +1 =

Original: X n +1 + (cid:15) be a clean test example
Revised: X n +1 + (cid:15) be a clean test example

Original: X n +1 with an additive corruption of an (cid:96) 2 -norm bounded adversarial noise (cid:107) (cid:15) (cid:107) 2 ≤ δ .
Revised: X n +1 with an additive corruption of an (cid:96) 2 -norm bounded adversarial noise (cid:107) (cid:15) (cid:107) 2 ≤ δ .

Original: Then, the prediction set C δ ( ˜ X n +1 ) deﬁned in (8) satisﬁes
Revised: Then, the prediction set C δ ( ˜ X n +1 ) deﬁned in (8) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Before presenting our framework to construct scores that rigorously satisfy (7), we pause to prove a lower bound on the coverage that the vanilla split conformal could attain in the adversarial setting.
Revised: Before presenting our framework to construct scores that rigorously satisfy (7), we pause to prove a lower bound on the coverage that the vanilla split conformal could attain in the adversarial setting.

Original: Theorem 2.
Revised: Theorem 2.

Original: Under the assumptions of Theorem 1, the prediction set C ( ˜ X n +1 ) deﬁned in (5) , applied with ˜ S in place of S , satisﬁes
Revised: Under the assumptions of Theorem 1, the prediction set C ( ˜ X n +1 ) deﬁned in (5) , applied with ˜ S in place of S , satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that τ can be simply computed by running a grid search on the sorted array of calibration scores.
Revised: Note that τ can be simply computed by running a grid search on the sorted array of calibration scores.

Original: It is important to observe that in contrast to Theorem 1, which guarantees at least 1 − α coverage for any user-speciﬁed level α , the worst coverage level τ of the vanilla split conformal is not controlled explicitly, i.e., τ is known only after looking at the training and calibration data.
Revised: It is important to observe that in contrast to Theorem 1, which guarantees at least 1 − α coverage for any user-speciﬁed level α , the worst coverage level τ of the vanilla split conformal is not controlled explicitly, i.e., τ is known only after looking at the training and calibration data.

Original: Observe also that, by construction, τ ≤ 1 − α
Revised: Observe also that, by construction, τ ≤ 1 − α

Original: (see Supplementary Section S1) and equality is met in a special case where δ = 0 , i.e., no attack is performed.
Revised: (see Supplementary Section S1) and equality is met in a special case where δ = 0 , i.e., no attack is performed.

Original: In this special case, M δ = 0 by deﬁnition, and both Theorem 1 and 2 converge to the classic coverage guarantee of the vanillasplit conformal algorithm.
Revised: In this special case, M δ = 0 by deﬁnition, and both Theorem 1 and 2 converge to the classic coverage guarantee of the vanilla conformal algorithm.

Original: 3. T HE RANDOMLY SMOOTHED SCORE
Revised: 3. T HE RANDOMLY SMOOTHED SCORE

Original: To apply the robust calibration procedure presented above, we need to have access to a constant M δ that satisﬁes (7) for a given non-conformity score function.
Revised: To apply the robust calibration procedure presented above, we need to have access to a constant M δ that satisﬁes (7) for a given non-conformity score function.

Original: In general, this constant is unknown for modern deep neural net classiﬁers as well as for complex non-conformity scores, such as the one deﬁned in (4).
Revised: In general, this constant is unknown for modern deep neural net classiﬁers as well as for complex non-conformity scores, such as the one deﬁned in (4).

Original: In what follows, we present a novel framework that takes inspiration from the literature on randomized smoothing (Duchi et al., 2012;
Revised: In what follows, we present a novel framework that takes inspiration from the literature on randomized smoothing (Duchi et al., 2012;

Original: Cohen et al., 2019;
Revised: Cohen et al., 2019;

Original: Salman et al., 2019), offering a theoretically grounded mechanism to bound the Lipschitz constant of any scoring function derived from any black-box classiﬁer.
Revised: Salman et al., 2019), offering a theoretically grounded mechanism to bound the Lipschitz constant of any scoring function derived from any black-box classiﬁer.

Original: Concretely, we suggest constructing a new score ˜ S from the base score S as follows:
Revised: Concretely, we suggest constructing a new score ˜ S from the base score S as follows:

Original: [Equation]
Revised: [Equation]

Original:  where ˜ S is a “smoothed” version of the base score S , obtained by averaging the value of S ( x + v, y ) over many independent samples v ∼ N (cid:0) 

Revised: (10) where ˜ S is a “smoothed” version of the base score S , obtained by averaging the value of S ( x + v, y ) over many independent samples v ∼ N (cid:0)

Original: d , σ 2 I d (cid:1) drawn from the multivariate normal distribution, and then applying Φ − 1 , the inverse of the cumulative distribution function (CDF) of the standard normal distribution.
Revised: d , σ 2 I d (cid:1) drawn from the multivariate normal distribution, and then applying Φ − 1 , the inverse of the cumulative distribution function (CDF) of the standard normal distribution.

Original: The hyper-parameter σ controls the level of smoothing, where a larger σ implies stronger smoothing.
Revised: The hyper-parameter σ controls the level of smoothing, where a larger σ implies stronger smoothing.

Original: (In Section 5.1, Figure 3, and Supplementary Section S5.1 we discuss the effect of this parameter on the statistical efﬁciency of the overall calibration procedure.)
Revised: (In Section 5.1, Figure 3, and Supplementary Section S5.1 we discuss the effect of this parameter on the statistical efﬁciency of the overall calibration procedure.)

Original: Assuming that the base score function S ( x, y ) ∈ [0 , 1] for any given ( x, y ) , which holds for most non-conformity scores in multi-class classiﬁcation problems, we can invoke a well-known result from the randomized smoothing literature (Salman et al., 2019, Lemma 2) and get the following upper-bound for the smoothed score:
Revised: Assuming that the base score function S ( x, y ) ∈ [0 , 1] for any given ( x, y ) , which holds for most non-conformity scores in multi-class classiﬁcation problems, we can invoke a well-known result from the randomized smoothing literature (Salman et al., 2019, Lemma 2) and get the following upper-bound for the smoothed score:

Original: [Equation]
Revised: [Equation]

Original: Importantly, (11) holds for every y ∈ Y , hence rigorously satisfying the relation (7) for the choice of M δ = δ/σ .
Revised: Importantly, (11) holds for every y ∈ Y , hence rigorously satisfying the relation (7) for the choice of M δ = δ/σ .

Original: Armed with ˜ S , we can invoke Theorems 1- 2 and construct an adversarially robust prediction set by following (8) or lower bound the coverage of the vanilla split conformal procedure.
Revised: Armed with ˜ S , we can invoke Theorems 1- 2 and construct an adversarially robust prediction set by following (8) or lower bound the coverage of the vanilla split conformal procedure.

Original: Corollary 1.
Revised: Corollary 1.

Original: Let ˜ S be the randomly smoothed score function (10) and set M δ
Revised: Let ˜ S be the randomly smoothed score function (10) and set M δ

Original: = δ/σ .
Revised: = δ/σ .

Original: The prediction set deﬁned in (8) satisﬁes 
Revised: The prediction set deﬁned in (8) satisﬁes P (cid:104) Y n

Revised: +1 ∈ C δ (cid:0) ˜

Revised: X n +1 (cid:1)(cid:105) ≥ 1 − α , and the worst coverage of (5) is given by P (cid:104) Y n +1 ∈ C (cid:0)

Revised: ˜

Revised: X n +1 (cid:1)(cid:105) ≥ τ, where τ = max (cid:110) τ (cid:48) : Q τ (cid:48) (cid:0)(cid:8)

Revised: ˜ S

Original: [Equation]
Revised: [Equation]

Original: and the worst coverage of (5) is given by
Revised: (cid:9) i ∈I cal (cid:1) ≤

Original: [Equation]
Revised: [Equation]

Original: with
Revised: 1 − α (cid:0)(cid:8) ˜ S

Original: [Equation]
Revised: [Equation]

Revised: (cid:9) i ∈I cal (cid:1) − δσ (cid:111) .

Original: The above leads to two possible use-cases of our proposal.
Revised: The above leads to two possible use-cases of our proposal.

Original: In the ﬁrst, we can implement our randomly smoothed conformal prediction procedure ( RSCP )
Revised: In the ﬁrst, we can implement our randomly smoothed conformal prediction procedure ( RSCP )

Original: (8) and construct valid prediction sets at any desired level, bearing in mind that these sets are likely to be larger due to the increased uncertainty induced by the adversarial perturbation.
Revised: (8) and construct valid prediction sets at any desired level, bearing in mind that these sets are likely to be larger due to the increased uncertainty induced by the adversarial perturbation.

Original: In the second case, we can deploy the vanilla split conformal algorithm with a smoothed version of the base score, providing important information about the worst coverage that might be obtained under an adversarial attack.
Revised: In the second case, we can deploy the vanilla split conformal algorithm with a smoothed version of the base score, providing important information about the worst coverage that might be obtained under an adversarial attack.

Original: In practice, the smoothed score cannot be evaluated explicitly as it requires to compute the expectation in (10).
Revised: In practice, the smoothed score cannot be evaluated explicitly as it requires to compute the expectation in (10).

Original: However, this smoothed score can be easily estimated by averaging over many i.i.d.
Revised: However, this smoothed score can be easily estimated by averaging over many i.i.d.

Original: Gaussian noise realizations in a Monte Carlo fashion (Cohen et al., 2019, Section 3.2), as described in Algorithm 1; see Section 5.1 and Supplementary Section S5.2 for more details about the effect of this approximation on the coverage and set-size.
Revised: Gaussian noise realizations in a Monte Carlo fashion (Cohen et al., 2019, Section 3.2), as described in Algorithm 1; see Section 5.1 and Supplementary Section S5.2 for more details about the effect of this approximation on the coverage and set-size.

Revised: Here, an interesting future direction is to develop a high probability bound for the smoothed score approximation error and integrate it with our method.

Original:  R ELATED W ORK 4. C ONFORMAL PREDICTION UNDER DISTRIBUTION SHIFT
Revised:  R ELATED W ORK 4. C ONFORMAL PREDICTION UNDER DISTRIBUTION SHIFT

Original: Previous work offers several generalizations of the vanilla conformal prediction framework to the case of distribution shift between the training and the test points, however, none of them addresses
Revised: Previous work offers several generalizations of the vanilla conformal prediction framework to the case of distribution shift between the training and the test points, however, none of them addresses

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: the adversarial setting speciﬁcally.
Revised: the adversarial setting speciﬁcally.

Original: Cauchois et al.
Revised: Cauchois et al.

Original: (2020) suggested a method to construct valid sets when the test distribution is in an f -divergence ball around the training distribution.
Revised: (2020) suggested a method to construct valid sets when the test distribution is in an f -divergence ball around the training distribution.

Original: While this approach shares the idea of inﬂating the value of Q 1 − α , the f -divergence measure is notoriously difﬁcult to estimate in practice (Nguyen et al., 2010; Tsybakov, 2009) and thus less applicable compared to our approach, which focuses on the addition of bounded (cid:96) 2 adversarial noise.
Revised: While this approach shares the idea of inﬂating the value of Q 1 − α , the f -divergence measure is notoriously difﬁcult to estimate in practice (Nguyen et al., 2010; Tsybakov, 2009) and thus less applicable compared to our approach, which focuses on the addition of bounded (cid:96) 2 adversarial noise.

Original: Gibbs &
Revised: Gibbs &

Original: Cand`es (2021) generalized the vanilla conformal prediction approach to an online setting where the data generating distribution varies over time by modifying the threshold adaptively as new labeled points are being observed, such that the long-term empirical coverage frequency would be at least 1 − α .
Revised: Cand ` es (2021) generalized the vanilla conformal prediction approach to an online setting where the data generating distribution varies over time by modifying the threshold adaptively as new labeled points are being observed, such that the long-term empirical coverage frequency would be at least 1 − α .

Original: This is of course different from our approach that works ofﬂine and assumes the same data generating process but with the addition of a malicious small perturbation induced by an attacker.
Revised: This is of course different from our approach that works ofﬂine and assumes the same data generating process but with the addition of a malicious small perturbation induced by an attacker.

Original: Tibshirani et al.
Revised: Tibshirani et al.

Original: (2019) studied the case of covariate shift and offered a weighted version of conformal prediction to address it.
Revised: (2019) studied the case of covariate shift and offered a weighted version of conformal prediction to address it.

Original: In this setting, the shift is only in P X while P Y | X remains intact.
Revised: In this setting, the shift is only in P X while P Y | X remains intact.

Original: Podkopaev & Ramdas (2021) modiﬁed this method to tackle the label-shift case,
Revised: Podkopaev & Ramdas (2021) modiﬁed this method to tackle the label-shift case,

Original: in which P X | Y remains the same but P Y varies between train and test time.
Revised: in which P X | Y remains the same but P Y varies between train and test time.

Original: Both approaches differ from ours, which handles a full distributional shift induced by the adversarial perturbation.
Revised: Both approaches differ from ours, which handles a full distributional shift induced by the adversarial perturbation.

Original: Lastly, Hechtlinger et al.
Revised: Lastly, Hechtlinger et al.

Original: (2018) suggested a non-conformity score that empirically tends to be more robust to adversarial attacks, replacing the classiﬁer’s class probability estimates of Y | X with a density estimator that approximates the conditional distribution of X | Y .
Revised: (2018) suggested a non-conformity score that empirically tends to be more robust to adversarial attacks, replacing the classiﬁer’s class probability estimates of Y | X with a density estimator that approximates the conditional distribution of X | Y .

Original: In contrast to our RSCP approach, this method is not supported by a theoretical guarantee for attaining the desired coverage in the adversarial setting.
Revised: In contrast to our RSCP approach, this method is not supported by a theoretical guarantee for attaining the desired coverage in the adversarial setting.

Original: 4. C ERTIFIED ADVERSARIAL ROBUSTNESS
Revised: 4. C ERTIFIED ADVERSARIAL ROBUSTNESS

Original: A different but related line of research is known as ‘robustness certiﬁcation’.
Revised: A different but related line of research is known as ‘robustness certiﬁcation’.

Original: Here, the goal is to provide an (cid:96) p radius around a test point, in which no adversarial perturbations exist, i.e, the predicted label provably remains the same inside this radius.
Revised: Here, the goal is to provide an (cid:96) p radius around a test point, in which no adversarial perturbations exist, i.e, the predicted label provably remains the same inside this radius.

Original: Currently, one of the mainstream approaches in robustness certiﬁcation is through randomized smoothing, which was proposed in (Lecuyer et al., 2019;
Revised: Currently, one of the mainstream approaches in robustness certiﬁcation is through randomized smoothing, which was proposed in (Lecuyer et al., 2019;

Original: Li et al., 2019;
Revised: Li et al., 2019;

Original: Cohen et al., 2019) to provide an analytic form of robustness certiﬁcate.
Revised: Cohen et al., 2019) to provide an analytic form of robustness certiﬁcate.

Original: More formally, given a base classiﬁer ˆ f ( x ) , one can construct a smoothed classiﬁer according to the following decision rule:
Revised: More formally, given a base classiﬁer ˆ f ( x ) , one can construct a smoothed classiﬁer according to the following decision rule:

Original: [Equation]
Revised: [Equation]

Original: In contrast to the base classiﬁer, the smoothed classiﬁer can be supported by a rigorous (cid:96) 2 certiﬁcation radius around the test point X n +

Revised: In contrast to the base classiﬁer, the smoothed classiﬁer can be supported by a rigorous (cid:96) 2 certiﬁcation radius around the test point X n +

Original: = x .
Revised: = x .

Original: It is self-evident that our work is different from robustness certiﬁcation, as we are not aiming to deliver a robustness certiﬁcate for a given classiﬁer over some test point; instead, we construct a valid prediction set whose size can vary with x .
Revised: It is self-evident that our work is different from robustness certiﬁcation, as we are not aiming to deliver a robustness certiﬁcate for a given classiﬁer over some test point; instead, we construct a valid prediction set whose size can vary with x .

Original: However, both works make use of the randomized smoothing technique, sharinga related set of ideas.
Revised: However, both works make use of the randomized smoothing technique, sharing related set of ideas.

Original: For example, practical evidence shows that for the smoothed model (13) to correctly classify the test point X n +1 , the base classiﬁer ˆ f needs to consistently label the noisy sample N (cid:0) X n +1 , σ 2 I (cid:1) as Y n +

Revised: For example, practical evidence shows that for the smoothed model (13) to correctly classify the test point X n +1 , the base classiﬁer ˆ f needs to consistently label the noisy sample N (cid:0) X n +1 , σ 2 I (cid:1) as Y n +

Original: (Cohen et al., 2019, Section 3.3).
Revised: (Cohen et al., 2019, Section 3.3).

Original: Training the base classiﬁer without accounting for the future smoothing operation would not necessarily satisfy this requirement, and, as a consequence, would lead to a very small certiﬁcation radius.
Revised: Training the base classiﬁer without accounting for the future smoothing operation would not necessarily satisfy this requirement, and, as a consequence, would lead to a very small certiﬁcation radius.

Original: We note that this observation is also relevant to our setting, where we found that using a classiﬁer that is not robust to Gaussian perturbations results in prediction sets that are often large and thus uninformative.
Revised: We note that this observation is also relevant to our setting, where we found that using a classiﬁer that is not robust to Gaussian perturbations results in prediction sets that are often large and thus uninformative.

Original: More details are provided in Section S4 of
Revised: More details are provided in Section S4 of

Original: Supplementary Material.
Revised: Supplementary Material.

Original: One way to improve the certiﬁed radius is by training the base classiﬁer on the labeled training pairs { ( N (cid:0) X i , σ 2 I (cid:1) , Y i ) } i ,
Revised: One way to improve the certiﬁed radius is by training the base classiﬁer on the labeled training pairs { ( N (cid:0) X i , σ 2 I (cid:1) , Y i ) } i ,

Original: i ∈
Revised: i ∈

Original: I tr as in Lecuyer et al.
Revised: I tr as in Lecuyer et al.

Original: (2019), where σ is equal to the one used for smoothing.
Revised: (2019), where σ is equal to the one used for smoothing.

Original: Later, Salman et al.
Revised: Later, Salman et al.

Original: (2019) further improved the robustness guarantee through adversarial training, where the adversary generates a bounded perturbation via the method of SMOOTHADV —an adversarial attack designed speciﬁcally against smoothed classiﬁers.
Revised: (2019) further improved the robustness guarantee through adversarial training, where the adversary generates a bounded perturbation via the method of SMOOTHADV —an adversarial attack designed speciﬁcally against smoothed classiﬁers.

Original: A more recent work by Salman et al.
Revised: A more recent work by Salman et al.

Original: (2020) suggested adding a denoising pre-processing layer preceding the classiﬁer, to remove the Gaussian noise used in randomized smoothing.
Revised: (2020) suggested adding a denoising pre-processing layer preceding the classiﬁer, to remove the Gaussian noise used in randomized smoothing.

Original: In our experiments, we ﬁnd that the training strategy proposed in Lecuyer et al.
Revised: In our experiments, we ﬁnd that the training strategy proposed in Lecuyer et al.

Original: (2019) works well in practice and results in relatively small prediction sets, although a better training scheme—such as the one from Salman et al.
Revised: (2019) works well in practice and results in relatively small prediction sets, although a better training scheme—such as the one from Salman et al.

Original: (2019) or Salman et al.
Revised: (2019) or Salman et al.

Original: (2020)—could be easily combined with our method and may further improve the overall statistical efﬁciency.
Revised: (2020)—could be easily combined with our method and may further improve the overall statistical efﬁciency.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate the performance of our proposed methods on three benchmark image classiﬁcation data sets: CIFAR10, CIFAR100 (Krizhevsky, 2009), and ImageNet
Revised: In this section, we evaluate the performance of our proposed methods on three benchmark image classiﬁcation data sets: CIFAR10, CIFAR100 (Krizhevsky, 2009), and ImageNet

Original: ILSVRC2012 (Deng et al., 2009), which are described in Supplementary Section S2.
Revised: ILSVRC2012 (Deng et al., 2009), which are described in Supplementary Section S2.

Original: We evaluate our methods as follows.
Revised: We evaluate our methods as follows.

Original: First, we ﬁt a model on the entire training set.
Revised: First, we ﬁt a model on the entire training set.

Original: Then, we split the remaining data into two equally sized disjoint subsets, one is used for calibration and the second for testing.
Revised: Then, we split the remaining data into two equally sized disjoint subsets, one is used for calibration and the second for testing.

Original: The calibration is done using the two base non-conformity scores mentioned in Section 2, i.e., HPS (3) and APS (4).
Revised: The calibration is done using the two base non-conformity scores mentioned in Section 2, i.e., HPS (3) and APS (4).

Original: Next, we attack each point in the test set by adding to it an adversarial perturbation whose (cid:96) 2 -norm is bounded by δ .
Revised: Next, we attack each point in the test set by adding to it an adversarial perturbation whose (cid:96) 2 -norm is bounded by δ .

Original: Lastly, for each adversarial test point, we construct a prediction set using three different conformal methods (described below), asking for a nominal coverage level of 1 − α .
Revised: Lastly, for each adversarial test point, we construct a prediction set using three different conformal methods (described below), asking for a nominal coverage level of 1 − α .

Original: Given the constructed sets, we test the validity of each method by reporting the marginal coverage rate, and the statistical efﬁciency by computing the average size of the prediction sets.
Revised: Given the constructed sets, we test the validity of each method by reporting the marginal coverage rate, and the statistical efﬁciency by computing the average size of the prediction sets.

Original: We repeat this process several times (50 for CIFAR10 and CIFAR100, and 20 for ImageNet) by randomly assigning data points to form the calibration and test sets.
Revised: We repeat this process several times (50 for CIFAR10 and CIFAR100, and 20 for ImageNet) by randomly assigning data points to form the calibration and test sets.

Original: In our experiments, we compare three different approaches: (i) vanilla split conformal prediction implemented with the two base-scores (HPS and APS), denoted by Vanilla CP ; (ii) vanilla split conformal prediction with our proposed smoothed version of the base scores, denoted by CP+SS ; and (iii) the proposed randomly smoothed conformal prediction (see Algorithm 1), denoted by
Revised: In our experiments, we compare three different approaches: (i) vanilla split conformal prediction implemented with the two base-scores (HPS and APS), denoted by Vanilla CP ; (ii) vanilla split conformal prediction with our proposed smoothed version of the base scores, denoted by CP+SS ; and (iii) the proposed randomly smoothed conformal prediction (see Algorithm 1), denoted by

Original: RSCP .
Revised: RSCP .

Original: Our experiments show that Vanilla CP constructs invalid prediction sets whose marginal coverage is way below the desired level.
Revised: Our experiments show that Vanilla CP constructs invalid prediction sets whose marginal coverage is way below the desired level.

Original: The smoothed score improves robustness and coverage but still does not achieve the nominal coverage rate.
Revised: The smoothed score improves robustness and coverage but still does not achieve the nominal coverage rate.

Original: The prediction sets constructed by our RSCP approach achieve the desired coverage level, aligning with our theoretical guarantees.
Revised: The prediction sets constructed by our RSCP approach achieve the desired coverage level, aligning with our theoretical guarantees.

Original: 5. I MPLEMENTATION DETAILS
Revised: 5. I MPLEMENTATION DETAILS

Original: Models For all data sets, we choose ResNet (He et al., 2016) to be the base architecture of the deep net classiﬁer and ﬁt two different models for each data set: one on clean training points and the second on points that are augmented with Gaussian noise of standard deviation σ that is equal to the smoothing parameter from (10).
Revised: Models For all data sets, we choose ResNet (He et al., 2016) to be the base architecture of the deep net classiﬁer and ﬁt two different models for each data set: one on clean training points and the second on points that are augmented with Gaussian noise of standard deviation σ that is equal to the smoothing parameter from (10).

Original: The former is used to explore the effect of adversarial examples on the validity of Vanilla CP .
Revised: The former is used to explore the effect of adversarial examples on the validity of Vanilla CP .

Original: The latter training strategy improves the robustness of the classiﬁer and, as a consequence, reduces the size of the prediction sets constructed by CP+SS and RSCP methods, as discussed in Section 4.2 and Supplementary Section S4.
Revised: The latter training strategy improves the robustness of the classiﬁer and, as a consequence, reduces the size of the prediction sets constructed by CP+SS and RSCP methods, as discussed in Section 4.2 and Supplementary Section S4.

Original: For CIFAR10 and ImageNet data sets, we use the pre-trained ResNet-110 and ResNet-50 models from Cohen et al.
Revised: For CIFAR10 and ImageNet data sets, we use the pre-trained ResNet-110 and ResNet-50 models from Cohen et al.

Original: (2019), and for CIFAR100 we ﬁt our own ResNet-110 model as the latter data set is not studied by Cohen et al.
Revised: (2019), and for CIFAR100 we ﬁt our own ResNet-110 model as the latter data set is not studied by Cohen et al.

Original: (2019), please see Supplementary Section S3 for additional details on the training procedure.
Revised: (2019), please see Supplementary Section S3 for additional details on the training procedure.

Original: Seealso Supplementary Section S8 for similar experiments in which we choose DenseNet (Iandola et al., 2014) and VGG (Simonyan & Zisserman, 2014) to be the base architectures. [Figure] [Figure]
Revised: See

Original: [Figure]
Revised: [Figure]

Original: A v e r a g

Original: [Figure]
Revised: [Figure]

Original:  e t S i z e
Revised: A v e r a g e S e t S i z e

Original: [Figure]
Revised: [Figure]

Original: The results correspond to the

Original: CIFAR100 data set, with δ = 0 .

Original: 125 and n s = 256 .

Original: 1 2 4 8 16 32 64 128 256 512 n s 0.
Revised: 1 2 4 8 16 32 64 128 256 512 n s 0.

Original: M a r g i n a l C o v e r a g e
Revised: M a r g i n a l C o v e r a g e

Original: Base Score APS
Revised: Base Score APS

Original: HPS 1 2 4 8 16 32 64 128 256 512 n s 2.
Revised: HPS 1 2 4 8 16 32 64 128 256 512 n s 2.

Original: A v e r a g e S e t S i z e The results correspond to the CIFAR10 data set, where we set δ and σ to be equal to 0 . 125 and
Revised: A v e r a g e S e t S i z e also Supplementary Section S8 for similar experiments in which we choose DenseNet (Iandola et al., 2014) and VGG (Simonyan & Zisserman, 2014) to be the base architectures.

Original: 25 , respectively. Attack algorithm For Vanilla CP , we generate adversarial noise using the projected gradient descent (PGD) attack (Madry et al., 2018), which is one of the strongest white-box attacks known in the literature to construct (cid:96) 2 bounded adversarial examples.
Revised: Attack algorithm For Vanilla CP , we generate adversarial noise using the projected gradient descent (PGD) attack (Madry et al., 2018), which is one of the strongest white-box attacks known in the literature to construct (cid:96) 2 bounded adversarial examples.

Original: For the other two methods that make use of randomized smoothing ( CP+SS and RSCP ), we apply SMOOTHADV (Salman et al., 2019)—a variation of PGD that is tailored for smoothed classiﬁers.
Revised: For the other two methods that make use of randomized smoothing ( CP+SS and RSCP ), we apply SMOOTHADV (Salman et al., 2019)—a variation of PGD that is tailored for smoothed classiﬁers.

Original: For CIFAR10 and CIFAR100, we attack the test points with an adversarial perturbation of energy δ = 0 .
Revised: For CIFAR10 and CIFAR100, we attack the test points with an adversarial perturbation of energy δ = 0 .

Original: 125 , and for ImageNet we use a larger perturbation of magnitude δ = 0 .
Revised: 125 , and for ImageNet we use a larger perturbation of magnitude δ = 0 .

Original: Both choices lead to a signiﬁcant reduction in the test accuracy of the base classiﬁer, as summarized in Supplementary Section S6.
Revised: Both choices lead to a signiﬁcant reduction in the test accuracy of the base classiﬁer, as summarized in Supplementary Section S6.

Original: Choosing σ for smoothing Recall Corollary 1 and observe that the coverage guarantee holds for any choice of the smoothing level σ .
Revised: Choosing σ for smoothing Recall Corollary 1 and observe that the coverage guarantee holds for any choice of the smoothing level σ .

Original: This parameter is intimately connected to the accuracy/robustness trade-off (Cohen et al., 2019, Section 4): a large σ will reduce the Lipschitz constant M δ = δ/σ , however, may also reduce the accuracy of the base classiﬁer (even if ﬁtted on noisy samples).
Revised: This parameter is intimately connected to the accuracy/robustness trade-off (Cohen et al., 2019, Section 4): a large σ will reduce the Lipschitz constant M δ = δ/σ , however, may also reduce the accuracy of the base classiﬁer (even if ﬁtted on noisy samples).

Original: Similar to previous work on randomized smoothing (Cohen et al., 2019; Salman et al., 2019), we treat σ as a hyper-parameter of our method which can be tuned, for example, by choosing σ that produces the smallest sets by cross-validation. {+
+}
Revised: Similar to previous work on randomized smoothing (Cohen et al., 2019; Salman et al., 2019), we treat σ as a hyper-parameter of our method which can be tuned, for example, by choosing σ that produces the smallest sets by cross-validation. {+
+}

Original: In practice, we ﬁnd that setting σ = 2 δ in (10) yields informative prediction sets, as illustrated in Figure 3.
Revised: In practice, we ﬁnd that setting σ = 2 δ in (10) yields informative prediction sets, as illustrated in Figure 3.

Original: This ﬁgure also demonstrates the accuracy/robustness trade-off discussed above: the size of the prediction sets reduces for 1 / 8 ≤ M δ ≤ 1 / 2 , and increases for M δ > 1 / 2 .
Revised: This ﬁgure also demonstrates the accuracy/robustness trade-off discussed above: the size of the prediction sets reduces for 1 / 8 ≤ M δ ≤ 1 / 2 , and increases for M δ > 1 / 2 .

Original: See Supplementary Section S5.1 for additional experiments that study the effect of σ on the marginal coverage and average set-size for all data sets.
Revised: See Supplementary Section S5.1 for additional experiments that study the effect of σ on the marginal coverage and average set-size for all data sets.

Original: Choosing the number of Monte Carlo iterations Following Algorithm 1, the smoothed score is evaluated by replacing the expectation (10) with the empirical mean (12), averaging over n s i.i.d.
Revised: Choosing the number of Monte Carlo iterations Following Algorithm 1, the smoothed score is evaluated by replacing the expectation (10) with the empirical mean (12), averaging over n s i.i.d.

Original: Gaussian noise samples.
Revised: Gaussian noise samples.

Original: This is similar to the approach Cohen et al.
Revised: This is similar to the approach Cohen et al.

Original: (2019, Section 3.2) used for approximating the smoothed classiﬁer deﬁned in (13).
Revised: (2019, Section 3.2) used for approximating the smoothed classiﬁer deﬁned in (13).

Original: In our experiments, we ﬁnd that our method is fairly robust to the choice of n s , and using more than n s = 256 realizations barely affects the marginal coverage and average set-size.
Revised: In our experiments, we ﬁnd that our method is fairly robust to the choice of n s , and using more than n s = 256 realizations barely affects the marginal coverage and average set-size.

Original: This phenomenon is demonstrated in Figure 4 for CIFAR10; see Supplementary Section S5.2 for additional experiments that illustrate the inﬂuence of n s on
Revised: This phenomenon is demonstrated in Figure 4 for CIFAR10; see Supplementary Section S5.2 for additional experiments that illustrate the inﬂuence of n s on

Original: CIFAR100 and ImageNet.
Revised: CIFAR100 and ImageNet.

Original: Therefore, in our experiments we set n s = 256 both for CIFAR10 and
Revised: Therefore, in our experiments we set n s = 256 both for CIFAR10 and

Original: CIFAR100, and n s = 64 for ImageNet due to limited GPU memory footprint; see Supplementary
Revised: CIFAR100, and n s = 64 for ImageNet due to limited GPU memory footprint; see Supplementary

Original: Section S7 for a discussion about the computational complexity  of RSCP .
Revised: Section S7 for a discussion about the computational complexity including the running time of RSCP .

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Figure 5 presents the marginal coverage obtained by Vanilla CP , CP+SS , and RSCP for the three data sets.
Revised: Figure 5 presents the marginal coverage obtained by Vanilla CP , CP+SS , and RSCP for the three data sets.

Original: As can be seen, Vanilla CP fails to construct valid prediction sets in the presenceof adversarial noise.
Revised: As can be seen, Vanilla CP fails to construct valid prediction sets in the presence

Original: For instance, consider the choice of HPS as the non-conformity score, and observe that the average coverage for CIFAR10, CIFAR100, and ImageNet is 27 .

Original: 24% , and

Original: [Figure]
Revised: [Figure]

Original: C P C P + SS [Figure]
Revised: of adversarial noise.

Original: S C P [Figure] CIFAR V a n ill a
Revised: For instance, consider the choice of HPS as the non-conformity score, and observe that the average coverage for CIFAR10, CIFAR100, and ImageNet is 27 .

Original: [-C P C P + SS R S C P   
 ImageNet-]
Revised: [-C P C P + SS R S C P   
 ImageNet-]24% , and 38 .

Original: 02% , respectively; all are much smaller than the desired 90% level, emphasizing the necessity of our proposal.
Revised: 02% , respectively; all are much smaller than the desired 90% level, emphasizing the necessity of our proposal.

Original: When using the same vanilla split conformal procedure, however with the smoothed version of the base non-conformity score (i.e., CP+SS ), we get a major improvement in terms of coverage.
Revised: When using the same vanilla split conformal procedure, however with the smoothed version of the base non-conformity score (i.e., CP+SS ), we get a major improvement in terms of coverage.

Original: Here, for the smoothed version of the HPS score, the average coverage levels obtained for CIFAR10, CIFAR100, and ImageNet are closer to 90% and are equal to 86 .
Revised: Here, for the smoothed version of the HPS score, the average coverage levels obtained for CIFAR10, CIFAR100, and ImageNet are closer to 90% and are equal to 86 .

Original: 10% , and 88 .
Revised: 10% , and 88 .

Original: 12% , respectively.
Revised: 12% , respectively.

Original: This serves as evidence that the new smoothed score is more robust to adversarial attacks.
Revised: This serves as evidence that the new smoothed score is more robust to adversarial attacks.

Original: Moreover, since we can derive the Lipschitz constant of the smoothed score, we can invoke the worst coverage bound from Theorem 2.
Revised: Moreover, since we can derive the Lipschitz constant of the smoothed score, we can invoke the worst coverage bound from Theorem 2.

Original: As illustrated, the (average) worst coverage is indeed below the empirical coverage level obtained by CP+SS .
Revised: As illustrated, the (average) worst coverage is indeed below the empirical coverage level obtained by CP+SS .

Original: While this lower bound is close to the empirical one (for both scores), there exists a gap between the two that can be possibly reduced by designing a better algorithm tailored to attack the smoothed score.
Revised: While this lower bound is close to the empirical one (for both scores), there exists a gap between the two that can be possibly reduced by designing a better algorithm tailored to attack the smoothed score.

Original: By contrast, our RSCP method generates valid prediction sets whose coverage is slightly above the nominal level, supporting our theoretical guarantee.
Revised: By contrast, our RSCP method generates valid prediction sets whose coverage is slightly above the nominal level, supporting our theoretical guarantee.

Original: Figure 5 also presents the average set-size constructed by all three methods.
Revised: Figure 5 also presents the average set-size constructed by all three methods.

Original: Observe that CP+SS generates larger sets compared to Vanilla CP since it relies on a more robust classiﬁer that leads to better coverage at cost of lower accuracy; see Supplementary Table S1.
Revised: Observe that CP+SS generates larger sets compared to Vanilla CP since it relies on a more robust classiﬁer that leads to better coverage at cost of lower accuracy; see Supplementary Table S1.

Original: Observe also that RSCP constructs larger sets than those of CP+SS due to the inﬂated threshold used in the calibration, accounting for the effect of the adversarial noise.
Revised: Observe also that RSCP constructs larger sets than those of CP+SS due to the inﬂated threshold used in the calibration, accounting for the effect of the adversarial noise.

Original: In addition, the sets obtained by the APS score are larger than those of HPS.
Revised: In addition, the sets obtained by the APS score are larger than those of HPS.

Original: This can be explained by (i) the coverage plots: the marginal coverage obtained when using the smoothed APS score tends to be larger than that of the smoothed HPS; and (ii) it is known that the APS score tends to construct larger sets even in the noiseless case since it aims at constructing sets of 1 − α coverage across different sub-populations; see  Romano et al.
Revised: This can be explained by (i) the coverage plots: the marginal coverage obtained when using the smoothed APS score tends to be larger than that of the smoothed HPS; and (ii) it is known that the APS score tends to construct larger sets even in the noiseless case since it aims at constructing sets of 1 − α coverage across different sub-populations; see Supplementary Section S11 for an experiment that illustrates this as well as Romano et al.

Original: (2020b)
Revised: (2020b).

Original: ; Angelopoulos et al.

Original: (2020) for more details.

Original:  C ONCLUSIONS AND FUTURE WORK
Revised:  C ONCLUSIONS AND FUTURE WORK

Original: In this paper, we generalized the technique of conformal prediction to the adversarial setting and introduced a model agnostic, scalable, and simple method for constructing valid prediction sets to multi-class classiﬁcation problems, accounting for the adversarial noise.
Revised: In this paper, we generalized the technique of conformal prediction to the adversarial setting and introduced a model agnostic, scalable, and simple method for constructing valid prediction sets to multi-class classiﬁcation problems, accounting for the adversarial noise.

Original: Our experiments show that our RSCP method is most effective when combined with robust classiﬁers, e.g., ones that are ﬁtted by augmenting Gaussian noise to the training data.
Revised: Our experiments show that our RSCP method is most effective when combined with robust classiﬁers, e.g., ones that are ﬁtted by augmenting Gaussian noise to the training data.

Original: However, this often leads to a degradation in test accuracy, a limitation that is not unique to our work.
Revised: However, this often leads to a degradation in test accuracy, a limitation that is not unique to our work.

Original: In fact, this is a matter of ongoing research in the robustness certiﬁcation literature, and note that any new development in that front (e.g., Salman et al.
Revised: In fact, this is a matter of ongoing research in the robustness certiﬁcation literature, and note that any new development in that front (e.g., Salman et al.

Original: (2020)) could be easily integrated with our method.
Revised: (2020)) could be easily integrated with our method as demonstrated in Supplementary Section S10.

Original: Consequently, an excitingdirection that we aim to explore is to design new non-conformity scores that are more robust to adversarial attacks, which could potentially improve the size of the constructed prediction sets. Another interesting future direction could be to extend the guarantees we provided beyond the assumption that the adversarial noise is (cid:96) 2 -norm bounded and consider different smoothing and calibration techniques that can handle the (cid:96) 0 , (cid:96) 1 , and (cid:96) ∞ cases (Lee et al., 2019; Teng et al., 2019; Zhang et al., 2019). 
Revised: An exciting future direction could be to extend the guarantees we provided beyond the assumption that the adversarial noise is (cid:96) 2 -norm bounded and consider different smoothing and calibration techniques that can handle the (cid:96) 0 , (cid:96) 1 , and (cid:96) ∞ cases (Lee et al., 2019; Teng et al., 2019; Zhang et al., 2019). Other directions could be to explore the use of our method to improve robustness to adversarial attacks in regression problems and to design calibration schemes to construct prediction sets that are valid conditionally on the calibration set (Bates et al., 2021; Park et al., 2021).

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: The ability to communicate the predictive uncertainty of any classiﬁer by constructing prediction sets of valid coverage is crucial to assess the performance of machine learning algorithms (Holland, 2020), improve their reliability, and avoid unwanted discrimination (Romano et al., 2020a).
Revised: The ability to communicate the predictive uncertainty of any classiﬁer by constructing prediction sets of valid coverage is crucial to assess the performance of machine learning algorithms (Holland, 2020), improve their reliability, and avoid unwanted discrimination (Romano et al., 2020a).

Original: Our proposed method further enriches the existing uncertainty quantiﬁcation toolbox, by providing a rigorous form of robustness to adversarial perturbations.
Revised: Our proposed method further enriches the existing uncertainty quantiﬁcation toolbox, by providing a rigorous form of robustness to adversarial perturbations.

Original: For example, one can promote fairness in high-stakes applications (in addition to robustness) by integrating RSCP within the framework of equalized coverage (Romano et al., 2020a).
Revised: For example, one can promote fairness in high-stakes applications (in addition to robustness) by integrating RSCP within the framework of equalized coverage (Romano et al., 2020a).

Original: Here, we stress that the validity of our methods relies on the i.i.d. assumption between the clean training and test points, and on the knowledge of the (cid:96) 2 -bound of the attack.
Revised: Here, we stress that the validity of our methods relies on the exchangeability assumption between the clean training and test points, and on the knowledge of the (cid:96) 2 -bound of the attack.

Original: If one of these assumptions is violated, the sets that RSCP constructs may not have the right coverage.
Revised: If one of these assumptions is violated, the sets that RSCP constructs may not have the right coverage.

Original: Therefore, it is recommended to assess whether the above assumptions are reasonable when using the proposed methods.
Revised: Therefore, it is recommended to assess whether the above assumptions are reasonable when using the proposed methods.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: The Supplementary Material includes a Python package that implements our methods, and code to reproduce our experiments.
Revised: The Supplementary Material includes a Python package that implements our methods, and code to reproduce our experiments.

Original: Complete proofs of the theorems presented in this paper can be found in Supplementary Section S1.
Revised: Complete proofs of the theorems presented in this paper can be found in Supplementary Section S1.

Original: A complete description of the data sets used in our experiments, including pre-processing steps, can be found in Supplementary Section S2.
Revised: A complete description of the data sets used in our experiments, including pre-processing steps, can be found in Supplementary Section S2.

Original: The details about the models used in our experiments and how they were trained are provided in Supplementary sections
Revised: The details about the models used in our experiments and how they were trained are provided in Supplementary sections

Original: S3 and S8.
Revised: S3 and S8.

Original: SUPPLEMENTARY M
Revised: S

Revised: UPPLEMENTARY M

Original: ATERIAL
Revised: ATERIAL

Original: : A DVERSARIALLY ROBUST C ONFORMAL P REDICTION S S UPPLEMENTARY P ROOFS
Revised: : A DVERSARIALLY R

Revised: OBUST C ONFORMAL P REDICTION S S UPPLEMENTARY P ROOFS

Original: In this section, we prove Theorem 1 and Theorem 2 presented in Section 3 of the main manuscript.
Revised: In this section, we prove Theorem 1 and Theorem 2 presented in Section 3 of the main manuscript.

Original: Proof of theorem 1.
Revised: Proof of theorem 1.

Original: From the deﬁnition of the prediction set in (8):
Revised: From the deﬁnition of the prediction set in (8):

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from the validity of the prediction set constructed by the vanilla split conformal procedure since ˜ S
Revised: where the last inequality follows from the validity of the prediction set constructed by the vanilla split conformal procedure since ˜ S

Original: ( X i , Y i ) , i ∈ {I cal ∪ { n + 1 }} are i.i.d. (Vovk et al., 2005); see also (Romano et al., 2019, Lemma 2).
Revised: ( X i , Y i ) , i ∈ {I cal ∪ { n + 1 }} are exchangeable (Vovk et al., 2005); see also (Romano et al., 2019, Lemma 2).

Original: Proof of theorem 2.
Revised: Proof of theorem 2.

Original: From the deﬁnition of the prediction set in (5):
Revised: From the deﬁnition of the prediction set in (5):

Original: [Equation]
Revised: [Equation]

Original: (using the deﬁnition of τ , deﬁned in (9))
Revised: (using the deﬁnition of τ , deﬁned in (9))

Original: [Equation]
Revised: [Equation]

Original:  where, similarly to the proof of Theorem 1, the last inequality follows from (Romano et al., 2019,
Revised: ≥ τ, where, similarly to the proof of Theorem 1, the last inequality follows from (Romano et al., 2019,

Original: Lemma 2).
Revised: Lemma 2).

Original: Lastly, we show that τ ≤ 1 − α .
Revised: Lastly, we show that τ ≤ 1 − α .

Original: From the deﬁnition of τ in (9):
Revised: From the deﬁnition of τ in (9):

Original: [Equation]
Revised: [Equation]

Original: S D ATA SETS
Revised: S D ATA SETS

Original: Overview We evaluate our methods on three widely used image classiﬁcation data sets: CIFAR10,
Revised: Overview We evaluate our methods on three widely used image classiﬁcation data sets: CIFAR10,

Original: CIFAR100 (Krizhevsky, 2009), and ImageNet (Deng et al., 2009).
Revised: CIFAR100 (Krizhevsky, 2009), and ImageNet (Deng et al., 2009).

Original: The CIFAR10 data set consists of 60,000 RGB color images of size 32 × 32 × 3 , corresponding to 10 different classes, with 6, images per class.
Revised: The CIFAR10 data set consists of 60,000 RGB color images of size 32 × 32 × 3 , corresponding to 10 different classes, with 6, images per class.

Original: In this data set, there are 50,000 training images and 10,000 test images.
Revised: In this data set, there are 50,000 training images and 10,000 test images.

Original: The
Revised: The

Original: CIFAR100 data set is identical in structure to CIFAR10, except it has 100 classes containingimages each.
Revised: CIFAR100 data set is identical in structure to CIFAR10, except it has 100 classes containingimages each.

Original: Overall, this data consists of 50,000 training images and 10,000 test images.
Revised: Overall, this data consists of 50,000 training images and 10,000 test images.

Original: In our experiments, we use the entire training set for ﬁtting the classiﬁers and split the rest 10,000 points into two sets of equal size (calibration and test) 50 times.
Revised: In our experiments, we use the entire training set for ﬁtting the classiﬁers and split the rest 10,000 points into two sets of equal size (calibration and test) 50 times.

Original: We use the ILSVRC2012 version of the
Revised: We use the ILSVRC2012 version of the

Original: ImageNet data set.
Revised: ImageNet data set.

Original: Here, the training set consists of 1.2 million color RGB images of different sizes, corresponding to 1,000 class labels.
Revised: Here, the training set consists of 1.2 million color RGB images of different sizes, corresponding to 1,000 class labels.

Original: To construct calibration and test sets, we (i) choose at random 10,000 points out of the 50,000 images from the validation set of ILSVRC2012, and (ii) split these into calibration and test sets of equal size, for 20 times.
Revised: To construct calibration and test sets, we (i) choose at random 10,000 points out of the 50,000 images from the validation set of ILSVRC2012, and (ii) split these into calibration and test sets of equal size, for 20 times.

Original: We use the entire ILSVRC2012 training set for ﬁtting the predictive models.
Revised: We use the entire ILSVRC2012 training set for ﬁtting the predictive models.

Original: Pre-processing The CIFAR10 and CIFAR100 images are normalized by (i) subtracting the training images mean value, equals to [0.4914, 0.4822, 0.4465], and (ii ) dividing the result by the images standard deviation, equals to [0.2023, 0.1994, 0.2010].
Revised: Pre-processing The CIFAR10 and CIFAR100 images are normalized by (i) subtracting the training images mean value, equals to [0.4914, 0.4822, 0.4465], and (ii ) dividing the result by the images standard deviation, equals to [0.2023, 0.1994, 0.2010].

Original: The ImageNet images are ﬁrst re-scaled to a size of 256 × 256 .
Revised: The ImageNet images are ﬁrst re-scaled to a size of 256 × 256 .

Original: Then, the central 224 × 224 pixels are taken and normalized by subtracting the mean value of the training images [0.485, 0.456, 0.406] and dividing the result by their standard deviation, given by [0.229, 0.224, 0.225].
Revised: Then, the central 224 × 224 pixels are taken and normalized by subtracting the mean value of the training images [0.485, 0.456, 0.406] and dividing the result by their standard deviation, given by [0.229, 0.224, 0.225].

Original: S M ODELS
Revised: S M ODELS

Original: As described in Section 5.1 of the main manuscript, for CIFAR10 and ImageNet data sets, we use the pre-trained ResNet-110 and ResNet-50 models from Cohen et al.
Revised: As described in Section 5.1 of the main manuscript, for CIFAR10 and ImageNet data sets, we use the pre-trained ResNet-110 and ResNet-50 models from Cohen et al.

Original: We use the models that were trained on clean training points, as well as models that were trained on Gaussian augmented training points.
Revised: We use the models that were trained on clean training points, as well as models that were trained on Gaussian augmented training points.

Original: These models are downloaded from https://github.com/ locuslab/smoothing .
Revised: These models are downloaded from https://github.com/ locuslab/smoothing .

Original: This link also provides the full details on how the models were trained.
Revised: This link also provides the full details on how the models were trained.

Original: Since the CIFAR100 data set was not studied by Cohen et al.
Revised: Since the CIFAR100 data set was not studied by Cohen et al.

Original: (2019), we ﬁt our own models by modifying the software package available at https://github.com/bearpaw/ pytorch-classification .
Revised: (2019), we ﬁt our own models by modifying the software package available at https://github.com/bearpaw/ pytorch-classification .

Original: Speciﬁcally, we ﬁt a residual network of depth 110 (ResNet110) by augmenting the training data with random crops of size 32 (after applying a zero padding of size 4) and random horizontal ﬂips.
Revised: Speciﬁcally, we ﬁt a residual network of depth 110 (ResNet110) by augmenting the training data with random crops of size 32 (after applying a zero padding of size 4) and random horizontal ﬂips.

Original: The images normalized as explained in Section S2.
Revised: The images normalized as explained in Section S2.

Original: For convenience, we added the normalization as a pre-processing layer to the ﬁtted model.
Revised: For convenience, we added the normalization as a pre-processing layer to the ﬁtted model.

Original: As explained in Section 5.1 of the main manuscript, we ﬁt two models: one on clean training points and another on points that are augmented with Gaussian noise of standard deviation σ that is equal to the one used for smoothing.
Revised: As explained in Section 5.1 of the main manuscript, we ﬁt two models: one on clean training points and another on points that are augmented with Gaussian noise of standard deviation σ that is equal to the one used for smoothing.

Original: The models are trained using the stochastic gradient descent optimizer with a momentum term of 0 .
Revised: The models are trained using the stochastic gradient descent optimizer with a momentum term of 0 .

Original: 9 for 164 epochs, using a batch size of 128 points.
Revised: 9 for 164 epochs, using a batch size of 128 points.

Original: The learning rate starts at 0 .
Revised: The learning rate starts at 0 .

Original: and is multiplied by a factor of 0 .
Revised: and is multiplied by a factor of 0 .

Original: 1 at epochs 81 and 122 .
Revised: 1 at epochs 81 and 122 .

Original: A weight decay regularization of 10 −is added to the loss function.
Revised: A weight decay regularization of 10 −is added to the loss function.

Original: We train the models using Pytorch, on a single Nvidia GEFORCE
Revised: We train the models using Pytorch, on a single Nvidia GEFORCE

Original: GTX 1080 Ti GPU.
Revised: GTX 1080 Ti GPU.

Original: S W HY DOES TRAINING WITH G AUSSIAN NOISE IMPORTANT ?
Revised: S W HY DOES TRAINING WITH G AUSSIAN NOISE IMPORTANT ?

Original: As stated in Section 4.2 of the main manuscript, while our method can work with any base classiﬁer ˆ f , in practice, we require ˆ f to have some degree of robustness in order to construct small prediction sets.
Revised: As stated in Section 4.2 of the main manuscript, while our method can work with any base classiﬁer ˆ f , in practice, we require ˆ f to have some degree of robustness in order to construct small prediction sets.

Original: Figure S6 demonstrates that our RSCP procedure constructs valid but very large prediction sets when deployed with a standard classiﬁer, ﬁtted without augmenting Gaussian noise to the CIFAR training points.
Revised: Figure S6 demonstrates that our RSCP procedure constructs valid but very large prediction sets when deployed with a standard classiﬁer, ﬁtted without augmenting Gaussian noise to the CIFAR training points.

Original: Here, the average size of the prediction sets is equal to 9 , a trivial result for the
Revised: Here, the average size of the prediction sets is equal to 9 , a trivial result for the

Original: CIFAR10 data set that has only 10 classes.
Revised: CIFAR10 data set that has only 10 classes.

Original: In this experiment, we use σ = 0 .
Revised: In this experiment, we use σ = 0 .

Original: 5 for smoothing, an attack of magnitude δ = 0 .
Revised: 5 for smoothing, an attack of magnitude δ = 0 .

Original: 125 , and n s = 

Revised: 125 , and n s =

Original: Gaussian noise samples for estimating the smoothed score.
Revised: Gaussian noise samples for estimating the smoothed score.

Original: [Figure]
Revised: [Figure]

Original: S H YPER P ARAMETERS S5. I NFLUENCE OF THE VALUE OF σ
Revised: S H YPER P ARAMETERS S5. I NFLUENCE OF THE VALUE OF σ

Original: Herein, we extend the discussion from Section 5.1 of the main manuscript and study the effect of the choice of the hyper-parameter σ used to compute the smoothed score.
Revised: Herein, we extend the discussion from Section 5.1 of the main manuscript and study the effect of the choice of the hyper-parameter σ used to compute the smoothed score.

Original: Speciﬁcally, we repeat the same experiment described in the context of Figure 3 for all data sets.
Revised: Speciﬁcally, we repeat the same experiment described in the context of Figure 3 for all data sets.

Original: Figure S7 shows how the ratio between σ and δ affects the average set-size and the average coverage obtained by our RSCP method.
Revised: Figure S7 shows how the ratio between σ and δ affects the average set-size and the average coverage obtained by our RSCP method.

Original: In this experiment, we use an adversarial perturbation of magnitude δ = 0 .
Revised: In this experiment, we use an adversarial perturbation of magnitude δ = 0 .

Original: 125 for CIFAR10 and
Revised: 125 for CIFAR10 and

Original: CIFAR100, and magnitude δ = 0 .
Revised: CIFAR100, and magnitude δ = 0 .

Original: 25 for ImageNet.
Revised: 25 for ImageNet.

Original: As can be seen, a ratio of σ/δ = 2 works well for all data sets and the two non-conformity scores.
Revised: As can be seen, a ratio of σ/δ = 2 works well for all data sets and the two non-conformity scores.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: = /
Revised: = /

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: ImageNet 

Revised: ImageNet

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: S5. I NFLUENCE OF THE VALUE OF n s
Revised: S5. I NFLUENCE OF THE VALUE OF n s

Original: As explained in Section 5.1 of the main manuscript, the smoothed score is evaluated by replacing the expectation (10) with the empirical mean (12), averaging over n s i.i.d.
Revised: As explained in Section 5.1 of the main manuscript, the smoothed score is evaluated by replacing the expectation (10) with the empirical mean (12), averaging over n s i.i.d.

Original: Gaussian noise samples.
Revised: Gaussian noise samples.

Original: Figure S8 shows the effect of the value of n s on the average set-size and the average coverage obtained by our RSCP method for all data sets.
Revised: Figure S8 shows the effect of the value of n s on the average set-size and the average coverage obtained by our RSCP method for all data sets.

Original: Following that ﬁgure, averaging more than n s =noise samples (or even n s = 64 ) barely affects the empirical coverage and size of the constructed sets.
Revised: Following that ﬁgure, averaging more than n s =noise samples (or even n s = 64 ) barely affects the empirical coverage and size of the constructed sets.

Original: M a r g i n a l C o v e r a g e
Revised: M a r g i n a l C o v e r a g e

Original: CIFAR  0.
Revised: CIFAR  0.

Original: CIFAR 1 2 4 8 16 32n s 0.
Revised: CIFAR 1 2 4 8 16 32n s 0.

Original: ImageNet 2.
Revised: ImageNet 2.

Original: A v e r a g e S e t S i z e
Revised: A v e r a g e S e t S i z e

Original: CIFAR
Revised: CIFAR

Original: Base Score APS
Revised: Base Score APS

Original: HPS
Revised: HPS

Original: CIFAR 1 2 4 8 16 32n s
Revised: CIFAR 1 2 4 8 16 32n s

Original: ImageNet
Revised: ImageNet

Original: S E FFECT OF THE ADVERSARIAL ATTACKS
Revised: S E FFECT OF THE ADVERSARIAL ATTACKS

Original: As stated in Section 5.1 of the main manuscript, in all our experiments we use an attack of (cid:96) 2 magnitude δ = 0 .
Revised: As stated in Section 5.1 of the main manuscript, in all our experiments we use an attack of (cid:96) 2 magnitude δ = 0 .

Original: 125 for CIFAR10 and CIFAR100, and δ = 0 .
Revised: 125 for CIFAR10 and CIFAR100, and δ = 0 .

Original: 25 for ImageNet.
Revised: 25 for ImageNet.

Original: Table S1 illustrates the effect of such adversarial perturbations on the test accuracy of the models used in the experiments presented in Section 5 of the main manuscript as well as in Supplementary Section S8 that is presented hereafter.
Revised: Table S1 illustrates the effect of such adversarial perturbations on the test accuracy of the models used in the experiments presented in Section 5 of the main manuscript as well as in Supplementary Section S8 that is presented hereafter.

Original: The PGD attack leads to a signiﬁcant reduction in the test accuracy of models that are ﬁtted without Gaussian noise augmentation (i.e., σ = 0 ).
Revised: The PGD attack leads to a signiﬁcant reduction in the test accuracy of models that are ﬁtted without Gaussian noise augmentation (i.e., σ = 0 ).

Original: By contrast, the models that are trained with Gaussian noise are more robust, as the decrease in test accuracy is smaller when attacked by
Revised: By contrast, the models that are trained with Gaussian noise are more robust, as the decrease in test accuracy is smaller when attacked by

Original: SMOOTHADV .
Revised: SMOOTHADV .

Original: This, however, comes at the expense of an accuracy reduction on the clean test points.
Revised: This, however, comes at the expense of an accuracy reduction on the clean test points.

Original: S C OMPUTATIONAL COMPLEXITY
Revised: S C OMPUTATIONAL COMPLEXITY

Original: Following Algorithm 1, at test time, the computational complexity of RSCP is dominated by the number of calls to the predictive model, required to compute the smoothed score for each of the L possible labels y ∈ Y .
Revised: Following Algorithm 1, at test time, the computational complexity of RSCP is dominated by the number of calls to the predictive model, required to compute the smoothed score for each of the L possible labels y ∈ Y .

Original: Hence, to construct aprediction set for a given test point, the above amounts to a total of n s · L calls to  the classiﬁer.
Revised: The number of calls to the predictive model for creating a set for a single test image is n s , and the number of classes L only affects the number of scores that need to be calculated using the model predictions.

Original: The training involves ﬁtting a classiﬁer, and the calibration requires computing the smoothed non-conformity scores for all calibration points, where both steps can be performed only once.
Revised: The training involves ﬁtting a classiﬁer, and the calibration requires computing the smoothed non-conformity scores for all calibration points, where both steps can be performed only once.

Original: [Table]
Revised: [Table]

Original: S R ESULTS WITH D ENS N ET AND VGG MODELS
Revised: Table S2 summarizes the average runtime for constructing a prediction set using Vanilla CP , CP+SS , and RSCP with the HPS score function (3), for a single image, where the average is taken over 100 realizations. We present the results obtained by two different implementation strategies of our methods: ‘single mode’ and ‘batch mode’. The ‘single’ implementation repeats the following two steps for n s times: (i) augment to the input image one realization of Gaussian noise, and (ii) feed this noisy image to the neural net model.

Revised: The ‘batch’ implementation is more efﬁcient since it parallelizes the inference step as follows: (i) augment n s noise realizations to the same input image, and (ii) pass all these noisy realizations together (as one batch) to the neural net model.

Revised: We follow the exact same setup from the experiments used to create Figure 5 of the main manuscript.

Original: In this section
Revised: Speciﬁcally, n s = 256 for CIFAR10 and CIFAR100, and n s = 64 for ImageNet. The runtimes are evaluated using Pytorch, on a single Nvidia GEFORCE GTX 1080Ti GPU. Following Table S2, we can see that when using the batch implementation the inference time of CP+SS and RSCP is approximately 3-4 times larger than Vanilla CP .

Revised: Also, the runtimes of

Revised: CP+SS and RSCP are almost identical, since the only difference between the two is the threshold used to construct the prediction sets.

Original: [Table]
Revised: [Table]

Original:  we study the performance of our method when applied with DenseNet (Iandola et al., 2014) and VGG (Simonyan & Zisserman, 2014) models on the CIFAR10 and CIFAR100 data sets, as opposed to experiments presented thus far that use only ResNet (He et al., 2016) classiﬁers.
Revised: S R ESULTS WITH D ENS N ET AND VGG MODELS In this section, we study the performance of our method when applied with DenseNet (Iandola et al., 2014) and VGG (Simonyan & Zisserman, 2014) models on the CIFAR10 and CIFAR100 data sets, as opposed to experiments presented thus far that use only ResNet (He et al., 2016) classiﬁers.

Original: The experiments here follow the protocol described in Section 5 of the main manuscript.
Revised: The experiments here follow the protocol described in Section 5 of the main manuscript.

Original: We use the same attack algorithms and magnitude δ = 0 .
Revised: We use the same attack algorithms and magnitude δ = 0 .

Original: 125 as in the ResNet-110 experiments for CIFAR  and CIFAR100.
Revised: 125 as in the ResNet-110 experiments for CIFAR  and CIFAR100.

Original: As for the hyper-parameters of our methods, we choose σ = 0 .
Revised: As for the hyper-parameters of our methods, we choose σ = 0 .

Original: 25 for smoothing, and n s = 256 for estimating the smoothed score.
Revised: 25 for smoothing, and n s = 256 for estimating the smoothed score.

Original: Figure S9 compares the coverage and size of the prediction sets constructed by combining ResNet-110, VGG, or DenseNet with Vanilla CP ,
Revised: Figure S9 compares the coverage and size of the prediction sets constructed by combining ResNet-110, VGG, or DenseNet with Vanilla CP ,

Original: CP+SS , and RSCP .
Revised: CP+SS , and RSCP .

Original: As can be seen, the difference between the three models is minor in terms of coverage for all calibration methods in both data sets.
Revised: As can be seen, the difference between the three models is minor in terms of coverage for all calibration methods in both data sets.

Original: We can also see that our theoretical guarantees hold: for CP+SS , the worst coverage bound is below the empirical coverage obtained, and RSCP yields valid prediction sets in contrast to the two other methods.
Revised: We can also see that our theoretical guarantees hold: for CP+SS , the worst coverage bound is below the empirical coverage obtained, and RSCP yields valid prediction sets in contrast to the two other methods.

Original: In terms of size, the difference between ResNet and DenseNet is minor, while VGG constructs larger sets, especially on CIFAR100.
Revised: In terms of size, the difference between ResNet and DenseNet is minor, while VGG constructs larger sets, especially on CIFAR100.

Original: Figure 1 of the main manuscript presents the results obtained by applying the VGG models to the
Revised: Figure 1 of the main manuscript presents the results obtained by applying the VGG models to the

Original: CIFAR10 data set, using the HPS score.
Revised: CIFAR10 data set, using the HPS score.

Original: In that ﬁgure, we additionally tested the vanilla conformal method on the clean test points, to show the validity of this method when applied to clean test data that satisfy the i.i.d. assumption.
Revised: In that ﬁgure, we additionally tested the vanilla conformal method on the clean test points, to show the validity of this method when applied to clean test data that satisfy the exchangeability assumption.

Original: Turning to the technical details about the training procedure.
Revised: Turning to the technical details about the training procedure.

Original: For each architecture, we ﬁt two models: one on clean training points and the second on points that are augmented with Gaussian noise, as explained in Section 5.1 and Supplementary Section S3.
Revised: For each architecture, we ﬁt two models: one on clean training points and the second on points that are augmented with Gaussian noise, as explained in Section 5.1 and Supplementary Section S3.

Original: The DenseNet architecture consists of 100 layers with a growth rate of 12 and a compression rate of 2 .
Revised: The DenseNet architecture consists of 100 layers with a growth rate of 12 and a compression rate of 2 .

Original: We use the same data augmentation and normalization described in Section S2.
Revised: We use the same data augmentation and normalization described in Section S2.

Original: The models are trained using stochastic gradient descent with a momentum term that equals 0 .
Revised: The models are trained using stochastic gradient descent with a momentum term that equals 0 .

Original: 9 , a batch size of 64 , and a total of 300 epochs.
Revised: 9 , a batch size of 64 , and a total of 300 epochs.

Original: The learning rate starts at 0 .
Revised: The learning rate starts at 0 .

Original: and is multiplied by a factor of 0 .
Revised: and is multiplied by a factor of 0 .

Original: 1 at epochs 150 and 225 .
Revised: 1 at epochs 150 and 225 .

Original: We also use a weight decay regularization that equals 10 − 4 .
Revised: We also use a weight decay regularization that equals 10 − 4 .

Original: The VGG architecture consists of 19 layers with batch normalization.
Revised: The VGG architecture consists of 19 layers with batch normalization.

Original: We use the same data augmentation and normalization from Section S2.
Revised: We use the same data augmentation and normalization from Section S2.

Original: The
Revised: The

Original: VGG models are trained using stochastic gradient descent: a momentum of 0 .
Revised: VGG models are trained using stochastic gradient descent: a momentum of 0 .

Original: 9 , batch size of 128 , a for a total of 164 epochs.
Revised: 9 , batch size of 128 , a for a total of 164 epochs.

Original: We set the initial learning rate to be equal to 0 .
Revised: We set the initial learning rate to be equal to 0 .

Original: 1 and multiplied by a factor of 0 .
Revised: 1 and multiplied by a factor of 0 .

Original: 1 at epochs 81 and 122 .
Revised: 1 at epochs 81 and 122 .

Original: We also use a weight decay regularization, which equals 10 − 4 .
Revised: We also use a weight decay regularization, which equals 10 − 4 .

Original: [Figure]
Revised: [Figure]

Revised: S R ESULTS ON C LEAN D ATA

Revised: In this section, we study the performance of our methods in a situation where no adversarial attack is performed.

Revised: We follow the setup from Figure 5 of the main manuscript, and compare the three methods of interest ( Vanilla CP , CP+SS , and RSCP ) on the three data sets, using the same models, smoothing parameter σ , and the number of noise realizations n s .

Revised: Importantly, in contrast to

Revised: Figure 5, the results that are presented in Figure S10 are obtained by applying these three methods on clean test points that are not corrupted by adversarial noise.

Revised: Following that ﬁgure, both Vanilla

Revised: CP and CP+SS achieve the desired coverage of 90% , which stands in line with the theoretical guarantee of split conformal prediction; see also Theorem 2.

Revised: By contrast, RSCP results in a higher coverage rate than the desired one, since this method always accounts for the uncertainty induced by a possible attack even if not applied.

Revised: Moving to the average set-size, notice that all the methods perform similarly to the case where the test data is corrupted, as depicted in Figure 5 of the main manuscript.

Revised: Speciﬁcally, the average size of the sets constructed by CP+SS is larger than that of

Revised: Vanilla CP , possibly due to smoothed model that is used to make the predictions.

Revised: However, it is important to stress that CP+SS has a signiﬁcantly smaller drop in coverage in the adversarial setting.

Revised: We also note that the average size of the sets constructed by RSCP is larger than those of CP+SS due to the inﬂation of threshold Q 1 − α .

Revised: As a concluding remark, we iterate here the discussion from Section 3.2 of the main manuscript, displaying the two possible use-cases of our proposal.

Revised: In the ﬁrst, one can implement RSCP and construct valid prediction sets at any desired level of coverage, while bearing in mind that these sets are likely to be larger even if no attack is performed.

Revised: In the second case, one can deploy CP+SS , guaranteeing that the sets will have the desired coverage in the case where no attack is performed while providing important information about the worst coverage that might be obtained under an adversarial attack.

Revised: Moreover, we believe that in practice these two use-cases can be combined via the following two-step procedure to improve statistical efﬁciency: (i) apply a method for detecting whether a given test point is corrupted by an adversarial noise (e.g., Metzen et al., 2017; Feinman et al., 2017; Cohen et al., 2020); and (ii) if an attack is detected, apply the RSCP method to construct a valid prediction set, otherwise use the vanilla CP or CP+SS since these methods are less conservative.

Revised: [Figure]

Revised: S R ESULTS WITH AN ADVERSARIALLY - TRAINED MODEL

Revised: In this section, we study the performance of our methods, integrated with the adversarial training scheme of Salman et al.

Revised: (2019) that was shown to improve the robustness to the speciﬁc

Revised: SMOOTHADV attack we applied at test time.

Revised: This new, more robust, predictive model replaces the one we used in Section 5 (see also Section S3), which is ﬁtted merely by adding Gaussian noise to the training points.

Revised: Recall that the SMOOTHADV attack has two parameters: the magnitude of the noise δ and the smoothing strength σ .

Revised: Here, we use a training procedure that is ideal in the sense that it has access to the same δ used for attacking the model, and the same σ used to compute the smoothed score.

Revised: We perform this experiment only on the CIFAR10 data set and ﬁtted a ResNet110 model using the code provided by Salman et al.

Revised: Their software package is available at https://github.com/Hadisalman/smoothing-adversarial ; this link also provides the full details on how the models are trained.

Revised: The results are depicted in Figure S11, which follows the exact same experimental protocol used to create Figure 5 of the main manuscript.

Revised: That ﬁgure compares two versions of CP+SS : the ﬁrst is implemented with a model ﬁtted on Gaussian augmented data, and the second is implemented with the adversarially trained model of Salman et al.

Revised: As can be seen, the latter model achieves a slightly better coverage, closer to the desired one, and also improves the statistical efﬁciency.

Revised: Interestingly, even in this optimistic case, CP+SS does not attain the desired coverage perfectly.

Revised: It is also important to stress that if the new model would be attacked by a different and more effective attack, the coverage can possibly drop even further.

Revised: It is worth noting that this experiment supports our discussion from Section 6, arguing that any new development for improving adversarial robustness could be easily integrated with our method to improve the overall performance.

Revised: [Figure]

Revised: S HPS VS . APS

Revised: In this section, we discuss the difference between the APS and HPS non-conformity scores and demonstrate the advantages that APS has over HPS.

Revised: Romano et al.

Revised: (2020b) carefully argued that while conformal prediction methods—which use the HPS score—are guaranteed to achieve the marginal coverage criterion (1), they may fail to achieve the stronger and more informative notion of conditional coverage, deﬁned as

Revised: [Equation]

Original: {+In plain words, the above statement requires the coverage to hold for a speciﬁc observation X n +
+}
Revised: {+In plain words, the above statement requires the coverage to hold for a speciﬁc observation X n +
+}

Revised: = x , and not merely on average over all test points.

Revised: While it is known that conditional coverage cannot be achieved in ﬁnite samples without imposing strong regularity conditions and/or modeling assumptions (Foygel Barber et al., 2021; Vovk, 2012), it was shown by Romano et al.

Revised: (2020b) that the APS score performs better in terms of conditional coverage compared to the HPS score.

Revised: In fact,

Revised: Romano et al.

Revised: (2020b) rigorously showed that by combining the APS score with an oracle classiﬁer that has access to the true conditional class probabilities P Y | X , one can exactly satisfy (S14).

Revised: To demonstrate the advantage of APS in the adversarial setting, we repeat the experiment from

Revised: Figure 5 of the main manuscript and present the coverage and the average set-size, conditional on each of the 10 classes of the CIFAR10 data set.

Revised: Following Figure S12, the APS prediction sets achieve a steady coverage across all classes, close to the desired level for the CP+SS , while the coverage rates of HPS vary greatly.

Revised: Moreover, the latter method tends to undercover the labels of the samples that belong to classes 2-5.

Revised: One can also see that the size of the prediction sets constructed by APS varies across the different classes, and therefore better reﬂecting the prediction uncertainty.

Revised: This stands in contrast with the sets constructed via the HPS score, whose sizes are more or less constant for all class labels.

Revised: M a r g i n a l

Revised: C o v e r a g e

Revised: C P +

Revised: SS Base Score

Revised: APS

Revised: HPS

Revised: Nominal Level

Revised: Class 0.

Revised: R S C

Revised: P 1.

Revised: A v e r a g e

Revised: S e t

Revised: S i z e

Revised: C P +

Revised: SS

Revised: Base Score

Revised: APS

Revised: HPS

Revised: Class

Revised: R S C

Revised: P