Original: U NCERTAINTY S ETS FOR I MAGE C LASSIFIERS USING C ONFORMAL P REDICTION
Revised: U NCERTAINTY S ETS FOR I MAGE C LASSIFIERS USING C ONFORMAL P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Convolutional image classiﬁers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings.
Revised: Convolutional image classiﬁers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings.

Original: Existing uncertainty quantiﬁcation techniques, such as Platt scaling, attempt to calibrate the network’s probability estimates, but they do not have formal guarantees.
Revised: Existing uncertainty quantiﬁcation techniques, such as Platt scaling, attempt to calibrate the network’s probability estimates, but they do not have formal guarantees.

Original: We present an algorithm that modiﬁes any classiﬁer to output a predictive set containing the true label with a user-speciﬁed probability, such as 90% .
Revised: We present an algorithm that modiﬁes any classiﬁer to output a predictive set containing the true label with a user-speciﬁed probability, such as 90% .

Original: The algorithm is simple and fast like Platt scaling, but provides a formal ﬁnite-sample coverage guarantee for every model and dataset.
Revised: The algorithm is simple and fast like Platt scaling, but provides a formal ﬁnite-sample coverage guarantee for every model and dataset.

Original: Furthermore, our method generates much smaller predictive sets than alternative methods, since we introduce a regularizer to stabilize the small scores of unlikely classes after Platt scaling.
Revised: Furthermore, our method generates much smaller predictive sets than alternative methods, since we introduce a regularizer to stabilize the small scores of unlikely classes after Platt scaling.

Original: In experiments on both Imagenet and Imagenet-V2 with
Revised: In experiments on both Imagenet and Imagenet-V2 with

Original: ResNet-152 and other classiﬁers, our scheme outperforms existing approaches, achieving exact coverage with sets that are often factors of 5 to 10 smaller.
Revised: ResNet-152 and other classiﬁers, our scheme outperforms existing approaches, achieving exact coverage with sets that are often factors of 5 to 10 smaller.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Imagine you are a doctor making a high-stakes medical decision based on diagnostic information from a computer vision classiﬁer.
Revised: Imagine you are a doctor making a high-stakes medical decision based on diagnostic information from a computer vision classiﬁer.

Original: What would you want the classiﬁer to output in order to make the best decision?
Revised: What would you want the classiﬁer to output in order to make the best decision?

Original: This is not a casual hypothetical; such classiﬁers are already used in medical settings (e.g., Razzak et al., 2018; Lundervold & Lundervold, 2019;
Revised: This is not a casual hypothetical; such classiﬁers are already used in medical settings (e.g., Razzak et al., 2018; Lundervold & Lundervold, 2019;

Original: Li et al., 2014).
Revised: Li et al., 2014).

Original: A maximumlikelihood diagnosis with an accompanying probability may be useful, but it may not be the most essential piece of information.
Revised: A maximumlikelihood diagnosis with an accompanying probability may be useful, but it may not be the most essential piece of information.

Original: As a doctor, your primary objective is to ensure the health of your patient, which requires ruling in or ruling out harmful diagnoses.
Revised: As a doctor, your primary objective is to ensure the health of your patient, which requires ruling in or ruling out harmful diagnoses.

Original: In other words, even if the most likely diagnosis is a stomach ache, it is equally or more important to rule out stomach cancer.
Revised: In other words, even if the most likely diagnosis is a stomach ache, it is equally or more important to rule out stomach cancer.

Original: Therefore, you would want the classiﬁer to give you—in addition to an estimate of the most likely outcome— actionable uncertainty quantiﬁcation, such as a set of predictions that provably covers the true diagnosis with a high probability (e.g., 90% ).
Revised: Therefore, you would want the classiﬁer to give you—in addition to an estimate of the most likely outcome— actionable uncertainty quantiﬁcation, such as a set of predictions that provably covers the true diagnosis with a high probability (e.g., 90% ).

Original: This is called a prediction set (see Figure 1).
Revised: This is called a prediction set (see Figure 1).

Original: Our paper describes a novel method for constructing prediction sets from any pre-trained image classiﬁer that are provably exact, relatively small, and practical to implement.
Revised: Our paper describes a novel method for constructing prediction sets from any pre-trained image classiﬁer that are provably exact, relatively small, and practical to implement.

Original: Formally, for a discrete response Y ∈ Y = { 1 , . . .
Revised: Formally, for a discrete response Y ∈ Y = { 1 , . . .

Original: , K } and a feature vector X ∈ R d , we develop a method for taking any ﬁtted classiﬁer and creating an uncertainty set function, C ( X ) , mapping a feature vector to a subset of { 1 , . .
Revised: , K } and a feature vector X ∈ R d , we develop a method for taking any ﬁtted classiﬁer and creating an uncertainty set function, C ( X ) , mapping a feature vector to a subset of { 1 , . .

Original: , K } such that
Revised: , K } such that

Original: [Equation]
Revised: [Equation]

Original: for a pre-speciﬁed conﬁdence level α such as 5%.
Revised: for a pre-speciﬁed conﬁdence level α such as 5%.

Original: We introduce a general technique that modiﬁes any black-box classiﬁer to output predictive sets that are rigorously guaranteed to satisfy the desired coverage property shown in Eq.
Revised: We introduce a general technique that modiﬁes any black-box classiﬁer to output predictive sets that are rigorously guaranteed to satisfy the desired coverage property shown in Eq.

Original: For evaluations, we focus on Imagenet classiﬁcation using convolutional neural networks (CNNs) as the base classiﬁers, since this is a particularly challenging testbed.
Revised: For evaluations, we focus on Imagenet classiﬁcation using convolutional neural networks (CNNs) as the base classiﬁers, since this is a particularly challenging testbed.

Original: In this setting, X would be the image and Y would be the class label.
Revised: In this setting, X would be the image and Y would be the class label.

Original: One approach might be to assemble the set by including classes from highest to lowest probability (e.g., after Platt scaling and a softmax function; see Platt et al., 1999; Guo et al., 2017) until their sum just exceeds the threshold 1 − α .
Revised: One approach might be to assemble the set by including classes from highest to lowest probability (e.g., after Platt scaling and a softmax function; see Platt et al., 1999; Guo et al., 2017) until their sum just exceeds the threshold 1 − α .

Original: We call this strategy naive and formulate it precisely in Algorithm 1.
Revised: We call this strategy naive and formulate it precisely in Algorithm 1.

Original: There are two problems with naive :
Revised: There are two problems with naive :

Original: ﬁrst, the probabilities output by CNNs are
Revised: ﬁrst, the probabilities output by CNNs are

Original: [Equation]
Revised: [Equation]

Original: known to be incorrect (Nixon et al., 2019)
Revised: known to be incorrect (Nixon et al., 2019)

Original: , so the sets from naive do not achieve coverage.
Revised: , so the sets from naive do not achieve coverage.

Original: Second, for examples where the model is not conﬁdent, naive must select many classes before it reaches the desired conﬁdence level, leading to a large set size.
Revised: Second, for examples where the model is not conﬁdent, naive must select many classes before it reaches the desired conﬁdence level, leading to a large set size.

Original: The coverage problem can be solved by picking a new threshold using holdout samples.
Revised: The coverage problem can be solved by picking a new threshold using holdout samples.

Original: For example, with α = 10%, if choosing sets that contain 93% estimated probability achieves 90% coverage on the holdout set, we use the 93% cutoff instead.
Revised: For example, with α = 10%, if choosing sets that contain 93% estimated probability achieves 90% coverage on the holdout set, we use the 93% cutoff instead.

Original: We refer to this algorithm, introduced in Romano et al.
Revised: We refer to this algorithm, introduced in Romano et al.

Original: (2020), as Adaptive Prediction Sets ( APS ).
Revised: (2020), as Adaptive Prediction Sets ( APS ).

Original: The APS procedure provides coverage but produces large sets because it is sensitive to the noisy probability estimates assigned to unlikely classes—see
Revised: The APS procedure provides coverage but produces large sets because it is sensitive to the noisy probability estimates assigned to unlikely classes—see

Original: Section 2.3 for details.
Revised: Section 2.3 for details.

Original: To ﬁx this, we introduce a regularization technique that tempers the inﬂuence of these noisy estimates, leading to smaller, more stable sets.
Revised: To ﬁx this, we introduce a regularization technique that tempers the inﬂuence of these noisy estimates, leading to smaller, more stable sets.

Original: We describe our proposed algorithm,
Revised: We describe our proposed algorithm,

Original: Regularized Adaptive Prediction Sets ( RAPS ), in Algorithms 2 and 3 (with APS as a special case).
Revised: Regularized Adaptive Prediction Sets ( RAPS ), in Algorithms 2 and 3 (with APS as a special case).

Original: As we will see in Section 2, both APS and RAPS are always guaranteed to satisfy Eq.
Revised: As we will see in Section 2, both APS and RAPS are always guaranteed to satisfy Eq.

Original: (1)—regardless of model and dataset.
Revised: (1)—regardless of model and dataset.

Original: Furthermore, we show that RAPS is guaranteed to have better performance than choosing a ﬁxed-size set.
Revised: Furthermore, we show that RAPS is guaranteed to have better performance than choosing a ﬁxed-size set.

Original: Both methods impose negligible computational requirements in both training and evaluation, and require only 1000 held-out data points in practice.
Revised: Both methods impose negligible computational requirements in both training and evaluation, and require only 1000 held-out data points in practice.

Original: We will also see in Section 3 that RAPS provides signiﬁcant practical gains.
Revised: We will also see in Section 3 that RAPS provides signiﬁcant practical gains.

Original: Conducting the most extensive evaluation of conformal prediction in deep learning to date, on Imagenet and Imagenet-V2, we ﬁnd that RAPS is uniformly preferable to naive and APS .
Revised: Conducting the most extensive evaluation of conformal prediction in deep learning to date, on Imagenet and Imagenet-V2, we ﬁnd that RAPS is uniformly preferable to naive and APS .

Original: For example, using a ResNeXt-101, naive does not achieve coverage, while APS and RAPS achieve it exactly.
Revised: For example, using a ResNeXt-101, naive does not achieve coverage, while APS and RAPS achieve it exactly.

Original: However, APS sets have an average size of 20, while RAPS sets have an average size of 2 at α = 10%
Revised: However, APS sets have an average size of 19, while RAPS sets have an average size of 2 at α = 10%

Original: (Figure 2 and Table 1).
Revised: (Figure 2 and Table 1).

Revised: We will provide an accompanying codebase that implements our method as a wrapper for any PyTorch classiﬁer, along with code to exactly reproduce all of our experiments.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Reliably estimating predictive uncertainty for neural networks is an unsolved problem.
Revised: Reliably estimating predictive uncertainty for neural networks is an unsolved problem.

Original: Historically, the standard approach has been to train a Bayesian neural network to learn a distribution over network weights (Quinonero-Candela et al., 2005; MacKay, 1992; Neal, 2012; Kuleshov et al., 2018;
Revised: Historically, the standard approach has been to train a Bayesian neural network to learn a distribution over network weights (Quinonero-Candela et al., 2005; MacKay, 1992; Neal, 2012; Kuleshov et al., 2018;

Original: Gal, 2016).
Revised: Gal, 2016).

Original: This approach incurs signiﬁcant computational expense and implementational complexity, which more recent approaches address with model ensembles (Lakshminarayanan et al., 2017; Jiang et al., 2018) or approximations of Bayesian inference for the parameters of the last layer only (Riquelme et al., 2018; Sensoy et al., 2018).
Revised: This approach incurs signiﬁcant computational expense and implementational complexity, which more recent approaches address with model ensembles (Lakshminarayanan et al., 2017; Jiang et al., 2018) or approximations of Bayesian inference for the parameters of the last layer only (Riquelme et al., 2018; Sensoy et al., 2018).

Original: These methods also have major practical limitations; for example, ensembling requires training many copies of a neural network adversarially.
Revised: These methods also have major practical limitations; for example, ensembling requires training many copies of a neural network adversarially.

Original: Therefore, the most widely used approach to quantifying uncertainty is the ad-hoc “calibration” of the softmax scores with Platt scaling (Platt et al., 1999; Guo et al., 2017; Nixon et al., 2019).
Revised: Therefore, the most widely used approach to quantifying uncertainty is the ad-hoc “calibration” of the softmax scores with Platt scaling (Platt et al., 1999; Guo et al., 2017; Nixon et al., 2019).

Original: We will refer to this approach as traditional calibration .
Revised: We will refer to this approach as traditional calibration .

Original: In this work, we develop an alternative method for uncertainty quantiﬁcation that is based on conformal prediction .
Revised: In this work, we develop an alternative method for uncertainty quantiﬁcation that is based on conformal prediction .

Original: Originating in the online learning literature, conformal prediction is a general approach for generating predictive sets that satisfy the coverage property in Eq.
Revised: Originating in the online learning literature, conformal prediction is a general approach for generating predictive sets that satisfy the coverage property in Eq.

Original: (1) (Vovk et al., 1999; 2005).
Revised: (1) (Vovk et al., 1999; 2005).

Original: Although the original approach to conformal prediction requires special algorithmic development for each predictor under study, a convenient data-splitting version known as split conformal prediction enables conformal prediction methods to be deployed for essentially any predictor (Papadopoulos et al., 2002; Lei et al., 2018).
Revised: Although the original approach to conformal prediction requires special algorithmic development for each predictor under study, a convenient data-splitting version known as split conformal prediction enables conformal prediction methods to be deployed for essentially any predictor (Papadopoulos et al., 2002; Lei et al., 2018).

Original: While mechanically very different from traditionalcalibration as discussed above, we will refer to our approach as conformal calibration to highlight that the two methodologies have overlapping but different goals.
Revised: While mechanically very different from traditional

Original: [Figure]
Revised: [Figure]

Original: All methods use a ResNet-152 as the base  and results are reported for  random  of 
Revised: All methods use a ResNet-152 as the base classiﬁer, and results are reported for 100 random splits of Imagenet-Val, each of size 20K. See Section 3.1 for full details.

Revised: calibration as discussed above, we will refer to our approach as conformal calibration to highlight that the two methodologies have overlapping but different goals.

Original: Note that conformal prediction is a general framework, not a speciﬁc algorithm—there are many conformal algorithms and important design decisions must be made to achieve the best performance for each context.
Revised: Note that conformal prediction is a general framework, not a speciﬁc algorithm—there are many conformal algorithms and important design decisions must be made to achieve the best performance for each context.

Original: To this end, Romano et al.
Revised: To this end, Romano et al.

Original: (2020) and Cauchois et al.
Revised: (2020) and Cauchois et al.

Original: (2020) introduce techniques aimed at achieving coverage that is similar across regions of feature space, whereas Hechtlinger et al.
Revised: (2020) introduce techniques aimed at achieving coverage that is similar across regions of feature space, whereas Hechtlinger et al.

Original: (2018) introduce a technique aimed at achieving equal coverage for each class.
Revised: (2018) introduce a technique aimed at achieving equal coverage for each class.

Original: While these methods have conceptual appeal, thus far there has been limited empirical evaluation of this general approach for state-of-the-art CNNs.
Revised: While these methods have conceptual appeal, thus far there has been limited empirical evaluation of this general approach for state-of-the-art CNNs.

Original: Concretely, the only works that we are aware of that include some evaluation of conformal methods on ImageNet—the gold standard for benchmarking computer vision methods—are Hechtlinger et al.
Revised: Concretely, the only works that we are aware of that include some evaluation of conformal methods on ImageNet—the gold standard for benchmarking computer vision methods—are Hechtlinger et al.

Original: , Park et al.
Revised: , Park et al.

Original: (2019), Cauchois et al.
Revised: (2019), Cauchois et al.

Original: (2020), and Messoudi et al.
Revised: (2020), and Messoudi et al.

Original: (2020), although in all four cases further experiments are needed to more fully evaluate their operating characteristics for practical deployment.
Revised: (2020), although in all four cases further experiments are needed to more fully evaluate their operating characteristics for practical deployment.

Original:  M ETHODS
Revised:  M ETHODS

Original: In developing uncertainty set methods to improve upon naive , we are guided by three desiderata.
Revised: In developing uncertainty set methods to improve upon naive , we are guided by three desiderata.

Original: First and most importantly, the coverage desideratum says the sets must provide exactly 1 − α coverage, as discussed above.
Revised: First and most importantly, the coverage desideratum says the sets must provide exactly 1 − α coverage, as discussed above.

Original: Secondly, the size desideratum says we want sets of small size, since these convey more detailed information and may be more useful in practice.
Revised: Secondly, the size desideratum says we want sets of small size, since these convey more detailed information and may be more useful in practice.

Original: Lastly, the adaptiveness desideratum says we want the sets to communicate instance-wise uncertainty: they should be smaller for easy test-time examples than for hard ones; see Figure 1 for an illustration.
Revised: Lastly, the adaptiveness desideratum says we want the sets to communicate instance-wise uncertainty: they should be smaller for easy test-time examples than for hard ones; see Figure 1 for an illustration.

Original: Coverage and size are obviously competing objectives, but size and adaptiveness are also often in tension.
Revised: Coverage and size are obviously competing objectives, but size and adaptiveness are also often in tension.

Original: The size desideratum seeks small sets, while the adaptiveness desideratum seeks larger sets for instances where the classiﬁer is very uncertain.
Revised: The size desideratum seeks small sets, while the adaptiveness desideratum seeks larger sets for instances where the classiﬁer is very uncertain.

Original: For example, always predicting a set of size ﬁve could achieve coverage, but it is not adaptive.
Revised: For example, always predicting a set of size ﬁve could achieve coverage, but it is not adaptive.

Original: We will investigate the behavior of various prediction sets along all three axes in our experiments.
Revised: We will investigate the behavior of various prediction sets along all three axes in our experiments.

Original: We now turn to the speciﬁcs of our proposed method.
Revised: We now turn to the speciﬁcs of our proposed method.

Original: We begin in Subsection 2.1 by describing an abstract data-splitting procedure called conformal calibration that enables the near-automatic construction of valid predictive sets (that is, sets satisfying Eq.
Revised: We begin in Subsection 2.1 by describing an abstract data-splitting procedure called conformal calibration that enables the near-automatic construction of valid predictive sets (that is, sets satisfying Eq.

Original: Subsequently, in Subsection 2.2, we provide a detailed presentation of a concrete procedure for performing conformal calibration.
Revised: Subsequently, in Subsection 2.2, we provide a detailed presentation of a concrete procedure for performing conformal calibration.

Original: In
Revised: In

Original: Subsection 2.3 we discuss the optimality of our procedure, proving that it is strictly better than any procedure that returns sets of a ﬁxed size, unlike alternative approaches.

Original: Algorithm 1 Naive Prediction Sets
Revised: Algorithm 1 Naive Prediction Sets

Original: Input: α , sorted scores scores , associated permutation of classes I , rand 1: procedure NAIVE ( α, scores, I, rand ) 2:
Revised: Input: α , sorted scores scores , associated permutation of classes I , rand 1: procedure NAIVE ( α, scores, I, rand ) 2:

Original: L ←3: while sum( scores [0 : L ]) < 1 − α do (cid:46) Stop if 1 − α probability exceeded 4:
Revised: L ←3: while sum( scores [0 : L ]) < 1 − α do (cid:46) Stop if 1 − α probability exceeded 4:

Original: L ← L +5: if rand then (cid:46) Break ties randomly (explained in Appendix C) 6:
Revised: L ← L +5: if rand then (cid:46) Break ties randomly (explained in Appendix C) 6:

Original: U ← Unif(0 , 1) 7:
Revised: U ← Unif(0 , 1) 7:

Original: V ← ( scores [0 : L ] − (1 − α )) /scores
Revised: V ← ( scores [0 : L ] − (1 − α )) /scores

Original: [ L ] (cid:46) Vectors are zero-indexed in pseudocode 8: if U ≤ V then 9:
Revised: [ L ] (cid:46) Vectors are zero-indexed in pseudocode 8: if U ≤ V then 9:

Original: L ← L −10: return I [0 : L ]
Revised: L ← L −10: return I [0 : L ]

Original: Output: The 1 − α prediction set, I [0 : L ] t ^ ccal25% 50% 75% 1− a ¢ 100% t : set size parameter e m p
Revised: Output: The 1 − α prediction set, I [0 : L ] t [Figure] ccal25% 50% 75% 1− a ¢ 100% t : set size parameter e m p

Original: i r i c a l c o v e r age (a) Conformal calibration 0.50.
Revised: i r i c a l c o v e r age ( [Figure] Conformal calibration 0.50.

Original: k reg t ^ ccal 0.
Revised: k reg t ^ ccal 0.

Original: c d f b e a class pena li z ed p r obab ili t y penalty probability
Revised: c d f b e a class pena li z ed p r obab ili t y penalty probability

Original: [Figure]
Revised: [Figure]

Revised: Subsection 2.3 we discuss the optimality of our procedure, proving that it is strictly better than any procedure that returns sets of a ﬁxed size, unlike alternative approaches.

Original: 2. C ONFORMAL CALIBRATION
Revised: 2. C ONFORMAL CALIBRATION

Original: We ﬁrst review a general technique for producing valid prediction sets.
Revised: We ﬁrst review a general technique for producing valid prediction sets.

Original: The idea is intuitive: consider a procedure that outputs a predictive set for each observation, and further suppose that this procedure has a tuning parameter τ that controls the size of the sets.
Revised: The idea is intuitive: consider a procedure that outputs a predictive set for each observation, and further suppose that this procedure has a tuning parameter τ that controls the size of the sets, as elaborated in Gupta et al.

Original: (In RAPS , τ is the cumulative sum of the sorted, penalized classiﬁer scores.)
Revised: (In RAPS , τ is the cumulative sum of the sorted, penalized classiﬁer scores.)

Original: We take a small independent conformal calibration set of data, and then choose the tuning parameter τ such that the predictive sets are large enough to achieve 1 − α coverage on this conformal calibration set.
Revised: We take a small independent conformal calibration set of data, and then choose the tuning parameter τ such that the predictive sets are large enough to achieve 1 − α coverage on this conformal calibration set.

Original: See Figure 3 for an illustration.
Revised: See Figure 3 for an illustration.

Original: This calibration step yields a choice of τ , and the resulting set is formally guaranteed to have coverage 1 − α on a future test point from the same distribution; see Theorem 1 below.
Revised: This calibration step yields a choice of τ , and the resulting set is formally guaranteed to have coverage 1 − α on a future test point from the same distribution; see Theorembelow.

Original: Formally, let ( X i , Y i ) i =1 ,...,n be an independent and identically distributed (i.i.d.)
Revised: Formally, let ( X i , Y i ) i =1 ,...,n be an independent and identically distributed (i.i.d.)

Original: set of variables that was not used for model training.
Revised: set of variables that was not used for model training.

Original: Further, let C ( x, u, τ ) : R d × [0 , 1] × R → 2 Y be a setvalued function that takes a feature vector x to a subset of the possible labels.
Revised: Further, let C ( x, u, τ ) : R d × [0 , 1] × R → 2 Y be a setvalued function that takes a feature vector x to a subset of the possible labels.

Original: The second argument u is included to allow for randomized procedures; let U 1 , . .
Revised: The second argument u is included to allow for randomized procedures; let U 1 , . .

Original: , U n be i.i.d.
Revised: , U n be i.i.d.

Original: uniform [0 , 1] random variables that will serve as the second argument for each data point.
Revised: uniform [0 , 1] random variables that will serve as the second argument for each data point.

Original: Suppose that the sets are indexed by τ such that they are nested , meaning larger values of τ lead to larger sets:
Revised: Suppose that the sets are indexed by τ such that they are nested , meaning larger values of τ lead to larger sets:

Original: [Equation]
Revised: [Equation]

Original: To ﬁnd a function that will achieve 1 − α coverage on test data, we select the smallest τ that gives at least 1 − α coverage on the conformal calibration set, with a slight correction to account for the ﬁnite sample size:
Revised: To ﬁnd a function that will achieve 1 − α coverage on test data, we select the smallest τ that gives at least 1 − α coverage on the conformal calibration set, with a slight correction to account for the ﬁnite sample size:

Original: [Equation]
Revised: [Equation]

Original: The set function C ( x, u, τ ) with this data-driven choice of τ is guaranteed to have correct ﬁnitesample coverage on a fresh test observation, as stated formally next.
Revised: The set function C ( x, u, τ ) with this data-driven choice of τ is guaranteed to have correct ﬁnitesample coverage on a fresh test observation, as stated formally next.

Original: Theorem 1 (Conformal calibration coverage guarantee) .
Revised: Theorem 1 (Conformal calibration coverage guarantee) .

Original: Suppose ( X i , Y i , U i ) i =1 ,...,n and ( X n +1 , Y n +1 , U n +1 ) are i.i.d.
Revised: Suppose ( X i , Y i , U i ) i =1 ,...,n and ( X n +1 , Y n +1 , U n +1 ) are i.i.d.

Original: and let C ( x, u, τ ) be a set-valued function satisfying the nesting property in Eq.
Revised: and let C ( x, u, τ ) be a set-valued function satisfying the nesting property in Eq.

Original: Suppose further that the sets C ( x, u, τ ) grow to include all labels for large enough τ : for all x ∈ R d , C ( x, u, τ )
Revised: Suppose further that the sets C ( x, u, τ ) grow to include all labels for large enough τ : for all x ∈ R d , C ( x, u, τ )

Original: =
Revised: =

Original: Y for some τ .
Revised: Y for some τ .

Original: Then for ˆ τ ccal deﬁned as in Eq.
Revised: Then for ˆ τ ccal deﬁned as in Eq.

Original: (3), we have the following coverage guarantee:
Revised: (3), we have the following coverage guarantee:

Original: [Equation]
Revised: [Equation]

Original: This is the same coverage property as Eq.
Revised: This is the same coverage property as Eq.

Original: (1) in the introduction, written in a more explicit manner.
Revised: (1) in the introduction, written in a more explicit manner.

Original: The result is not new—a special case ﬁrst appears in the regression setting in Papadopoulos et al.
Revised: The result is not new—a special case ﬁrst appears in the regression setting in Papadopoulos et al.

Original: (2002), and subsequent work adapts the idea to classiﬁcation (Hechtlinger et al., 2018).
Revised: (2002), and subsequent work adapts the idea to classiﬁcation (e.g., Hechtlinger et al., 2018).

Original: As a technical remark, for most families of set-valued functions C ( x, u, τ ) there is a matching upper bound:
Revised: As a technical remark, for most families of set-valued functions C ( x, u, τ ) there is a matching upper bound:

Original: [Equation]
Revised: [Equation]

Original: Roughly speaking, this will hold whenever the sets grow smoothly in τ .
Revised: Roughly speaking, this will hold whenever the sets grow smoothly in τ .

Original: See Lei et al.
Revised: See Lei et al.

Original: (2018) for a formal statement of the required conditions.
Revised: (2018) for a formal statement of the required conditions.

Original: 2. O UR METHOD
Revised: 2. O UR METHOD

Original: Conformal calibration is a powerful general idea, allowing one to achieve the coverage desideratum for any choice of sets C ( x, u, τ ) .
Revised: Conformal calibration is a powerful general idea, allowing one to achieve the coverage desideratum for any choice of sets C ( x, u, τ ) .

Original: Nonetheless, this is not yet a full solution, since the quality of the resulting prediction sets can vary dramatically depending on the design of C ( x, u, τ ) .
Revised: Nonetheless, this is not yet a full solution, since the quality of the resulting prediction sets can vary dramatically depending on the design of C ( x, u, τ ) .

Original: In particular, we recall the size and adaptiveness desiderata from Section 3—we want our uncertainty sets to be as small as possible while faithfully articulating the instance-wise uncertainty of each test point.
Revised: In particular, we recall the size and adaptiveness desiderata from Section 1—we want our uncertainty sets to be as small as possible while faithfully articulating the instance-wise uncertainty of each test point.

Original: In this section, we explicitly give our algorithm, which can be viewed as a special case of conformal calibration with the uncertainty sets C designed to extract information from CNNs.
Revised: In this section, we explicitly give our algorithm, which can be viewed as a special case of conformal calibration with the uncertainty sets C designed to extract information from CNNs.

Original: Our algorithm has three main ingredients.
Revised: Our algorithm has three main ingredients.

Original: First, for a feature vector x , the base model computes class probabilities ˆ π x ∈ R k , and we order the classes from most probable to least probable.
Revised: First, for a feature vector x , the base model computes class probabilities ˆ π x ∈ R k , and we order the classes from most probable to least probable.

Original: Then, we add a regularization term to promote small predictive sets.
Revised: Then, we add a regularization term to promote small predictive sets.

Original: Finally, we conformally calibrate the penalized prediction sets to guarantee exact coverage on future test points.
Revised: Finally, we conformally calibrate the penalized prediction sets to guarantee exact coverage on future test points.

Original: Formally, let ρ x ( y )
Revised: Formally, let ρ x ( y )

Original: =
Revised: =

Original: (cid:80) Ky (cid:48)
Revised: (cid:80) Ky (cid:48)

Original: =
Revised: =

Original: ˆ π x
Revised: ˆ π x

Original: ( y (cid:48) )
Revised: ( y (cid:48) )

Original: I { ˆ π x ( y (cid:48) )
Revised: I { ˆ π x ( y (cid:48) )

Original: >
Revised: >

Original: ˆ π x ( y ) } be the total probability mass of the set of labels that are more likely than y .
Revised: ˆ π x ( y ) } be the total probability mass of the set of labels that are more likely than y .

Original: These are all the labels that will be included before y is included.
Revised: These are all the labels that will be included before y is included.

Original: In addition, let o x ( y ) =
Revised: In addition, let o x ( y ) =

Original: |{ y (cid:48)
Revised: |{ y (cid:48)

Original: ∈ Y :
Revised: ∈ Y :

Original: ˆ π x ( y (cid:48) ) ≥ ˆ π x ( y ) }| be the ranking of y among the label based on the probabilities
Revised: ˆ π x ( y (cid:48) ) ≥ ˆ π x ( y ) }| be the ranking of y among the label based on the probabilities

Original: ˆ π .
Revised: ˆ π .

Original: For example, if y is the third most likely label, then o x ( y ) = 3 .
Revised: For example, if y is the third most likely label, then o x ( y ) = 3 .

Original: We take
Revised: We take

Original: [Equation]
Revised: [Equation]

Original: regularization [Equation] , (4) where ( z ) + denotes the positive part of z and λ, k reg ≥ 0 are regularization hyperparameters that are introduced to encourage small set sizes.
Revised: where ( z ) + denotes the positive part of z and λ, k reg ≥ 0 are regularization hyperparameters that are introduced to encourage small set sizes.

Original: See Figure 3 for a visualization of a RAPS predictive set and Appendix B for a discussion of how to select k reg and λ .
Revised: See Figure 3 for a visualization of a RAPS predictive set and Appendix B for a discussion of how to select k reg and λ .

Original: Since this is the heart of our proposal, we carefully parse each term.
Revised: Since this is the heart of our proposal, we carefully parse each term.

Original: First, the ρ x ( y ) term increases as y ranges from the most probable to least probable label, so our sets will prefer to include the y that are predicted to be the most probable.
Revised: First, the ρ x ( y ) term increases as y ranges from the most probable to least probable label, so our sets will prefer to include the y that are predicted to be the most probable.

Original: The second term, ˆ π x ( y ) · u , is a randomized term to handle the fact that the value will jump discretely with the inclusion of each new y .
Revised: The second term, ˆ π x ( y ) · u , is a randomized term to handle the fact that the value will jump discretely with the inclusion of each new y .

Original: The randomization term can never impact more than one value of y : there is at most one value of y such that y ∈ C ( x, 0 , τ ) but y / ∈ C
Revised: The randomization term can never impact more than one value of y : there is at most one value of y such that y ∈ C ( x, 0 , τ ) but y / ∈ C

Original: ( x, 1 , τ ) .
Revised: ( x, 1 , τ ) .

Original: These ﬁrst two terms can be viewed as the CDF transform after arranging the classes from most likely to least likely, randomized in the usual way to result in a continuous uniform random variable (cf.
Revised: These ﬁrst two terms can be viewed as the CDF transform after arranging the classes from most likely to least likely, randomized in the usual way to result in a continuous uniform random variable (cf.

Original: Romano et al., 2020).
Revised: Romano et al., 2020).

Original: We discuss randomization further in Appendix C.
Revised: We discuss randomization further in Appendix C.

Original: Lastly, the regularization promotes small set sizes: for values of y that occur farther down the ordered list of classes, the term λ · ( o x ( y ) − k reg ) + makes that value of y require a higher value of τ before it is included in the predictive set.
Revised: Lastly, the regularization promotes small set sizes: for values of y that occur farther down the ordered list of classes, the term λ · ( o x ( y ) − k reg ) + makes that value of y require a higher value of τ before it is included in the predictive set.

Original: For example, if k reg = 5 , then the sixth most likely value of y has an extra penalty of size λ , so it will never be included until τ exceeds ρ x ( y ) +
Revised: For example, if k reg = 5 , then the sixth most likely value of y has an extra penalty of size λ , so it will never be included until τ exceeds ρ x ( y ) +

Original: ˆ π x ( y ) · u + λ , whereas it enters when τ exceeds ρ x ( y )
Revised: ˆ π x ( y ) · u + λ , whereas it enters when τ exceeds ρ x ( y )

Original: +
Revised: +

Original: ˆ π x ( y ) · u in the nonregularized version.
Revised: ˆ π x ( y ) · u in the nonregularized version.

Original: Our method has the following coverage property:
Revised: Our method has the following coverage property:

Original: Proposition 1 ( RAPS coverage guarantee) .
Revised: Proposition 1 ( RAPS coverage guarantee) .

Original: Suppose ( X i , Y i , U i ) i =1 ,...,n and ( X n +1 , Y n +1 , U n +1 ) are i.i.d.
Revised: Suppose ( X i , Y i , U i ) i =1 ,...,n and ( X n +1 , Y n +1 , U n +1 ) are i.i.d.

Original: and let C ∗ ( x, u, τ ) be deﬁned as in Eq.
Revised: and let C ∗ ( x, u, τ ) be deﬁned as in Eq.

Original: Suppose further that ˆ π x ( y ) > 0 for all x and y .
Revised: Suppose further that ˆ π x ( y ) > 0 for all x and y .

Original: Then for ˆ τ ccal deﬁned as in Eq.
Revised: Then for ˆ τ ccal deﬁned as in Eq.

Original: (3), we have the following coverage guarantee:
Revised: (3), we have the following coverage guarantee:

Original: [Equation]
Revised: [Equation]

Original: Note that the ﬁrst inequality is a corollary of Theorem 1, and the second inequality is a special case of the remark in Section 2.1.
Revised: Note that the ﬁrst inequality is a corollary of Theorem 1, and the second inequality is a special case of the remark in Section 2.1.

Original: The restriction that ˆ π x ( y ) > 0 is not necessary for the ﬁrst inequality.
Revised: The restriction that ˆ π x ( y ) > 0 is not necessary for the ﬁrst inequality.

Original: 2. O PTIMALITY CONSIDERATIONS

Original: In our experiments, the procedure that always returns the ﬁve most likely classes often has coverage above 90%, yet APS produces very large sets on average (see Table 1).

Original: To explain this phenomenon, note that small probability estimates (say, after the tenth most likely score) are dominated by noise.

Original: This leads to a permutation problem of the unlikely classes, whose ordering is determined mostly by random chance.

Original: If 1% of the true classes from the calibration set are deep in the tail due to the permutation problem, APS will choose large 99% predictive sets; see Figure 2.

Original: The inclusion of

Original: Input: α ; n tuples ( scores i , I i , y i ) ni
Revised: Input: α ; n tuples ( scores i , I i , y i ) ni

Original: =1 containing sorted scores, the associated permutation of indexes, and labels for each example in the calibration set; k reg ; λ ; rand 1: procedure RAPSC ( α , ( scores i , I i , y i , k reg ) ni
Revised: =1 containing sorted scores, the associated permutation of indexes, and labels for each example in the calibration set; k reg ; λ ; rand 1: procedure RAPSC ( α , ( scores i , I i , y i , k reg ) ni

Original: =1 , λ ) 2: for i ∈ { 1 , · · · , n } do 3:
Revised: =1 , λ ) 2: for i ∈ { 1 , · · · , n } do 3:

Original: L i ← { j : I i [ j ] =
Revised: L i ← { j : I i [ j ] =

Original: y i } 4:
Revised: y i } 4:

Original: E i ← Σ L i j =0 scores i [ j ] + λ
Revised: E i ← Σ L i j =0 scores i [ j ] + λ

Original: ( L i − k reg + 1) + 5: if rand then 6:
Revised: ( L i − k reg + 1) + 5: if rand then 6:

Original: U ∼ Unif(0 , 1) 7:
Revised: U ∼ Unif(0 , 1) 7:

Original: E i ← E i − scores i
Revised: E i ← E i − scores i

Original: [ L i ] + U ∗ scores
Revised: [ L i ] + U ∗ scores

Original: i [ L i ] 8: ˆ τ ccal ← the (cid:100) (1 − α )(1 + n ) (cid:101) largest value in { E i } ni = 9: return ˆ τ ccal
Revised: i [ L i ] 8: ˆ τ ccal ← the (cid:100) (1 − α )(1 + n ) (cid:101) largest value in { E i } ni = 9: return ˆ τ ccal

Original: Output: The generalized quantile, ˆ τ ccal (cid:46) The value in Eq.
Revised: Output: The generalized quantile, ˆ τ ccal (cid:46) The value in Eq.

Original: Algorithm
Revised: Algorithm

Original: Input: α , sorted scores and the associated permutation of classes
Revised: Input: α , sorted scores and the associated permutation of classes

Original: I for a test-time example, ˆ τ ccal from Algorithm 2, k reg , λ , rand 1: procedure RAPS ( α, scores, I, ˆ τ ccal , k reg , λ, rand ) 2:
Revised: I for a test-time example, ˆ τ ccal from Algorithm 2, k reg , λ , rand 1: procedure RAPS ( α, scores, I, ˆ τ ccal , k reg , λ, rand ) 2:

Original: L ← | j ∈ Y : Σ ji =0 scores [ i ] + λ
Revised: L ← | j ∈ Y : Σ ji =0 scores [ i ] + λ

Original: ( L − k reg )
Revised: ( L − k reg )

Original: + ≤ ˆ τ ccal | +3:
Revised: + ≤ ˆ τ ccal | +3:

Original: V ← (ˆ τ ccal − Σ L − 1 i =

Revised: V ← (ˆ τ ccal − Σ L − 1 i =

Original: scores [ i ] − λ
Revised: scores [ i ] − λ

Original: ( L − k reg )
Revised: ( L − k reg )

Original: +
Revised: +

Original: + scores [ L − 1]) /scores
Revised: + scores [ L − 1]) /scores

Original: [ L − 1] 4: if rand & V ≤ U ∼ Unif(0 , 1) then 5:
Revised: [ L − 1] 4: if rand & V ≤ U ∼ Unif(0 , 1) then 5:

Original: L ← L −6: return C
Revised: L ← L −6: return C

Original: = I [0 : L ] (cid:46)
Revised: = I [0 : L ] (cid:46)

Original: The L most likely classes
Revised: The L most likely classes

Original: Output: The 1 − α conﬁdence set, C (cid:46)
Revised: Output: The 1 − α conﬁdence set, C (cid:46)

Original: The set in Eq.
Revised: The set in Eq.

Revised: 2. O PTIMALITY CONSIDERATIONS

Revised: In our experiments, the procedure that always returns the ﬁve most likely classes often has coverage above 90%, yet APS produces very large sets on average (see Table 1).

Revised: To explain this phenomenon, note that small probability estimates (say, after the tenth most likely score) are dominated by noise.

Revised: This leads to a permutation problem of the unlikely classes, whose ordering is determined mostly by random chance.

Revised: If 1% of the true classes from the calibration set are deep in the tail due to the permutation problem, APS will choose large 99% predictive sets; see Figure 2.

Original: (4) the RAPS regularization effectively eliminates this behavior; see Figure 4.
Revised: The inclusion of the RAPS regularization effectively eliminates this behavior; see Figure 4.

Original: To complement these experimental results, we now formally prove that RAPS with the correct regularization parameters will always dominate the simple procedure that returns a ﬁxed set size.
Revised: To complement these experimental results, we now formally prove that RAPS with the correct regularization parameters will always dominate the simple procedure that returns a ﬁxed set size.

Original: As we show in Appendix B, these parameters are easy to select and the method is not sensitive to their values.
Revised: As we show in Appendix B, these parameters are easy to select and the method is not sensitive to their values.

Original: For a feature vector x , let ˆ y (1) ( x ) be the label with the highest predicted probability, ˆ y (2) ( x ) be the label with the second highest predicted probability, and so on.
Revised: For a feature vector x , let ˆ y (1) ( x ) be the label with the highest predicted probability, ˆ y (2) ( x ) be the label with the second highest predicted probability, and so on.

Original: We deﬁne the top-k predictive sets to be { ˆ y (1) ( x ) , . . .
Revised: We deﬁne the top-k predictive sets to be { ˆ y (1) ( x ) , . . .

Original: , ˆ y ( k ) ( x ) } .
Revised: , ˆ y ( k ) ( x ) } .

Original: Proposition 2 ( RAPS dominates top-k sets) .
Revised: Proposition 2 ( RAPS dominates top-k sets) .

Original: Suppose ( X i , Y i , U i ) i =1 ,...,n and ( X n +1 , Y n +1 , U n +1 ) are i.i.d.
Revised: Suppose ( X i , Y i , U i ) i =1 ,...,n and ( X n +1 , Y n +1 , U n +1 ) are i.i.d.

Original: draws.
Revised: draws.

Original: Let k ∗ be the smallest k such that the top-k predictive sets have coverage at least (cid:100) ( n + 1)(1 − α ) (cid:101) /n
Revised: Let k ∗ be the smallest k such that the top-k predictive sets have coverage at least (cid:100) ( n + 1)(1 − α ) (cid:101) /n

Original: on the conformal calibration points ( X i , Y i ) i =1 ,...,n .
Revised: on the conformal calibration points ( X i , Y i ) i =1 ,...,n .

Original: Take C ∗ ( x, u, τ ) as in Eq.
Revised: Take C ∗ ( x, u, τ ) as in Eq.

Original: (4) with  k reg = k ∗ andany λ = 1 .
Revised: (4) with any k reg ≤ k ∗ and λ = 1 .

Original: Then with ˆ τ ccal chosen as in Eq.
Revised: Then with ˆ τ ccal chosen as in Eq.

Original: (3), we have
Revised: (3), we have

Original: C

Original: [Equation]
Revised: [Equation]

Original: [Equation] n + [Equation] , U n
Revised: τ ccal ) ⊂ {

Original: [Equation]
Revised: [Table]

Original: , ˆ τ ccal ) ⊂ { ˆ y (1) ( x ) , . . .
Revised: y (1) ( x ) , . . .

Original: , ˆ y ( k ∗ ) ( x ) } .
Revised: , ˆ y ( k ∗ ) ( x ) } .

Original: In words, the RAPS procedure  will be strictly better than the top- k procedure in the sense that it has smaller sets while maintaining the desired coverage level.
Revised: In words, the RAPS procedure with heavy regularization will be strictly better than the top- k procedure in the sense that it has smaller sets while maintaining the desired coverage level.

Original: This is not true of either the naive baseline or the APS procedure; Figure 4 shows that these two procedures sometimes return predictive sets with size much larger than k ∗ .
Revised: This is not true of either the naive baseline or the APS procedure; Figure 4 shows that these two procedures sometimes return predictive sets with size much larger than k ∗ .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we report on experiments that study the performance of the predictive sets from naive , APS , and RAPS , evaluating each based on the three desiderata above. 
Revised: In this section we report on experiments that study the performance of the predictive sets from naive , APS , and RAPS , evaluating each based on the three desiderata above. We begin with a brief [Table]

Original: We begin with a brief preview of the experiments.
Revised: [Table] [Table] preview of the experiments.

Original: In Experiment 1 , we evaluated naive , APS , and RAPS on ImagenetVal.
Revised: In Experiment 1 , we evaluated naive , APS , and RAPS on ImagenetVal.

Original: Both APS and RAPS provided exact coverage, while naive sets had coverage slightly below the speciﬁed level.
Revised: Both APS and RAPS provided exact coverage, while naive sets had coverage slightly below the speciﬁed level.

Original: APS had larger sets on average than naive and RAPS .
Revised: APS had larger sets on average than naive and RAPS .

Original: RAPS had a much smaller average set size than APS and naive .
Revised: RAPS had a much smaller average set size than APS and naive .

Original: In Experiment 2 , we repeated Experiment 1 on
Revised: In Experiment 2 , we repeated Experiment 1 on

Original: Imagenet-V2, and the conclusions still held.
Revised: Imagenet-V2, and the conclusions still held.

Original: In Experiment 3 , we produced histograms of set sizes for naive , APS , and RAPS for several different values of λ , illustrating a simple tradeoff between set size and adaptiveness.
Revised: In Experiment 3 , we produced histograms of set sizes for naive , APS , and RAPS for several different values of λ , illustrating a simple tradeoff between set size and adaptiveness.

Original: Finally, in Experiment 4 , we computed histograms of RAPS sets stratiﬁed by image difﬁculty, showing that RAPS sets are smaller for easier images than for difﬁcult ones.[Table]
Revised: Finally, in Experiment 4 , we computed histograms of RAPS sets stratiﬁed by image difﬁculty, showing that RAPS sets are smaller for easier images than for difﬁcult ones.

Original: 5.52 ShufﬂeNet . In our experiments, we use nine standard, pretrained Imagenet classiﬁers from the torchvision open source repository (Paszke et al., 2019) with standard normalization, resize, and crop parameters, facilitating reproducibility and avoiding bias.
Revised: In our experiments, we use nine standard, pretrained Imagenet classiﬁers from the torchvision open source repository (Paszke et al., 2019) with standard normalization, resize, and crop parameters, facilitating reproducibility and avoiding bias.

Original: Before applying naive , APS , or RAPS , we calibrated the classiﬁers using the standard temperature scaling/Platt scaling procedure described in Guo et al.
Revised: Before applying naive , APS , or RAPS , we calibrated the classiﬁers using the standard temperature scaling/Platt scaling procedure described in Guo et al.

Original: Thereafter, naive , APS , and RAPS were applied, with RAPS using parameters k reg = 5 and λ = 0 .
Revised: Thereafter, naive , APS , and RAPS were applied, with RAPS using parameters k reg = 5 and λ = 0 .

Original: We used the randomized versions of these algorithms—see Appendix C for a discussion.
Revised: We used the randomized versions of these algorithms—see Appendix C for a discussion.

Original: 3. E XPERIMENT 1: COVERAGE VS SET SIZE ON I MAGENET
Revised: 3. E XPERIMENT 1: COVERAGE VS SET SIZE ON I MAGENET

Original: In this experiment, we calculated the coverage and mean set size of each procedure for two different choices of α .
Revised: In this experiment, we calculated the coverage and mean set size of each procedure for two different choices of α .

Original: Over 100 trials, we randomly sampled two subsets of Imagenet-Val: one conformal calibration subset of size 20K and one evaluation subset of size 20K.
Revised: Over 100 trials, we randomly sampled two subsets of Imagenet-Val: one conformal calibration subset of size 20K and one evaluation subset of size 20K.

Original: The median-of-means over trials for both coverage and set size are reported in Table 1.
Revised: The median-of-means over trials for both coverage and set size are reported in Table 1.

Original: Figure 2 illustrates the performances of naive , APS , and RAPS ; RAPS has much smaller sets than both naive and APS , while achieving exact coverage.
Revised: Figure 2 illustrates the performances of naive , APS , and RAPS ; RAPS has much smaller sets than both naive and APS , while achieving exact coverage.

Original: 3. E XPERIMENT 2: COVERAGE VS SET SIZE ON I MAGENET -V 
Revised: 3. E XPERIMENT 2: COVERAGE VS SET SIZE ON I MAGENET -V

Original: The same procedure as Experiment 1 was repeated on Imagenet-V2, with exactly the same normalization, resize, and crop parameters.
Revised: The same procedure as Experiment 1 was repeated on Imagenet-V2, with exactly the same normalization, resize, and crop parameters.

Original: The size of the calibration and evaluation sets was 5K, since
Revised: The size of the calibration and evaluation sets was 5K, since

Original: Imagenet-V2 is a smaller dataset.
Revised: Imagenet-V2 is a smaller dataset.

Original: The result shows that our method can still provide exact coverage under a signiﬁcant distribution shift, as long as the conformal calibration set comes from the new distribution.
Revised: The result shows that our method can still provide exact coverage under a signiﬁcant distribution shift, as long as the conformal calibration set comes from the new distribution.

Original: The variance of the coverage is higher due to the smaller calibration and test set sizes
Revised: The variance of the coverage is higher due to the smaller calibration and test set sizes.

Original: . 3. E XPERIMENT 3: S ET SIZES OF NAIVE , APS , AND RAPS ON I MAGENET
Revised: 3. E XPERIMENT 3: S ET SIZES OF NAIVE , APS , AND RAPS ON I MAGENET

Original: We demonstrate that regularization leads to smaller set sizes in more detail.
Revised: We demonstrate that regularization leads to smaller set sizes in more detail.

Original: For three values of λ , we collected the set sizes produced by each of naive , APS , and RAPS and plotted them as separate histograms on the same axis.
Revised: For three values of λ , we collected the set sizes produced by each of naive , APS , and RAPS and plotted them as separate histograms on the same axis.

Original: The left panels of Figure 4 display the results of this experiment.
Revised: The left panels of Figure 4 display the results of this experiment.

Original: As λ grows, the average set size decreases.
Revised: As λ grows, the average set size decreases

Original:  3. E XPERIMENT 4: A DAPTIVENESS OF RAPS ON I MAGENET
Revised: . 3. E XPERIMENT 4: A DAPTIVENESS OF RAPS ON I MAGENET

Original: Lastly, we show that RAPS sets are smaller for easy images than hard ones, addressing the adaptiveness desideratum.
Revised: Lastly, we show that RAPS sets are smaller for easy images than hard ones, addressing the adaptiveness desideratum.

Original: We stratiﬁed the evaluation set into easy, medium, and hard images.
Revised: We stratiﬁed the evaluation set into easy, medium, and hard images.

Original: Easy images are instances the classiﬁer got correct.
Revised: Easy images are instances the classiﬁer got correct.

Revised: Medium difﬁculty images are examples where the

Original: Medium difﬁculty images are examples where the classiﬁer’s second or third highest classes were correct.
Revised: [Table] [Figure] [Figure] classiﬁer’s second or third highest classes were correct.

Original: Hard images are those remaining.
Revised: Hard images are those remaining.

Original: Then, we ran RAPS over each of these strata and plotted their set sizes as histograms in the right panels of Figure 4.
Revised: Then, we ran RAPS over each of these strata and plotted them as violins in the right panels of Figure 4.

Original: When λ is small, RAPS allows sets to be large.
Revised: When λ is small, RAPS allows sets to be large.

Original: But when λ is very large, RAPS essentially clips sets to be a maximum of size 5.
Revised: But when λ is very large, RAPS essentially clips sets to be a maximum of size 5.

Original: Experiments 3 and 4 together illustrate the tradeoff between the adaptiveness and size desiderata: as the average set size decreases, the RAPS procedure truncates sets larger than the smallest ﬁxed set that provides coverage.
Revised: Experiments 3 and 4 together illustrate the tradeoff between the adaptiveness and size desiderata: as the average set size decreases, the RAPS procedure truncates sets larger than the smallest ﬁxed set that provides coverage.

Original: Loosely speaking, this happens because hard examples are undercovered; since large sets are penalized, the conformal calibration procedure must increase set sizes on easy examples to ensure the 1 − α coverage guarantee is satisﬁed over the dataset as a whole.However, the size only increases for a small fraction of easy images, since they [Table]
Revised: Loosely speaking, this happens because hard examples are undercovered; since large sets are penalized, the conformal calibration procedure must increase set sizes on easy examples to ensure the 1 − α coverage guarantee is satisﬁed over the dataset as a whole.

Original: [Figure] [Figure] are much more common than hard ones. It also only increases by a small amount, since  the second most likely class will have non-negligible mass.
Revised: However, the size only increases slightly for a small fraction of easy images, since they are more common than hard ones, and the total probability mass can often exceed ˆ τ ccal by including only one more class.

Original: The difference is barely noticeable in Figure [Figure]
Revised: We further explore this experimentally in Tables 4 and 5 (see Appendix D).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Our method enables a researcher to take any base classiﬁer and return predictive sets guaranteed to achieve a pre-speciﬁed error level, such as 90%, while retaining small average size.
Revised: Our method enables a researcher to take any base classiﬁer and return predictive sets guaranteed to achieve a pre-speciﬁed error level, such as 90%, while retaining small average size.

Original: It is simple to deploy, so it is an attractive, automatic way to quantify the uncertainty of image classiﬁers—an essential task in such settings as medical diagnostics, self-driving vehicles, and ﬂagging dangerous internet content.
Revised: It is simple to deploy, so it is an attractive, automatic way to quantify the uncertainty of image classiﬁers—an essential task in such settings as medical diagnostics, self-driving vehicles, and ﬂagging dangerous internet content.

Original: Predictive sets in computer vision have many further uses, since they systematically identify hard test-time examples.
Revised: Predictive sets in computer vision have many further uses, since they systematically identify hard test-time examples.

Original: Finding such examples is useful in active learning where one only has resources to label a small number of points.
Revised: Finding such examples is useful in active learning where one only has resources to label a small number of points.

Original: In a different direction, one can improve efﬁciency of a classiﬁer by using a cheap classiﬁer outputting a prediction set ﬁrst, and an expensive one only when the cheap classiﬁer outputs a large set (a cascade ; see, e.g., Li et al. 2015).
Revised: In a different direction, one can improve efﬁciency of a classiﬁer by using a cheap classiﬁer outputting a prediction set ﬁrst, and an expensive one only when the cheap classiﬁer outputs a large set (a cascade ; see, e.g., Li et al. 2015).

Original: Finally, one can use predictive sets during model development to identify failure cases and outliers.
Revised: Finally, one can use predictive sets during model development to identify failure cases and outliers.

Original: This helps probe the model’s weaknesses, suggesting strategies for improving its performance.
Revised: This helps probe the model’s weaknesses, suggesting strategies for improving its performance.

Original: A P ROOFS
Revised: A P ROOFS

Original: Theorem 1.
Revised: Theorem 1.

Original: Let s ( x, u, y ) =
Revised: Let s ( x, u, y ) =

Original: inf τ { y ∈ C ( x, u, τ ) }, and let s i
Revised: inf τ { y ∈ C ( x, u, τ ) } and let s i

Original: =
Revised: =

Original: s ( X i , U
Revised: s ( X i , U

Original: i , Y i ) for i = 1 , . .
Revised: i , Y i ) for i = 1 , . .

Original: . , n .
Revised: . , n .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: because C ( x, u, τ ) is a ﬁnite set growing in τ by the assumption in Eq.
Revised: because C ( x, u, τ ) is a ﬁnite set growing in τ by the assumption in Eq.

Original: Thus, {
Revised: Thus, {

Revised: [Equation]

Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Original: { i
Revised: s i

Original: [Equation]
Revised: [Equation]

Original:  Considering the left expression, the inﬁmum over τ of the set on the left hand side is the (cid:100) (1 − α )( n + 1) (cid:101) smallest value of the s i , so this is the value of ˆ τ ccal .
Revised: n [Equation] Considering the left expression, the inﬁmum over τ of the set on the left hand side is the (cid:100) (1 − α )( n + 1) (cid:101) smallest value of the s i , so this is the value of ˆ τ ccal .

Original: Since s 1 , . . .
Revised: Since s 1 , . . .

Original: , s n , s ( X n +1 , U n +1 , Y n +1 ) are exchangeable random variables, |{ i : s ( X n +1 , U n +1 , Y n +1 )
Revised: , s n , s ( X n +1 , U n +1 , Y n +1 ) are exchangeable random variables, |{ i : s ( X n +1 , U n +1 , Y n +1 )

Original: >
Revised: >

Original: s i }| is stochastically dominated by the discrete uniform distribution on { 0 , 1 , . . . , n } .
Revised: s i }| is stochastically dominated by the discrete uniform distribution on { 0 , 1 , . . . , n } .

Original: We thus have that
Revised: We thus have that

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: The lower bound follows from Theorem 1.
Revised: The lower bound follows from Theorem 1.

Original: To prove the upper bound, using the result from Theorem 2.2 of Lei et al.
Revised: To prove the upper bound, using the result from Theorem 2.2 of Lei et al.

Original: it sufﬁces to show that the variables s ( X i ,
Revised: it sufﬁces to show that the variables s ( X i ,

Original: U i , Y i )
Revised: U i , Y i )

Original: =
Revised: =

Original: inf { τ :
Revised: inf { τ :

Original: Y i ∈ C
Revised: Y i ∈ C

Original: ( X i , U i , τ ) } are almost surely distinct.
Revised: ( X i , U i , τ ) } are almost surely distinct.

Original: To this end, note that that
Revised: To this end, note that that

Original: [Equation]
Revised: [Equation]

Original: and due to the middle term of the sum, these values are distinct almost surely provided ˆ π X
Revised: and due to the middle term of the sum, these values are distinct almost surely provided ˆ π X

Original: i
Revised: i

Original: ( Y i ) > 0 .
Revised: ( Y i ) > 0 .

Revised: k reg | λ

Revised: [Table]

Revised: [Table]

Original: Proposition 2.
Revised: Proposition 2.

Original: We ﬁrst show that ˆ τ ccal ≤ 1  .
Revised: We ﬁrst show that ˆ τ ccal ≤ 1 + k ∗ − k reg .

Original: Note that since at least (cid:100) (1 − α )( n +1) (cid:101) of the conformal calibration points are covered by a set of size k ∗ , at least (cid:100) (1 − α )( n + 1) (cid:101) of the E i in Algorithmare less than or equal to 1.
Revised: Note that since at least (cid:100) (1 − α )( n + 1) (cid:101) of the conformal calibration points are covered by a set of size k ∗ , at least (cid:100) (1 − α )( n + 1) (cid:101) of the E i in Algorithm 2 are less than or equal to 1 + k ∗ − k reg .

Original: Thus, by the deﬁnition of ˆ τ ccal , we have that it is less than or equal to 1.
Revised: Thus, by the deﬁnition of ˆ τ ccal , we have that it is less than or equal to 1 + k ∗ − k reg .

Original: Then, note that by the deﬁnition of C ∗ in Eq.
Revised: Then, note that by the deﬁnition of C ∗ in Eq.

Original: (4), we have that |C ∗ ( X n +1 , U n +1 , ˆ τ ccal ) | ≤ k reg .
Revised: (4), we have that [Equation]

Original: as long as ˆ τ ccal ≤ 1 , since for the k reg + 1 most likely class, the sum will exceed λ ≥ 1 ≥ ˆ τ ccal .
Revised: as long as ˆ τ ccal ≤ 1+ k ∗ − k reg , since for the k ∗ +1 most likely class, the sum in Eq.

Original:  k reg | λ
Revised: (4) will exceed λ · (1 + k ∗ − k reg ) =

Original: [Table] [Table]
Revised: (1 + k ∗ − k reg ) ≥ ˆ τ ccal , and so the k ∗ + 1 class will not be in the set.

Original: B P ICKING k reg AND λ
Revised: B P ICKING k reg AND λ

Original: While any value of the tuning parameters λ and k reg lead to exact coverage (Proposition 1), some values will lead to smaller sets.
Revised: While any value of the tuning parameters λ and k reg lead to exact coverage (Proposition 1), some values will lead to smaller sets.

Original: We proved that with k reg =
Revised: We proved that with k reg =

Original: k ∗ and λ ≥ 1 , RAPS outputs sets no larger than k ∗ (Proposition 2).
Revised: k ∗ and λ ≥ 1 , RAPS outputs sets no larger than k ∗ (Proposition 2).

Original: In practice, the sets are often much smaller, as we show across our experiments.
Revised: In practice, the sets are often much smaller, as we show across our experiments.

Original: Furthermore, in our experiments we chose k reg = 5 and λ = 0 .
Revised: Furthermore, in our experiments we chose k reg = 5 and λ = 0 .

Original: 2 , which achieved strong results for many models (which have varying accuracy) and choices of the coverage level, illustrating our method’s robustness to these parameter choices.
Revised: 2 , which achieved strong results for many models (which have varying accuracy) and choices of the coverage level, illustrating our method’s robustness to these parameter choices.

Original: We report on a wider set of parameter values in Table 3, and ﬁnd that the performance is strong for many parameter choices; indeed, our ﬁxed choice of parameters presented in Table 1 is not even the optimal set of parameters, it was just taken as a reasonably good representative value.
Revised: We report on a wider set of parameter values in Table 3, and ﬁnd that the performance is strong for many parameter choices; indeed, our ﬁxed choice of parameters presented in Table 1 is not even the optimal set of parameters, it was just taken as a reasonably good representative value.

Original: To build intuition for the role of k reg , consider the following scenarios where we set λ
Revised: To build intuition for the role of k reg , consider the following scenarios where we set λ

Original: = 1 .
Revised: = 1 .

Original: If k reg ≤ k ∗ − 2 , the sets output by RAPS will always be of size k ∗ or k ∗ − 1 ; these sets are still small, but not adaptive.
Revised: If k reg ≤ k ∗ − 2 , the sets output by RAPS will always be of size k ∗ or k ∗ − 1 ; these sets are still small, but not adaptive.

Original: If k reg =
Revised: If k reg =

Original: k ∗ + m , m ∈ N , then RAPS would only penalize sets of size k ∗ + m and above, losing the guarantee described in Proposition 2.
Revised: k ∗ + m , m ∈ N , then RAPS would only penalize sets of size k ∗ + m and above, losing the guarantee described in Proposition 2.

Original: Therefore, it is natural to pick k reg ≈ k ∗ .
Revised: Therefore, it is natural to pick k reg ≈ k ∗ .

Original: For example, if the desired coverage is 90% and the top-5 accuracy of the model is around 90%, we would pick k reg = 5 .
Revised: For example, if the desired coverage is 90% and the top-5 accuracy of the model is around 90%, we would pick k reg = 5 .

Original: Turning to λ , RAPS with a larger λ tends to cover fewer difﬁcult examples and more easy ones.
Revised: Turning to λ , RAPS with a larger λ tends to cover fewer difﬁcult examples and more easy ones.

Original: Small values of λ like 0 .
Revised: Small values of λ like 0 .

Original: 01 will work reasonably well in many settings, which makes them a attractive default choice, but tuning both k ∗ and λ can lead to even better performance.
Revised: 01 will work reasonably well in many settings, which makes them a attractive default choice, but tuning both k ∗ and λ can lead to even better performance.

Original: C R ANDOMIZED PREDICTORS
Revised: C R ANDOMIZED PREDICTORS

Original: The reader may wonder why we choose to use a randomized procedure.
Revised: The reader may wonder why we choose to use a randomized procedure.

Original: The randomization is needed to achieve 1 − α coverage exactly, which we will explain via an example.
Revised: The randomization is needed to achieve 1 − α coverage exactly, which we will explain via an example.

Original: Note that the randomization is of little practical importance, since the predictive set output by the randomized procedure will differ from the that of the non-randomized procedure by at most one element.
Revised: Note that the randomization is of little practical importance, since the predictive set output by the randomized procedure will differ from the that of the non-randomized procedure by at most one element.

Original: Turning to an example, assume for a particular input image we expect a set of size k to have 91% coverage, and a set of size k − 1 to have 89% coverage.
Revised: Turning to an example, assume for a particular input image we expect a set of size k to have 91% coverage, and a set of size k − 1 to have 89% coverage.

Original: In order to achieve our desired coverage of 90% , we randomly choose size k or k − 1 with equal probability.
Revised: In order to achieve our desired coverage of 90% , we randomly choose size k or k − 1 with equal probability.

Original: In general, the probabilities will not be equal, but rather chosen so the weighted average of the two coverages is exactly 90% .
Revised: In general, the probabilities will not be equal, but rather chosen so the weighted average of the two coverages is exactly 90% .

Original: If a user of our method desires deterministic sets, it is easy to turn off this randomization with a single ﬂag, resulting in slightly conservative sets. 
Revised: If a user of our method desires deterministic sets, it is easy to turn off this randomization with a single ﬂag, resulting in slightly conservative sets. λ = [Table] λ =

Revised: [Table]

Revised: λ = 0 .

Revised: [Table]

Revised: λ = 0 .

Revised: [Table]

Revised: λ = 0 .λ

Revised: = 0 .λ =

Revised: [Table]

Revised: λ = 0 is the APS procedure).

Revised: The desired coverage level is 90%.

Revised: The ‘difﬁculty’ is the ranking of the true class’s estimated probability.

Revised: [Table]

Revised: D C OVERAGE AND SIZE CONDITIONAL ON IMAGE DIFFICULTY

Revised: In order to probe the adaptiveness properties of APS and RAPS we stratiﬁed coverage and size by image difﬁculty (the position of the true label in the list of most likely to least likely classes, based on the classiﬁer predictions) in Table 4.

Revised: With increasing λ , coverage decreases for more difﬁcult images and increases for easier ones.

Revised: In the most difﬁcult regime, even though APS can output large sets, those sets still rarely contain the true class.

Revised: This suggests regularization is a sensible way to stabilize the sets.

Revised: As a ﬁnal word on Table 4, notice that as λ increases, coverage improves for the more common medium-difﬁculty examples, although not for very rare and difﬁcult ones.

Revised: In Table 5, we stratiﬁed coverage and image difﬁculty by the set size produced by RAPS .

Revised: Turning our attention to the λ = 0 column, we see that when APS outputs a set of size 101 − 1000 , APS has higher than 1 − α coverage.

Revised: If the probabilities output by the CNN were correct, APS would have 1 − α coverage in each stratum, so this indicates the probabilities output by the CNN are wrong.

Revised: This provides further evidence for the the permutation problem discussed in the main text.

Revised: The APS procedure still achieves coverage by outputting excessively large sets for certain examples (as in the last row of the table).

Revised: Alternatively, RAPS can be used to regularize the set sizes—for λ = .

Revised: 001 to λ = .

Revised: 01 the coverage stratiﬁed by set size is more balanced.