Original: T RADING C OVERAGE FOR P RECISION : C ONFORMAL P REDICTION WITH L IMITED F ALSE D ISCOVERIES
Revised: T RADING C OVERAGE FOR P RECISION : C ONFORMAL P REDICTION WITH L IMITED F ALSE P OSITIVES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we develop a new approach to conformal prediction in which we aim to output a precise set of promising prediction candidates that is guaranteed to contain a limited number of incorrect answers.
Revised: In this paper, we develop a new approach to conformal prediction in which we aim to output a precise set of promising prediction candidates that is guaranteed to contain a limited number of incorrect answers.

Original: Standard conformal prediction provides the ability to adapt to model uncertainty by constructing a calibrated candidate set in place of a single prediction, with guarantees that the set contains the correct answer with high probability.
Revised: Standard conformal prediction provides the ability to adapt to model uncertainty by constructing a calibrated candidate set in place of a single prediction, with guarantees that the set contains the correct answer with high probability.

Original: In order to obey this coverage property, however, conformal sets can often become inundated with noisy candidates—which can render them unhelpful in practice.
Revised: In order to obey this coverage property, however, conformal sets can often become inundated with noisy candidates—which can render them unhelpful in practice.

Original: This is particularly relevant to large-scale settings where the cost (monetary or otherwise) of false positives is substantial, such as for in-silico screening for drug discovery, where any positively identiﬁed molecular compound is then manufactured and tested.
Revised: This is particularly relevant to large-scale settings where the cost (monetary or otherwise) of false positives is substantial, such as for in-silico screening for drug discovery, where any positively identiﬁed molecular compound is then manufactured and tested.

Original: We propose to trade coverage for precision by enforcing that the presence of incorrect candidates in the predicted conformal sets (i.e., the total number of false discoveries) is bounded according to a user-speciﬁed tolerance.
Revised: We propose to trade coverage for precision by enforcing that the presence of incorrect candidates in the predicted conformal sets (i.e., the total number of false positives) is bounded according to a user-speciﬁed tolerance.

Original: Subject to this constraint, our algorithm then optimizes for a generalized notion of set coverage (i.e., the true discovery rate) that allows for any number of true answers for a given query (including zero).
Revised: Subject to this constraint, our algorithm then optimizes for a generalized notion of set coverage (i.e., the true positive rate) that allows for any number of true answers for a given query (including zero).

Original: We demonstrate the effectiveness of this approach across a number of classiﬁcation tasks in natural language processing, computer vision, and computational chemistry.
Revised: We demonstrate the effectiveness of this approach across a number of classiﬁcation tasks in natural language processing, computer vision, and computational chemistry.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: For many classiﬁcation problems, returning a set of plausible responses instead of  single prediction is a useful way of representing uncertainty (Gammerman and Vovk, 2007; Lei, 2014; Bates et al., 2020).
Revised: For many classiﬁcation problems, returning a set of plausible responses instead of a single prediction is a useful way of representing uncertainty (Gammerman and Vovk, 2007; Lei, 2014; Bates et al., 2020).

Original: Aligned with this goal, conformal prediction (Vovk et al., 2005) is an increasingly popular method for creating conﬁdent prediction sets that provably contain the correct answer with high probability.
Revised: Aligned with this goal, conformal prediction (Vovk et al., 2005) is an increasingly popular method for creating conﬁdent prediction sets that provably contain the correct answer with high probability.

Original: Unfortunately, these guarantees do not come for free; in order to achieve proper coverage on difﬁcult tasks, conformal predictors are often unable to rule out an overwhelming number of candidates—making their prediction sets large and inefﬁcient (Angelopoulos et al., 2021c).
Revised: Unfortunately, these guarantees do not come for free; in order to achieve proper coverage on difﬁcult tasks, conformal predictors are often unable to rule out an overwhelming number of candidates—making their prediction sets large and inefﬁcient (Angelopoulos et al., 2021c).

Original: This can make conformal predictors unusable in settings in which the cost of false positives is substantial.
Revised: This can make conformal predictors unusable in settings in which the cost of false positives is substantial.

Original: Consider the example of in-silico screening for drug discovery (see Figure 1).
Revised: Consider the example of in-silico screening for drug discovery (see Figure 1).

Original: In-silico screening uses computational tools to identify potentially viable molecular compounds for a particular purpose of interest.
Revised: In-silico screening uses computational tools to identify potentially viable molecular compounds for a particular purpose of interest.

Original: For instance, Stokes et al.
Revised: For instance, Stokes et al.

Original: (2020) recently performed a high-throughput screen of compounds from the multi-million molecule ZINC15  (Sterling and Irwin, 2015)database using models predictive of E. coli inhibition in order to ﬁnd a new antibiotic, and were successful.
Revised: (2020) recently performed a high-throughput screen of compounds from the multi-million molecule ZINC15 database (Sterling and Irwin, 2015) using models predictive of E. coli inhibition in order to ﬁnd a new antibiotic, and were successful.

Original: Clearly, guaranteeing high hit rates for this task with conformal prediction is an attractive proposition.
Revised: Clearly, guaranteeing high hit rates for this task with conformal prediction is an attractive proposition.

Original: The catch, however, is that each returned candidate must be veriﬁed experimentally—and too many false positives can quickly consume the available budget (e.g., of time, funding, or other limited resources).
Revised: The catch, however, is that each returned candidate must be veriﬁed experimentally—and too many false positives can quickly consume the available budget (e.g., of time, funding, or other limited resources).

Original: This is especially relevant when a valid answer—in this case, an effective drug—might not even exist.
Revised: This is especially relevant when a valid answer—in this case, an effective drug—might not even exist.

Original: In this paper, we are interested in creating conﬁdent prediction sets that trade off coverage for precision in order to provably constrain the total number of false discoveries (FD).
Revised: In this paper, we are interested in creating conﬁdent prediction sets that trade off coverage for precision in order to provably constrain the total number of false positives (FP).

Original: In other words, we shift the focus of our performance guarantee to be on limiting the number of incorrect answers in our outputs (e.g., to ≤ k on average), with the understanding that we will potentially fail to recover some proportion of the true answers—i.e., we will obtain a lower true discovery rate (TDR).
Revised: In other words, we shift the focus of our performance guarantee to be on limiting the number of incorrect answers in our outputs (e.g., to ≤ k on average), with the understanding that we can potentially fail to recover some proportion of the true answers—i.e., we may obtain a lower true positive rate (TPR).

Original: Concretely, suppose we have been given n (potentially) multi-label classiﬁcation examples ( X i , Z i )
Revised: Concretely, suppose we have been given n (possibly) multi-label classiﬁcation examples ( X i , Z i )

Original: ∈
Revised: ∈

Original: X × 2 Y , i = 1 , . . .
Revised: X × 2 Y , i = 1 , . . .

Original: n as calibration data, that have been drawn exchangeably from some underlying distribution
Revised: n as calibration data, that have been drawn exchangeably from some underlying distribution

Original: P XZ .
Revised: P XZ .

Original: Here we consider the generalized scenario where each observation X i may have any number of corresponding correct labels (including zero, in the case of having no answer at all).
Revised: Here we consider the generalized scenario where each observation X i may have any number of corresponding correct labels (including zero, in the case of having no answer at all).

Original: That is, the response Z i can be any subset of the full label space Y .
Revised: That is, the response Z i can be any subset of the full label space Y .

Original: 1 Let X n +1 ∈
Revised: 1 Let X n +1 ∈

Original: X be a new exchangeable test example for which we would like to predict Z n +1 ⊆ Y .
Revised: X be a new exchangeable test example for which we would like to predict Z n +1 ⊆ Y .

Original: Our goal is to construct a set predictor
Revised: Our goal is to construct a set predictor

Original: C k ( X n +1 ) that maximizes recall with respect to the true Z n +1 , but more importantly has an expected false discovery budget that is controlled at a user-deﬁned tolerance level k ∈ R > 0 , i.e.,
Revised: C k ( X n +1 ) that maximizes recall with respect to the true Z n +1 , but more importantly has an expected false positive budget that is controlled at a user-deﬁned tolerance level k ∈ R > 0 , i.e.,

Original: [Equation]
Revised: [Equation]

Original: As an additional alternative to bounding the expected number of false discoveries to k , we also seek a set predictor C k,δ that has more direct control of the probability of exceeding k false discoveries:
Revised: As an additional alternative to bounding the expected number of false positives to k , we also seek a set predictor C k,δ that has more direct control of the probability of exceeding k false positives:

Original: [Equation]
Revised: [Equation]

Original: where δ ∈ (0 , 1) is another user-deﬁned tolerance level.
Revised: where δ ∈ (0 , 1) is another user-deﬁned tolerance level.

Original: Both constructions deﬁne different, but useful, operating conditions; the ﬁrst is more straightforward (e.g., for the general practitioner), while the second offers a ﬁner, two-parameter level of control.
Revised: Both constructions deﬁne different, but useful, operating conditions; the ﬁrst is more straightforward (e.g., for the general practitioner), while the second offers a ﬁner, two-parameter level of control.

Original: Note that both are well-known concepts in the context of multiple hypothesis testing—i.e.,  the PFER and k -FWER (Spjøtvoll, 1972; Romano and 
Revised: Both are well-known concepts in the context of multiple hypothesis testing—i.e., PFER, the per-family error rate, and k -FWER, the familywise error rate (Spjøtvoll, 1972; Romano and Wolf, 2007)—though

Original: Wolf, 2007), though the setting (multi-label classiﬁcation) and available tools (conformal prediction) considered here differ substantially (see §2).
Revised: the setting (multi-label classiﬁcation) and available tools (conformal prediction) considered here differ substantially (see §2).

Original: Here, and in the rest of the main paper, both constraints are marginal over the choice of calibration and testing data points ( X i , Z i ) , i = 1 , . . . , n + 1 ; though enforcing(relatively stronger) calibration-conditional constraintsonly requires  minor modifcation.
Revised: Note that both constraints are marginal over the choice of calibration and test data { ( X i , Z i ) } n +1 i =1 , however, enforcing calibration-conditional constraints requires only minor modiﬁcation (Bates et al., 2020).

Original: In order to achieve the desired levels of false discovery control, we present an approach that is based on set classiﬁcation , combined with a form of conformal calibration.
Revised: In order to achieve the desired levels of false positive control, we present an approach that is based on set classiﬁcation , combined with a form of conformal calibration.

Original: Speciﬁcally, we use a set nonconformity measure F : X × 2 Y
Revised: Speciﬁcally, we use a set nonconformity measure F : X × 2 Y

Original: → R to score candidate output sets, S ∈ 2 Y .
Revised: → R to score candidate output sets, S ∈ 2 Y .

Original: We learn this function from separate multi-label classiﬁcation training data.
Revised: We learn this function from separate multi-label classiﬁcation training data.

Original: Intuitively, a high nonconformity score should reﬂect the conﬁdence that the candidate set might contain a high number of false discoveries, and vice-versa.
Revised: Intuitively, a high nonconformity score should reﬂect the conﬁdence that the candidate set might contain a high number of false positives, and vice-versa.

Original: To empirically maximize the true discovery rate, we then return the largest set among all candidates that have nonconformity scores below a threshold that we set such that our false positive constraints are satisﬁed.
Revised: To empirically maximize the true positive rate, we then return the largest set among all candidates that have nonconformity scores below a threshold that we set such that our false positive constraints are satisﬁed.

Original: Candidate sets are explored greedily with a best-ﬁrst strategy that adds top-ranked individual labels to a growing, nested output set S until our nonconformity threshold is met.
Revised: Candidate sets are explored greedily with a best-ﬁrst strategy that adds top-ranked individual labels to a growing, nested output set S until our nonconformity threshold is met.

Original: This both allows us to ﬁnd efﬁcient solutions for large label spaces Y , and to leverage prior theory for calibrating expectations of monotonic losses for set predictors (Angelopoulos et al., 2021b).
Revised: This both allows us to ﬁnd efﬁcient solutions for large label spaces Y , and to leverage prior theory for calibrating expectations of monotonic losses for set predictors (Angelopoulos et al., 2021b).

Original: Contributions.
Revised: Contributions.

Original: In summary, our main results are as follows:
Revised: In summary, our main results are as follows:

Original: • A theoretical reframing of conformal prediction that provides rigorous false positive control; • A simple and effective strategy for constructing valid output sets with high true discovery rates; • A demonstration of the practical utility of our framework across a range of classiﬁcation tasks.
Revised: • A theoretical reframing of conformal prediction that provides rigorous false positive control; • A simple and effective strategy for constructing valid output sets with high true positive rates; • A demonstration of the practical utility of our framework across a range of classiﬁcation tasks.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Uncertainty estimation.
Revised: Uncertainty estimation.

Original: In recent years, there has been a growing interest in estimating model uncertainty.
Revised: In recent years, there has been a growing interest in estimating model uncertainty.

Original: A large body of work focuses on calibrating the model posterior, p θ (ˆ y n +1 | x n +1 ) , such that the accuracy, y n [-+
-] =
Revised: A large body of work focuses on calibrating model-based conditional probabilities, p θ (ˆ y n +1 | x n +1 ) , such that the accuracy, y n [-+
-]+1 =

Original: ˆ y n +1 , is equal to the estimated probability (Niculescu-Mizil and Caruana, 2005; Kuleshov et al., 2018; Kumar et al., 2019).
Revised: ˆ y n +1 , is equal to the estimated probability (NiculescuMizil and Caruana, 2005; Kuleshov et al., 2018; Kumar et al., 2019).

Original: In theory, these estimates could be used to create prediction sets with few false discoveries, but they are not always guaranteed to be accurate (Guo et al., 2017; Ashukha et al., 2020; Hirschfeld et al., 2020).
Revised: In theory, these estimates could be used to create prediction sets with few false positives, but they are not always guaranteed to be accurate (Guo et al., 2017; Ashukha et al., 2020; Hirschfeld et al., 2020).

Original: In a similar vein, Bayesian formalisms underlie several popular approaches to quantifying predictive uncertainty via computing the posterior predictive distribution over model parameters (Neal, 1996; Graves, 2011;Hernández-Lobato and
Revised: In a similar vein, Bayesian formalisms underlie several popular approaches to quantifying predictive uncertainty via computing the posterior predictive distribution over model parameters (Neal, 1996; Graves, 2011;

Original:  Adams, 2015; Gal and Ghahramani, 2016).
Revised: Hernández-Lobato and Adams, 2015; Gal and Ghahramani, 2016).

Original: Like calibration, however, the quality of these methods can vary depending on the suitability of the presumed prior, as well as on any approximation error.
Revised: However, the quality of these methods can vary depending on the suitability of the presumed prior and on any approximation error.

Original: Conformal prediction.
Revised: Conformal prediction.

Original: As introduced in §1, conformal prediction (Vovk et al., 2005) provides a ﬁnite-sample, distribution-free method for obtaining prediction sets C with guarantees on the event 1 { Y n +1 ∈ C ( X n +1 ) } .
Revised: As introduced in §1, conformal prediction (Vovk et al., 2005) provides a ﬁnite-sample, distribution-free method for obtaining prediction sets C with guarantees on the event 1 { Y n +1 ∈ C ( X n +1 ) } .

Original: Most efforts in CP focus on improving the predictive efﬁciency, E [ |C ( X n +1 ) | ] , of the conformal sets (Vovk et al., 2016;
Revised: Most efforts in CP focus on improving the predictive efﬁciency, E [ |C ( X n +1 ) | ] , of the conformal sets (Vovk et al., 2016;

Original: Sadinle et al., 2019; Romano et al., 2020; Angelopoulos et al., 2021c; Fisch et al., 2021; Hoff, 2021).
Revised: Sadinle et al., 2019; Romano et al., 2020; Angelopoulos et al., 2021c; Fisch et al., 2021; Hoff, 2021).

Original: As coverage is guaranteed by design, improving efﬁciency will naturally lead to more precise sets with fewer false positives—but not to a speciﬁable level.
Revised: As coverage is guaranteed by design, improving efﬁciency will naturally lead to more precise sets with fewer false positives—but not to a speciﬁable level.

Original: Cauchois et al.
Revised: Cauchois et al.

Original: (2021) develop a conformal approach to multi-label classiﬁcation that can guarantee that the prediction set only contains true labels (i.e., FD = 0), but does not offer ﬁne-grained control.
Revised: (2021) develop a conformal approach to multi-label classiﬁcation that can guarantee that the prediction set only contains true labels (i.e., FP = 0), but does not offer ﬁne-grained control.

Original: Most relevant to our work, Bates et al.
Revised: Most relevant to our work, Bates et al.

Original: (2020) gives a ﬂexible framework for controlling the risk, E [ L ( Y, T ( X ))] , of a set-valued predictor T with an arbitrary loss function L —as long the loss respects a monotonic nesting property, S ⊂ S (cid:48)
Revised: (2020) gives a ﬂexible framework for controlling the risk, E [ L ( Y, T ( X ))] , of a set-valued predictor T with an arbitrary loss function L —as long the loss respects a monotonic nesting property, S ⊂ S (cid:48)

Original: ⇒ L ( S ) ≥ L ( S (cid:48) ) , for any two prediction sets S and
Revised: ⇒ L ( S ) ≥ L ( S (cid:48) ) , for any two prediction sets S and

Original: S (cid:48) .
Revised: S (cid:48) .

Original: The calibration strategy we use here for marginal expectations is based on the recent extension in
Revised: The calibration strategy we use here for marginal expectations is based on the recent extension in

Original: Angelopoulos et al.
Revised: Angelopoulos et al.

Original: (2021b).
Revised: (2021b).

Original: Concurrent to this work, Angelopoulos et al.
Revised: Concurrent to this work, Angelopoulos et al.

Original: (2021a) proposed methods to rigorously control non-monotonic losses, including the related false discovery rate (FDR), which normalizes the number of false discoveries over the size of the prediction set.
Revised: (2021a) proposed methods to rigorously control non-monotonic losses, including the related false discovery rate (FDR), which normalizes the number of false positives over the size of the prediction set.

Original: However, as most of our target applications have relatively few true positives, FDR control can be somewhat volatile and lead to many empty predictions (making controlling total false discoveries a more natural ﬁt).
Revised: However, as most of our target applications have relatively few true positives, FDR control can be somewhat volatile and lead to many empty predictions (making controlling total false positives a more natural ﬁt for this work).

Original: Multiple testing.
Revised: Multiple testing.

Original: Controlling the number of false discoveries over a collection of hypothesis tests is a well-studied problem in statistics (Dunn, 1961; Benjamini and Hochberg, 1995; Lehmann and Romano, 2005; Romano and Wolf, 2007, etc. ).
Revised: Controlling the number of false positives/discoveries over a collection of hypothesis tests is a well-studied problem in statistics (Dunn, 1961; Benjamini and Hochberg, 1995; Lehmann and Romano, 2005; Romano and Wolf, 2007, etc. ).

Original: Recently, FDR control has also been studied for outlier detection in a conformal inference setting (Bates et al., 2021).
Revised: Recently, FDR control has also been studied for outlier detection in a conformal inference setting (Bates et al., 2021).

Original: Most statistical approaches to false discovery control operate over p-values for each hypothesis test that have speciﬁc dependency structures (e.g., independent or positively dependent), or otherwise use more conservative corrections.
Revised: Most statistical approaches to false discovery control operate over p-values for each hypothesis test that have speciﬁc dependency structures (e.g., independent or positively dependent), or otherwise use more conservative corrections.

Original: Though similar, our multi-label classiﬁcation setting is slightly different from standard multiple testing in that there is (1) an unknown dependency structure between candidate labels for the same query, but also (2) an extra layer of exchangeability over the n + 1 queries.
Revised: Though similar, our multi-label classiﬁcation setting is slightly different from standard multiple testing in that there is (1) an unknown dependency structure between candidate labels for the same query, but also (2) an extra layer of exchangeability over the n + 1 queries.

Original: Our approach is able to ignore (1) by leveraging (2) within a conformal calibration framework to obtain desirable guarantees.
Revised: Our approach is able to ignore (1) by leveraging (2) within a conformal calibration framework to obtain desirable guarantees.

Original: Selective classiﬁcation.
Revised: Selective classiﬁcation.

Original: Our work also bears some relation to selective classiﬁcation (El-Yaniv and
Revised: Our work also bears some relation to selective classiﬁcation (El-Yaniv and

Original: Wiener, 2010), where models have the option to abstain from answering.
Revised: Wiener, 2010), where models have the option to abstain from answering.

Original: In particular, Geifman and El-Yaniv (2017) propose a strategy for ﬁnding classiﬁers with speciﬁc selective risks (i.e., the expected accuracy over answered examples).
Revised: In particular, Geifman and El-Yaniv (2017) propose a strategy for ﬁnding classiﬁers with speciﬁc selective risks (i.e., the expected accuracy over answered examples).

Original: In our setting, this is analogous to controlling false discoveries at k ≈ 0 .
Revised: In our setting, this is analogous to controlling false positives using k ≈ 0 .

Original: If uncertain, the model would have to “abstain” by outputting an empty set.
Revised: If uncertain, the model would have to “abstain” by outputting an empty set.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We begin with abrief review of conformal prediction (see Shafer and Vovk, 2008).
Revised: We begin with a review of conformal prediction (see Shafer and Vovk, 2008).

Original: Here, and in the rest of the paper, upper-case letters ( X ) denote random variables; lower-case letters ( x ) denote scalars, and script letters ( X ) denote sets, unless otherwise speciﬁed.
Revised: Here, and in the rest of the paper, upper-case letters ( X ) denote random variables; lower-case letters ( x ) denote constants, and script letters ( X ) denote sets, unless otherwise speciﬁed.

Original: Proofs are deferred to the appendix.
Revised: Proofs are deferred to the appendix.

Original: Given a new example x , for every candidate label y ∈ Y , single label conformal prediction (where y n +1 is a scalar) either accepts or rejects the null hypothesis that the pairing ( x, y ) is correct.
Revised: Given a new example x , for every candidate label y ∈ Y , single label conformal prediction (where y n +1 is a scalar) either accepts or rejects the null hypothesis that the pairing ( x, y ) is correct.

Original: The test statistic for this test is a nonconformity measure , M (( x, y ) , D ) , where D is a dataset of exchangeable, labeled examples.
Revised: The test statistic for this test is a nonconformity measure , M (( x, y ) , D ) , where D is a dataset of exchangeable, labeled examples.

Original: Informally, a lower value of M reﬂects that point ( x, y ) “conforms” to D , whereas a higher value of M reﬂects that ( x, y ) is atypical relative to D .
Revised: Informally, a lower value of M reﬂects that point ( x, y ) “conforms” to D , whereas a higher value of M reﬂects that ( x, y ) is atypical relative to D .

Original: A practical choice for M is  model-based likelihood, e.g., − log p θ ( y | x ) , where θ is a model ﬁt to D .
Revised: A practical choice for M is a model-based loss, e.g., − log p θ ( y | x ) , where θ is a model ﬁt to D .

Original: For conformal prediction to work, is important that M preserves the exchangeability of its inputs: i.e., M should be symmetric with respect to permutations.
Revised: For conformal prediction to work, is important that M preserves the exchangeability of its inputs: i.e., M should be symmetric with respect to permutations.

Original: In order to avoid retraining M every time a new candidate label y is considered, “split” conformal prediction (Papadopoulos, 2008) uses a proper training set D train to learn a ﬁxed M .
Revised: In order to avoid retraining M every time a new candidate label y is considered, “split” conformal prediction (Papadopoulos, 2008) uses a proper training set D train to learn a ﬁxed M .

Original: This preserves exchangeability of the calibration and test points without further modiﬁcation—and is a popular and computationally efﬁcient strategy (which we follow in this work).
Revised: This preserves exchangeability of the calibration and test points without further modiﬁcation—and is a popular and computationally efﬁcient strategy (which we follow in this work).

Original: To construct a prediction set for a new test point x n +1 , the conformal classiﬁer outputs all y for which the null hypothesis (that pairing ( x, y ) is correct) is not rejected.
Revised: To construct a prediction set for a new test point x n +1 , the conformal classiﬁer outputs all y for which the null hypothesis (that pairing ( x, y ) is correct) is not rejected.

Original: This is achieved by comparing the scores of the test candidates to the scores computed over the ﬁrst n calibration examples.
Revised: This is achieved by comparing the scores of the test candidates to the scores computed over the ﬁrst n calibration examples.

Original: Theorem 3.1 (Split CP, Vovk et al.
Revised: Theorem 3.1 (Split CP, Vovk et al.

Original: ; Papadopoulos (2008)) .
Revised: ; Papadopoulos (2008)) .

Original: Assume that examples ( X i , Y i ) ∈
Revised: Assume that examples ( X i , Y i ) ∈

Original: X ×Y , i = 1 , . .
Revised: X ×Y , i = 1 , . .

Original: . , n +1 are drawn exchangeably from a distribution P XY .
Revised: . , n +1 are drawn exchangeably from a distribution P XY .

Original: For a ﬁxed nonconformity measure M , let random variable V i
Revised: For a ﬁxed nonconformity measure M , let random variable V i

Original: = M ( X i , Y i ) be the nonconformity score of ( X i , Y i ) .
Revised: = M ( X i , Y i ) be the nonconformity score of ( X i , Y i ) .

Original: For (cid:15) ∈ (0 , 1) , deﬁne the conformal set (based on the ﬁrst n examples) at x ∈ X as
Revised: For (cid:15) ∈ (0 , 1) , deﬁne the conformal set (based on the ﬁrst n examples) at x ∈ X as

Original: C [Equation]
Revised: C (cid:15) ( x ) :

Original: [Equation] := (cid:110) y ∈ Y : M ( x n +1 , y ) ≤ Quantile(1 − (cid:15) ; V 1: n ∪ {∞} ) (cid:111) .
Revised: = (cid:110) y ∈ Y : M ( x, y ) ≤ Quantile(1 − (cid:15) ; V 1: n ∪ {∞} ) (cid:111) .

Original: Then C (cid:15) ( X n + ) satisﬁes P ( Y n +1 ∈ C (cid:15) ( X n +1 )) ≥ 1 − (cid:15) .
Revised: Then C (cid:15) ( X n +1 ) satisﬁes P ( Y n +1 ∈ C (cid:15) ( X n +1 )) ≥ 1 − (cid:15) .

Original: Remark 3.2.
Revised: Remark 3.2.

Original: Cauchois et al.
Revised: Cauchois et al.

Original: (2021) recently extended conformal prediction to the multi-label case to obtain guarantees of the form P ( C inner (cid:15)
Revised: (2021) recently extended conformal prediction to the multi-label case to obtain guarantees of the form P ( C inner (cid:15)

Original: ( X n +1 ) ⊆
Revised: ( X n +1 ) ⊆

Original: Z n +1 ⊆
Revised: Z n +1 ⊆

Original: C outer (cid:15) ( X n +1 )) ≥ 1 − (cid:15) .
Revised: C outer (cid:15) ( X n +1 )) ≥ 1 − (cid:15) .

Original: Their approach follows a modiﬁed version of thealgorithm for the single label case.
Revised: Their approach constructs bounding sets using a modiﬁed version of the single label algorithm (i.e., Eq.

Original: The motivation for our work is quickly evident from Eq.
Revised: The motivation for our work is quickly evident from Eq.

Original: (3): if we are unable to reject most candidates  based on their nonconformity scores M ( x n +1 , y ) , then C (cid:15) ( x ) can contain many false positives.
Revised: (3): if we are unable to reject most candidates y ∈ Y based on their nonconformity scores M ( x n +1 , y ) , then C (cid:15) ( x ) can contain many false positives.

Original:  C ONFORMAL PREDICTION WITH FALSE DISCOVERY CONTROL
Revised:  C ONFORMAL PREDICTION WITH FALSE POSITIVE CONTROL

Original: We now introduce our strategy for trading off coverage for precision by imposing constraints on the number of false positives that are contained in our output sets.
Revised: We now introduce our strategy for trading off coverage for precision by imposing constraints on the number of false positives that are contained in our output sets.

Original: To brieﬂy remind the reader of our setting, we assume that we have been given n exchangeable multi-label classiﬁcation examples, ( X i , Z i )
Revised: To brieﬂy remind the reader of our setting, we assume that we have been given n exchangeable multi-label classiﬁcation examples, ( X i , Z i )

Original: ∈
Revised: ∈

Original: X × 2 Y , i = 1 , . . .
Revised: X × 2 Y , i = 1 , . . .

Original: n as calibration data, that are drawn from a distribution P XZ .
Revised: n as calibration data, that are drawn from a distribution P XZ .

Original: The response Z i is treated as a generalized set of correct labels for input X i , and is a subset of Y , where Y is ﬁnite.
Revised: The response Z i is treated as a generalized set of correct labels for input X i , and is a subset of Y , where Y is ﬁnite.

Original: For a prediction C ( x n +1 ) ⊆
Revised: For a prediction C ( x n +1 ) ⊆

Original: Y evaluated at a point x n +1 ∈ X with true label set Z n +1 ⊆
Revised: Y evaluated at a point x n +1 ∈ X with true label set z n +1 ⊆

Original: Y , we deﬁne the true discovery proportion (TDP) as the ratio of correct labels that are recovered:
Revised: Y , we deﬁne the true positive proportion (TPP) as the ratio of correct labels that are recovered:

Original: [Equation]
Revised: [Equation]

Original: (Note that TDR := E [TDP] ) , (4) and the number of false discoveries (FD) as the total count of incorrect labels in C ( x n +1 ) :
Revised: (Note that TPR := E [TPP] ) , (4) and the number of false positives (FP) as the total count of incorrect labels in C ( x n +1 ) :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Our goal, as stated in §1, is to maximize the expected TDP (in standard classiﬁcation, this would equate to accuracy; or power in statistics), while constraining the FD in either of two ways:
Revised: Our goal, as stated in §1, is to maximize the expected TPP (in standard classiﬁcation, this would equate to accuracy; or power in statistics), while constraining the FP in either of two ways:

Original: Deﬁnition 4.1 ( k -FD validity) .
Revised: Deﬁnition 4.1 ( k -FP validity) .

Original: A conformal classiﬁer C k evaluated at test point  X n +1 is k -FD valid if the expectation of its FD satisﬁes E [FD( Z n +1 ,
Revised: A conformal classiﬁer C k evaluated at test point ( X n +1 , Z n +1 ) is k -FP valid

Original:  C k ( X n +1 ))] ≤ k .
Revised: if the expectation of its FP satisﬁes E [FP( Z n +1 , C k ( X n +1 ))] ≤ k .

Original: Deﬁnition 4.2 ( ( k, δ ) -FDP validity) .
Revised: Deﬁnition 4.2 ( ( k, δ ) -FP validity) .

Original: A conformal classiﬁer C k,δ evaluated at test point  X n +1  is ( k, δ ) -FD valid if the CDF of its FD satisﬁes P (FD( Z n +1 , C k,δ ( X n +1 )) ≤ k ) ≥ 1 − δ .
Revised: A conformal classiﬁer C k,δ evaluated at test point ( X n +1 , Z n +1 ) is ( k, δ ) -FP valid if the CDF of its FP satisﬁes P (FP( Z n +1 , C k,δ ( X n +1 )) ≤ k ) ≥ 1 − δ .

Original: 4. A N ORACLE SET PREDICTOR
Revised: 4. A N ORACLE SET PREDICTOR

Original: To motivate our approach, imagine an oracle with access to P Z | X , the conditional distribution of the multi-label set Z given the input X .
Revised: To motivate our approach, imagine an oracle with access to P Z | X , the conditional distribution of the multi-label set Z given the input X .

Original: Given this information, for any input x ∈ X and candidate set
Revised: Given this information, for any input x ∈ X and candidate set

Original: S ∈ 2 Y , such an oracle would be able to exactly calculate both the expectation and the conditional distribution of the number of false (and true) discoveries in S given x .
Revised: S ∈ 2 Y , such an oracle would be able to exactly calculate both the expectation and the conditional distribution of the number of false (and true) positives in S given x .

Original: In order to maximize the TDR while meeting k -FD and ( k, δ ) -FD validity, it could then yield the following set predictions:
Revised: In order to maximize the TPR while meeting k -FP and ( k, δ ) -FP validity, it could then yield the following set predictions:

Original: [Equation]
Revised: [Equation]

Original: where ties in the arg max are broken by set size (smaller is better).
Revised: where ties in the arg max are broken by set size (smaller is better).

Original: This oracle has the advantage of not only being marginally valid as required, but also being conditionally valid given the  input X n +1  . 
Revised: This oracle has the advantage of not only being marginally valid, but also being conditionally valid given the test input X n +1 = x n +1 . [Table]

Original: Deﬁnitions: x n+1 is the point, D train is a set, D cal is is the
Revised: [Table] [Table] [Table]

Original: , and is parameter for considering only ranked candidates y ∈ Y . 1: function PREDICT ( x n +1 , D train , D cal , k , B = |Y| ) 2: D (1)train , D (2)train ← SPLIT ( D train ) 3: # Use part of the training
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: to learn individual label likelihood p θ ( y i ∈ Z | x ) . 4: p θ (
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: i
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: Z | x ) ← TRAIN ( MPNN , D (1)train ) 5:
Revised: [Table] [Table] [Table] [Table]

Original: # Use the
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: smaller) part to learn set function F ( x, S ) .
Revised: [Table] [Table] [Table] [Table]

Original: F ( x, S ) ←
Revised: [Table] [Table] [Table] [Table]

Original: ( DeepSets , D (2)
Revised: [Table] [Table] [Table] [Table]

Original: ) 7: # Compile
Revised: [Table]

Original: T
Revised: [Table] [Table] on this ordering. MFP( x i , z i ) ← INIT ( x i , z i , (cid:101) F , {S 1 , . . .

Original: = ∅ 9:
Revised: , S B } ) 17:

Original: FOR
Revised: T cal ← T cal ∪ { MFP i i ) } 18: t k ←

Original: DO 10: #
Revised: _ THRESHOLD

Original: 11: {S 1 , .
Revised: Eq

Original: S B } = { SORT ( Y , pθ ( y ∈
Revised: 19: return t k 20: function PREDICT ( x p

Original: Z i | x i )) 1: j : j ∈ { 1 ,. . .
Revised: (cid:101) F ,

Original: , B }} 12: # MFD( x i , Z
Revised: B ) 21: # candidates line 14). 22: {S

Original: i
Revised: B c

Original: INIT ( i , i
Revised: )) 1: j j

Original: , S B } ) 14:
Revised: B }} 23: # C k (

Original: T 16: t 
Revised: ) ← arg max S∈{S n +1 , 1 ..., S n +1 ,B } {|S| : (cid:101) F ( x n +1 , S ) ≤ t k } 25: return C k ( x n +1 )

Original: 4. A PPROXIMATING THE ORACLE USING D EEP S ETS
Revised: 4. A PPROXIMATING THE ORACLE USING SET FUNCTIONS

Original: Of course, computing such an oracle is not pactical, as P Z | X is not known.
Revised: Of course, computing such an oracle is not practical, as P Z | X is not known.

Original: Instead, let F be a set function F : X × 2 Y
Revised: Instead, let F be a set function F : X × 2 Y

Original: →
Revised: →

Original: R that generates a score for the candidate set S given x n +1 .
Revised: R that generates a score for the candidate set S given x n +1 .

Original: Importantly, F can be any function (as we will later show), but to best replicate the oracle’s behavior for k -FD and ( k, δ ) -FD valid predictions, F should be a good estimator of the distribution of false positives in S .
Revised: Importantly, F can be any function (as we will later show), but to best replicate the oracle’s behavior for k -FP and ( k, δ ) -FP valid predictions, F should be a good estimator of the distribution of false positives in S .

Original: We develop a simple approach to modelingand learning F via a deep sets model (Zaheer et al., 2017). 
Revised: We now describe a simple approach to modeling F using DeepSets (Zaheer et al., 2017). DeepSets is a popular method which is known to be a universal approximator for continuous set functions.

Original: Let { φ ( x, y 1 ) , . ..
Revised: Let { φ ( x, y 1 ) , . .

Original: , φ ( x, y l ) } featurize a candidate set S ⊆ Y , where φ ( x, y i ) ∈ R
Revised: , φ ( x, y s ) } featurize a candidate set S ⊆ Y , where φ ( x, y c ) ∈ R

Original: d isthe representation of an individual label y i ∈ S computed as a function of ( x, y i )  .
Revised: d is a function of ( x, y c ) , for y c ∈ S .

Original: In practice, we ﬁnd that taking φ ( x, y i ) to be a (ﬁxed)scalar estimate  p θ ( y i ∈ Z | x ) —the marginal likelihood of y i being a correct label for x —performs reasonably well and is simple to implement.
Revised: In practice, we ﬁnd that taking φ ( x, y c ) to be a (ﬁxed) estimate of p θ ( y c ∈ Z | x ) , the marginal likelihood of y c being a correct label, performs well and is simple to implement.

Original: This means that the features to the set predictor model need only be deﬁned bythe (uncalibrated) conﬁdence thresholds of any backbone model.
Revised: These (uncalibrated) prediction scores can be provided by any backbone model.

Original: For example, in our chemistry we use a directed MPNN (Yang et al., 2019) that independently classiﬁes molecules for being active/inactive for the target property.
Revised: For example, in our in-silico screening task, we deﬁne φ using a directed MPNN (Yang et al., 2019) that independently classiﬁes molecules as active or inactive for the target property.

Original: φ ( x, y i ) is then simply a sigmoidal output.

Original: Given φ , the deep sets model is deﬁned as
Revised: Given φ , the DeepSets model is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where enc( · ) and dec( · ) are two neural  models parameterized by θ 1 and θ 2 , respectively, and softmax( · ) is taken over the total possible number of false positives, { 0 , . .
Revised: where enc( · ) and dec( · ) are two neural encoder/decoder models parameterized by θ 1 and θ 2 , respectively, and softmax( · ) is taken over the total possible number of false positives, { 0 , . .

Original: . , |Y|} .
Revised: . , |Y|} .

Original: Ψ is trained to predict the number of false positives in S via cross entropy, using labeled sets sampled from auxiliary training data.
Revised: Ψ is trained to predict the number of false positives in S via cross entropy, using labeled sets sampled from held-out training data.

Original: We then compute F k and F k,δ (for either k -FD or ( k, δ ) -FD validity) as
Revised: We then compute F k and F k,δ (for either k -FP or ( k, δ ) -FP validity) as

Original: F k ( x, S ) := |S| (cid:88) i = i · Ψ( x, S )i and
Revised: F k ( x, S ) := |S| (cid:88) η = η · Ψ( x, S )

Original:  F k,δ ( x, S ) :
Revised: η and F k,δ ( x, S ) :

Original: = 1 − min( k, |S| ) (cid:88) i =
Revised: = 1 − min( k, |S| ) (cid:88) η

Original:  Ψ( x, S ) i , (9) where Ψ( x, S )
Revised: = Ψ( x, S ) η , (9) where Ψ( x, S )

Original: i denotes the i th index of the softmax (i.e., the estimated probability that FD = i ).
Revised: η denotes the η -th index of the softmax (i.e., the estimated probability that FP = η ).

Original: In the next section we will refer only to F , with the understanding it differs based on the validity goal.
Revised: In the next section we will refer only to F , with the understanding it differs based on the validity goal.

Original: 4. I DENTIFYING VALID CANDIDATE SETS
Revised: 4. I DENTIFYING VALID CANDIDATE SETS

Original: Although our set predictor F is trained to model either the expected FD or its CDF, it is not necessarily accurate.
Revised: Although our set predictor F is trained to model either the expected FP or its CDF, it is not necessarily accurate.

Original: If F were simply substituted into Eq.
Revised: If F were simply substituted into Eq.

Original: (6) or Eq.
Revised: (6) or Eq.

Original: (7), it would not produce  set predictions that are guaranteed to be valid.
Revised: (7), it may not produce valid set predictions.

Original: To remedy this, we must calibrate a threshold forrejecting or accepting candidate sets based on F .
Revised: To remedy this, we carefully calibrate a threshold for accepting candidate sets based on F .

Original: To do so, weﬁrst greedily identify a sequence of nested candidate sets, S 1 ⊂ S 2 ⊂ . . .
Revised: First, we greedily identify a sequence of nested candidate sets, S 1 ⊂ S 2 ⊂ . . .

Original: ⊂ Y , by ranking individual labels y  ∈ Y by their estimated likelihoods of being true positives , i.e., p θ ( y  ∈ Z | x ) , and including them one by one.
Revised: ⊂ Y , by ranking individual labels y c ∈ Y by their estimated likelihoods of being true positives, i.e., p θ ( y c ∈ Z | x ) , and include them one by one.

Original: Each of these candidate sets isthen given a score by F , which is taken to be a nonconformity score .
Revised: Next, each of these candidate sets is given a score by F , which we treat as a “nonconformity score.”

Original: We manually enforce that the scores produce a ranking consistent with their size (though this is rarely necessary).
Revised: We manually enforce that the scores produce a ranking consistent with their size: for set S j

Original: Speciﬁcally, for set S i we compute its adjusted nonconformity score ˜ F ( x, S i ) as ˜ F ( x, S i ) :
Revised: we compute its adjusted nonconformity score (cid:101) F ( x, S j ) as (cid:101) F ( x, S j ) :

Original: = max {F ( x, S 1 ) , . .
Revised: = max {F ( x, S 1 ) , . .

Original: , F ( x, S i ) } .
Revised: , F ( x, S j ) } .

Revised: For efﬁciency (see Remark 4.4), we only take the top B ≤ |Y| sets (although this doesn’t come for free: a smaller B may result in fewer true positives).

Original: Let MFD( x, Z )( t ) := max (cid:8) FD( Z , S i ): ˜ F ( x, S i ) ≤ t (cid:9) denote the maximum number of false discoveries over all nested sets S i for an input x that have nonconformity scores less than t .
Revised: Let MFP( x, z, t ) := max (cid:8) FP( z, S j ): (cid:101) F ( x, S j ) ≤ t (cid:9) denote the maximum number of false positives over all nested sets S j for an input x that have nonconformity scores less than t .

Original: If this thresholded set is empty, then the MFD is deﬁned to be 0 .
Revised: If this thresholded set is empty, then the MFP is deﬁned to be 0 .

Original: Since the sets are ordered by size, this quantity is guaranteed to be monotonically non-decreasing in t , i.e., 
Revised: Since the sets are ordered by size, this quantity is guaranteed to be monotonically non-decreasing in t , i.e., t ≤ t (cid:48) =

Original: [Equation]
Revised: ⇒ MFP( x, z, t ) ≤ MFP( x, z, t (cid:48) ) .

Original: Using this property, we can calibrate t to create a set ﬁlter with controlled MFD, as formalized next:
Revised: Using this property, we can calibrate t to create a set ﬁlter with controlled MFP, as formalized next:

Original: Theorem 4.3 (FD-CP) .
Revised: Theorem 4.3 (FP-CP) .

Original: Assume that examples ( X i , Z i ) ∈
Revised: Assume that examples ( X i , Z i ) ∈

Original: X × 2 Y , i = 1 , . .
Revised: X × 2 Y , i = 1 , . .

Original: . , n + 1 are drawn exchangeably from a distribution P XY .
Revised: . , n + 1 are drawn exchangeably from a distribution P XY .

Original: For each example i , let S i,j , j = 1 , . . 
Revised: For each example i , let S i,j , j = 1 , . . .

Original: . , |Y| ,where |Y| < B , be nestedcandidate sets withmonotonically non-decreasing, lower-bounded nonconformity scores ˜ F ( X i , S i,j ) .
Revised: , B (where B ≤ |Y| is a ﬁnite hyper-parameter) be nested sets with non-decreasing, lower-bounded nonconformity scores (cid:101) F ( X i , S i,j ) .

Original: For k ∈ R > 0 and δ ∈ (0 , 1) deﬁne the random variables T k and T k,δ asT k := sup (cid:26) t ∈ R : n [Equation] ≤ k
Revised: For k ∈ R > 0 and δ ∈ (0 , 1) deﬁne the random variables T k and T k,δ as

Original: [Equation]
Revised: [Equation]

Original: and [Equation] [Equation] Thenwe have that for any j ∈ { 1 , . . 
Revised: Then for any j ∈ { 1 , . . .

Original: . , |Y|} ,  E [FD( Z n +1 , S n +1 ,j ) | 
Revised: , B } , we have that E [FP( Z n +1 , S n +1 ,j ) | (cid:101) F

Original: ˜ F ( X i , S n +1 ,j ) ≤ T k ] ≤ k , and
Revised: ( X i , S n +1 ,j ) ≤ T k ] ≤ k , and

Original: P (FD( Z n +1 , S n +1 ,j ) ≤ k | ˜ F ( X i , S n +1 ,j ) ≤ T k,δ ) ≥ 1 − δ .
Revised: P (FP( Z n +1 , S n +1 ,j ) ≤ k | (cid:101) F ( X i , S n +1 ,j ) ≤ T k,δ ) ≥ 1 − δ .

Original: The proof, given in Appendix A, usesthe results from marginal RCPS (Angelopoulos et al., 2021b).
Revised: The proof, given in Appendix A, uses results from marginal RCPS (Angelopoulos et al., 2021b).

Original: Remark 4.4.
Revised: Remark 4.4.

Revised: The hyper-parameter B plays an important role.

Original: T k may be very conservative if  |Y| is very large, to the point where T k =
Revised: T k may be very conservative if B = |Y| and |Y| is very large, to the point where T k =

Original: −∞ always if |Y| > k ∗ ( n + 1) .
Revised: −∞ always if |Y| > k ∗ ( n + 1) .

Original: To account for this, it can be beneﬁcial to truncate the considered label space Y for an example x i to only the top m (cid:28) k ∗ ( n + 1) individual candidates, { y 1 , . . .
Revised: It can be beneﬁcial to truncate the considered label space Y for an example x i to only the top B (cid:28) k ∗ ( n + 1) individual candidates, { y 1 , . . .

Original: , y m } ∈ Y m .
Revised: , y B } ∈ Y B .

Original: For example, for text generation tasks (like NMT) Y is inﬁnite, but we can  restrict our predictions to a subset of the top m beach search candidates (where m can still be large to a reasonable degree).
Revised: For example, for text generation tasks (like machine translation), the label-space Y is inﬁnite, but we can instead restrict our predictions to a subset of the top B beam search candidates (where B can still be large to a reasonable degree).

Original: Remark 4.5.
Revised: Remark 4.5.

Original: Note that no constraints are put on the underlying ˜ F in Theorem 4.3 (it is modelagnostic).
Revised: Note that no constraints are put on the underlying (cid:101) F in Theorem 4.3; it is modelagnostic, and need not even be a DeepSets architecture of the form of Eq.

Original: If, however, ˜ F is a good estimator of FD( Z, S ) | X , then T k and T k,δ are likely to identify sets that are approximately valid conditional on X n +1 , which we show empirically  in §6.
Revised: If, however, (cid:101) F is a good estimator of FP( Z, S ) | X , then T k and T k,δ are likely to identify sets that are approximately valid conditional on X n +1 , which we show empirically for our choice of DeepSets models in §6.

Original: 4. S ELECTING THE FINAL OUTPUT SET
Revised: 4. S ELECTING THE FINAL OUTPUT SET

Original: The main result of Theorem 4.3 is that, using the calibrated nonconformity threshold T k or T k,δ = t ∗ , we can construct a collection of sets that are simultaneously valid.
Revised: The main result of Theorem 4.3 is that, using the calibrated nonconformity threshold T k or T k,δ = t ∗ , we can construct a collection of sets that are simultaneously valid.

Original: In other words, we are free to select any setcontained in the ﬁltered set of candidates { S i : ˜ F ( x, S i )
Revised: In other words, we are free to select any set in the ﬁltered set of candidates {S n +1 ,j : (cid:101) F ( x, S n +1 ,j )

Original: ≤ t ∗ } as a valid output.
Revised: ≤ t ∗ } as a valid output.

Original: Ideally, we would be able to follow the the oracle strategy in returning the smallest set with the highest true positive proportion.

Original: This would make our predictions efﬁcient , in the sense that we are not including more false positives than necessary (even if the total number is still ≤ k ).

Original: A reasonable choice is to then choose S i ∗ where i ∗ = arg max

Original: i |S i | − F ( x, S i ) ; but this can be sub-optimal if F is not accurate.

Original: As a robust and straightforward alternative, we simply take the largest set in our ﬁltered set of candidates as our ﬁnal output.
Revised: As a greedy, but simple and robust, approach we simply take the largest set in our ﬁltered set of candidates as our ﬁnal output.

Original: That is, for an input x ∈ X and calibrated threshold t ∗ ∈ R
Revised: That is, for an input x ∈ X and calibrated threshold t ∗ ∈ R

Original: [Equation]
Revised: [Equation]

Original: (13) where C ∗ is either C k or C k,δ (i.e., they are constructed the same way, but for different thresholds t ∗ ), and where we deﬁne arg max to return ∅ if the set {S ∈ {S 1 , . ..
Revised: (12) where C ◦ is either C k or C k,δ (i.e., they are constructed the same way, but for different thresholds t ∗ ), and where we deﬁne arg max to return ∅ if the set {S ∈ {S 1 , . .

Original: , S |Y| } : ˜ F ( x, S ) ≤ t ∗ } is empty.
Revised: S B } : (cid:101) F ( x, S ) ≤ t ∗ } is empty.

Original: Proposition 4.6 (Greedy FD-CP) .
Revised: Proposition 4.6 (Greedy FP-CP) .

Original: As deﬁned in Eq.
Revised: As deﬁned in Eq.

Original: (13) , the set C k ( X n +1 , T k ) is k -FD valid, and the set C k,δ ( X n +1 , T k,δ ) is ( k, δ ) -FD valid.
Revised: (12) , the set C k ( X n +1 , T k ) is k -FP valid, and the set C k,δ ( X n +1 , T k,δ ) is ( k, δ ) -FP valid.

Original: Furthermore, among sets S n +1 ,j ∈ {S n +1 , 1 , . . .
Revised: Furthermore, among sets S n +1 ,j ∈ {S n +1 , 1 , . . .

Original: , S n +1 , |Y| } that have nonconformity scores ≤ T  k or T k,δ , the set C ∗ ( X n +1 , T ∗ ) maximizes the TDP.
Revised: , S n +1 , |Y| } that have nonconformity scores ≤ T ◦ = T k or T k,δ , the set C ◦ ( X n +1 , T ◦ ) maximizes the TPR.

Original: The proof follows from Theorem 3.1, and from the fact that the TDP is monotonically non-decreasing in set size when S n +1 ,j are nested.

Original: Example pseudo-code for our procedure (speciﬁcally as would be applied to in-silico screening with the goal of k -FD control) is given in Algorithm 1.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: We empirically evaluate our method on three different tasks covering computational chemistry, computer vision, and natural language processing.

Original: In this section, we brieﬂy outline each task andits associated modeling choices.
Revised: In this section, we brieﬂy outline our evaluation tasks and associated models.

Original: We also describe our implementation, evaluation, and baselines.
Revised: We also describe our evaluation and baselines.

Original: For ease of experimentation (and performance reasons, see Remark 4.4), we choose to cap the number of considered labels per example to100 for each task (but the full label space Y can be much larger).
Revised: For all experiments, we set B = 100 (recall Remark 4.4), which caps the number of considered labels per example to the top 100 . Appendix C contains additional details.

Original: 5. T ASKS
Revised: T ASKS

Original: In-silico screening for drug discovery.
Revised: In-silico screening for drug discovery.

Original: As introduced in §1, the goal of in-silico screening is to identify potentially effective drugs to manufacture and test.
Revised: As introduced in §1, the goal of in-silico screening is to identify potentially effective drugs to manufacture and test.

Original: We use the ChEMBL database (Mayr et al., 2018) to screen molecules for combinatorial constraint satisfaction, where given a constraint such as “ has property A but not property B ”, we want to identify the subset of molecules from a given set of candidates that have the desired attributes.
Revised: We use the ChEMBL database (Mayr et al., 2018) to screen molecules for combinatorial constraint satisfaction, where given a constraint such as “ has property A but not property B ,” we want to identify the subset of molecules from a given set of candidates that have the desired attributes.

Original: We partition the dataset both by molecules and property combinations, so that at test time the model must make predictions on combinations it has never been tested on before, over a pool of molecules that it has never seen before.
Revised: We partition the dataset both by molecules and property combinations, so that at test time the model must make predictions on combinations it has never been tested on before, over a pool of molecules that it has never seen before.

Original: Scores for individual candidate molecules are obtained via combining independent property assignment probabilities predicted by an ensemble of directed MPNNs (Yang et al., 2019) using a naive independence assumption.
Revised: Scores for individual candidate molecules are obtained via combining independent property assignment probabilities predicted by an ensemble of directed MPNNs (Yang et al., 2019) using a naive independence assumption.

Original: Our training, validation, and test sets consist of 4 . 4 k , 797 , and 5 k screening examples, respectively.

Original: We reuse the training set to learn both the base MPNN model and the deep sets model F .

Original: Object detection.
Revised: Object detection.

Original: The goal of object detection is to place bounding boxes around all objects  in an image (of which there may be many).
Revised: The goal of object detection is to place bounding boxes around all objects of a certain type that are present in an image (of which there may be few, many, or none).

Original: We use the MS-COCO dataset (Lin et al., 2014), a large scale object detection dataset with images of complex everyday scenes containing 80 object categories.
Revised: We use the MS-COCO dataset (Lin et al., 2014), a large scale object detection dataset with images of complex everyday scenes containing 80 object categories (such as person, bicycle, dog, car, etc).

Original: We extract typed bounding box candidates (i.e., tuples of both location and category) using an EfﬁcientDet model (Tan et al., 2020) with non-maximum suppression.
Revised: We extract typed bounding box candidates (i.e., tuples of both location and category) using an EfﬁcientDet model (Tan et al., 2020) with non-maximum suppression.

Original: True positives are deﬁned as boxes that have an intersection over union (IoU) > 0 .
Revised: True positives are deﬁned as boxes that have an intersection over union (IoU) > 0 .

Original: 5 with a gold annotation of the same type.
Revised: 5 with a gold annotation of the same type.

Original: Since the ofﬁcial test set is hidden, we use the 5 k examples from the validation set and randomly partition them into 1 / 1 / 3 k sized splits for training F , validation, and testing.

Original: Entity extraction.
Revised: Entity extraction.

Original: In entity extraction, we are interested in identifying all named entities that appear in a  sentence x of length l , x = { w 1 , . . .
Revised: In entity extraction, we are interested in identifying all named entities that appear in a tokenized sentence x of length l , x = { w 1 , . . .

Original: , w l } .
Revised: , w l } .

Original:  We report results on theCoNLL NER dataset (Tjong
Revised: A named entity is a contiguous span { w start , . . . , w end } ⊆ x of the input sentence that refers to a real-world object (such as a person, location, organization, or product) that can be denoted with a proper name. We report results on the

Original:  Kim Sang and De Meulder, 2003), where we use the PURE span-based entity extraction model of Zhong and Chen (2021) to predict scores for all O ( l 2 ) candidate spans.
Revised: CoNLL NER dataset (Tjong Kim Sang and De Meulder, 2003), where we use the PURE span-based entity extraction model of Zhong and Chen (2021) to predict scores for all O ( l 2 ) candidate spans.

Original: We consider exact span predictions of the correct category to be true positives, and all others to be false positives.
Revised: We consider exact span predictions of the correct category to be true positives, and all others to be false positives.

Original: Not all sentences are guaranteed to have entities (in fact, many do not contain any).
Revised: Not all sentences are guaranteed to have entities (in fact, many do not contain any).

Original: We extract 1 k examples for training F out of the 3 .

Original: 3 k validation set, and report test results on the original 3 . 5 k test set.

Original: 5. E VALUATION
Revised: 5. E VALUATION

Original: We use a proper training, validation, and test set for each task.
Revised: We use a proper training, validation, and test set for each task.

Original: We use the training set to learn all models, i.e., p θ ( y i ∈ Z | x ) and F . {+
+}
Revised: We use the training set to learn all models, i.e., p θ ( y c ∈ Z | x ) and F . {+
+}

Original: We perform model selection on the validation set, and report ﬁnal numbers as the average over 1000 random trials on the test set, where in each trial we partition the data into 80% calibration ( x 1: n ) and 20% prediction points ( x n +1 ).
Revised: We do model selection on the validation set, and report ﬁnal numbers as the average over 1000 random trials on the test set, where in each trial we partition the data into 80% calibration ( x 1: n ) and 20% prediction points ( x n +1 ).

Original: To compare aggregate performance across tolerance levels, we plot each metric as a function of k/ |Y| (up to k/ |Y| = 0 . 5 ), and compute the AUC.
Revised: To compare aggregate performance across tolerance levels, we plot each metric as a function of k (up to k = B ), and compute the area under the curve (AUC).

Original: Shaded regions show the 16 - 84 th percentiles across trials.
Revised: Shaded regions show the 16 - 84 th percentiles across trials.

Original: In addition to TDR and FD metrics, inspired by Angelopoulos et al. (2021c), we compute the size-stratiﬁed k -FD violation :[Equation]
Revised: In addition to TPR and FP, inspired by Angelopoulos et al. (2021c), we compute the size-stratiﬁed k -FP violation :

Original: [Equation]
Revised: [Equation]

Original:  where {A} aj =1 forms a partition of { 1 , . .
Revised: (13) where {A} as =1 forms a partition of { 1 , . .

Original: . , |Y|} , and (cid:98) E denotes the empirical average over our test trials.
Revised: . , |Y|} , and (cid:98) E denotes the empirical average over our test trials.

Original: SSFD k,δ for ( k, δ ) -FD violation is deﬁned similarly when compared to the target 1 − δ level.
Revised: SSFP k,δ for ( k, δ ) -FP violation is deﬁned similarly, where we compare the worst-case average number of predictions with more than k false positives to the tolerance δ .

Original: As argued in Angelopoulos et al.
Revised: As argued in Angelopoulos et al.

Original: (2021c), a lower size-stratiﬁed violation suggests that a classiﬁer has better conditional coverage.
Revised: (2021c), a lower size-stratiﬁed violation suggests that a classiﬁer has better conditional coverage.

Original: As marginal validity is  theoretically guaranteed by our procedure (and holds empirically in our experiments), we focus on this metric in our results.
Revised: As marginal validity is already theoretically guaranteed by our procedure, we focus on this metric.

Original: 5. B ASELINES
Revised: 5. B ASELINES

Original: For all experiments, we compare our conformalized deep sets model (NN), to the following baselines:
Revised: For all experiments, we compare our conformalized DeepSets model (NN), to the following baselines:

Original: [Figure]
Revised: Top-

Original: • Top k. We naively take the top k (cid:48) ﬁxed predictions for any x n +1 , where k (cid:48) is found using average performance on the calibration set (without any correction factors, so it is not generally guaranteed to be valid).
Revised: k . We naively take the top k (cid:48) ﬁxed predictions for any x n +1 , where k (cid:48) is found using average performance on the calibration set (without any correction factors, so it is not generally guaranteed to be valid).

Original: Note that k (cid:48) can be (and mostly is) different than the user-speciﬁed k for FD
Revised: Note that k (cid:48) can be (and mostly is) different than the user-speciﬁed k for FP. [Figure]

Original: • Inner Sets.
Revised: Outer Sets @ 90.

Original: We use the method of Cauchois et al.
Revised: We use the (one-sided) multi-label conformal prediction technique of Cauchois et al.

Original: (2021) to bound P (  C (cid:15) ( 
Revised: (2021) to bound P ( Z n +1 ⊆ C (cid:15) ( X n +1 )) ≥ 0 .

Revised: Though not directly comparable, we use this to benchmark our method against sets that preserve marginal coverage (at a typical level).

Revised: For simplicity, we use the direct inner/outer method without dynamic CQC quantiles.

Original:  X n +1 ) ⊆
Revised: Inner Sets. Again, we use the (one-sided) method of Cauchois et al. (2021), this time to bound P ( C (cid:15) ( X n +1 ) ⊆

Original: Z n +1 ) ≥ 1 − (cid:15) at level (cid:15) =
Revised: Z n +1 ) ≥ 1 − (cid:15) at level (cid:15) =

Original: k/ |Y|  for k -FD control and at level (cid:15) =
Revised: k/B (recall that B ≤ |Y| is the truncation parameter, and the FP upper bound) for k -FP control and at level (cid:15) =

Original: δ for ( k, δ ) -FD control.
Revised: δ for ( k, δ ) -FP control.

Original: It is straightforward to show that these levels of (cid:15) conservatively achieve k -FD and ( k, δ ) -FD control.
Revised: It is straightforward to show that these levels of (cid:15) conservatively achieve k -FP and ( k, δ ) -FP control.

Original: For simplicity, we use a one-sided version of the direct inner/outer method described in Cauchois et al.

Original: , and do not use conformalized quantile regression for computing thresholds

Original: • Independent scoring (max).
Revised: Independent scoring (max).

Original: We take F ( x, S ) to be the maximum individual label uncertainty in S , max { 1 − p θ ( y i ∈ Z | x ): y i ∈ S} .
Revised: We take F ( x, S ) to be the maximum individual label uncertainty in S , max { 1 − p θ ( y c ∈ Z | x ): y c ∈ S} .

Original: This is equivalent to choosing labels independently.
Revised: This is equivalent to choosing labels independently.

Original: The full model is calibrated using the same FD-CP algorithm (it is a drop-in replacement for the NN).
Revised: The full model is calibrated using the same FP-CP algorithm (it is a drop-in replacement for the NN).

Original: Average scoring (avg).
Revised: Average scoring (avg).

Original: We take F ( x, S ) to be the average individual label uncertainty in S , |S| − 1 (cid:80) y i ∈S 1 − p θ ( y i ∈ Z | x ) .
Revised: We take F ( x, S ) to be the average individual label uncertainty in S , |S| − 1 (cid:80) y c ∈S 1 − p θ ( y c ∈ Z | x ) .

Original: We calibrate p θ ( y i ∈ Z | z ) using Platt scaling.
Revised: We calibrate p θ ( y c ∈ Z | z ) using Platt scaling (Platt, 1999).

Original: As with the max score, the full model is calibrated using the same FD-CP algorithm.
Revised: As with the max score, the full model is calibrated using the same FP-CP algorithm

Revised: Baseline (1) contrasts our approach with what is normally a “ﬁrst thought” in practice; (2) and (3) test the efﬁcacy of our system over existing techniques; (4) and (5) compare to simpler scoring variants.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In the following, we present our main results for conformal prediction with false discovery control.

Original: We evaluate both k -FD and ( k, δ ) -FD approaches.

Original: In all of the following experiments, we report results on the test set, using conﬁgurations selected based on validation set performance.

Original: Constraining false discoveries.
Revised: Constraining false positives.

Original: The top row of Figure 2 and 3 shows the size-stratiﬁed violation (Eq. 14) for ( k, δ ) -FD and k -FD, respectively.
Revised: The top rows of Figures 2 and 3 show the size-stratiﬁed violation (Eq. (13)) for ( k, δ ) -FP and k -FP, respectively.

Original: Our deep set NN is able to predict the FD of a set better than  the max or average baselines, and therefore, the model has better conditional coverage, meaning fewer excessive false discoveries. Other than the in-silico screening with ( k, δ ) -FD where the Inner Sets method is very conservative, the Top k and the Inner Sets baselines allow very small violations.
Revised: Across levels of k , our approach based on DeepSets (NN) achieves substantially lower worst-case violations than either of the max or average score baselines.

Original: However, this resultby itself is not  impressive—as simply always abstaining (returning an empty set) will lead to zero SSFD.
Revised: The Top- k and Inner Sets approaches also prevent large violations (though, by itself, this result is not necessarily impressive—as simply always returning an empty set will lead to zero SSFP ).

Original: When also considering the TDR (bottom row in the ﬁgures), we see that our FD-CP model is superior (often considerably so).
Revised: When accounting for TPR (bottom rows), we see that our FP-CP model does considerably better.

Original: Maximizing true discovery rates.
Revised: Maximizing true positive rates.

Original:  Table 1 summarizes the empirical results for the in-silico screening task with several representative conﬁgurations.
Revised: The top rows of Figures 2 and 3 plot TPR rates and AUC across many values of k , while Table 1 details results for the in-silico screening task for several representative conﬁgurations.

Original: We see that our FD-CP (NN) method provides signiﬁcantly higher true discovery rates while meeting the speciﬁed false discovery bounds.
Revised: On the screening task, we see that our FP-CP (NN) method provides signiﬁcantly higher TPR than other baselines.

Original: For example, allowing no more than 5 false discoveries leads to a TDR of 36 .
Revised: For example, allowing no more than 5 false positives leads to a TPR of 36 .

Original: 1% with k -FD.
Revised: 1% with k -FP.

Original: In 
Revised: In comparison, the TPR of Top k is only 29 . Differences in TPR are less pronounced on the object detection and entity extraction tasks, however, all FP-CP methods (with max, avg, or NN scoring) provide high TPR (exceeding non FP-CP methods) even at low values of k .

Original: [Figure] [Table]
Revised: [Figure] [Figure]

Revised: FP

Original: [Figure]
Revised: [Figure]

Revised: k

Original: [Table]
Revised: [Figure]

Revised: Avg.

Original: [Figure]
Revised: [Figure]

Revised: TPR

Original: [Table]
Revised: Avg. FP TPR Avg.

Original: [Table]
Revised: FP TPR Avg. FP

Original: [Table]
Revised: TPR Avg. FP TPR k =4.

Original: [Table]
Revised: 36.1 k =14.

Original: comparison, the TDR of Top k is only 29 .
Revised: 59.9 k =24. 73.2 k =34. FP ≤ k TPR

Original: The advantage of the deep set neural network for estimating the number of false discoveries is most pronounced in the challenging in-silico task.
Revised: FP ≤ k TPR FP ≤ k TPR

Revised: FP ≤ k

Revised: TPR

Revised: FP ≤ k

Revised: TPR k =100.

Revised: 31.6 k =94.

Revised: [Table]

Revised: Comparison to conformal coverage methods.

Original: The two other tasks are overall easier, as suggested by their higher TDR (Figures 2 and 3).
Revised: Table B.1 gives the results of the coverage-seeking Outer Sets method at level 0 . 90 (a typical tolerance). Indeed, we achieve strong TPR ( 97 .

Revised: 2% for the in-silico screening task), but also incur a high false positive cost in the process ( 63 .

Original: Yet, our FD-CP (NN) consistently provides high TDR compared to all baselines for both k -FD and ( k, δ ) -FD.
Revised: 6 average FP for in-silico screening). In contrast, our method allows us to directly modulate false positives, without losing high TPR (e.g., equivalently controlling for ≤ 63 . 6 FP, we acheive 97 . 0% TPR on screening).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Conformal prediction, in its standard formulation, already grants theoretical performance guarantees that can bea critical in many machine learning applications.
Revised: Conformal prediction, in its standard formulation, already grants theoretical performance guarantees that can be critical in many machine learning applications.

Original: Naively applying CP, however, can yield disappointing results.
Revised: Naively applying CP, however, can yield disappointing results.

Original: Even if the target coverage is indeed upheld, the predicted sets may be too large,  include too many false positives to be practical.
Revised: Even if the target coverage is indeed upheld, the predicted sets may be too large, and include too many false positives to be practical.

Original: In this paper, we proposed a method for trading coverage guarantees in favor of precision guarantees, where we enforce a limit to the number of false positives that are contained in our prediction sets.
Revised: In this paper, we proposed a method for trading coverage guarantees in favor of precision guarantees, where we enforce a limit to the number of false positives that are contained in our prediction sets.

Original: Our results show that our method yields classiﬁers that (1) still achieve strong true discovery rates compared to their coverage-seeking counterparts, and (2) provide meaningful output sets with effectively controlled total false discovery counts.
Revised: Our results show that our method yields classiﬁers that (1) still achieve strong true positive rates compared to their coverage-seeking counterparts, and (2) provide meaningful output sets with effectively controlled total false positive counts.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Our FP control methods are general and can be applied to many applications and on top of any model for computing nonconformity scores.
Revised: Our FP control methods are general and can be applied to many applications and on top of any model for computing nonconformity scores.

Original: It’s worth noting that any undesirable biases exhibited by underlying models can still propagate to the prediction sets of our methods.
Revised: It’s worth noting that any undesirable biases exhibited by underlying models can still propagate to the prediction sets of our methods.

Original: While our methods provide marginal performance guarantees, we recommend  any application to perform controlled evaluation across target populations to ensure fairness.
Revised: While our methods provide marginal performance guarantees, we recommend that any application to perform controlled evaluation across target populations to ensure fairness.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: All datasets used in this paper are publicly available (see §5.1).
Revised: All datasets used in this paper are publicly available (see §5.1).

Original: Also, we use publicly available models for computing the nonconformity scores.
Revised: Also, we use publicly available models for computing the nonconformity scores.

Original: For drug discovery, we use the ensemble of directed MPNNs from ChemProp ( https://github.com/chemprop/chemprop ).
Revised: For drug discovery, we use the ensemble of directed MPNNs from chemprop ( https://github.com/chemprop/chemprop ).

Original: For object detection, we use tf_efﬁcientdet_d2 from https://github.com/rwightman/ efficientdet-pytorch .
Revised: For object detection, we use tf_efficientdet_d2 from https://github.com/rwightman/ efficientdet-pytorch .

Original: For entity extraction, we train the PURE entity model ( https: //github.com/princeton-nlp/PURE ) on the ConLL03 dataset using a context window of 100, batch size 32, albert-base-v2 encoding model, and the suggested learning rate of 1 e − 5 , and task learning rate 5 e − 4 .
Revised: For entity extraction, we train the PURE entity model ( https: //github.com/princeton-nlp/PURE ) on the ConLL03 dataset using a context window of 100, batch size 32, albert-base-v2 encoding model, and the suggested learning rate of 1 e − 5 , and task learning rate 5 e − 4 .

Original: Default values were used for all other parameters.
Revised: Default values were used for all other parameters.

Original: The results in Sectionare all based on the experimental setting described in Section 5.
Revised: The results in Section 6 are all based on the experimental setting described in Section 5.

Original: We will release our code for running these experiments and for reproducing all plots and tables.
Revised: We will release our code for running these experiments and for reproducing all plots and tables.

Original: Appendix A P ROOFS A. P ROOF OF T HEOREM 3. 
Revised: Appendix A P ROOFS A. P ROOF OF T HEOREM 3.

Original: Proof.
Revised: Proof.

Original: This is a well-known result; we prove it here for completeness.
Revised: This is a well-known result; we prove it here for completeness.

Original: Since the nonconformity scores V i are constructed symmetrically, then
Revised: Since the nonconformity scores V i are constructed symmetrically, then

Original: [Equation]
Revised: [Equation]

Original: for all permutations ( σ (1) , . . .
Revised: for all permutations ( σ (1) , . . .

Original: σ ( n + 1)) .
Revised: σ ( n + 1)) .

Original: Therefore, if { ( X i , Y i ) } n +1 i =1 are exchangeable, then so too are their nonconformal scores { V i } n +1 i =1 .
Revised: Therefore, if { ( X i , Y i ) } n +1 i =1 are exchangeable, then so too are their nonconformal scores { V i } n +1 i =1 .

Original: By the construction of C , we have
Revised: By the construction of C , we have

Original: [Equation]
Revised: [Equation]

Original: This implies that V n +1 is ranked among the (cid:98) (1 − (cid:15) ) · ( n + 1) (cid:99) smallest of V 1 , . . .
Revised: This implies that V n +1 is ranked among the (cid:100) (1 − (cid:15) ) · ( n + 1) (cid:101) smallest of V 1 , . . .

Original: , V n , ∞ .
Revised: , V n , ∞ .

Original: Since V i are exchangeable, this happens with probability at least 1 − (cid:15) .
Revised: Since V i are exchangeable, this happens with probability at least 1 − (cid:15) .

Original: A. P ROOF OF T HEOREM 4. 
Revised: A. P ROOF OF T HEOREM 4.

Original: Our proof will use Theorem 1 of Angelopoulos et al.
Revised: Our proof will use Theorem 1 of Angelopoulos et al.

Original: (2021a), which we state here:
Revised: (2021a), which we state here:

Original: Theorem A.1 (Marginal RCPS) .
Revised: Theorem A.1 (Marginal RCPS, monotonic non-increasing case) .

Original: Let L i : R → R , i = 1 , . . .
Revised: Let L i : R → R , i = 1 , . . .

Original: , n + 1 be exchangeableloss functions, where L i ( t ) is non-increasing in t .
Revised: , n +be exchangeable functions, where L i ( t ) is non-increasing in t .

Original: Also, take g : R → R where g ( x ) is non-decreasing in x .
Revised: Also, take g : R → R where g ( x ) is non-decreasing in x .

Original: Further assume that g ◦ L i is right-continuous, and 
Revised: Further assume that g ◦ L i is right-continuous, and inf t g ( L i ( t )) < γ, sup t g ( L i

Original: [Equation]
Revised: [Equation]

Original: ∞ almost surely .
Revised: ∞ almost surely .

Original: [Equation]
Revised: [Equation]

Original: For any γ ≥ 0 , deﬁne the random variable T ( γ, g ) as 
Revised: For any γ ≥ 0 , deﬁne the random variable T ( γ, g ) as [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then E [ g ◦ L n +

Revised: Then E [ g ◦ L n +

Original: ( T ( γ ; g ))] ≤ γ + Bn +1 .
Revised: ( T ( γ ; g ))] ≤ γ + Bn +1 .

Original: Corollary A.2 (Marginal RCPS, adjusted) .
Revised: Corollary A.2 (Marginal RCPS, adjusted) .

Original: Under the same setting as in Theorem A.1,
Revised: Under the same setting as in Theorem A.1,

Original: [Equation]
Revised: [Equation]

Original: 17) where
Revised: 16) where

Original: [Equation]
Revised: [Equation]

Original: Following their analysis, we give a second corollary for lower boundingthe “reward” of function 
Revised: Following their analysis, we give a second corollary for lower bounding function R n +1 , where

Original: Rn +1 , where R 1 , . .
Revised: R 1 , . .

Original: , R n +1 are now non-decreasing functions.
Revised: , R n +1 are now non-decreasing functions.

Original: Corollary A.3 (Marginal RCPS, monotonic non-decreasing case) .
Revised: Corollary A.3 (Marginal RCPS, monotonic non-decreasing case) .

Original: Similar to the setting in Theorem A.1, let R i
Revised: Similar to the setting in Theorem A.1, let R i

Original: R → R , i = 1 , . .
Revised: R → R , i = 1 , . .

Original: . , n + 1 be exchangeablereward functions, where R i ( t ) is non-decreasing in t .
Revised: . , n + 1 be exchangeable functions, where R i ( t ) is non-decreasing in t .

Original: Also, take g : R → R where g ( x ) is non-decreasing in x .
Revised: Also, take g : R → R where g ( x ) is non-decreasing in x .

Original: Further assume that g ◦ R i is right-continuous, and 
Revised: Further assume that g ◦ R i is right-continuous, and inf

Original: [Equation] 
Revised: [Equation] [Equation]

Original: almost surely .
Revised: almost surely .

Original: [Equation]

Original: For any γ ≤ 0 , deﬁne the random variable T ( γ, g ) as
Revised: For any γ ≤ 0 , deﬁne the random variable T ( γ, g ) as

Original: [Equation]
Revised: [Equation]

Original: (20) where we deﬁne inf ∅ = ∞ .
Revised: where we deﬁne inf ∅ = ∞ .

Original: Then E [ g ◦ R n +1 ( T ( γ ; g ))] ≥ γ .
Revised: Then E [ g ◦ R n +1 ( T ( γ ; g ))] ≥ γ .

Original: Proof.
Revised: Proof.

Original: Let
Revised: Let

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: [Equation] Since inf t g ( R i ( t )) ≥ 0 , sup t g ( R i ( t )) > C ≥ γ , T (cid:48) ( γ ; g ) and T ( γ, g ) are both well-deﬁned almost surely.
Revised: Since inf t g ( R i ( t )) ≥ 0 , sup t g ( R i ( t )) > C ≥ γ , T (cid:48) ( γ ; g ) and T ( γ, g ) are both well-deﬁned almost surely.

Original: Since inf t g ( R i ( t )) ≥ 
Revised: Since inf t g ( R i ( t )) ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original:  Thus, T (cid:48) ( γ ; g ) ≤ T ( γ ; g ) .
Revised: [Equation] Thus, T (cid:48) ( γ ; g ) ≤ T ( γ ; g ) .

Original: Since g ( R i ( t )) is non-decreasing in t ,
Revised: Since g ( R i ( t )) is non-decreasing in t ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Let E f be the unordered set (bag) of { R 1 , . .
Revised: Let E f be the unordered set (bag) of { R 1 , . .

Original: , R n +1 } .
Revised: , R n +1 } .

Original: Then T (cid:48)
Revised: Then T (cid:48)

Original: ( γ ; g ) is a function of E f , and is a constant conditional on E f .
Revised: ( γ ; g ) is a function of E f , and is a constant conditional on E f .

Original: Exchangeability of R i and right-continuity of g ◦ R i imply
Revised: Exchangeability of R i and right-continuity of g ◦ R i imply

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The proof is completed by  Eq.
Revised: The proof is completed by taking the expectation over E f and then applying Eq.

Original: We now prove our main theorem.
Revised: We now prove our main theorem.

Original: Proof.
Revised: Proof.

Original: w.l.o.g. let us the ﬂip signs of our non-conformity measures such that
Revised: W.l.o.g. let us the ﬂip signs of our non-conformity measures such that

Original: [Equation]
Revised: [Equation]

Original: and T k , T k,δ are instead deﬁned via inf .
Revised: and T k , T k,δ are instead deﬁned via inf .

Original: We are given that the sets S are nested, and it is clear from its deﬁnition that FD is non-decreasing for S i ⊂ S j .
Revised: We are given that the sets S are nested, and it is clear from its deﬁnition that FP is non-decreasing for S i ⊂ S j .

Original: Since the ranking of − ˜ F ( x, S ) is consistent with that of −|S| and monotonically non-increasing, MFD is therefore also monotonically non-increasing in t.
Revised: Since the ranking of − (cid:101) F ( x, S ) is consistent with that of −|S| and monotonically non-increasing, MFP is therefore also monotonically non-increasing in t .

Original: We prove T k ﬁrst.
Revised: We prove T k ﬁrst.

Original: Since Y is ﬁnite by assumption, we have that sup t MFD( x, Z , t ) ≤ |Y| < ∞ .
Revised: Since B is constrained to be ﬁnite, we have that sup t MFP( x, Z , t ) ≤ B < ∞ .

Original: Furthermore, since − ˜ F ( X i , S i,j ) is upper-bounded by assumption, we have that inf t MFD( x, Z , t ) = 0 < k ∈ R > 0 .
Revised: Furthermore, since − (cid:101) F ( X i , S i,j ) is upper-bounded by assumption, we have that inf t MFP( x, Z , t ) = 0 < k ∈ R > 0 .

Original: Finally, as evident from its deﬁnition in Eq.
Revised: Finally, as evident from its deﬁnition in Eq.

Original: (25), MFD( x, Z , t ) is right-continuous.
Revised: (24), MFP( x, Z , t ) is right-continuous.

Original: Let L i ( t ) =
Revised: Let L i ( t ) =

Original: MFD( X i , Z i , t ) and g ( x ) = x .
Revised: MFP( X i , Z i , t ) and g ( x ) = x .

Original: We can then directly apply Corollary A.2 to obtain
Revised: We can then directly apply Corollary A.2 to obtain

Original: [Equation]
Revised: [Equation]

Original: The proof for T k is completed by the fact that MFD( X n +1 , Z n +1 , T k ) ≥[Equation] Z n [Equation]
Revised: The proof for T k is completed by the fact that MFP( X n +1 , Z n +1 , T k ) ≥

Original: [Equation] S n + [Equation] j ) conditioned on  S n +1 ,j  ≤ T k .
Revised: FP( Z n +1 , S n +1 ,j ) conditioned on (cid:101) F ( X n +1 , S n +1 ,j ) ≤ T k,δ .

Original: We proceed similarly for T k,δ .
Revised: We proceed similarly for T k,δ .

Original: Let L i ( t ) = 1 { MFD( x, Z , t ) ≤ k } .
Revised: Let L i ( t ) = 1 { MFP( x, Z , t ) ≤ k } .

Original: Since MFD is non-increasing and right-continuous, L i ( t ) is non-decreasing and right-continuous.
Revised: Since MFP is non-increasing and right-continuous, L i ( t ) is non-decreasing and right-continuous.

Original: Let γ = 1 − δ ∈ (0 , 1) .
Revised: Let γ = 1 − δ ∈ (0 , 1) .

Original: By the same argument as before about MFD and from the fact that L i ( t ) ∈ { 0 , 1 } , we can apply Corollary A.3 to get
Revised: By the same argument as before concerning MFP , and from the fact that L i ( t ) ∈ { 0 , 1 } , we can apply Corollary A.3 to get

Original: [Equation]
Revised: E [ 1 { MFP( X n +1 , Z n +1 , T k,δ ) ≤ k } ] = P (MFP( X n +1 , Z n +1 , T k,δ ) ≤ k ) ≥ 1 − δ.

Original: This gives P (MFD( X n +1 , Z n +1 , T k,δ ) ≤ k ) ≥ 1 − δ .
Revised: This gives P (MFP( X n +1 , Z n +1 , T k,δ ) ≤ k ) ≥ 1 − δ .

Original: The proof for T k,δ is then completed by the fact that MFD( X n +1 , Z n +1 , T k,δ ) ≥ FD( Z n +1 , S n +1 ,j ) conditioned on  S n +1 ,j  ≤ T k,δ .
Revised: The proof for T k,δ is then completed by the fact that MFP( X n +1 , Z n +1 , T k,δ ) ≥ FP( Z n +1 , S n +1 ,j ) conditioned on (cid:101) F ( X n +1 , S n +1 ,j ) ≤ T k,δ .

Original: A. P ROOF OF P ROPOSITION 4. 
Revised: A. P ROOF OF P ROPOSITION 4.

Original: Proof.
Revised: Proof.

Original: By deﬁnition, we have F ( X n +1 , C ∗ ( X n +1 , T ∗ ))
Revised: By deﬁnition, we have F ( X n +1 , C ◦ ( X n +1 , T ◦ ))

Original: ≤ T ∗ .
Revised: ≤ T ◦ , where ◦ is a placeholder for k or k, δ .

Original: By Theorem 4.3, we can then conclude that
Revised: By Theorem 4.3, we can then conclude that

Original: [Equation]
Revised: [Equation]

Original: ) and
Revised: (27) and

Original: [Equation]
Revised: [Equation]

Original: Finally, if S ⊂ S (cid:48) then y i ∈ S ⇒ y i ∈ S (cid:48) for any y i
Revised: Finally [Table] if S ⊂ S (cid:48) then y c ∈ S ⇒ y c ∈ S (cid:48) for any y c ∈ z ⊆ Y .

Original: ∈ Z .

Original: Since the candidate sets are nested,
Revised: Since the candidate sets are nested,

Original: [Equation]
Revised: [Equation]

Original: Thus picking the biggest candidate set maximizes the TDP.
Revised: Thus picking the biggest candidate set maximizes the TPP for a given Z = z [Table]

Revised: as well as its expectation over random Z ,

Revised: i

Revised: [Table]

Revised: e., the TPR.

Revised: B C ONFORMAL COVERAGE RESULTS

Revised: [Table]

Revised: C I MPLEMENTATION AND DATASET DETAILS

Revised: [Table]

Revised: We construct a molecular property screening task using the ChEMBL dataset (see Mayr et al., 2018).

Revised: Given a speciﬁed constraint such as “ is active for property A but not property B ,” we want to retrieve at least one molecule from a given set of candidates that satisﬁes this constraint.

Revised: The input for each molecule is its SMILES string, a notational format that speciﬁes its molecular structure.

Revised: The motivation of this task is to simulate in-silico screening for drug discovery, where it is often the case where chemists are searching for a new molecule that satisﬁes several constraints (such as toxicity and efﬁcacy limits), out of a pool of many possible candidates.

Revised: We split the ChEMBL dataset into a 60-20-20 split of molecules, where 60% of molecules are separated into a train set, 20% into a validation set, and 20% into a test set.

Revised: Next, we take all properties that have at least 50 positive and negative examples (to avoid highly imbalanced properties).

Revised: Of these properties, we take all N choose K combinations that have at least 100 molecules with all K properties labelled (ChEMBL has many missing values).

Revised: We set K to 2.

Revised: For each combination, we randomly sample an assignment for each property (i.e., { active , inactive } K ).

Revised: We discard combinations for which more than 90% of labeled molecules satisfy the constraint.

Revised: We keep 5000 combinations for the test set, 767 for validation, and 4375 for training.

Revised: The molecules for each of the combinations are only sourced from their respective splits (i.e., molecular candidates for constraints in the property combination validation split only come from the molecule validation split).

Revised: Therefore, at inference time, given a combination we have never seen before, on a molecules we have never seen before, we must try to retrieve at least one molecule that has the desired combination assignment.

Revised: Our directed Message Passing Neural Network (MPNN) is implemented using the chemprop repository (Yang et al., 2019).

Revised: The MPNN model uses graph convolutions to learn a deep molecular representation, that is shared across property predictions.

Revised: Each property value (active/inactive) is predicted using an independent classiﬁer head.

Revised: The ﬁnal prediction is based on an ensemble of 5 models, trained with different random seeds.

Revised: Given a combination assignment ( Z 1 = z 1 , . . .

Revised: , Z k = z k ) , we compute the joint likelihood independently, i.e.,

Revised: [Equation]

Revised: Object detection.

Revised: As discussed in §5, we use the MS-COCO dataset (Lin et al., 2014) to evaluate our conformal object detection.

Revised: MS-COCO consists of images of complex everyday scenes containingobject categories (such as person, bicycle, dog, car, etc.), multiple of which may be contained in any given example.

Revised: Since the ofﬁcial test set is hidden, we use the 5 k examples from the development set and randomly partition them into sets of size 1 k , 1 k , and 3 k for calibration, validation, and testing, respectively.

Revised: The EfﬁcientDet model (Tan et al., 2020) 4 for extracting bounding boxes uses a pipeline of three neural networks to extract deep features, and then predict candidates.

Revised: The model also uses a non-maximum suppression (NMS) post-processing step to reduce the total number of predictions by keeping only the one with the maximum score across highly overlapping prediction boxes.

Revised: We merge the predictions of all classes into a uniﬁed set, where each element is a tuple of (class, bounding box).

Revised: This means that multiple class predictions can be included for the same bounding box (i.e., there is class uncertainty), and multiple bounding boxes can be found for the same class (i.e., there are multiple objects in one image).

Revised: We deﬁne true positives as predictions that have an intersection over union (IoU) value > 0 .

Revised: 5 with a gold bounding box annotation, and that match the annotation’s class.

Revised: Entity extraction.

Revised: Entity extraction is a popular task in natural language processing.

Revised: Given a sentence, such as “ Barack Obama was born in Hawaii ,” the goal is to identify and classify all named entities that appear—i.e., (“Barack Obama”, Person) and (“Hawaii”, Location).

Revised: We use the CoNLL

Revised: NER dataset (Tjong Kim Sang and De Meulder, 2003), and extract 1 k examples for calibration out of the 3 . 3 k development set, and report test results on the 3 . 5 k test set.

Revised: For our base model, we use the entity extraction module of PURE (Zhong and Chen, 2021), that predicts span scores with a classiﬁer head on top of Albert-base (Lan et al., 2020) contextual embeddings.

Revised: The classiﬁcation head has two non-linear layers and uses the learned contextual embeddings of the span start and end tokens, concatenated with a learned span width embedding.

Revised: We train the model on the training set of the

Revised: CoNLL NER dataset.

Revised: We use the ofﬁcial code repository 5 and the following parameters: 1 e −learning rate, 5 e − 4 task learning rate, 32 train batch size, and 100 context window.

Revised: Similar to our object detection task, we treat exact span predictions of the correct category as true positives, and any other entity predictions as false positives.

Revised: As illustrated in Table C.1, a fairly large number of sentences do not contain any entities at all, while other sentences may contain several.

Revised: D P RACTICAL CONSIDERATIONS

Revised: In this section we address a number of practical considerations for our FP-CP method.

Revised: D. C HOOSING A SUITABLE k

Revised: An outstanding question a practitioner faces is how to choose the value of k for k -FP and ( k, δ ) -FP objectives.

Revised: The value of k in our method has a reliable and easy interpretation: it is the total number of incorrect answers.

Revised: For many tasks, such as in-silico screening, there is a direct relation between the number of noisy predictions (e.g., failed experiments conducted during wet-lab validation) and total “wasted” cost.

Revised: Therefore, for example, given some approximate budget Q and cost per noisy prediction c , a reasonable approach is to then set k ≈ Q/c .

Revised: Of course, the advantage of our approach is that the user may set k to whatever they wish—this might change based on their needs.

Revised: D. C HOOSING BETWEEN k - FWER AND FDR CONTROL

Revised: A related question to D.1 is when to target k -FWER (i.e., our k -FP and ( k, δ ) -FP objectives) or

Revised: FDR (e.g., using Angelopoulos et al. (2021a)).

Revised: This choice is well discussed in the multilple testing literature (Lehmann and Romano, 2005; Romano and Wolf, 2007; Gold et al., 2009).

Revised: An important aspect to consider is the size of the label space Y , natural rate of true and false positives, and the efﬁciency of the base model at separating true positives from false positives.

Revised: When the total number of true positives is large and |Y| is large then it is reasonable to control the FDR.

Revised: If, however, the natural rate of true positives is low, or they are not well separated from false positives, then the FDR can be quite high and hard to control, especially for smaller prediction sets (as the ratio of positive to negative labels can be quickly driven down even with the addition of only a few false positives).

Revised: As a intuition illustration, suppose for a given example there is one true positive that is ranked 10th by the base model.

Revised: For many applications, 10 total predictions (with 9 false positives) is not overly burdensome.

Revised: However, the lowest FDR cutoff that allows for this positive to be discoverable is 0 .(which, for other examples, may allow for hundreds of false positives—an outcome which may not be so desirable for some applications, even given a high number of accompanying true positives).

Revised: To satisfy a lower FDR rate, the algorithm must output an empty set (with FDR = 0).

Revised: This remains true even if there are a few (but not many) other true positives: for instance, in the previous example, if predictions 10-20 were also all true positives then the lowest FDR is still only 0.5—specifying a FDR tolerances any lower than this would force an empty set prediction.

Revised: D. L EARNING MORE EXPRESSIVE SET FUNCTIONS

Revised: Our choice of DeepSets model is motivated by its property of being a universal approximator for continuous set functions.

Revised: Of course, its realized accuracy depends on its exact parameterization and optimization.

Revised: In terms of input features, in §4.2, we chose a simplistic φ ( x, y c ) for two reasons: (1) we view it’s simplicity as an advantage (practitioners can simply plug-in individual multi-label probabilities, or other scalar conformity scores, that most out-of-the-box methods provide into a general framework without having to do any more work for providing additional features), and (2) it is easy to train this light-weight model on smaller amounts of data.

Revised: This approach, however, can discard potentially helpful information about the underlying input x , and any dependencies between labels y c and y (cid:48) c .

Revised: For example, if y c and y (cid:48) c are mutually exclusive, then the number of false positives if both are included in S has to be at least 1 .

Revised: Using more expressive φ that is able to capture and take advantage of this sort of side information about x and y c is a subject for future work.

Revised: D. C ONSTRUCTING NON - NESTED CANDIDATE SETS

Revised: Our careful construction of nested prediction sets that have bounded, non-decreasing, and rightcontinuous risks is key to our calibration procedure.

Revised: It is, however, limited by the underlying ranking of labels y c ∈ Y by the individual likelihood model p θ ( y c ∈ Z | x ) .

Revised: In theory, the set function F may be able to identify higher quality outputs sets S ∈ 2 Y by jointly considering all of the included elements (rather than ranking them one-by-one).

Revised: That said, the search process over 2 Y is expensive, and the FP risk is no longer non-decreasing, which necessitates more involved calibration methods that can account for non-monotonic risks, such as that of Angelopoulos et al.

Revised: (2021a)).

Revised: Nevertheless, this is a promising subject for future work, and one that can potentially be combined with efﬁcient search or candidate space pruning methods (e.g., such as in Fisch et al.

Revised: D. F AILURE CASES OF GREEDY OUTPUT SELECTION

Revised: In §4.4 we greedily select the largest candidate set that passes our ﬁlter threshold.

Revised: A downside of this approach is that it may include more false positives than necessary (i.e., to achieve the same

Revised: TPR).

Revised: Ideally, we would be able to follow the oracle strategy in returning the smallest set with the highest true positive proportion.

Revised: This would make our predictions efﬁcient , in the sense that we are not including more false positives than necessary (even if the total number is still ≤ k ).

Revised: A reasonable choice is to then choose S j ∗ where j ∗ = arg max j |S j | − F ( x, S j ) ; but this can be sub-optimal if F is not accurate, which motivates our greedy strategy.

Revised: Nevertheless, non-greedy set selection may be more efﬁcient if F is indeed accurate—this can be tested on a validation set.