L EARNING O PTIMAL C ONFORMAL C LASSIFIERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Modern deep learning based classiﬁers show very high accuracy on test data but this does not provide sufﬁcient guarantees for safe deployment, especially in highstake AI applications such as medical diagnosis.
Usually, predictions are obtained without a reliable uncertainty estimate or a formal guarantee.
Conformal prediction (CP) addresses these issues by using the classiﬁer’s predictions, e.g ., its probability estimates, to predict conﬁdence sets containing the true class with a user-speciﬁed probability.
However, using CP as a separate processing step after training prevents the underlying model from adapting to the prediction of conﬁdence sets.
Thus, this paper explores strategies to differentiate through CP during training with the goal of training model with the conformal wrapper end-to-end .
In our approach, conformal training (ConfTr) , we speciﬁcally “simulate” conformalization on mini-batches during training.
Compared to standard training, ConfTr reduces the average conﬁdence set size ( inefﬁciency ) of state-of-the-art CP methods applied after training.
Moreover, it allows to “shape” the conﬁdence sets predicted at test time, which is difﬁcult for standard CP.
On experiments with several datasets, we show ConfTr can inﬂuence how inefﬁciency is distributed across classes, or guide the composition of conﬁdence sets in terms of the included classes, while retaining the guarantees offered by CP.
I NTRODUCTION
In classiﬁcation tasks, for input x , we approximate the posterior distribution over classes y ∈
[ K ] := { 1 , . . .
, K } , denoted π y ( x ) ≈ P ( Y =
y
| X = x ) .
Following Bayes’ decision rule, the single class with highest posterior probability is predicted for optimizing a 0-1 classiﬁcation loss.
This way, deep networks π θ,y ( x ) with parameters θ achieve impressive accuracy on held-out test sets.
However, this does not guarantee safe deployment.
Conformal prediction (CP) (Vovk et al., 2005) uses a posttraining calibration step to guarantee a user-speciﬁed coverage : by allowing to predict conﬁdence sets C ( X ) ⊆
[ K ] , CP guarantees the true class Y to be included with conﬁdence level α , i.e . P ( Y ∈ C ( X )) ≥ 1 − α when the calibration examples ( X i , Y i ) , i ∈ I cal are drawn exchangeably from the test distribution.
This is usually achieved in two steps: In the prediction step , so-called conformity scores (w.r.t.
to a class k ∈ [ K ] ) are computed to construct the conﬁdence sets C ( X ) .
During the calibration step , these conformity scores on the calibration set w.r.t.
the true class Y i are ranked to determine a cut-off threshold τ for the predicted probabilities π θ ( x ) guaranteeing coverage 1 − α .
This is called marginal coverage as it holds only unconditionally, i.e ., the expectation is being taken not only w.r.t.
( X, Y ) but also over the distribution of all possible calibration sets, rather than w.r.t.
the conditional distribution p ( Y | X ) .
CP also outputs intuitive uncertainty estimates: larger conﬁdence sets | C ( X ) | generally convey higher uncertainty.
Although CP is agnostic to details of the underlying model π θ ( x ) , the obtained uncertainty estimates depend strongly on the model’s performance.
If the underlying classiﬁer is poor, CP results in too large and thus uninformative conﬁdence sets.
“Uneven” coverage is also a common issue, where lower coverage is achieved on more difﬁcult classes.
To address such problems, the threshold CP method of (Sadinle et al., 2019) explicitly minimizes inefﬁciency.
Romano et al.
(2020) and Cauchois et al.
(2020) propose methods that perform favorably in terms of (approximate) conditional coverage.
The adaptive prediction sets ( APS ) method of Romano et al.
(2020) is further extended by Angelopoulos et al.
(2021) to return smaller conﬁdence sets.
These various objectives are typically achieved by changing the deﬁnition of the conformity scores.
In all cases, CP is used as a post-training calibration step.
In contrast, our work does not focus on advancing CP a Conformal training:
Model πθ inputs
[Figure]
πθ [Figure] πθ [Figure]
[Figure]
pred.probs
[Figure]
pred Calibration Prediction τ = SmoothCal ( { ( πθ ( xi ) , yi ) } i ∈ B cal, α )
Cθ ( xi ; τ )= SmoothPred ( πθ ( xi ), τ ) diff.
w.r.t.
θ through πθ ( xi ) diff.
w.r.t.
θ through πθ ( xi ) and τ class
[Figure]
L [Figure] Indeed, while the ﬂexibility of CP regarding the underlying model appears attractive, it is also a severe limitation: Learning the model parameters θ is not informed about the post-hoc “conformalization”, i.e ., they are are not tuned towards any speciﬁc objective such as reducing expected conﬁdence set size ( inefﬁciency ).
During training, the model will typically be trained to minimize cross-entropy loss.
At test time, in contrast, it is used to obtain a set predictor C ( X ) with speciﬁc properties such as low inefﬁciency.
In concurrent work, Bellotti (2021) addresses this issue by learning a set predictor C ( X ) through thresholding logits: Classes with logits exceeding 1 are included in C ( X ) and training aims to minimize inefﬁciency while targeting coverage 1 − α .
In experiments using linear models only, this approach is shown to decrease inefﬁciency.
However, (Bellotti, 2021) ignores the crucial calibration step of CP during training and does not allow to optimize losses beyond marginal coverage or inefﬁciency.
In contrast, our work subsumes (Bellotti, 2021), but additionally considers the calibration step during training, which is crucial for further decreasing inefﬁciency.
Furthermore, we aim to allow ﬁne-grained control over class-conditional inefﬁciency or the composition of the conﬁdence sets by allowing to optimize arbitrary losses deﬁned on conﬁdence sets.
Our contributions can be summarized as follows:
We propose conformal training (ConfTr) , a procedure allowing to train model and conformal wrapper end-to-end .
This is achieved by developing smooth implementations of recent CP methods for use during training.
On each mini-batch, ConfTr “simulates” conformalization, using half of the batch for calibration, and the other half for prediction and loss computation, c.f . Fig.
1 a (cid:13) .
After training, any existing CP method can provide a coverage guarantee.
In experiments, using ConfTr for training consistently reduces the inefﬁciency of conformal predictors such as threshold CP ( T HR ) (Sadinle et al., 2019) or APS (Romano et al., 2020) applied after training.
We further improve over (Bellotti, 2021), illustrating the importance of the calibration step during training.
Using carefully constructed losses, ConfTr allows to “shape” the conﬁdence sets obtained at test time: We can reduce class-conditional inefﬁciency or “coverage confusion”, i e ., the likelihood of two or more classes being included in the same conﬁdence sets, c.f . Fig.
1 b (cid:13) .
Generally, in contrast to (Bellotti, 2021), ConfTr allows to optimize arbitrary losses on the conﬁdence sets.
Because ConfTr is agnostic to the CP method used at test time, our work is complementary to most related work, i.e ., any advancement in terms of CP is directly applicable to ConfTr.
For example, this might include conditional or application-speciﬁc guarantees as in (Sadinle et al., 2016; Bates et al., 2021).
Most importantly, ConfTr preserves the coverage guarantee obtained through CP.
CP Baseline Comparison by Ineff
Dataset, α
T
[Figure]
RAPS
CIFAR10, 0 .2.
CIFAR10, 0 .3.
CIFAR100, 0 . 01 19.22 10.63 16.62 14.
8 Class
I n e ff i
c i e n c y
CIFAR10:
Inefficiency by Class for Baseline+Thr
D IFFERENTIABLE C ONFORMAL P REDICTORS
We are interested in training the model π θ end-to-end with the conformal wrapper in order to allow ﬁne-grained control over the conﬁdence sets C ( X ) .
Before developing differentiable CP methods for training in Sec. 2.2, we review two recently proposed conformal predictors that we use at test time.
These consist of two steps, see Sec. 2.1: for prediction (on the test set) we need to deﬁne the conﬁdence sets C θ ( X ; τ ) which depend on the model parameters θ through the predictions π θ
and where the threshold τ is determined during calibration on a held-out calibration set ( X i , Y i ) , i ∈ I cal in order to obtain coverage.
2. C ONFORMAL P REDICTORS
The threshold conformal predictor (T HR ) (Sadinle et al., 2019) constructs the conﬁdence sets by thresholding probabilities: C θ ( x ; τ ) :
= { k : π θ,k ( x ) =:
E θ ( x, k ) ≥ τ } .
Here, the subscript
C θ makes the dependence on the model π θ and its parameters θ explicit.
During calibration, τ is computed as the α (1 + 1 / | I cal | )
-quantile of the so-called conformity scores E θ ( x i , y i )
=
π θ,y i ( x i ) .
The conformity scores indicate, for each example, the threshold that ensures coverage.
Marginal coverage of (1 − α ) is guaranteed on a test example ( X, Y ) .
In practice, T HR can also be applied on logits (T HR L) or log-probabilities (T HR LP) instead of probabilities.
Adaptive Prediction Sets (APS) (Romano et al., 2020) constructs conﬁdence sets based on the ordered probabilities.
Speciﬁcally, C θ ( x ; τ ) :
= { k : E θ ( x, k ) ≤ τ } with:
[Equation]
where π θ,y (1) ( x ) ≥ . .
≥ π θ,y ( K ) ( x ) and U is a uniform random variable in [0 , 1] to break ties.
Similar to T HR , the conformity scores E θ ( x i , y i ) w.r.t.
the true classes
y i are used for calibration, but the (1 − α )(1 + 1 / | I cal | ) -quantile is required to ensure marginal coverage on test examples.
Performance of CP is then measured using two metrics: (empirical and marginal) coverage (Cover) as well as inefﬁciency (Ineff) .
Letting I test be a test set of size | I test | , these metrics are computed as
Cover :=
| I test | (cid:88) i ∈ I test δ
[ y i ∈ C ( x i )] and Ineff := | I test | (cid:88) i ∈ I test | C (
[Equation]
where δ denotes an indicator function that is 1 when its argument is true and 0 otherwise.
Due to the marginal coverage guarantee provided by CP ( c.f .
(Romano et al., 2020) or App.
C), the empirical coverage, when averaged across several calibration/test splits, is Cover ≈ 1 − α .
Thus, we concentrate on inefﬁciency as the main metric to compare across CP methods and models.
With accuracy , we refer to the (top-1) accuracy with respect to the arg max -predictions, i.e ., arg max k π θ,k ( x ) , obtained by the underlying model π .
As shown in Fig.
2 (left), T HR clearly outperforms T HR L and APS w.r.t.
inefﬁciency (lower is better) averaged across random I cal /
I test splits (details in Sec. F).
CP is intended to be used as a “wrapper” around π θ .
“Better” CP methods generally result in lower inefﬁciency for a ﬁxed model π θ .
For example, following Fig.
2 (left), regularized APS (RAPS)
(Angelopoulos et al., 2021) recently showed how to improve inefﬁciency compared to APS by modifying the conformity score – without outperforming T HR , however.
Fine-grained control over inefﬁciency, e.g ., conditioned on the class or the composition of the C ( X ) is generally not possible.
Integrating CP into the training procedure promises a higher degree of control, however, requires differentiable CP implementations, e.g ., for T HR or APS.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
2. D IFFERENTIABLE P REDICTION AND C ALIBRATION S TEPS
Differentiating through CP involves differentiable prediction and calibration steps: We want
C θ ( x ; τ ) to be differentiable w.r.t.
the predictions π θ ( x ) , and τ to be differentiable w.r.t.
to the predictions π θ ( x i ) , i ∈
I cal used for calibration.
We emphasize that, ultimately, this allows to differentiate through both calibration and prediction w.r.t.
the model parameters θ , on which the predictions π θ ( x ) and thus the conformity scores E θ ( x, k ) depend.
For brevity, we focus on T HR , see Alg. 1 and discuss APS in App.
D.
Prediction involves thresholding the conformity scores E θ ( x, k ) , which can be smoothed using the sigmoid function σ ( z ) = 1 / 1+exp( − z ) and a temperature hyper-parameter T :
C θ,k ( x ; τ ) :
= σ ( ( E θ ( x,k ) − τ ) / T ) .
Essentially, C θ,k ( x ; τ ) ∈ [0 , 1] represents a soft assignment of class k to the conﬁdence set, i.e ., can be interpreted as the probability of k being included.
For T → 0 , the “hard” conﬁdence set will be recovered, i.e ., C θ,k ( x ; τ ) = 1 for k ∈ C θ ( x ; τ ) and 0 otherwise.
For T HR , the conformity scores are naturally differentiable w.r.t.
to the parameters θ because E ( x, k ) = π θ,k ( x ) .
As the conformity scores are already differentiable, calibration merely involves a differentiable quantile computation.
This can be accomplished using any smooth sorting approach (Blondel et al., 2020; Cuturi et al., 2019; Williamson, 2020).
These often come with a “dispersion” hyper-parameter (cid:15) such that smooth sorting approximates “hard” sorting for (cid:15) → 0 .
Overall, this results in the threshold τ being differentiable w.r.t.
the predictions of the calibration examples { ( π θ ( x i ) , y i } i ∈
I cal and the model’s
parameters θ .
As this approximation is using smooth operations, the coverage guarantee seems lost.
However, in the limit of T, (cid:15) → 0 we recover the original non-smooth computations and the corresponding coverage guarantee.
Thus, it is reasonable to assume that, in practice, we empirically obtain coverage close to (1 − α ) .
We found that this is sufﬁcient because these smooth variants are only used during training.
At test time, we use the original (non-smooth) implementations and the coverage guarantee follows directly from (Romano et al., 2020; Sadinle et al., 2019).
C ONFORMAL T RAINING (C ONF T R ): Learning C ONFORMAL P REDICTION
The key idea of conformal training (ConfTr) is to “simulate” CP during training, i.e ., performing both calibration and prediction steps on each mini-batch.
This is accomplished using the differentiable conformal predictors as introduced in Sec. 2.2.
ConfTr can be viewed as a generalization of (Bellotti, 2021) that just differentiates through the prediction step with a ﬁxed threshold, without considering the crucial calibration step, see App.
E. In both cases, only the training procedure changes. After training, standard (non-smooth) conformal predictors are applied.
3. C ONF T R BY O PTIMIZING I NEFFICIENCY
ConfTr performs (differentiable) CP on each mini-batch during stochastic gradient descent (SGD) training.
In particular, as illustrated in Fig.
1 a (cid:13) , we split each mini-batch B in half: the ﬁrst half is used for calibration, B cal , and the second one for prediction and loss computation, B pred .
That is, on B cal , we calibrate τ by computing the α (1 + 1 / | B cal | )
-quantile of the conformity scores in a differentiable manner.
It is important to note that we compute C θ ( x i ; τ ) only for i ∈ B pred and not for i ∈ B cal .
Then, in expectation across mini-batches and large enough | B cal | , for T, (cid:15) → 0 , CP guarantees coverage 1 − α on B pred .
Assuming empirical coverage to be close to (1 − α ) in practice, we only need to minimize inefﬁciency during training: [Equation] with Ω( C [Equation]
x
[Equation]
We emphasize that ConfTr optimizes the model parameters θ on which the conﬁdence sets C θ depend through the model predictions π θ .
Here, Ω is a “smooth” size loss intended to minimize the expected inefﬁciency, i.e ., E [ | C θ ( X ; τ ) | ] , not to be confused with the statistic in Eq.
(2) used for evaluation.
Remember that C π,k ( x ; τ ) can be understood as a soft assignment of class k to the conﬁdence set C θ ( x ; τ ) .
By default, we use κ = 1 in order to not penalize singletons.
However, κ ∈ { 0 , 1 } can generally be treated as hyper-parameter.
After training, any CP method can be applied to re-calibrate τ on a held-out calibration set I cal as usual, i.e ., the thresholds τ obtained during training are not kept.
This ensures that we obtain a coverage guarantee of CP.
3. C ONF T R WITH C LASSIFICATION L OSS
In order to obtain more control over the composition of conﬁdence sets C θ ( X ; τ ) at test time, ConfTr can be complemented using a generic loss L :
[Equation]
While L can be any arbitrary loss deﬁned directly on the conﬁdence sets C θ , we propose to use a “conﬁgurable” classiﬁcation loss L class .
This classiﬁcation loss is intended to explicitly enforce coverage, i.e ., make sure the true label Y is included in C θ
( X ; τ ) , and optionally penalize other classes k not to be included in C θ , as illustrated in Fig.
1 b (cid:13) .
To this end, we deﬁne
L
[Equation]
[Equation] [Equation] As above, C θ,k ( x ; τ ) ∈ [0 , 1] such that 1 − C θ ( x ; τ ) can be understood as the likelihood of k not being in C θ ( x ; τ ) .
In Eq.
(5), the ﬁrst term is used to encourage coverage, while the second term can be used to avoid predicting other classes.
This is governed by the loss matrix L : For L = I K , i.e ., the identity matrix with K rows and columns, this loss simply enforces coverage (perfect coverage if L class = 0 ).
However, setting any L y,k > 0 for y (cid:54) =
k penalizes the model from including class k in conﬁdence sets with ground truth y .
Thus, cleverly deﬁning L allows to deﬁne rather complex objectives, as we will explore next.
ConfTr with (optional) classiﬁcation loss is summarized in
Alg.
1 (right) and Python code can be found in App.
P.
3. C ONF T R WITH G ENERAL AND A PPLICATION -S PECIFIC L OSSES
We consider several use cases motivated by medical diagnosis, e.g ., breast cancer screening (McKinney et al., 2020) or classiﬁcation of dermatological conditions (Liu et al., 2020; Roy et al., 2021;
Jain et al., 2021).
In skin condition classiﬁcation, for example, predicting sets of classes, e.g ., the top- k conditions, is already a common strategy for handling uncertainty.
In these cases, we not only care about coverage guarantees but also desirable characteristics of the conﬁdence sets.
These constraints in terms of the predicted conﬁdence sets can, however, be rather complicated and pose difﬁculties for standard CP.
We explore several exemplary use cases to demonstrate the applicability of ConfTr, that are also relevant beyond the considered use cases in medical diagnosis.
First, we consider “shaping” class-conditional inefﬁciency, formally deﬁned as
[Equation]
Similarly, we can deﬁne inefﬁciency conditional on a group of classes.
For example, we could reduce inefﬁciency, i.e ., uncertainty, on “low-risk” diseases at the expense of higher uncertainty on “high-risk” conditions.
This can be thought of as re-allocating time spent by a doctor towards high-risk cases.
Using ConfTr, we can manipulate group- or class-conditional inefﬁciency using a weighted size loss ω · Ω( C ( X ; τ )) with ω := ω ( Y ) depending on the ground truth Y in Eq.
Next, we consider which classes are actually included in the conﬁdence sets.
CP itself does not enforce any constraints on the composition of the conﬁdence sets.
However, with ConfTr, we can penalize the “confusion” between pairs of classes: for example if two diseases are frequently confused by doctors, it makes sense to train models that avoid conﬁdence sets that contain both diseases.
To control such cases, we deﬁne the coverage confusion matrix as
[Equation]
The off-diagonals, i.e ., Σ y,k for y (cid:54) =
k , quantify how often class k is included in conﬁdence sets with true class y .
Reducing Σ y,k can be accomplished using a positive entry L y,k > 0 in Eq.
Finally, we explicitly want to penalize “overlap” between groups of classes in conﬁdence sets.
For example, we may not want to concurrently include very high-risk conditions among low-risk ones in conﬁdence sets, to avoid unwanted anxiety or tests for the patient.
Letting K 0 (cid:93) K 1 being two disjoint sets of classes, we deﬁne mis-coverage as
[Equation]
Reducing MisCover
→ 1 means avoiding classes K 1 being included in conﬁdence sets of classes K 0 .
Again, we use L y,k > 0 for y ∈ K 0 , k ∈ K 1 to approach this problem.
MisCover 1 → 0 is deﬁned analogously and measures the opposite, i.e ., classes K 0 being included in conﬁdence sets of K 1 .
E XPERIMENTS
We present experiments in two parts: First, in Sec.
4.1, we demonstrate that ConfTr can reduce inefﬁciency of T HR and APS compared to CP applied to a baseline model trained using crossentropy loss separately (see Tab. 1 for the main results).
Thereby, we outperform concurrent work of Bellotti (2021).
Second, in Sec. 4.2, we show how ConfTr can be used to “shape” conﬁdence sets, i.e ., reduce class-conditional inefﬁciency for speciﬁc (groups of) classes or coverage confusion of two or more classes, while maintaining the marginal coverage guarantee.
This is impossible using (Bellotti, 2021) and rather difﬁcult for standard CP.
We consider several benchmark datasets as well as architectures, c.f . Tab.
A, and report metrics averaged across 10 random calibration/test splits for 10 trained models for each method.
We focus on (non-differentiable) T HR and APS as CP methods used after training and, thus, obtain the corresponding coverage guarantee.
T HR , in particular, consistently achieves lower inefﬁciency for a ﬁxed conﬁdence level α than, e.g ., T HR L ( i.e ., T HR on logits) or RAPS, see Fig. 2 (left).
We set α = 0 .
01 and use the same α during training using ConfTr.
Hyper-parameters are optimized for T HR or APS individually.
We refer to App. F for further details on datasets, models, evaluation protocol and hyper-parameter optimization.
4. R EDUCING I NEFFICIENCY WITH C ONF T R
In the ﬁrst part, we focus on the inefﬁciency reductions of ConfTr in comparison to a standard crossentropy training baseline and (Bellotti, 2021) (Bel).
After summarizing the possible inefﬁciency reductions, we also discuss which CP method to use during training and how ConfTr can be used for ensembles and generalizes to lower α .
[Table] Main Results: In Tab. 1, we summarize the inefﬁciency reductions possible through ConfTr (trained with T HR LP) in comparison to Bel (trained with T HR L) and the baseline.
Bel does not consistently improve inefﬁciency on all datasets.
Speciﬁcally, on MNIST, EMNIST or CIFAR100, inefﬁciency actually worsens .
Our ConfTr, in contrast, reduces inefﬁciency consistently, not only for T HR but also for APS.
Here, improvements on CIFAR for T HR are generally less pronounced.
This is likely because we train linear models on top of a pre-trained ResNet (He et al., 2016) where features are not taking into account conformalization at test time, see App.
J. For APS, in contrast, improvements are still signiﬁcant.
Across all datasets, training with L class generally performs slightly better, especially for datasets with many classes such as EMNIST ( K =52 ) or CIFAR100 ( K =100 ).
Overall, ConfTr yields signiﬁcant inefﬁciency reductions, independent of the CP method used at test time.
Conformal Predictors for Training:
In Tab. 1, we use T HR LP during training, irrespective of the
CP method used at test time.
This is counter-intuitive when using, e.g ., APS at test time.
However, training with T HR and APS is rather difﬁcult, as discussed in App.
I. This is likely caused by limited gradient ﬂow as both T HR and APS are deﬁned on the predicted probabilities instead of log-probabilities as used for T HR LP or in cross-entropy training.
Moreover, re-formulating the conformity scores of APS in Eq.
(11) to use log-probabilities is non-trivial.
In contrast, Bel has to be trained using T HR L as a ﬁxed threshold τ is used during training. This is because the calibration step is ignored during training.
Also, ﬁxing τ is not straightforward for T HR due to the limited range of the predicted probabilities π θ,k ( x ) ∈ [0 , 1] , see App.
E. We believe that this contributes to the poor performance of Bel on several datasets.
Finally, we found that Bel or ConfTr do not necessarily recover the accuracy of the baseline.
Remember that we refer to the accuracy in terms of the arg max -prediction of π θ .
When training from scratch, accuracy can be 2-6% lower while still reducing inefﬁciency.
This is interesting because ConfTr is still able to improve inefﬁciency, highlighting that cross-entropy training is not appropriate for CP.
Further Results: Tab. 2 includes additional results for ConfTr to “conformalize” ensembles on
CIFAR10 (left) and with lower conﬁdence levels α on EMNIST (right).
In the ﬁrst example, we consider applying CP to an ensemble of models.
Ensemble CP methods such as (Yang & Kuchibhotla, 2021) cannot improve Ineff over the best model of the ensemble, i [Table] e ., 3.10 for T HR .
Instead, [Table] [Figure] training an MLP on top of the ensemble’s logits can improve Ineff to 2.40 and additionally using
ConfTr to 2.35.
The second example shows that ConfTr, trained for α =0 .
01 , generalizes very well to signiﬁcantly smaller conﬁdence levels, e.g ., α =0 .
001 on EMNIST.
In fact, the improvement of
ConfTr (without L class ) in terms of inefﬁciency is actually more signiﬁcant for lower conﬁdence levels.
We also found ConfTr to be very stable regarding hyper-parameters, see App. H. Only too small batch sizes ( e.g ., | B | =100 on MNIST) prevents convergence.
This is likely because of too few examples ( | B cal | =50 ) for calibration with α =0 .
01 during training.
More results, e.g ., on binary datasets or including additional hyper-parameter ablation can be found in App.
J.
C ONFORMAL T RAINING FOR A PPLICATIONS : C ASE S TUDIES
For the second part, we focus on ConfTr trained with T HR LP and evaluated using T HR .
We follow
Sec.
3.3 and start by reducing class- or group-conditional inefﬁciency using ConfTr (without L class ), before demonstrating reductions in coverage confusion of two or more classes and avoiding miscoverage between groups of classes (with L class ).
Because this level of control over the conﬁdence sets is not easily possible using Bel or standard CP, we concentrate on ConfTr only:
Shaping Conditional Inefﬁciency: We use ConfTr to reduce class-conditional inefﬁciency for speciﬁc classes or a group of classes, as deﬁned in Eq.
In Fig.
2, inefﬁciency is shown to vary widely across classes:
On CIFAR10, the more difﬁcult class 3 (“cat”) obtains higher inefﬁciency than the easier class 1 (“automobile”).
Thus, in Fig.
3, we use ω =10 as described in Sec.
3.3 to reduce class- or group-conditional inefﬁciency.
We report the relative change in percentage, showing that inefﬁciency reductions of 20% or more are possible for many classes, including “cat” on
CIFAR10 (left, blue).
This is also possible for two groups of classes, “vehicles” vs. “animals” (middle).
However, these reductions usually come at the cost of a slight increase in average inefﬁciency across all classes (green).
On CIFAR100, we consider 20 coarse classes, each containing 5 of the 100 classes (right).
Again, signiﬁcant inefﬁciency reductions per coarse class are possible.
These observations generalize to all other considered datasets and different class groups, see App.
L.
Avoiding Coverage Confusion: Next, we use ConfTr to manipulate the coverage confusion matrix as deﬁned in Eq.
Speciﬁcally, we intend to reduce coverage confusion of selected sets of classes.
[Figure]
penalize = 6 .
Here, L y,k > 0 , but L y,k = 0 , i.e ., Cover confusion is not reduced symmetrically.
[Figure]
“cat”)
[Figure]
of higher inefﬁciency for class
(“good”) and boost class-conditional coverage for class 0.
Using a non-zero entry L y,k > 0 , y (cid:54)
= k in L class , as described in Sec. 3.3, Fig. 4 (left) shows that coverage confusion can be reduced signiﬁcantly for large enough L y,k on Fashion-MNIST: Considering classes 4 and 6 (“coat” and “shirt”) confusion can be reduced by roughly 1% . However, as accuracy stays roughly the same and coverage is guaranteed, this comes at the cost of increasing coverage confusion for other class pairs, e.g ., 2 (“pullover”) and 4.
ConfTr can also be used to reduce coverage confusion of multiple class pairs (middle) or a whole row in the coverage confusion matrix Σ y,k with ﬁxed y and y (cid:54) =
k ∈ [ K ] .
Fig.
4 (right) shows the results for class 6: coverage confusion with, e.g ., classes 0 (“t-shirt”), 2 or 4 (blue, green and violet) is reduced roughly 0 .
5% each at the cost of increased confusion of classes 2 and 4 (in gray).
These experiments can be reproduced on other datasets, e.g ., MNIST or CIFAR10 in App.
M.
Reducing Mis-Coverage: We can also address unwanted “overlap” of two groups of classes using
ConfTr and L class .
In Fig.
5 (left) we explicitly measure mis-coverage as deﬁned in Eq.
First, on
CIFAR10, we consider a singleton group K 0 = { 3 } (“cat”) and K 1 =[ K ] \ { 3 } : The ConfTr baseline
MisCover
→ 1 tells us that 98.92% of conﬁdence sets with true class 3 also contain other classes.
Given an average inefﬁciency of 2 . 84 this is reasonable.
Using L 3 ,k = 1 , k (cid:54) =
3 , this can be reduced to 91.6%.
Vice-versa, the fraction of conﬁdence sets of class y (cid:54) =3 containing class 3 can be reduced from 36.52% to 26.43%.
On CIFAR100, this also allows to reduce overlap between “human-made” (35 classes) and “natural” (65 classes) things, e.g ., MisCover
→ 1 reduces from 40.09% to 15.77%, at the cost of a slight increase in inefﬁciency.
See App.
N for additional results.
Binary Datasets: Finally, in Fig.
5 (right), we illustrate that the above conclusions generalize to the binary case: On WineQuality, we can control inefﬁciency of class 0 (“bad wine”, minority class with ∼ 37% of examples) at the expense of increased inefﬁciency for class 1 (“good wine”, top).
Similarly, we can (empirically) improve class-conditional coverage for class 0 (bottom) or manipulate coverage confusion of both classes, see App.
O.
C ONCLUSION
We introduced conformal training (ConfTr) , a novel method to train conformal predictors endto-end with the underlying model.
This addresses a major limitation of conformal prediction (CP) in practice: The model is ﬁxed, leaving CP little to no control over the predicted conﬁdence sets.
In thorough experiments, we demonstrated that ConfTr can improve inefﬁciency of state-of-the-art
CP methods such as T HR (Sadinle et al., 2019) or APS (Romano et al., 2020).
More importantly, motivated by medical diagnosis, we highlighted the ability of ConfTr to manipulate the predicted conﬁdence sets in various ways.
First, ConfTr can “shape” the class-conditional inefﬁciency distribution, i.e ., reduce inefﬁciency on speciﬁc classes at the cost of higher inefﬁciency for others.
Second, ConfTr allows to control the coverage-confusion matrix by, e.g ., reducing the probability of including classes other than the ground truth in conﬁdence sets.
Finally, this can be extended to explicitly reduce “overlap” between groups of classes in the predicted conﬁdence sets.
In all cases,
ConfTr does not lose the (marginal) coverage guarantee provided by CP.
E THICS S TATEMENT
Recent deep learning based classiﬁers, as used in many high-stakes applications, achieve impressive accuracies on held-out test examples.
However, this does not provide sufﬁcient guarantees for safe deployment.
Conformal prediction (CP), instead, predicts conﬁdence sets equipped with a guarantee that the true class is included with speciﬁc, user-speciﬁed probability.
These conﬁdence sets also provide intuitive uncertainty estimates.
We speciﬁcally expect CP to be beneﬁcial in the medical domain, improving trustworthiness among doctors and patients alike by providing performance guarantees and reliable uncertainty estimates.
Yet, the current work does not contain experiments with personal/sensitive medical data.
The presented results are on standard benchmark datasets only.
However, these beneﬁts of CP may not materialize in many applications unless CP can be better integrated into existing classiﬁers.
These are predominantly deep networks, trained end-to-end to, e.g ., optimize classiﬁcation performance.
CP, in contrast, is agnostic to the underlying model, being applied as “wrapper” post-training, such that the obtained conﬁdence sets may not be optimal, e.g ., in terms of size ( inefﬁciency ) or composition ( i.e ., the included classes).
Especially in the medical domain, constraints on the conﬁdence sets can be rather complex.
Our conformal training (ConfTr) integrates CP into the training procedure, allowing to optimize very speciﬁc objectives deﬁned on the predicted conﬁdence sets – without losing the guarantees.
In medical diagnosis, smaller conﬁdence sets may avoid confusion or anxiety among doctors or patients, ultimately leading to better diagnoses.
For example, we can reduce inefﬁciency ( i.e ., the ambiguity of predicted conditions) for conditions that are particularly difﬁcult for doctors to diagnose.
Alternatively, ConfTr allows to avoid confusion between low- and high-risk conditions within the conﬁdence sets.
Generally, beyond medical diagnosis, we believe ConfTr to have positive impact in settings where additional constraints on conﬁdence sets are relevant in addition to the guarantees and uncertainty estimates provided by CP.
R EPRODUCIBILITY S TATEMENT
In order to ensure reproducibility, we include a detailed description of our experimental setup in App.
F. We discuss all necessary information for conformal training (ConfTr) as well as our baselines.
This includes architectures, training procedure and hyper-parameters, as well as preprocessing/data augmentation if applicable.
Furthermore, we describe our evaluation procedure which includes multiple calibration/test splits for conformal prediction (CP) at test time as well as multiple training runs to capture randomness in the used calibration examples and during training.
To this end, Tab.
A reports the training/calibration/test splits of all used datasets and Tab.
B the used hyper-parameters for ConfTr.
While Alg. 1 already summarizes the used (smooth) threshold CP methods and our ConfTr, App.
P (speciﬁcally Alg. B) lists the corresponding Python implementation of these key components.
A O VERVIEW AND O UTLINE
In the appendix, we discuss an additional baseline, called coverage training (CoverTr) , provide additional details on our experimental setup and include complementary results.
Speciﬁcally, the appendix includes:
• Additional discussion of related work in Sec.
B; • Formal statement of the coverage guarantee obtained through conformal prediction (CP) in Sec.
C; • Differentiable implementation of APS in Sec.
D; • Discussion of coverage training (CoverTr) and (Bellotti, 2021) in Sec.
E; • Details on our experimental setup, including dataset splits, model details and used hyperparameters for ConfTr, in Sec.
F; • Experiments regarding random training and test trials in Sec.
G; • Hyper-parameter ablation on MNIST in Sec.
H; • CoverTr and ConfTr ablation on MNIST and Fashion-MNIST in Sec.
I; • Complete inefﬁciency (Ineff) results on all datasets in Sec.
J; • Effect of (standard) ConfTr on class-conditional inefﬁciency and coverage (Cover) confusion in Sec.
K; • Additional results for ConfTr shaping the class-conditional inefﬁciency distribution in Sec.
L; • More experiments for ConfTr manipulating coverage confusion in Sec.
M; • Complementary results for ConfTr reducing mis-coverage (MisCover) in Sec.
N; • Class-conditional inefﬁciency and coverage confusion on binary datasets in Sec.
O; • Python- and Jax (Bradbury et al., 2018) code for ConfTr in Sec.
P.
B R ELATED W ORK
Conformal prediction (CP) builds on early work by Vovk et al.
(2005) considering both regression, see e.g ., (Romano et al., 2019) for references, and classiﬁcation settings, e.g .
(Romano et al., 2020; Angelopoulos et al., 2021; Cauchois et al., 2020; Hechtlinger et al., 2018).
Most of these approaches follow a split CP approach (Lei et al., 2013) where a held-out calibration set is used, as in the main paper, however, other variants based on cross-validation (Vovk, 2013) or jackknife ( i.e ., leave-oneout) (Barber et al., 2019a) are available.
These approaches mostly provide marginal coverage.
Vovk (2012); Barber et al. (2019b) suggest that it is generally difﬁcult or impossible to obtain conditional coverage.
However, Romano et al.
(2020) work towards empirically better conditional coverage and Sadinle et al.
(2019) show that efﬁcient class-conditional coverage is possible.
Angelopoulos et al.
(2021) extend the work by Romano et al.
(2020) to obtain smaller conﬁdence sets at the expense of the obtained empirical conditional coverage.
CP has also been studied in the context of ensembles (Yang & Kuchibhotla, 2021), allowing to perform model selection based on inefﬁciency while keeping coverage guarantees.
The work of Bates et al.
(2021) can be seen as a CP extension in which a guarantee on an arbitrary, user-speciﬁed risk can be obtained, using a conformal predictor similar to (Sadinle et al., 2019).
Our conformal training (ConfTr) follows the split CP approach and is speciﬁcally targeted towards classiﬁcation problems.
Nevertheless, extensions to regression, or other CP formulations such as (Bates et al., 2021) during training, are possible.
Beyond that, ConfTr is agnostic to the CP method used at test time and can thus be seen as complementary to the CP methods discussed above.
This means that ConfTr can easily be combined with approaches such as (Bates et al., 2021) or class-conditional conformal predictors (Sadinle et al., 2019) at test time.
In terms of learning to predict conﬁdence sets, our approach has similarities to the multiple choice learning of Guzm´an-Rivera et al.
(2012) which yields multiple possible outputs in structured prediction settings ( e.g ., image segmentation).
However, the obtained prediction sets are ﬁxed size and no coverage guarantee is provided.
Concurrent work by Bellotti (2021) is discussed in detail in App.
E.
C C OVERAGE G UARANTEE
Following Romano et al.
, we brieﬂy state the coverage guarantee obtained by CP in formal terms: Given that the learning algorithm used is invariant to permutations of the training examples, and the calibration examples { ( X i , Y i ) } i ∈ I cal are exchangeably drawn from the same distribution encountered at test time, the discussed CP methods satisfy
[Equation]
As highlighted in (Romano et al., 2020), this bound is near tight if the scores E ( x i ) are almost surely distinct:
[Equation]
Note that this is the case for APS due to the uniform random variable U in Eq.
(Romano et al., 2020) notes that there is generally no guarantee on conditional coverage, as this requires additional assumptions.
However, class-conditional coverage can be obtained using T HR as outlined in (Sadinle et al., 2019).
Moreover, Sadinle et al.
(2019) show that T HR is the most efﬁcient conformal predictor given a ﬁxed model π θ , i.e ., minimizes inefﬁciency.
We refer to (Sadinle et al., 2019) for exact statements of the latter two ﬁndings.
D D IFFERENTIABLE APS
Our differentiable implementation closely follows the one for T HR outlined in Sec. 2.2.
The main difference is the conformity score E ( x, k ) computation, i.e .,
[Equation]
[Equation]
where π θ,y (1) ( x ) ≥ . .
≥ π θ,y ( K ) ( x ) and U is a uniform random variable in [0 , 1] to break ties.
As in the calibration step, we use an arbitrary smooth sorting approach for this.
This implementation could easily be extended to include the regularizer of Angelopoulos et al.
(2021), as well. E C OVERAGE T RAINING As intermediate step towards conformal training (ConfTr) , we can also ignore the calibration step and just differentiate through the prediction step, i.e ., C θ
( X ; τ ) .
This can be accomplished by ﬁxing the threshold τ .
Then, π θ essentially learns to produce probabilities that yield “good” conﬁdence sets C θ ( X ; τ ) for the chosen threshold τ .
Following Alg.
A, coverage training (CoverTr) computes
C θ ( X ; τ ) on each mini-batch using a ﬁxed τ .
The model’s parameters θ are obtained by solving
[Equation]
Again, L is the classiﬁcation loss from Eq.
(5) and Ω the size loss from Eq.
The classiﬁcation loss has to ensure that the true label y is in the predicted conﬁdence set C
θ ( X ; τ ) as the calibration step is missing.
In contrast to ConfTr, CoverTr strictly requires both classiﬁcation and size loss during training.
This is because using a ﬁxed threshold τ yields trivial solutions for both classiﬁcation and size loss when used in isolation ( i.e ., L is minimized for C θ
( X ; τ ) =
[ K ] and Ω is minimized for C θ
( X ; τ ) = ∅ ).
Thus, balancing both terms in Eq.
(12) using λ is crucial during training.
As with ConfTr, the threshold τ is re-calibrated at test time to obtain a coverage guarantee.
Choosing τ for training, in contrast, can be difﬁcult: First, τ will likely evolve during training (when π θ gets [Table] function C OVERAGE T RAINING ( τ [Equation]
λ ) [Table] for mini-batch B [Equation]
[Table]
C (
[Equation]
[Equation]
=
[Equation]
MOOTH P RED ( π θ ( x
[Equation]
[Equation]
[Equation]
∈ B 4: L B := (cid:80)
i
∈ B L ( C θ ( x i ; τ ) , y i ) 5: Ω B := (cid:80) i ∈
B Ω( C θ ( x i ; τ )) 6: ∆ :
= ∇ θ 1 / | B | ( L B + λ Ω B ) 7: update parameters θ using ∆
Algorithm A: Coverage Training (CoverTr) :
Compared to Alg. 1 for ConfTr, CoverTr simpliﬁes training by not differentiating through the calibration step and avoiding splitting the batch B in half.
However, ﬁxing the threshold τ can be a problem and training requires both coverage and size loss.
[Table]
more and more accurate) and, second, the general ballpark of reasonable thresholds τ depends on the dataset as well as model and is difﬁcult to predict in advance.
In concurrent work by Bellotti (2021) (referred to as Bel), the problem with ﬁxing a threshold τ is circumvented by using T HR L during training, i.e ., T HR on logits.
As the logits are unbounded, the threshold can be chosen arbitrarily, e.g ., τ = 1 .
As Bel also follows the formulation of Eq.
(12), the approach can be seen as a special case of CoverTr.
However, a less ﬂexible coverage loss is used during training: Instead of L class , the loss is meant to enforce a speciﬁc coverage level (1 − α ) on each mini-batch.
This is done using a squared loss on coverage:
[Equation]
for a mini-batch B of examples.
In contrast to Eq.
(12), L cov is applied per batch and not per example.
For the size loss, Bellotti (2021) uses κ = 0 in Eq.
Besides not providing much control over the conﬁdence sets, L cov also encourages coverage (1 − α ) instead of perfect coverage.
Nevertheless, this approach is shown to improve inefﬁciency of T HR L on various UCI datasets (Dua & Graff, 2017) using linear logistic regression models.
The experiments in the main paper show that this generalizes to non-linear models and more complex datasets.
Nevertheless, Bel is restricted to T HR L which is outperformed signiﬁcantly by both T HR and APS.
Thus, Bel is consistently outperformed by ConfTr in terms of inefﬁciency improvements.
Moreover, the approach cannot be used for any of the studied use cases in Sec. 3.3.
Using CoverTr with T HR and APS remains problematic.
While we found τ ∈ [0 . 9 , 0 . 99] (or [ − 0 .
1 , − 0 .
01 for T HR LP) to work reasonably on some datasets, we had difﬁculties on others, as highlighted in Sec.
I. Moreover, as CoverTr requires balancing coverage L and size loss Ω , hyperparameter optimization is more complex compared to ConfTr.
By extension, these problems also limit the applicability of Bel.
Thus, we would ideally want to re-calibrate the threshold τ after each model update.
Doing calibration on a larger, held-out calibration set, however, wastes valuable training examples and compute resources.
Thus, ConfTr directly calibrates on each mini-batch and also differentiates through the calibration step itself to obtain meaningful gradients.
F E XPERIMENTAL S ETUP
Datasets and Splits: We consider Camelyon
(Bejnordi et al., 2017), GermanCredit (Dua &
Graff, 2017), WineQuality (Cortez et al., 2009), MNIST (LeCun et al., 1998), EMNIST (Cohen et al., 2017), Fashion-MNIST (Cohen et al., 2017) and CIFAR (Krizhevsky, 2009) with a ﬁxed split of training, calibration and test examples.
Tab.
A summarizes key statistics of the used datasets which we elaborate on in the following.
Except Camelyon, all datasets are provided by Tensorﬂow (Abadi et al., 2015) 1 .
For Camelyon, we use the pre-computed features of Wilder et al.
(2020) which are based on open source code from the Camelyon2016 challenge 2 .
For datasets providing a default training/test split, we take the last 10% of training examples as calibration set.
On Camelyon, we use the original training set, but split test examples into 100 validation and 17 test examples.
This is because less than 100 calibration examples are not meaningful for α =0 .
As we evaluate 10 random calibration/test splits, the few test examples are not problematic in practice.
On GermanCredit and
WineQuality, we manually created training/calibration/test splits, roughly matching 70%/10%/20%.
We use the “white wine” subset for WineQuality; to create a binary classiﬁcation problem, wine with quality 6 or higher is categorized as “good wine” (class 1), following (Bellotti, 2021).
Finally, for EMNIST, we consider a subset of the “byClass” split that contains 52 = 2 · 26 classes comprised of all lower and upper case letters.
We take the ﬁrst 122.8k examples, split as in Tab.
A.
Models and Training: We consider linear models, multi-layer perceptrons (MLPs) and ResNets
(He et al., 2016) as shown in Tab.
A. Speciﬁcally, we use a linear model on MNIST and GermanCredit, 1- or 2-layer MLPs on Camelyon2016, WineQuality and Fashion-MNIST, and ResNet-34/ (He et al., 2016) on CIFAR10/100.
Models and training are implemented in Jax (Bradbury et al., 2018) 3 and the ResNets follow the implementation and architecture provided by Haiku (Hennigan et al., 2020) 4 .
Our l -layer MLPs comprise l hidden layers.
We use 32 , 256 , 128 , 64 units per hidden layer on Camelyon, WineQuality, EMNIST and Fashion-MNIST, respectively.
These were chosen by grid search over { 16 , 32 , 64 , 128 , 256 } .
In all cases, we use ReLU activations (Nair & Hinton) and batch normalization (Ioffe & Szegedy, 2015).
We train using stochastic gradient descent (SGD) with momentum 0 . 0005 and Nesterov gradients.
The baseline models are trained with cross-entropy loss, while ConfTr follows Alg. 1 and CoverTr follows Alg.
A. Learning rate and batch size are optimized alongside the ConfTr hyper-parameters using grid search, see below.
The number of epochs are listed in Tab.
A
and we follow a multi-step learning rate schedule, multiplying the initial learning rate by 0 .
1 after 2 / 5 , 3 / 5 and 4 / 5 of the epochs.
We use Haiku’s default initializer.
On CIFAR, we apply whitening using the per-channel mean and standard deviation computed on the training set.
On the non-image datasets (Camelyon, GermanCredit, WineQuality), we whiten each feature individually.
On MNIST, EMNIST and Fashion-MNIST, the input pixels are just scaled to [ − 1 , 1] .
Except on CIFAR, see next paragraph, we do not use any data augmentation.
Finally, we do not use
Platt scaling (Guo et al., 2017) as used in (Angelopoulos et al., 2021).
Fine-Tuning on CIFAR:
On CIFAR10 and CIFAR100, we train base ResNet-34/ResNet-50 models which are then ﬁne-tuned using Bel, CoverTr or ConfTr.
We speciﬁcally use a ResNet-34 with only 4 base channels to obtain an accuracy of 82.6%, using only random ﬂips and crops as data augmentation.
The rationale is to focus on the results for CP at test time, without optimizing accuracy of the base model.
On CIFAR100, we use 64 base channels for the ResNet-50 and additionally employ
AutoAugment (Cubuk et al., 2018) and Cutout (Devries & Taylor, 2017) as data augmentation.
This model obtains 73.64% accuracy.
These base models are trained on 100% of the training examples (without calibration examples).
For ﬁne-tuning, the last layer ( i.e ., logit layer) is re-initialized and trained using the same data augmentation as applied for the base model, subject to the random training trials described below.
We also consider “extending” the ResNet by training a 2-layer MLP with 128 units per hidden layer on top of the features (instead of re-initializing and ﬁne-tuning the logit layer).
All reported results either correspond to ﬁne-tuned ( i.e ., linear model on features) or extended models ( i.e ., 2-layer MLP on features) trained on these base models.
Hyper-Parameters:
The ﬁnal hyper-parameters selected for ConfTr (for T HR at test time) on all datasets are summarized in Tab.
B. These were obtained using grid search over the following hyper-parameters: batch size in { 1000 , 500 , 100 } for WineQuality, MNIST, EMNIST,
Fashion-MNIST and CIFAR, { 300 , 200 , 100 , 50 } on GermanCredit and { 80 , 40 , 10 } on Camelyon learning
Grid search was done for dataset individually on 100% of the training examples ( Tab.
A).
That is, for hyper-parameter optimization we did not perform random training trials as described next.
The best hyper-parameters according to inefﬁciency after evaluating 3 random calibration/test splits were selected, both for T HR and APS at test time, with and without L class . [Table] [Table]
Tab.
B allows to make several observations.
First, on the comparably small (and binary) datasets
Camelyon and GermanCredit, the size weight λ
= 5 is rather high.
For ConfTr without L class , this just indicates that a higher learning rate could be used.
Then using L class , however, this shows that the size loss is rather important for ConfTr, especially on binary datasets.
Second, we found the temperature T to have low impact on results, also see Sec.
H.
On multiclass datasets, the size weight λ is usually higher when employing L class .
Finally, especially with L class , using “valid” size loss, i.e ., κ = 1 , to not penalize conﬁdence sets of size 1, works better than κ = 0 .
Random Training and Test Trials: For statistically meaningful results, we perform random test and training trials.
Following common practice (Angelopoulos et al., 2021), we evaluate CP methods at test time using 10 random calibration/test splits.
To this end, we throw all calibration and test examples together and sample a new calibration/test split for each trial, preserving the original calibration/test composition which is summarized in Tab. A. Metrics such as coverage and inefﬁciency are then empirically evaluated as the average across all test trials.
Additionally, and in contrast to (Bellotti, 2021), we consider random training trials: After hyper-parameters optimization on all training examples, we train 10 models with the ﬁnal hyper-parameters on a new training set obtained by sampling the original one with up to 5 replacements.
For example, on MNIST, with 55k training [Table] examples, we randomly sample 10 training sets of same size with each, on average, containing only ∼ 68% unique examples from the original training set.
Overall, this means that we report, e.g ., inefﬁciency as average over a total of 10 · 10 = 100 random training and test trials.
As a consequence, our evaluation protocol accounts for randomness at test time ( i.e ., regarding the calibration set) and at training time ( i.e ., regarding the training set, model initialization, etc .).
G I MPORTANCE OF R ANDOM T RIALS
In Tab.
C we highlight the importance of random training and test trials for evaluation.
On the left, we show the impact of trials at test time, i.e ., 10 random calibration/test splits, for a ﬁxed model on
MNIST.
While the standard deviation of coverage is comparably small, usually ≤ 0 .
2% , standard deviation of inefﬁciency is higher in relative terms.
This makes sense as coverage is guaranteed, while inefﬁciency depends more strongly on the sampled calibration set.
The right table, in contrast, shows that training trials exhibit lower standard deviation in terms of inefﬁciency.
However, training with, e.g ., APS will mainly result in high inefﬁciency, on average, because of large standard deviation.
In fact, ConfTr with APS or T HR at training time results in worse inefﬁciency mainly because training is less stable.
This supports the importance of running multiple training trials for ConfTr.
H I MPACT OF H YPER -P ARAMETERS
In Tab.
D, we conduct ablation for individual hyper-parameters of ConfTr with T HR LP and without
L class on MNIST.
The hyper-parameters used in the main paper, c.f .
Tab.
B, are highlighted in bold .
As outlined in Sec.
F, hyper-parameter optimization was conducted on 100% training examples with only 3 random test trials, while Tab.
D shows results using random training and test trials.
We found batch size and learning rate to be most impactful.
While batch sizes 1000 and 500 both work, batch size 100 prevents ConfTr from converging properly.
This might be due to the used α = 0 .
01 which might be too low for batch size 100 where only 50 examples are available for calibration during training.
Without L class , the size weight λ merely scales the learning rate and, thus, has little to no impact.
For ConfTr with L class , we generally found the size weight λ to be more important for balancing classiﬁcation loss L and size loss Ω in Eq.
Temperature has no signiﬁcant impact, although a temperature of 0.5 or 1 works best.
Finally, the hyper-parameters do generalize to a lower conﬁdence level α = 0 .
Signiﬁcantly lower values, e.g ., α = 0 . 001 , are, however, [Table] [Table]
[Table] [Table] [Table] not meaningful due to the batch size of 500 .
However, signiﬁcantly higher conﬁdence levels, e.g ., α = 0 .
1 or α = 0 .
05 , require re-optimizing the other hyper-parameters.
I C OVER T R AND C ONF T R A BLATION ON MNIST AND F ASHION -MNIST
In Tab.
E, we present an ablation for CoverTr, see Sec.
E, and ConfTr on MNIST, using a linear model, and Fashion-MNIST, using a 2-layer MLP.
Bel is generally able to improve inefﬁciency of
T HR L. Using T HR , however, Bel worsens inefﬁciency on MNIST signiﬁcantly, while improving slightly over the baseline on Fashion-MNIST.
As a result, the improvement of ConfTr over Bel is also less signiﬁcant on Fashion-MNIST.
Using CoverTr with T HR or APS during training works poorly.
As described in Tab.
C, this is mainly due to a high variation across training runs, i.e ., individual models might work well, but training is not stable enough to get consistent improvements.
Thus, on MNIST, inefﬁciency for CoverTr with T HR and APS is very high.
Moreover, on FashionMNIST, we were unable to train CoverTr with T HR and APS.
Using T HR LP, training with CoverTr works and is reasonably stable, but does not improve over the baseline.
It does improve over Bel on
MNIST though.
As described in the main paper, we suspect the ﬁxed threshold τ to be problematic.
Overall, however, only ConfTr is able to outperform the T HR baseline on both datasets.
Here,
ConfTr with L class works slightly better than without.
J A LL I NEFFICIENCY R ESULTS
Tab.
F shows complementary results for ConfTr on CIFAR10, EMNIST and CIFAR100.
For results on MNIST and Fashion-MNIST, see Tab.
D.
On CIFAR10, we also include ConfTr using a 2-layer
MLP on top of ResNet features – instead of the linear model used in the main paper.
In Tab.
F, this is referred to as “extending”.
However, inefﬁciency increases slightly compared to re-initializing and training just the (linear) logit layer.
This shows that the smaller inefﬁciency improvements on CIFAR shown in the main paper are not due to the linear model used, but rather caused by the features themselves.
We suspect that this is because the features are trained to optimize cross-entropy loss, leaving ConfTr less ﬂexibility to optimize inefﬁciency.
In Tab.
G, we consider three binary datasets, i.e ., WineQuality, GermanCredit and Camelyon.
On binary datasets, T HR L, T HR and APS perform very similar.
This already suggests that there is little room for inefﬁciency improvements.
Indeed,
ConfTr is not able to improve inefﬁciency signiﬁcantly.
However, this is partly due to our thorough evaluation scheme: On Camelyon (using α =0 . 05 ), we do not report averages across all training trials, but the results corresponding to the best model.
This is because sub-sampling the training examples is unreasonable given that there are only 280 of them.
Thus, Camelyon shows that ConfTr can improve inefﬁciency.
On WineQuality or GermanCredit, however, this is “hidden” in reporting averages across 10 training runs. [Table]
K E FFECT OF C ONF T R ON C LASS -C ONDITIONAL I NEFFICIENCY AND C OVERAGE C ONFUSION
Fig.
G shows that standard ConfTr (without L class ) does not have a signiﬁcant inﬂuence on the classconditional inefﬁciency distribution compared to the baseline.
Similarly, ConfTr with L class and identity loss matrix L
= I K does not inﬂuence coverage confusion besides reducing overall inefﬁciency.
Speciﬁcally, on MNIST, Fashion-MNIST and CIFAR10, we show the class-conditional inefﬁciency distribution (left) as well as the coverage confusion matrices (middle and right) for the baseline and ConfTr.
On the left, we consider ConfTr without L class , and on the right with L class .
As can be seen, only an overall reduction of inefﬁciency is visible, the distribution of Ineff [ y ] , c.f .
Eq.
(6), across classes y remains roughly the same.
For coverage confusion Σ from Eq.
(7), the same observation can be made, i.e ., an overall reduction of inefﬁciency also reduces confusion, but the spatial pattern remains the same.
Thus, in the main paper and the following experiments, we always highlight the improvement over standard ConfTr, without L class for reducing class-conditional inefﬁciency and with L class for changing coverage confusion or improving MisCover.
[Table]
L S HAPING C LASS -C ONDITIONAL I NEFFICIENCY ON O THER D ATASETS
Fig.
A and B provide complementary results demonstrating the ability of ConfTr to shape the classor group-conditional inefﬁciency distribution.
First, Fig.
A plots inefﬁciency for individual classes on CIFAR10 and coarse classes on CIFAR100.
In both cases, signiﬁcant inefﬁciency reductions are possible for high weights ω in Eq.
(3), irrespective or whether the corresponding (coarse) class has above-average inefﬁciency to begin with.
This means that inefﬁciency reduction is possible for easier and harder classes alike.
Second, Fig.
B plots the relative inefﬁciency changes, in percentage, possible per-class or group on MNIST, Fashion-MNIST and CIFAR100.
For CIFAR100, we show only the ﬁrst 10 classes for brevity.
In all cases, signiﬁcant inefﬁciency reductions are possible, at the expense of a slight increases in average inefﬁciency across all classes.
Here, MNIST is considerably [Figure] Reducing Class- and Group-Conditional Inefﬁciency on CIFAR. Results, complementary to Fig. 3, showing the impact of higher size weights ω in Eq.
(3) for classes 0 and 3 (“airplane” and “cat”) on CIFAR10 and coarse classes 9 and 15 (“large man-made outdoor things” and “reptiles”) on CIFAR100.
ConfTr allows to reduce inefﬁciency (blue) in all cases, irrespective of whether inefﬁciency is generally above or below average (green).
[Figure]
[Table]
[Figure]
[Figure]
[Table]
[Figure]
easier than Fashion-MNIST: higher inefﬁciency reductions are possible per class and the cost in terms of average inefﬁciency increase is smaller.
On CIFAR100, inefﬁciency reductions of 40% or more are possible.
This is likely because of the high number of classes, i.e ., ConfTr has a lot of ﬂexibility to ﬁnd suitable trade-offs during training.
M M ANIPULATING C OVERAGE C ONFUSION ON O THER D ATASETS
Fig.
C to E provide additional results for reducing coverage confusion using ConfTr.
First, in Fig.
C we show the full coverage confusion matrices for the ConfTr baseline (with L class , left) and ConfTr with L y,k = 1 , y (cid:54) =
k ∈ { 4 , 5 , 7 } (right, marked in red) on CIFAR10.
This allows to get the complete picture of how coverage confusion changes and the involved trade-offs.
As demonstrated in the main paper, coverage confusion for, e.g ., classes 4 and 5 (“deer” and “dog”) reduces.
However, coverage confusion for other class pairs might increase slightly.
Then, supplementary to Fig.
4 in the main paper, we provide the actual numbers in Fig.
D. In particular, we visualize how the actual coverage confusion entries (left) or rows (right) change depending on the off-diagonal weights L y,k .
Finally, Fig.
E presents additional results on MNIST and CIFAR10.
From these examples it can be seen that reducing coverage confusion is easier on MNIST, reducing linearly with the corresponding penalty L y,k .
Moreover, the achieved reductions are more signiﬁcant.
On CIFAR10, in contrast, coverage confusion reduces very quickly for small L y,k before stagnating for larger L y,k .
At the same time, not all targeted class pairs might yield signiﬁcant coverage confusion reductions. [Figure] [Table] [Table]
N M IS C OVER R ESULTS ON A DDITIONAL D ATASETS
Tab.
H provides mis-coverage results for different settings on MNIST, Fashion-MNIST and CIFAR10.
As in the main paper, we are able to reduce mis-coverage signiﬁcantly on MNIST and
Fashion-MNIST.
Only on CIFAR10, considering “vehicles” vs. “animals” as on CIFAR100 in the main paper, we are unable to obtain signiﬁcant reductions.
While, we are able to reduce
MisCover
→ 1 slightly from 22.22% to 20%, MisCover 1 → 0 increases slightly from 16.45% to 16.73% even for high off-diagonal weights used in L .
Compared to CIFAR100, this might be due to less ﬂexibility to ﬁnd suitable trade-offs as CIFAR10 has only 10 classes.
Moreover, mis-coverages on CIFAR10 are rather small to begin with, indicating that vehicles and animals do not overlap much by default.
O A DDITIONAL R ESULTS ON B INARY D ATASETS
Fig.
F shows results complementing Fig.
5 (right) in the main paper.
Speciﬁcally, we show that reducing inefﬁciency for class 1 (“good wine”) is unfortunately not possible.
This might also be due to the fact that class 1 is the majority class, with ∼ 63% of examples.
However, in addition to improving coverage conditioned on class 0, we are able to reduce coverage confusion Σ 0 , 1 , c.f .
Sec. 3.3.
We found that these results generalize to GermanCredit, however, being less pronounced, presumably because of signiﬁcantly fewer training and calibration examples. [Figure] Fig.
5 (right) in the main paper, we plot the possible inefﬁciency reduction for class 1 (“good wine”, left) and full coverage confusion matrices for increased L 0 , 0 > 1 and L 1 , 0 > 0 (right, top and bottom, respectively).
While we can reduce inefﬁciency for class 0 (“bad wine”), this is not possible for class 1.
However, class-conditional coverage for class 0 can be improved signiﬁcantly and we can reduce coverage confusion Σ 0 , 1 .
P P SEUDO C ODE
Alg.
B presents code in Python, using Jax (Bradbury et al., 2018),
Haiku (Hennigan et al., 2020) and Optax (Hessel et al., 2020).
We assume access to a smooth sorting routine that allows to compute quantiles in a differentiable way: smooth_quantile .
Speciﬁcally, Alg.
B provides an exemplary implementation of ConfTr with (smooth) T HR and L class as outlined in Alg. 1 in the main paper.
smooth_predict_threshold and smooth_calibrate_threshold implement differentiable prediction and calibration steps for T HR .
These implementations are used in compute_loss_and_error to “simulate” CP on mini-batches during training.
Size loss Ω from Eq.
(3) and classiﬁcation loss from Eq.
(5) are implemented in compute_size_loss and compute_general_classification_loss .
Note that the deﬁnition of compute_loss_and_error distinguishes between trainable_params and fixed_params , allowing to ﬁne-tune a pre-trained model.
[Figure]
[Table]
[Figure]
I n e ff i
c i e n c y
F-MNIST: Baseline+Thr
[Figure]
I n e ff i
c i e n c y
F-MNIST:
ConfTr+Thr
[Table]
[Figure]
[Table]
n e ff i
c i e n c y
[Table]
I n e i
[Table] [Table]
[Figure]
on MNIST and
Fashion-MNIST, class-conditional inefﬁciency is also lower, on average.
But the distribution across classes remains similar.
The same holds for coverage confusion, where lower overall inefﬁciency reduces confusion across the matrix, but the “pattern” remains roughly the same.
On CIFAR10,
ConfTr does not improve average inefﬁciency signiﬁcantly, such that the confusion matrix remains mostly the same.
Algorithm B Python Pseudo-Code for ConfTr:
We present code based on our Python and Jax implementation of ConfTr.
In particular, we include smooth calibration and prediction steps for
T HR as well as the classiﬁcation loss L class and the size loss Ω .
Instead of including a full training loop, compute_loss_and_error shows how to compute the loss which can then be called using jax.value_and_grad(compute_loss_and_error, has_aux=True) and used for training using Optax.
Hyper-parameters, including alpha , dispersion , size_weight , temperature , loss_matrix , size_weights and weight_decay , are not deﬁned explicitly for brevity.
smooth_quantile is assumed to be a provided differentiable quantile computation method.
Finally, model can be any Jax/Haiku model
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
classification
[Table]
on
[Table]
"""one_hot
[Table]
one
[Table]
hot
[Table] [Table]
one
[Table]
hot
[Table] [Table]
one
[Table] [Table]
[Table]
sum
[Table] [Table]
return jnp.
[Table]
[Table] [Table]
compute
[Table] [Table]
int
[Table]
weights
[Table]
jnp
[Table]
> jnp
[Table]
ndarray
[Table] [Table]
Compute
[Table]
"""return jnp
[Table]
mean
[Table]
sum
[Table] [Table]
size
[Table] [Table]
[Table]
compute_
[Table]
_and_error(
[Table] [Table]
[Table] [Table]
>
[Table] [Table]
params
[Table] [Table]
[Table]
logits, new
[Table] [Table]
[Table] [Table]
])val
[Table]
]val
[Table]
labels
[Table]
labels
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
return loss, new_model_state