T OWARDS TRUSTWORTHY PREDICTIONS FROM DEEP NEURAL NETWORKS WITH FAST ADVERSARIAL CALIBRATION
Anonymous authors
Paper under double-blind review
A BSTRACT
To facilitate a wide-spread acceptance of AI systems guiding decision making in real-world applications trustworthiness of deployed models is key.
That is, it is crucial for predictive models to be uncertainty-aware and yield well-calibrated and thus trustworthy) predictions for both in-domain samples as well as under domain shift.
Recent efforts to account for predictive uncertainty include post-processing steps for trained neural networks
Bayesian neural networks as well as alternative non
-Bayesian approaches such as ensemble approaches and evidential deep learning
Here, we propose an efﬁcient yet general modelling approach for obtaining well-calibrated trustworthy probabilities for samples obtained after a domain shift
We introduce a new training strategy combining an entropy-encouraging loss term with an adversarial calibration loss term and demonstrate that this results in well-calibrated and technically trustworthy predictions for a wide range of perturbations
We comprehensively evaluate previously proposed approaches on different data modalities
, a large range of data sets including sequence data network architectures and perturbation strategies and observe that our modelling approach substantially outperforms existing state-of-the-art approaches, yielding well calibrated predictions under domain drift.
I NTRODUCTION
To facilitate a wide-spread acceptance of AI systems guiding decision making in real-world applications, trustworthiness of deployed models is key.
Not only in safety-critical applications such as autonomous driving or medicine (Helldin et al., 2013; Caruana et al., 2015; Leibig et al., 2017), but also in dynamic open world systems in industry it is crucial for predictive models to be uncertaintyaware.
Only if predictions are calibrated in the case of any gradual domain shift, covering the entire spectrum from in-domain ("known unknowns") to truly out-of-domain samples ("unknown unknowns"), they can be trusted.
In particular in industrial and IoT settings, deployed models may encounter erroneous and inconsistent inputs far away from the input domain throughout the life-cycle; in addition, the distribution of the input data may gradually move away from the distribution of the training data (e.g. due to wear and tear of the assets, maintenance procedures or change in usage patterns).
Avarietyof approaches to account for predictive uncertainty exist.
They include post-processing steps for trained neural networks, where for example a validation set, drawn from the same distribution as the training data, is used to rescale the logit vectors returned by a trained neural network such that in-domain predictions are well calibrated (Platt, 1999; Guo et al., 2017).
Orthogonal approaches have been proposed where trust scores and other measures for out-of-distribution (OOD) detection are derived, typically also based on trained networks (Liang et al., 2018; Jiang et al., 2018; Papernot &
McDaniel, 2018); however these latter approaches are designed to detect only truly OOD samples and do not consider the continuum of domain shifts from in-domain to truly OOD.
Alternative avenues towards intrinsically uncertainty-aware networks have been followed by training probabilistic models.
In particular, a lot of research effort has been put into training Bayesian neural networks, where typically a prior distribution over the weights is speciﬁed and, given the training data, a posterior distribution over the weights is inferred.
This distribution can then be used to quantify predictive uncertainty.
Since exact inference is untractable, a range of approaches for approximate inference has been proposed.
In particular approaches based on variational approximations have recently received a lot of attention and range from estimators of the fully factorized posterior (Blundell et al., 2015), to the interpretation of Gaussian dropout as performing approximate Bayesian inference (Gal & Ghahramani, 2016) and facilitating a complex posterior using normalising ﬂows (Louizos & Welling, 2017).
Since such Bayesian approaches often come at a high computational cost, alternative non-Bayesian approaches have been proposed, that can also account for predictive uncertainty.
These include ensemble approaches, where smooth predictive estimates can be obtained by training ensembles of neural networks using adversarial examples (Lakshminarayanan et al., 2017), and evidential deep learning, where predictions of a neural net are modelled as subjective opinions by placing a Dirichlet distribution on the class probabilities (Sensoy et al., 2018).
Both for Bayesian and non-Bayesian approaches, uncertainty-awareness and the quality of predictive uncertainty are typically evaluated by analysing the behaviour of the predictive entropy for out-of-domain predictions in form of gradual perturbations (e.g. rotation of an image), adversarial examples or held-out classes.
However, while an increasing predictive entropy for increasingly strong perturbations can be an indicator for uncertainty-awareness, simply high predictive entropy is not sufﬁcient for trustworthy predictions.
Models can only be trusted if the conﬁdence of predictions is calibrated, that is if the entropy matches the actual accuracy of the model.
For example, if the entropy is too high, the model will yield under-conﬁdent predictions and similarly, if the entropy is too low, predictions will be over-conﬁdent.
Notably, the focus of related work introduced above has been on image data and it remains unclear how these approaches perform for other data modalities, in particular when modelling sequences with long-range dependencies using complex architectures such as LSTMs (Hochreiter & Schmidhuber, 1997) or GRUs (Cho et al., 2014).
Here, we propose an efﬁcient yet general modelling approach for obtaining calibrated, trustworthy probabilities for both in-domain samples as well as under domain shift that can readily be applied to a wide range of data modalities and model architectures.
More speciﬁcally, we ﬁrst introduce a simple loss function to encourage high entropy on wrong predictions and combine this with an adversarial calibration loss term.
Since in practical applications it is a priori not clear what type or magnitude of domain drift will occur, we evaluate calibration under doamin drift for 10 different perturbations and 10 different noise levels not seen during training.
Our contribution in this paper is three-fold.
(i) we illustrate the limitations of entropy as measure for trustworthy predictions and motivate the use of the expected calibration error for quantifying technical robustness (Dawid, 1982; DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005; Naeini et al., 2015; Guo et al., 2017).
(ii) we introduce a new training strategy combining an entropy-encouraging loss with an adversarial calibration loss term and demonstrate that this results in better calibration and technical trustworthiness of predictions for diverse types of out-of-domain samples and perturbations, compared to the state-of-the-art.
(iii)
We apply the concept of uncertainty-awareness and trustworthiness to sequence models and demonstrate that our approach substantially improves predictive uncertainty over existing approaches when classifying long sequences.
While previous studies only compared predictive entropy for one simple architecture (LeNet) and typically one type of domain shift (Sensoy et al., 2018; Louizos & Welling, 2017), we here present an extensive comparison ofdifferent architectures across 10 different perturbation strategies.
T OWARDS TECHNICALLY TRUSTWORTHY PREDICTIONS 2. L IMITATIONS OF ENTROPY AS MEASURE FOR UNCERTAINTY - AWARENESS
Recent efforts in terms of evaluating predictive uncertainty have focused on entropy as measure for uncertainty-awareness for predictions under domain shift.
While entropy quantiﬁes the uncertainty encoded in the model output, it is not clear what absolute entropy is required for a model to be reliable, given a set of samples from an out-of-domain distribution.
For example, a popular evaluation strategy consists of computing the absolute entropy for out-of-domain samples generated using perturbation strategies based on the images in the test set (e.g. gradual rotation of images) (Sensoy et al., 2018; Louizos & Welling, 2017).
In this case, the entropy should increase with rotation angle, as the accuracy decreases in a coordinated fashion (since the model was not trained with rotated images) (Fig. 1).
However, such evaluations alone are not sufﬁcient to determine whether model predictions are technically reliable (or trustworthy), since it is not clear whether accuracy and model conﬁdence/uncertainty are coupled in a meaningful way.
Building on prior work utilising the concept of calibration for in-domain predictions, this coupling can be quantiﬁed using reliability diagrams
(Guo et al., 2017), where the model conﬁdence (i e. the probability associated with the predicted class label) is linked to accuracy in a stratiﬁed manner.
That is, for a given set of samples obtained under domain shift, conﬁdence and accuracy should match for all conﬁdence levels between 1 /n classes and 1.0.
For example, for the subset of samples with conﬁdence between e.g. 60% and 70% the average accuracy should lie in that same range; this relationship should hold for all intervals.
Figure 1 illustrates that the accuracy decreases, while the entropy increases if perturbed images are fed to a trained neural network (top right); however, additional information directly linking the uncertainty or conﬁdence of a model to its accuracy is required to establish whether predictions are calibrated.
This is illustrated by reliability diagrams in ﬁgure 1 (bottom row), showing accuracy as function of binned conﬁdence and the expected calibration error (ECE) curve, summarizing the calibration gap by covering the entire spectrum of domain shifts.
(DeGroot & Fienberg, 1983; Niculescu-Mizil &
Caruana, 2005).
changes with increasing perturbation strength.
2.1. Q UANTIFYING CALIBRATION UNDER DOMAIN SHIFT USING THE ECE CURVE
Let X ∈ R D and Y ∈ { 1 , . .
, C } be random variables that denote the D -dimensional input and labels in a classiﬁcation task with C classes, respectively.
Let h ( X ) =
( ˆ Y , ˆ P ) be the output of a neural network classiﬁer
h predicting a class ˆ Y and associated conﬁdence ˆ P based on X .
We follow
Guo et al.
(2017) and formally deﬁne perfect calibration such that conﬁdence and accuracy match for all conﬁdence levels:
[Equation]
This directly leads to a deﬁnition of miss-calibration mca as the difference in expectation between conﬁdence and accuracy, that is
[Equation]
mca can be estimated from ﬁnite samples, by partitioning predictions into M equally-spaced bins and computing a weighted average of the bins’ difference between accuracy and conﬁdence.
The resulting
[Equation]
with B m being the set of indices of samples whose prediction conﬁdence falls into its associated interval I m .
conf( B m ) and acc( B m ) are the average conﬁdence and accuracy associated to B m respectively and n the number of samples in the dataset.
It can be shown that ECE is directly connected to miss-calibration as ECE using M bins converges to the M-term Riemann-Stieltjes sum of eq.
2 Guo et al.
Finally, ECE can also be interpreted as a summary measures of reliability diagrams since it quantiﬁes the calibration gap (red bars in ﬁgure 1).
ECE was recently popularised when used quantify in-domain calibration for modern neural networks
Guo et al.
Here, we use an ECE based measure to evaluate the calibration of a predictive model under domain drift.
Since the type of domain drift that may occur after training is generally not known a priori, we deﬁne a range of distinct perturbation types not seen during training.
Each perturbation strategy mimics a scenario where the data a deployed model encounters stems from a distribution that gradually shifts away from the training distribution in a different manner.
For each perturbation type we compute the ECE for a range of perturbation strengths.
We then generate a
ECE-perturbation curve and summarize overall calibration by computing a micro-averaged ECE across all perturbation strengths.
ECE curves and micro-averaged ECE are primarily designed to evaluate calibration in domain-drift scenarios where the distribution the inputs are drawn from changes, but the set of labels remains unchanged.
For truly OOD samples with labels that are not part of the train classes, entropy-based measure are a useful complementary approach to evaluate predictive uncertainty.
2. A SIMPLE APPROACH FOR CALIBRATED PREDICTIVE UNCERTAINTY ESTIMATION 2.2. P REDICTIVE ENTROPY
To mitigate overconﬁdent predictions displayed by conventional deep neural networks, we ﬁrst introduce a loss term encouraging a uniform distribution of the scores in case the model "does not know".
That is, we distribute the probability mass of false predictions uniformly over C classes:
[Equation]
with p ij being the conﬁdence associated to the j th class of sample i , y ij its one-hot encoded label.
This simple loss term increases uncertainty-awareness by encouraging an increased entropy ( S ) in the presence of high predictive uncertainty without directly affecting reconstruction loss (categorical cross-entropy) and thus accuracy.
This has the advantage that our approach - in contrast to stateof-the-art Bayesian neural networks such as those based on multiplicative normalizing ﬂows or evidential deep learning - can be readily applied to complex architectures based on LSTMs or GRUs.
In addition, the loss term is parameter free and thus does not require hyperparameter tuning, again facilitating easy usage.
2.2. A DVERSARIAL CALIBRATION
While the entropy-based loss term does encourage uncertainty-awareness, we found that it is beneﬁcial to introduce an additional loss term addressing model calibration directly.
Explicitly encouraging calibration for out-of-domain samples, however - e.g. via an ECE-based measure - requires knowledge on the type of perturbed or erroneous samples the model is expected to encounter.
In many realworld applications it is not clear from which distribution these samples will be drawn and, for model predictions to be truly trustworthy requires robustness against all such potential out-ofdomain samples.
That is, we would like our model to be technically robust for inputs around an (cid:15) -neighbourhood of the in-domain training samples, for a wide range of (cid:15) and for all 2 D directions in {− 1 , 1 } D .
While inputs from a random direction are unlikely to be representative examples for generic out-of-domain samples, by deﬁnition adversarial examples are generated along a dimension where the loss is high.
Lakshminarayanan et al.
(2017) show that adversarial training can improve the smoothness of predictions, in particular when training an ensemble of 5 neural networks in an adversarial fashion.
Here, we demonstrate that using adversarial samples to directly optimise model calibration (rather than the squared error of one-hot encoded labels (Lakshminarayanan et al., 2017)) results in substantially more trustworthy predictions for out-of-domain samples from a large number of unrelated directions.
We implement an ECE-inspired calibration loss by minimizing miss-calibration for samples generated using the fast gradient sign method (FGSM) (Goodfellow et al., 2014), with (cid:15) ranging from 0 to 0.
(sampled at 10 equally spaced bins at random).
To this end we minimise the L2 norm of the difference between the predicted conﬁdence of a sample i , which we denote as conf( i ) , and its corresponding binned accuracy acc( B m i ) , for all samples.
This is directly motivated by the deﬁnition of ECE (eq. 2.1.1) and as for the computation of ECE, we partition the predictions of the network into M equally-spaced bins, with m i ∈ { 1 , . . .
, M } being bin into which sample i falls.
As for ECE, B m i is the set of indices of samples falling in bin m i and acc( B m i ) the average accuracy of samples B m i .
We set M = 10 for all experiments.
[Equation]
The ﬁnal loss balancing a standard reconstruction loss (categorical cross entropy (CCE)) against the entropy and adversarial calibration loss can then be written as L =
L CCE + λ adv L adv + λ S L S .
The choice of hyperparameters λ adv and λ S is described in the appendix along with a robustness analysis, an ablation study for both loss terms and a summary of the algorithm.
Note that we do not use the FGSM samples for adversarial training in the sense that we do not try to minimize the reconstruction error (cross entropy) for those samples.
E XPERIMENTAL RESULTS
We compare our approach for fast adversarial calibration to both Bayesian and non-Bayesian work and perform an extensive set of experiments.
We evaluate model trustworthiness by quantifying model calibration for 10 distinct strategies to generate out-of-domain samples.
We show that our approach is able to yield technically trustworthy predictions across 4 datasets, 4 model architectures and three data modalities.
We assess 9 distinct image-based perturbation types including left rotation, right rotation, shift in x direction, shift in y direction, xy shift, shear, zoom in x direction, zoom in y direction and xy zoom for image data.
In addition, we investigate robustness to random word swaps for text data.
More speciﬁcally, a perturbation is generated by ﬁrst drawing a random set of words in a corpus.
Next each of these words is replaced by a word drawn at random from the vocabulary.
For all perturbation strategies, perturbed samples were generated at 10 different levels, starting at no perturbation, until accuracy reached random levels; relative perturbation strength is denoted by epsilon.
The micro-averaged ECE for a speciﬁc perturbation strategy was computed by ﬁrst perturbing each sample in the test set at 10 different levels and then calculating the overall ECE across all samples.
By computing this micro-averaged ECE for 100 different perturbation scenarios, we quantify the ability of neural networks to to yield well-calibrated, technically robust predictions in diverse circumstances.
We ﬁrst show that our modelling approach substantially outperforms existing approaches for sequence models and then illustrate improved performance for image data.
To evaluate our modelling approach for sequence data, we ﬁt models on the following datasets:
Sequential MNIST.
10 classes of handwritten digits.
Images are converted to pixel-wise sequences of length 28x28.
20 Newsgroups.
News articles partitioned into 20 classes.
News classes are modelled as sequences of words using word embeddings.
We used the 20,000 most common words as vocabulary and a maximum word length of 2500.
We ﬁtted LSTM and GRU models with one hidden layer for all sequence modelling tasks.
For the image classiﬁcation tasks, we ﬁtted a LeNet model to MNIST data in order to establish a fair comparison to the state-of-the-art (Guo et al., 2017; Sensoy et al., 2018).
To evaluate the performance for more complex architectures, we further ﬁtted a deep neural net with VGG19 architecture on the
CIFAR10 dataset.
We used standard splits into training and test set for all datasets.
We compared the following modelling approaches: (i) L2-Dropout , referring to a standard neural
[Figure]
net with L2 regularisation as baseline, (ii) MC-Dropout corresponding to the modelling approach presented by Gal & Ghahramani (2016), (iii) Deep Ensembles referring to an approach based on an ensemble of neural nets trained using adversarial examples (Lakshminarayanan et al., 2017), (iv)
EDL referring to Evidential Deep Learning (Sensoy et al., 2018), (v) MNF referring to a Bayesian neural network trained using multiplicative normalising ﬂows (Louizos & Welling, 2017) and (vi)
FALCON , which is our method based on Fast AdversariaL CalibratiON.
Additional comparisons to temperature scaling (Guo et al., 2017) and stochasctic variational inference (SVI) based on Flipout (Wen et al., 2018) are shown in the Appendix A.6.
3. P REDICTIVE UNCERTAINTY FOR SEQUENCE MODELING
We trained LSTM models with one hidden layer of 130 hidden units using the RMSPROP optimizer.
GRU models were trained with one hidden layer of 250 hidden units to reﬂect the reduced complexity of GRU cells compared to LSTM cells.
The Bayesian neural network based on multiplicative normalizing ﬂows (MNF) was developed for convolutional neural networks; since the transfer of such a complex modelling approach from convolutional neural networks to recurrent neural networks is out of the scope of this work, we omitted MNF in our comparison of sequence models.
Sequential MNIST For deep ensembles of LSTMs trained on sequential MNIST we found that models did not converge when training the networks with adversarial examples; we therefore also trained ensembles with a reduced (cid:15) of 0.005 and report performance for this modiﬁed Deep Ensemble approach.
For the deep ensemble of GRUs on sequential MNIST and the deep ensemble of LSTMs on the 20 Newsgroups data, we report performance with standard adversarial training ( (cid:15) = 0 . 01 ).
Fitting LSTM models on sequential MNIST is a challenging task (Bai et al., 2018), and it was only possible to achieve state-of-the-art predictive power with EDL for shorter sequences (downsampling of images before conversion to sequence).
While performance of GRUs was better for all modelling approaches, EDL also did not achieve a competitive accuracy (Table S2 in appendix A.7).
We found that our approach achieved competitive predictive power for LSTM and GRU models and substantially improved calibration of the predictive uncertainty for both models (Figure 2, Table 1).
This illustrates that in contrast to existing approaches FALCON is able to yield well-calibrated and trustworthy predictions without compromising on accuracy, even for challenging tasks such as classifying long sequences with LSTMs.
20 Newsgroups To further evaluate the ability of FALCON to model sequence data, we compared the performance of FALCON to existing approaches for an NLP task.
To this end, we trained LSTMs to classify news articles into one of 20 classes.
We generated vector representations of words using the pre-trained GLOVE embedding (length 100) and used the ﬁrst 2500 words of an article as input for an LSTM.
We trained LSTMs with one hidden layer of 130 hidden units and evaluated it on a perturbation strategy based on random word swaps.
To establish a perturbation strategy with gradually increasing perturbations, we varied the fraction of words drawn from each sample between 0% and 45% in 5% steps (gradually decreasing accuracy to random levels).
Similar to the LSTM model trained on sequential MNIST, we found that EDL did not achieve competitive predictive power, with an accuracy of 49.3% only.
In contrast, FALCON resulted in well-calibrated predictions while maintaining a competitive accuracy of 75.7%, compared to 75.9%, 72.8% and 77.3% for L2-Dropout, MC-Dropout and Deep Ensemble respectively.
As before, the model conﬁdence of FALCON was substantially better calibrated than existing methods (Figure 3).
3. P REDICTIVE UNCERTAINTY FOR IMAGE CLASSIFICATION
[Figure]
bration error for 20 Newsgroups data.
We next evaluated the trustworthiness of predictions for image classiﬁcation tasks.
To establish a fair comparison with state-of-the-art models, including Bayesian neural networks, we ﬁrst trained the 5 existing approaches and evaluated them on 9 different perturbation strategies (not used during training).
While with increasingly strong perturbations the predictive entropy increased for all models, this was not necessarily matched by a good calibration across the range of the perturbation.
At the typical example of the perturbation y-zoom, it becomes clear that for most methods entropy did not increase sufﬁciently fast to match the decrease in accuracy, resulting in increasingly overconﬁdent predictions and an increasing ECE for stronger perturbations (Fig. 4).
While FALCON and EDL yielded well-calibrated predictions that were robust across all perturbation levels, it is worth noting that EDL has a substantially higher ECE for in-domain predictions, reﬂecting under-conﬁdent predictions on the test set (see also Fig.
S4, S5, appendix).
We observed this tendency of EDL towards under-conﬁdence when faced with new samples drawn from the same distribution as the training data (known unknowns) also for a different dataset and architecture (VGG19 on CIFAR10; ECE FALCON = 0 .
107 , ECE EDL = 0 .
125 on the test set).
While predictions with FALCON were also slightly under-conﬁdent for low levels of perturbation, this has arguably a smaller practical relevance than the substantial overconﬁdence of other baseline methods (c.f appendix A.6 for a more in-depth analysis).
We observed a similar behaviour across all other 8 perturbation strategies, which was reﬂected in the lowest micro-averaged ECE for FALCON, followed by EDL (Figure 5; Table 1).
[Figure]
To evaluate the technical robustness and calibration of FALCON on a more complex architecture for image classiﬁcation, we trained a VGG19 model on the CIFAR10 dataset.
We again observed a similar trend as for the MNIST data, with FALCON yielding well calibrated predictions across all perturbation strategies (Figure 5).
The considerable overconﬁdence of most baseline methods (except
EDL) when making predictions on OOD samples (epsilon 90) is also reﬂected by a substantially lower maximum entropy (Fig.
S7, Appendix A.7) and the overall distribution of conﬁdence scores (Fig. S8).
Note that we omitted MNF due to the large memory requirements stemming from the use of multiplicative normalising ﬂows.
[Table]
D ISCUSSION AND C ONCLUSION
We presented a fast, simple and generalizable approach for encouraging well-calibrated uncertaintyawareness of deep neural networks.
To this end, we combine an entropy encouraging loss-term with an adversarial calibration loss and show on diverse data modalities and model architectures that our approach yields well-calibrated predictions for both in-domain and out-of-domain samples generated based on 10 distinct perturbations.
We present a detailed analysis of calibration under domain drift for recurrent neural networks and identify major drawbacks of existing methods that were developed for (and evaluated on) image classiﬁcation tasks.
Thus, EDL was only able to result in networks with a high accuracy when trained on short sequences; both for the sequential MNIST and and 20 Newsgroups data, the EDL approach resulted in a substantially lower accuracy compared to baseline LSTM and GRU models.
While MC dropout is easy to ﬁt and fast, it results only in small improvements over the L2-Dropout baseline, especially for sequence data.
In contrast, our modeling approach is fast and robust, with well-calibrated predictive uncertainty across 10 perturbations,datasets, 4 model architectures and three data modalities.
A A PPENDIX A. P ARAMETER AND HYPERPARAMETER SETTINGS
Deep Ensembles, MNF and EDL were trained with default values for method-speciﬁc hyperparameters (e g. number of neural networks in a Deep Ensemble).
In addition, the following hyperparameters were picked using hyperparameter searches.
For all methods, the learning rate was chosen from { 1 e − 5 , 5 e − 5 , 1 e − 4 , 5 e − 4 , 1 e − 3 , 5 e − 3 } .
In addition, for the baseline method (L2), our method (FALCON), Deep Ensembles and EDL, dropout was chosen from { 0 , 0 . 5 } and
L2-regularisation from { 0 . 0 , 0 .
For EDL we chose the KL regularisation from { 0 . 5 , 1 ., 5 ., 15 ., 10 ., 30 ., 50 . } .
For a fair comparison with this state-of-the art model, we chose λ S from this same set of values for FALCON and λ adv from { 0 . 25 , 1 e − 1 , 1 e − 2 , 1 e − 3 , 1 e − 4 } .
We also assessed the effect of the individual loss terms λ adv and λ S and the robustness of FALCON with respect to the choice of λ S and λ adv (section A.4).
We used a batch size of 128 for all models and standard splits in train and test data for all datasets.
For the 20 Newsgroups dataset we used the keras tokenizer to format text samples, converting words into lower case, removing punctuation and special characters !"
#$%&()*+,-./:;<=>?@[\\]^_‘{}~\t\n’.
A. P ERTURBATION STRATEGIES
In practice it is not clear what type of perturbation a model may encounter.
To assess how neural networks cope in diverse settings, we generated out-of-domain samples based on 10 different perturbation strategies.
Each perturbation strategy mimics a scenario where the data a deployed model encounters stems from a distribution that gradually shifts away from the training distribution in a different manner.
Samples generated with maximum perturbation strength correspond for example to corrupted or erroneous samples a deployed model may face, unperturbed samples correspond to those drawn from the same distribution as the training data ("known unknowns").
Trustworthy AI models should yield well-calibrated conﬁdence scores in all those settings that it may encounter throughout its life-cycle.
We quantify this based on the expected calibration error, micro-averaged across all perturbation strengths, including no perturbation (Tables S6-S10).
For all perturbation strategies we chose 10 levels of perturbation, starting at no perturbation, such that accuracy levels were close to random for maximum perturbation strength (Table S5).
Speciﬁc levels of perturbation are listed in Table S4; for visualisation purposes we re-scaled all perturbation-speciﬁc parameters to range from 0 to 90 (in steps of 10) and denote this general perturbation strength as epsilon.
Perturbations include image transformations (rotation, shift, zoom, shear) as well as a word perturbation (word swap).
For sequential MNIST, perturbations were performed on the image before transforming the image to a sequence.
A. T RAINING ALGORITHM
Training was performed following Algorithm 1, summarizing the description in section 2.2.
Algorithm 1 FALCON with set of perturbation levels
E = { 0 , 0 . 05 , 0 . 1 , 0 . 15 , 0 . 2 , 0 . 25 , 0 . 3 , 0 . 35 , 0 . 4 , 0 . 45 } (n.b. (cid:15) =
encourages in-domain calibration) , mini batch size b , number of ECE bins M and training set ( X, Y ) .
m i ∈ { 1 , . . .
, M } denotes the bin into which sample i falls and B
m i is the set of indices of samples falling in bin m i .
acc
( B m i ) the average accuracy of samples B m i .
1: repeat 2:
Read minibatch MB
= ( { X 1 , . . .
, X b } , { Y 1 , . . .
, Y b } ) from training set 3:
Randomly sample (cid:15) MB from E 4:
Generate FGSM minibatch MB adv of size b from samples in MB using (cid:15) MB 5:
Compute L CCE and L S and do one training step using mini batch MB 6:
Compute predictions for all samples in MB and partition into M equally spaced bins 7:
Compute binned accuracy acc( B m i ) for all samples i in MB adv 8:
Compute L adv based on MB adv and do one training step using MB adv 9: until training converged
A. A BLATION STUDY AND SENSITIVITY ANALYSIS
In order to investigate the inﬂuence of the individual loss terms on calibration, we performed an ablation study, omitting one of the two loss terms, L S and L adv respectively.
While either loss term results in an improved calibration compared to the L2-dropout baseline, combining both terms yields consistently better results.
[Table]
t
[Table]
[Figure]
We further performed a robustness analysis in order to quantify the dependence of model performance on calibration under domain drift.
To this end, we trained LeNet models on MNIST varying both hyperparameters over wide ranges.
We ﬁrst ﬁxed λ S at the optimal value of 50 and varied λ adv between 0.0005 and 0.03 and computed the micro-averaged ECE for perturbation yzoom for all hyperparameter combinations.
Next, we ﬁxed λ
adv at the optimal value of 0.02 and varied λ S between 10 and 100.
We found that even when varying both hyperparameters over a wide range, ECE remained robust and varied by less than 0.04 for λ adv and less than 0.06 for λ S .
Accuracy was not affected by the choice of either λ and remained between 0.985 and 0.991.
[Figure]
A. A DDITIONAL BASELINES
In addition to the 5 baselines evaluated in depth in the main text
[Figure]
we also assessed two additional baselines, namely SVI based on Flipout and temperature scaling.
We found that temperature scaling, while improving in-domain calibration, is not beneﬁcial for calibration under domain-drift (Fig. S3).
In addition
[Figure]
we observe that SVI results in a decreased accuracy compared to other baselines and while calibration under domain drift improves in comparison to L2-dropout, it is substantially worse than FALCON and also EDL (Table S1)
[Figure]
L2-TS stands for temperature scaling, stands for Stochastic Variational Inference.
[Table]
A. U NDER - CONFIDENCE OF FALCON AND EDL FOR SMALL PERTURBATIONS
We observe e.g. in Fig.
4 (middle) [Table] that FALCON and EDL have a tendency to make slightly underconﬁdent predictions for small perturbations.
To further investigate the extend and practical relevance of this under-conﬁdence, we generated an empirical CDF of the conﬁdence scores for all approaches.
This conﬁrms the issue [Table] but also illustrates that FALCON is consistently less under-conﬁdent than
EDL: While for FALCON 13.3% of predictions on the test set are made with a conﬁdence of less than 90%, this increases to 30.5% for EDL, which is substantially higher than the 7.7% we observe for the well-calibrated Deep Ensembles.
In other words, only a minority of 5.6% of all predictions made by FALCON are made with a conﬁdence of less than 90% and should have been made with a higher conﬁdence.
This can also be seen at the level of individual samples, as illustrated in ﬁgure
[Figure]
[Figure] [Figure]
[Figure]
S5.
EDL makes noticeably under-conﬁdent predictions for small domain shifts before the entropy increases and conﬁdence scores match accuracy.
While FALCON also makes slightly under-conﬁdent predictions for in-domain samples, the corresponding conﬁdence scores are still substantially closer to 1 (matching the near perfect test accuracies for MNIST).
Like EDL, FALCON does not make over-conﬁdent predictions when moving further away from the training domain (epsilon greater 40).
This substantial over-conﬁdence e.g. illustrated in ﬁgure S5 has arguably a much bigger effect on a decision making process than the slight under-conﬁdence of FALCON affecting only a small fraction of samples: an underconﬁdent prediction of 89% rather than well-calibrated 95% is unlikely to affect a decision making process in a real world application to the same extent as an overconﬁdence of 80% rather than well-calibrated 10% (maximum entropy).
[Figure]
A. A DDITIONAL F IGURES AND T ABLES
[Figure]
[Figure]
[Figure]
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table]
[Table]
[Table]
FALCON 0.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table] [Table]