A Novel Regression Loss for Non-Parametric Uncertainty Optimization
Abstract
Quantiﬁcation of uncertainty is one of the most promising approaches to establish safe machine learning.
Despite its importance, it is far from being generally solved, especially for neural networks.
One of the most commonly used approaches so far is Monte Carlo dropout, which is computationally cheap and easy to apply in practice.
However, it can underestimate the uncertainty.
We propose a new objective, referred to as second-moment loss (SML), to address this issue.
While the full network is encouraged to model the mean, the dropout networks are explicitly used to optimize the model variance.
We intensively study the performance of the new objective on various UCI regression datasets.
Comparing to the state-of-the-art of deep ensembles, SML leads to comparable prediction accuracies and uncertainty estimates while only requiring a single model.
Under distribution shift, we observe moderate improvements.
As a side result, we introduce an intuitive Wasserstein distance-based uncertainty measure that is non-saturating and thus allows to resolve quality diﬀerences between any two uncertainty estimates.
1. Introduction
Having attracted great attention in both academia and digital economy, deep neural networks (DNNs, Goodfellow et al.
(2016)) are about to become vital components of safetycritical applications.
Examples are autonomous driving (Pomerleau, 1989; Bojarski et al., 2016) or medical diagnostics (Liu et al., 2014), where prediction errors potentially put humans at risk.
These systems require methods that are robust not only under lab conditions (i.i.d. data sampling), but also under continuous domain shifts, think e.g. of adults on e-scooters or growing varieties of mobile health sensors.
Besides shifts in the data, the data distribution itself poses further challenges.
Critical situations are (fortunately) rare and thus strongly under-represented in datasets.
Despite their rareness, these critical situations have a signiﬁcant impact on the safety of operations.
This calls for comprehensive selfassessment capabilities of DNNs and recent uncertainty mechanisms can be seen as a step in that direction.
While a variety of uncertainty approaches have been established, stable quantiﬁcation of uncertainty is still an open problem.
Many recent machine learning applications are e.g. equipped with Monte Carlo (MC) dropout (Gal and Ghahramani, 2016) that oﬀers conceptual simplicity and scalability.
However, it tends to underestimate uncertainties thus bearing disadvantages compared to more recent approaches such as deep ensembles (Lakshminarayanan et al., 2017).
We propose an alternative uncertainty mechanism.
It builds on dropout sub-networks and explicitly optimizes variances (see Fig.
1 for an illustrative example).
Technically, this is realized by a simple additive loss term, the second-moment loss .
To address the above outlined requirements for safety-critical systems, we evaluate our approach systematically w.r.t.
continuous data shifts.
In detail, our contribution is the introduction of a novel regression loss for better calibrated uncertainties applicable to dropout networks, reaching state-of-the-art performance in an empirical study and improving on it when considering data shifts.
2. Related work
Approaches to estimate predictive uncertainties can be broadly categorized into three groups:
Bayesian approximations, ensemble approaches and parametric models.
Monte Carlo dropout (Gal and Ghahramani, 2016) and its variants (see e.g. Gal et al.
(2017); Kendall and Gal (2017); Postels et al.
(2019)) are prominent representatives of the ﬁrst group.
They are theoretically well understood (see e.g. Sicking et al.
, oﬀer a Bayesian motivation, conceptual simplicity and scalability to application-size neural networks (NNs).
This combination distinguishes MC dropout from other Bayesian neural network (BNN) approximations like Blundell et al.
(2015) and Ritter et al.
Note that dropout training is also used—independent from an uncertainty context—for better model generalization (Srivastava et al., 2014).
Ensembles of neural networks, so-called deep ensembles (Lakshminarayanan et al., 2017), pose another popular approach to uncertainty modelling.
Comparative studies of uncertainty mechanisms (Snoek et al., 2019; Gustafsson et al., 2020) highlight their advantageous uncertainty quality, making deep ensembles a state-of-the-art method.
Fort et al.
(2019) argue that deep ensembles capture multi-modality of loss landscapes and thus yield potentially more diverse sets of solutions.
The third group are parametric modelling approaches that extend point estimations by adding a model output that is interpreted as variance or covariance (Nix and Weigend, 1994;
Heskes, 1997).
Typically, these approaches optimize a (Gaussian) negative log-likelihood (NLL, Nix and Weigend (1994)).
A more recent representative of this group is, e.g., Kendall and Gal (2017), for a review see Khosravi et al.
A closely related model class is deep kernel learning combining NNs and Gaussian processes (GPs) in various ways (see e.g.
(Wilson et al., 2016; Iwata and Ghahramani, 2017; Garnelo et al., 2018; Qiu et al.)).
The quality of uncertainties is typically evaluated using negative log-likelihood (Blei et al., 2006; Walker et al., 2016; Gal and Ghahramani, 2016), expected calibration error (ECE) (Naeini et al., 2015; Snoek et al., 2019), its variants, and by considering relations between uncertainty estimates and model errors (Sicking et al., 2019).
3. Second-moment loss
Monte Carlo (MC) dropout was proposed as a computationally cheap approximation of performing Bayesian inference in neural networks (Gal and Ghahramani, 2016).
Given a neural network f θ : R d → R m with parameters θ , MC dropout samples sub-networks f ˜ θ by randomly dropping nodes from the main model f θ .
During MC dropout inference the prediction is given by the mean estimate over the predictions of a given sample of subnetworks, while the uncertainty associated with this prediction can be estimated, e.g., in terms of the sample variance.
During MC dropout training the objective function, in our case the mean squared error (MSE), is applied to the sub-networks separately.
Due to this training procedure, all sub-network predictions are shifted towards the same training targets, which can result in overconﬁdent predictions, i.e. in an underestimation of prediction uncertainty.
Based on this observation, we propose to use the sub-networks f ˜ θ in a diﬀerent way: they are explicitly not encouraged to ﬁt the data mean directly.
This is the task of the full network f θ .
The sub-networks f ˜ θ instead model aleatoric uncertainty and prediction residuals if the prediction of the full network f θ is incorrect.
Thus, we deliberately assign diﬀerent ‘jobs’ to the main network f θ on the one hand and its sub-networks on the other hand.
Formalizing this idea into an optimization objective yields
[Equation]
[Equation]
[Table]
where the sum runs over a mini-batch of size M <N taken from the set of observed samples
D = { ( x i , y i ) }
Ni =1 , x i ∈ R
d denotes the input, y i ∈
R m the ground-truth label, and β >is a hyper-parameter that weights both terms.
The ﬁrst term, L regr , is the MSE w.r.t.
the full network f θ .
The second term, L sml , seeks to optimize 2 the sub-networks f ˜ θ .
It aims at ﬁnding sub-networks such that the distance | f ˜ θ − f θ | matches the aleatoric uncertainty or the prediction residual which is quantiﬁed by | f θ ( x i ) − y
i | . This leads to a signiﬁcant increase in the variance of the sub-networks, i.e. the second moment of f ˜ θ , compared to standard MC dropout, which is why we name L sml the second-moment loss (SML).
The standard deviations σ total of the predictions of the sub-networks w.r.t.
the prediction of the mean network induced by the SML have two components: the spread σ drop of the subnetworks and an oﬀset (cid:12)(cid:12)
f θ − (cid:104)
f
˜ θ (cid:105) (cid:12)(cid:12) between the full network and the sub-network mean that our loss might cause, concretely, σ total = σ drop + | f θ − (cid:104)
f
˜ θ (cid:105)| .
While | f θ − (cid:104) f ˜ θ (cid:105)| is reminiscent of residual matching, σ drop seems to be more closely related to modelling uncertainties.
We show in appendix A.2 that σ drop accounts on average for more than 80% of σ total in our experiments.
Note that while we investigate the proposed objective in terms of dropout sub-networks in this paper, our arguments as well as the actual approach are generally applicable to other models that allow to formulate sub-networks given some kind of mean model.
Besides the regression tasks considered here our approach could be useful for other objectives which use or beneﬁt from an underlying distribution, e.g. uncertainty quantiﬁcation in classiﬁcation.
4. Experiments
We study uncertainty quality on UCI regression datasets, where we extend the dataset selection in Gal and Ghahramani (2016) by adding three further datasets: ‘diabetes’, ‘california’, and ‘superconduct’.
Apart from i.i.d.
train- and test-data results, we study regression performance and uncertainty quality under data shift .
Such distributional changes and uncertainty quantiﬁcation are closely linked since the latter ones are rudimentary “selfassessment” mechanisms that help to judge model reliability.
These judgements gain importance for model inputs that are structurally diﬀerent from train data.
Appendix B. elaborates on our ways of splitting the data, namely pca-based splits in input space (using the ﬁrst principal component) and label-based splits.
We assess uncertainty performance in terms of the expected calibration error (ECE) and Wasserstein distance (WS) and regression performance using root-mean-squared error (RMSE) and negative log-likelihood (NLL).
All measures are described in detail in appendix B.1, where you can also ﬁnd more details on the network, the implementation of the methods and the training procedure.
For brevity of exposition, we limit our discussion here largely to the ECE.
An evaluation of the other measures can be found in appendix B.2.
All presented results are 5- or 10-fold cross validated.
Fig.
2 provides ECEs for 13 UCI datasets that are sorted by dataset size on the x-axis.
The top panel shows train- (green) and test-set (blue) ECEs, the bottom panel test-set ECEs under two pca-based data shifts (yellow-green, orange) and two label-based data splits (red, light red), for inter- and extrapolation respectively.
Uncertainty methods are encoded via plot markers, e.g. PU-DE as ‘star’ and SML-trained networks (‘ours’) as ‘square’.
We summarize these dataset-speciﬁc results on the right hand side of the ﬁgure (light grey background).
The columns ‘mean’ and ‘median’ of this summary show that on training sets, ECEs are smallest for PU, followed by PU-DE and the SML network.
On test data, however, PU, PU-DE and the SML network share the ﬁrst place.
Looking at the stability w.r.t.
data shift, i.e. the ability to extra- or interpolate to “unseen” data, PU loses in performance while PU-DE and SML reach the smallest calibration errors in three out of four cases, compare the lower panel in Fig.
Summarizing these evaluations, we ﬁnd SML to be as strong as the state-of-the-art method of PU-DEs while using only a single network compared to an ensemble of 5 networks.
We moreover observe advantages for SML under PCA- and label-based data shifts.
Three datasets lead to overestimated uncertainties for the SML, see discussion in appendix B.2.
A visual tool to further inspect uncertainty quality are residual-uncertainty scatter plots as shown in appendix B.3.
For a reﬂection on NLL and comparisons of the diﬀerent uncertainty measures see appendix B.2.
5. Conclusion
We approach dropout-based uncertainty quantiﬁcation from a new direction: sub-networks are explicitly not encouraged to model the data mean, they capture aleatoric uncertainties and potential ﬁtting residuals of the full network instead.
Technically, this is realized by an additional loss term that accompanies the standard regression objective: the secondmoment loss .
Our loss enables stable training.
Training complexity and runtime behavior at inference are comparable to MC dropout.
Task performances and uncertainty qualities of these models are on par with (parametric) deep ensembles, the widely used state-of-the-art for uncertainty quantiﬁcation.
However, unlike deep ensembles, we use single networks.
In practice, this might allow to reduce training eﬀort signiﬁcantly compared to deep ensembles, especially for application-scale networks.
Moreover, a single network requires only a fraction of the storage of a deep ensemble, making models with competitive uncertainties more accessible for mobile or embedded applications.
An extensive study of uncertainties under data shift revealed advantages of SML-trained models compared to deep ensembles: while both methods on average provide comparable results, we ﬁnd a higher stability across a variety of datasets and data shifts for the SML.
Technically, we attribute this gain in stability to our sub-network-based approach: like
MC dropout, we integrate uncertainty estimates into the very structure of the network, rendering it more robust towards unseen inputs than a parameter estimate.
Moreover, the second-moment loss can serve as a drop-in replacement for MC dropout on regression tasks.
For already trained MC dropout models, post-training with the secondmoment loss might suﬃce to improve uncertainty quality.
As an outlook, our ﬁrst such post-training experiments are encouraging.
Another interesting variant is the combination of SML with last-layer dropout (MC-LL) as it enables sampling-free inference (Postels et al., [Figure] [Table] [Figure]
Preliminary experiments show clearly improved uncertainty qualities compared to standard MC-LL.
A potentially interesting avenue for near real-time applications.
The simple additive structure of the second-moment loss makes it applicable to a variety of optimization objectives.
For classiﬁcation, we might be able to construct a nonparametric counterpart to prior networks (Malinin and Gales, 2018).
Taking a step back, we demonstrated an easily feasible approach to inﬂuence and train sub-network distributions.
This could be a promising avenue, for distribution matching but also for theoretical investigations.
Acknowledgments
The research of J. Sicking and M. Akila was funded by the German Federal Ministry for Economic Aﬀairs and Energy within the project “KI Absicherung – Safe AI for Automated Driving”.
Said authors would like to thank the consortium for the successful cooperation.
The work of T. Wirtz was funded by the German Federal Ministry of Education and Research, ML2R - no. 01S18038B. S. Wrobel contributed as part of the Fraunhofer Center for Machine Learning within the Fraunhofer Cluster for Cognitive Internet Technologies.
The work of A. Fischer was supported by the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) under Germany’s Excellence Strategy – EXC-2092 CaSa – 390781972.
Appendix A. Mechanics of the second-moment loss
We analytically study the optimization landscape evoked by the second-moment loss in
A.1.
This analysis provides building blocks to better understand the composition of the
SML-uncertainties as detailed in the remainder of this section.
A.1. Analytical properties of the second-moment loss
In the following, we look closer at the behaviour of the second-moment loss with respect to aleatoric uncertainty.
For this, we assume that the residuals, compare eq.
(1), are given by a Gaussian distribution with, for simplicity, µ Res.
= 0 and
σ Res.
= 1.
We want to determine the resulting loss for the L sml term in eq.
(1) governing the uncertainty estimation of the model.
It depends on the underling distribution of the eﬀective MC dropout distribution, which me model as N ( µ drop , σ drop ) such that:
[Equation]
where p 1 and p 2 are the Gaussian distributions discussed above.
After some calculation this yields:
[Equation]
which is visualized in Fig.
The two global minima can be found for σ drop = 0 and µ drop = ± (cid:112) 2 /π .
However, as we model a randomized residual y 1 these minima do not reach zero.
We ﬁnd that it is favourable to move µ drop away from the network prediction of µ Res.
= 0, the mean of the underlying data distribution.
But, this is only the case as long as the inherent uncertainty in the dropout distribution can be brought below σ drop < 2 /π , which is still smaller than the uncertainty of σ Res.
= 1 assumed within the training data distribution.
Otherwise, it is more favourable to have µ drop = µ Res.
= 0.
Decomposing the uncertainty for the UCI datasets in section A.2 showed mixed behaviour with indications for bi-modal shifts in µ drop as well as improved values of σ drop .
We already showed the eﬀect of this bi-modality in Fig.
1 at the beginning of the paper, where various sub-networks where sampled.
Clearly visible is a stronger variation between the networks compared to MC, but also a concentration around the two possible minima.
While this Fig.
provides a good visual estimate of σ drop the total uncertainty σ total would additionally contain the systematic shift | f θ − (cid:104)
f
˜ θ (cid:105)| .
Given the roughly symmetric distribution of the sub-networks we can expect it to be comparatively small.
A.2. Composition of the uncertainty estimate
The uncertainty estimate of the second-moment loss is comprised of two parts: σ total = σ drop + | f θ − (cid:104)
f
˜ θ (cid:105)| .
Fig.
4 reveals that σ drop contributes to more than 80% of σ total for the three presented datasets and for all applied data splits.
A highly similar behavior can be observed for all other datasets.
The analytical consideration in appendix A.1 suggests that for cases without aleatoric uncertainty the SML provides no incentive for | f θ − (cid:104)
f
˜ θ (cid:105)| > 0.
The same holds true in the presence of aleatoric uncertainty as long as σ drop is comparably large.
For aleatoric uncertainty and small σ drop larger | f θ − (cid:104)
f
˜ θ (cid:105)| are favorable.
However, as our loss is radial symmetric, all directions are equivalent and initialization and randomness determine the direction of the spread | f θ − (cid:104)
f
˜ θ
(cid:105)| for each individual sub-network.
This symmetry leads again to a small averaged | f θ − (cid:104)
f
˜ θ (cid:105)| .
σ drop on the contrary describes the width of a bi-modal set of sub-networks in these cases.
A.3. Detailed analysis of the two loss components
A deeper look into the structure of the second-moment loss is possible if we investigate its behaviour component-wise.
To clarify the results presented in Fig.
5, we recall the loss structure as
[Equation]
with a i = f θ ( x i ) − y i
and b i = f
˜ θ ( x i ) − f θ ( x i ).
Histograms of the a i (Fig. 5, ﬁrst column) enable a detailed view on network performance.
The uncertainty quality of the networks can be judged by studying the L 2 loss term more closely, namely by visualizing histograms of
[Figure]
Qualitatively, we observe that both the a i ’s and b i ’s are centered around zero which hints at successful optimization of regression performance and of uncertainty quality.
Details on how the optimization is technically realistic, can be gained from the scatter plots.
They show two qualitative shapes: a ‘line’ (ﬁrst row) and a ‘blob’ (second and third row).
For an in-detail discussion of the uni- and bi-modality of the second-moment loss landscape see A.1.
A ‘line’ shape reﬂects that all sub-networks occupy the same minimum given a bi-modal case.
Following appendix A.1, a ‘blob’ indicates a uni-modal case that might be evoked by large standard deviations σ drop .
Appendix B. Extension to the empirical study
Accompanying to the evaluation sketched in the body of the paper, section 4, we provide more details on the setup, used benchmarks and measures in the following sub-section.
Further information on the experiments are given in section B.2, which we extend by the measures skipped in the main text, and include a description on the used label splits.
We close with a look at the predicted uncertainties (per method) via scatter plots in section
B.3.
B.1. Experimental setup
The experimental setup used for the experiments is presented in three parts: the benchmark approaches we compare with, the evaluation measures we apply to quantify uncertainty, and a description of the neural networks and training procedures we employ.
Benchmark approaches
We compare dropout networks trained with the SML to archetypes of uncertainty modelling, namely approximate Bayesian techniques, parametric uncertainty, and ensembling approaches.
From the ﬁrst group, we pick MC dropout (abbreviated as MC ) and its variant last-layer MC dropout ( MC-LL ).
While these dropout approaches integrate uncertainty estimation into the very structure of the network, parametric approaches model the variance directly as the output of the neural network (Nix and
Weigend, 1994).
Such networks typically output mean and variance of a Gaussian distribution ( µ, σ ) and are trained by likelihood maximization.
This approach is denoted as PU for parametric uncertainty.
Ensembles of PU-networks (Lakshminarayanan et al., 2017), referred to as deep ensembles, pose a widely used state-of-the-art method for uncertainty estimation (Snoek et al., 2019).
Moreover, we consider ensembles of non-parametric standard networks.
We refer to the latter ones as DEs while we call those using PU PU-DEs .
All considered types of networks provide estimates ( µ i , σ i ) where σ i is obtained either analytically (PU), by sampling (MC, MC-LL, SML) or as an ensemble aggregate (DE, PU-DE).
Evaluation measures
In all experiments we evaluate both regression performance and uncertainty quality.
Regression performance is quantiﬁed by the root-mean-square error ( RMSE ), (cid:112) (1 /N (cid:80) i ( µ i − y i ) 2 (Bishop, 2006).
Another established metric in the uncertainty community is the (Gaussian) negative log-likelihood ( NLL ), 1 /N (cid:80) i (cid:0) log σ i + ( µ i − y i ) 2 / (2 σ 2 i ) + c (cid:1) , a hybrid between performance and uncertainty measure (Gneiting and Raftery, 2007), see appendix D.2 for a discussion.
The expected calibration error ( ECE , Kuleshov et al.
in contrast is not biased towards well-performing models and in that sense is a pure uncertainty measure.
It reads ECE =
(cid:80)
Bj =1 | ˜ p
j − 1 /B | for B equally spaced bins in quantile space and ˜ p
j = |{ r i | q j ≤ ˜ q
( r i ) <
q j +1 }| /N
the empirical frequency of data points falling into such a bin.
The normalized prediction residuals r i are deﬁned as r i =
( µ i − y i ) /σ
i .
Additionally, we propose to consider the Wasserstein distance of normalized prediction residuals ( WS ).
The Wasserstein distance (Villani, 2008), also known as earth mover’s distance (Rubner et al., 1998), is a transport-based measure denoted by ( d WS ) between two probability densities, with Wasserstein GANs (Arjovsky et al., 2017) as its most prominent application in ML.
For ideally calibrated uncertainties, we expect y i ∼ N ( µ i , σ i ) and therefore r i ∼ N (0 , 1).
Thus we use d WS ( { r i } i , N (0 , 1)) to measure deviations from this ideal behavior.
As ECE, this is a pure uncertainty measure.
However, it does not use binning and can therefore resolves deviations on all scales.
For example, two strongly ill-calibrated uncertainties ( r 1 , r 2 (cid:29) 1, r 1 < r 2 ) would result in (almost) identical
ECE values while WS would resolve this diﬀerence in magnitude.
Technical details
All investigated neural networks have the same architecture, 2 hidden layers of width 50, and ReLu activations (Glorot et al., 2011).
For all dropout-based methods (MC, MC-LL, SML) we set the drop rate to p = 0 .
Like MC, SML-trained networks apply Bernoulli dropout to all hidden activations.
In the case of MC-LL the dropout is only applied to the last hidden layer.
For ensemble methods (DE, DE-PU) we employ 5 networks.
For PE networks, we normalize the σ value using softplus (Glorot et al., 2011) and optimzie the NLL instead of the MSE.
For the optimization of all NNs we use the ADAM-optimizer (Kingma and Ba) with a learning rate of 0 . 001.
For ‘california’, the learning rate is reduced to 0 .
0001 as training of PU and PU-DE is unstable using the standard setup.
Additionally, we apply standard normalization to the input and output features of all datasets to enable better comparability.
Number of epochs trained and amount of cross validation diﬀers by the training-set size.
We categorize the datasets as follows: small datasets { yacht, diabetes, boston, energy, concrete, wine-red } , large datasets { abalone, power, naval, california, superconduct, protein } and very large datasets { year } .
For small datasets, NNs are trained for 1 , 000 epochs using mini-batches of size 100.
All results are 10-fold cross validated.
For large datasets, we train for 150 epochs and apply 5-fold cross validation.
We keep this large-dataset setting for the very large ‘year’ dataset but increase mini-batch size to 500.
All experiments are conducted on Core Intel(R)
Xeon(R) Gold 6126 CPUs.
Conducting the described experiments with cross validation on one CPU takes 80 h .
For SML it turns out that as long as 0 <β < 1, the actual value of β has only a limited inﬂuence on the optimization result, see appendix C for details.
Larger β -values can however favour uncertainty optimization at an expanse of task performance.
Throughout the body of the paper we use a conservative value of β = 0 .
B.2. RMSEs, NLLs and systematic evaluation
This sub-section provides further details on our experiments covering: an overview on the datasets and splits used for the data-shift studies, further uncertainty measure evaluations (RMSE, NLL, WS), and close with a discussion of the weaker SML results.
Datasets and data splits
For the regression data, Table 1 provides details on dataset references, preprocessing and basic statistics.
Extrapolation and interpolation data-shifts are, technically, introduced by applying non-i.i.d.
(independent and identically distributed) data splits.
Natural candidates for such non-i.i.d.
splits are splits along the main directions of data in input and output space, respectively.
Here, we consider 1D regression tasks.
Therefore, output-based splits are simply done on a scalar label variable (see Fig.
6, right).
We call such a split label-based (for a comparable split, see, e.g., Foong et al.
In input space, the ﬁrst component of a principal component analysis (PCA) provides a natural direction (see Fig.
6, left).
The actual PCA-split is then based on projections of the data points onto this ﬁrst PCA-component.
6 Splitting data along such an direction in input or output space in e.g. 10 equally large chunks, creates 2 outer data chunks andinner data chunks.
Training a model on 9 of these chunks such that the remaining chunk for evaluation is an inner chunk is called data interpolation .
If the remaining test chunk is an outer chunk, it is data extrapolation .
We introduce this distinction as extrapolation is expected to be considerably more diﬃcult than ‘bridging’ between feature combinations that were seen during training.
Regression quality
First, we consider regression performance (see top panel of Fig. 7).
Averaging the RMSE values over the considered 13 datasets (‘mean’ column) yields almost identical results for all uncertainty methods.
The only exceptions pose PU and PU-DE with larger train data RMSEs which could be due to NLL optimization favoring to adapt variance rather than mean.
However, this regularizing NLL-training comes along with a smaller generalization gap, leading to competitive test RMSEs.
Next, we investigate model performance under data shift, visualized in the bottom panel of Fig.
Again, regression quality is comparable between all methods.
As expected, performances under data shift are worse compared to those on i.i.d.
test sets.
Negative log-likelihoods
For NLL, results are less balanced compared to RMSE (see
Fig.
PU-DE and the SML-trained network reach comparably small average values, followed by MC and DE.
The average NLL values of MC-LL and PU are above the upper
[Table]
[Figure]
plot limit indicating a rather weak stability of these methods.
On PCA-interpolate and
PCA-extrapolate test sets, again PU-DE and SML-trained networks perform best.
On label-interpolate and label-extrapolate test sets, SML-trained networks take the ﬁrst place with a large margin.
The mean NLL values of most other approaches are above the upper plot limit.
Note that median results (the column next to ‘mean’) are not as widely spread and PU-DE and SML perform comparably well.
These qualitative diﬀerences between mean and median behavior indicate that most methods perform poorly ‘once in a while’.
A noteworthy observation as stability across a variety of data shifts and datasets can be seen
[Table]
[Figure]
as a crucial requirement for an uncertainty method.
SML-based models yield the highest stability in that sense w.r.t.
NLL.
Wasserstein distances
Studying Wasserstein distances, we again observe equally strong results for PU-DE and SML on train and test data (see column ‘mean’ in top panel of
Fig. 9).
PU in contrast possesses a large generalization gap thus yielding weak test set performances.
MC, MC-LL, and DE behave consistently weak on train and test sets with
MC-LL even falling out of plot range.
Under data shift (bottom panel of Fig. 9), the picture remains similar.
PU-DE and SML are in the lead and comparably strong with the exception of PU-DE on label-interpolate and label-extrapolate test data (‘mean’ column).
As for NLL, we ﬁnd these mean values of PU-DE to be signiﬁcantly above the respective median values indicating again weaknesses in the stability of parametric ensembles.
[Table]
[Figure]
[Table]
[Figure]
c
[Figure] [Figure]
Slight overestimation of small uncertainties for SML
The second-moment loss yields weak results on ‘yacht’, ‘energy’ and ‘naval’, the three easiest datasets if measured by test set RMSE, compare Fig. 8.
On these datasets neither aleatoric uncertainty nor modelling residuals play a mayor role.
In such cases, the second-moment loss seems to cause slightly overshooting uncertainty estimates (compare edges of Fig.
1 for a visual clue), likely due to its sub-network ‘repulsion’.
Back-propagating not only through f ˜ θ but also through the full network f θ in L sml might mitigate this eﬀect.
In practice, slight overestimations of small uncertainties might be acceptable.
In contrast, our method performs consistently strong on all more challenging datasets (‘california’, ‘superconduct’, ‘protein’, ‘year’).
A beneﬁcial characteristic for virtually any real-world task.
B.3. Residual-uncertainty scatter plots
Visual inspection of uncertainties can be helpful to understand their qualitative behaviour.
We scatter model residuals µ i − y
i
(respective x-axis in Fig.
11) against model uncertainties σ i (resp.
y-axis in Fig.
For a hypothetical ideal uncertainty mechanism, we expect ( y i − µ i ) ∼ N (0 , σ i ), i.e. model residuals following the predictive uncertainty distribution.
More concretely, 68 .
3% of all ( y i − µ i ) would lie within the respective interval [ − σ i , σ i ] and 99.7% of all ( y i − µ i ) within [ − 3 σ i , 3 σ i ].
Fig.
10 visualizes this hypothetical ideal.
Geometrically, the described Gaussian properties imply that 99 .
7% of all scatter points,
[Figure]
e.g. in Fig.
11 should lie above the blue 3 σ lines and 68 .
3% of them above the yellow 1 σ lines.
For ‘abalone’ test data (third row of Fig. 11), PU and SML qualitatively fulﬁl this requirement while MC and DE tend to underestimate uncertainties.
This ﬁnding is in accordance with our systematic evaluation.
For abalone and superconduct, we qualitatively ﬁnd PU, PU-DE and SML-trained networks to provide more realistic uncertainties compared to MC, MC-LL and DE (see Fig.
The naval dataset poses an exception in this regard as all uncertainty methods lead to comparably convincing uncertainty estimates.
The small test RMSEs of all methods on naval (see Fig. 7) indicate relatively small aleatoric
[Table]
uncertainties and model residuals.
Epistemic uncertainty might thus be a key driving factor and coherently MC, MC-LL and DE perform well.
The hypothetical ideal residual-uncertainty scatter plot we use in Fig.
10 is generated as follows: We draw 3000 standard deviations σ i ∼ U (0 , 2) and sample residuals r i from the respective normal distributions, r i ∼ N (0 , σ i ).
The pairs ( r i , σ i ) are visualized.
By construction, uncertainty estimates now ideally match residuals in a distributional sense.
But even in this perfect case, Pearson correlation between uncertainty estimates and absolute residuals is only approximately 55%.
Appendix C. Stability w.r.t. hyper-parameter β
Here, we analyze the impact of the SML-parameter β on the uncertainty quality of accordingly trained models.
For β = 0 .
9, we observe only relatively small diﬀerences in both ECE (see Fig. 12) and Wasserstein distance (see Fig. 13).
β
= 0 .
5 provides (by a small margin) the best average test set performance in both scores.
However, the best-performing β -value for an individual dataset can vary.
Experiments with β (cid:29) 1 (not shown here) cause non-convergent training in many cases as primarily uncertainty quality is optimized at the expense of task performance.
The opposite extreme case is β = 0, i.e. network optimization without any dropout mechanism.
Applying dropout at inference will therefore cause uncontrolled random ﬂuctuations around the network prediction.
[Figure] [Figure]
a n c
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
n d
[Figure]
Appendix D. In-depth investigation of uncertainty measures D.1. Dependencies between uncertainty measures
All uncertainty-related measures (NLL, ECE, Wasserstein distance) relate predicted uncertainties to actually occurring model residuals.
Each of them putting emphasize on different aspects of the considered samples: NLL is biased towards well-performing models,
ECE measures deviations within quantile ranges, Wasserstein distance resolves distances between normalized residuals.
The empirically observed dependencies between these uncertainty measures are visualized in Fig.
Additionally to Wasserstein distances, we consider
Kolmogorov-Smirnov (KS) distances (Stephens, 1974) on normalized residuals there.
It estimates a distance between the sample of normalized residuals and a standard Gaussian.
Diﬀerent from the Wasserstein distance, the KS-distance is not transport-based but determined by the largest distance between the empirical CDFs of the two samples.
It is therefore bounded to [0 , 1] and unable to resolve diﬀerences between samples that strongly deviate from a standard Gaussian one.
While all these scores are expectably correlated, noteworthy deviations from ideal correlation occur.
Therefore, we advocate for uncertainty evaluations based on various measures to avoid overﬁtting to a speciﬁc formalization of uncertainty.
The data splits in Fig.
14 are color-coded as follows: train is green, test is blue, pcainterpolate is green-yellow, pca-extrapolate is orange-yellow, label-interpolate is red and label-extrapolate is light red.
The mapping between uncertainty methods and plot markers reads: MC is ‘diamond’, MC-LL is ‘thin diamond’, DE is ‘cross’, PU is ‘point’, PU-DE is ‘pentagon’ and second-moment loss is ‘square’.
Some Wasserstein distances lie above the x-axis cut-oﬀ and are thus not visualized.
D.2. Discussion of NLL as a measure of uncertainty
Typically, DNNs using uncertainty are often evaluated in terms of their negative loglikelihood (NLL).
This property is aﬀected not only by the uncertainty, but also by the
DNNs performance.
Additionally, it is diﬃcult to interpret, sometimes leading to contraintuitive results, which we want to elaborate on here.
As a ﬁrst example, take the likelihood of two datasets x 1 = { 0 } and x 2 = { 0 . 5 } , each consisting of a single point, with respect to a normal distribution N (0 , 1).
Naturally, we ﬁnd x 1 to be located at the maximum of the considered normal distribution and deem it the more likely candidate.
But, if we extend these datasets to more than single points, i.e. ˜ x 1 = { 0 , 0 . 1 , 0 , − 0 .
1 , 0 } and ˜ x 2 = { 0 .
5 , − 0 .
4 , 0 , − 1 .
9 , − 0 .
7 } , it becomes obvious that ˜ x 2 is much more likely to follow the intended Gaussian distribution.
Nonetheless, NLL(˜ x 2 ) ≈ 1 . 4 > 0 .
9 ≈ NLL(˜ x 1 ), where
[Equation]
This may be seen as a direct consequence of the point-wise deﬁnition of NLL, which does not consider the distribution of the elements in ˜ x i .
From this observation also follows that a model with high prediction accuracy will have a lower NLL score as a worse performing one if uncertainties are predicted in the same way.
Independent of whether those reﬂected
the
[Figure]
“true” uncertainty in either case.
This issue can be further substantiated on a second example.
Consider two other datasets z 1 , z 2 drawn i.i.d.
from Gaussian distributions N (0 , σ i ) with two diﬀering values σ 1 <σ 2 .
If we determine the NLL of each with respect to its own distribution the oﬀset term in equation (5) leads to NLL( z 2 )
=
NLL( z 1 ) + log ( σ 2 /σ 1 ) with log ( σ 2 /σ 1 ) > 0.
Although both accurately reﬂect their own distributions, or uncertainties so to speak, the narrower z 1 is more “likely”.
This oﬀset makes it diﬃcult to assess reported NLL values for systems with heteroskedastic uncertainty.
While smaller is typically “better”, it is highly data- (and prediction-) dependent which value is good in the sense of a reasonable correlation between performance and uncertainty.