D EEP E VIDENTIAL R EGRESSION
Anonymous authors
Paper under double-blind review
A BSTRACT
Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust and efﬁcient measures of uncertainty are crucial.
While it is possible to train regression networks to output the parameters of a probability distribution by maximizing a Gaussian likelihood function, the resulting model remains oblivious to the underlying conﬁdence of its predictions.
In this paper, we propose a novel method for training deterministic NNs to not only estimate the desired target but also the associated evidence in support of that target.
We accomplish this by placing evidential priors over our original
Gaussian likelihood function and training our NN to infer the hyperparameters of our evidential distribution.
We impose priors during training such that the model is penalized when its predicted evidence is not aligned with the correct output.
Thus the model estimates not only the probabilistic mean and variance of our target but also the underlying uncertainty associated with each of those parameters.
We observe that our evidential regression method learns well-calibrated measures of uncertainty on various benchmarks, scales to complex computer vision tasks, and is robust to adversarial input perturbations.
I NTRODUCTION
[Figure]
Recent advances in deep supervised learning have yielded super human level performance and precision.
While these models empirically generalize well when placed into new test enviornments, they are often easily fooled by adversarial perturbations (Goodfellow et al., 2014), and have difﬁculty understanding when their predictions should not be trusted.
Today, regression based neural networks are being deployed in safety critical domains of computer vision (Godard et al., 2017) as well as in robotics and control (Bojarski et al., 2016) where the ability to infer model uncertainty is crucial for eventual wide-scale adoption.
Furthermore, precise uncertainty estimates are useful both for human interpretation of conﬁdence and anomaly detection, and also for propagating these estimates to other autonomous components of a larger, connected system.
Existing approaches to uncertainty estimation are roughly split into two categories: (1) learning aleatoric uncertainty (uncertainty in the data) and (2) epistemic uncertainty (uncertainty in the prediction).
While representations for aleatoric uncertainty can be learned directly from data, approaches for estimating epistemic uncertainty focus on placing probabilistic priors over the weights and sampling to obtain a measure of variance.
In practice, many challenges arise with this approach, such as the computational expense of sampling during inference, how to pick an appropriate weight prior, or even how to learn such a representation given your prior.
Instead, we formulate learning as an evidence acquisition process, where the model can acquire evidence during training (Sensoy et al., 2018;
Malinin & Gales, 2018).
Every training example adds support to a learned higher-order, evidential distribution.
Sampling from this evidential distribution yields instances of lower-order, likelihood functions from which the data was drawn (cf.
Fig. 1).
We demonstrate that, by placing priors over our likelihood function, we can learn a grounded representation of epistemic and aleatoric uncertainty without sampling during inference.
In summary, this work makes the following contributions:
A novel and scalable method for learning representations of epistemic and aleatoric uncertainty, speciﬁcally on regression problems, by placing evidential priors over the likelihood; 2.
Evaluation of learned epistemic uncertainty on benchmark regression tasks and comparison against other state-of-the-art uncertainty estimation techniques for neural networks; 3.
Robustness evaluation against out of distribution and adversarially perturbed test data.
M ODELLING UNCERTAINTIES FROM DATA 2. P RELIMINARIES
Consider the following supervised optimization problem: given a dataset, D , of N paired training examples, ( x 1 , y 1 ) , . .
, ( x N , y N ) , we aim to learn a function f , parameterized by a set of weights, w , which approximately solves the following optimization problem:
[Equation]
where L i ( · ) describes a loss function.
In this work, we consider deterministic regression problems, which commonly optimize the sum of squared errors, L i ( w ) = 12 (cid:107)
y
i − f ( x i ; w ) (cid:107) 2 .
In doing so, the model is encouraged to learn the average correct answer for a given input, but does not explicitly model any underlying noise or uncertainty in the data when making its estimation.
2. M AXIMUM LIKELIHOOD ESTIMATION
Alternatively, we can approach our optimization problem from a maximum likelihood perspective, where we learn model parameters that maximize the likelihood of observing the particular set of training datapoints.
In the context of deterministic regression, if we assume our targets, y i , were drawn i.i.d.
from a Gaussian distribution with mean and variance parameters θ = ( µ, σ 2 ) , then the likelihood of observing a single target, y i , can be expressed as
[Equation]
In maximum likelihood estimation, we aim to learn a model to infer the ( µ, σ 2 ) that maximize the likelihood of observing our targets, y .
Equivalently, instead of maximizing the likelihood function, in practice it is common to instead minimize the negative log likelihood by setting
[Equation]
In learning the parameters θ , this likelihood function allows us to successfully model the uncertainty of our data, also known as the aleatoric uncertainty.
However, our model remains oblivious to the predictive model uncertainty.
This metric, known as epistemic uncertainty, corresponds to the model’s uncertainty in its own output prediction (Kendall & Gal, 2017).
In this paper, we present a novel approach for estimating the evidence in support of network predictions by directly learning both the inferred aleatoric uncertainty as well as the underlying epistemic uncertainty over its predictions.
We achieve this by placing higher-order prior distributions over the learned parameters governing the distribution from which our observations are drawn.
E VIDENTIAL UNCERTAINTY FOR REGRESSION 3. P ROBLEM SETUP
We consider the problem where our observed targets, y i , are drawn i.i.d.
from a Gaussian distribution with unknown mean and variance ( µ, σ 2 ) , which we seek to probabilistically estimate.
We model this by placing a conjugate prior distribution on ( µ, σ 2 ) .
If we assume our observations are drawn
C
Higher Order Distribution:
Lower Order Distribution:
[Figure]
HighVariance
Low Variance
A iii ii i iii ii i
B from a Gaussian, this leads to placing a Gaussian prior on our unknown mean and an Inverse-Gamma prior on our unknown variance:
[Equation]
where Γ( · ) is the gamma function, and γ ∈ R , λ > 0 , α > 0 , β > 0 .
From a variational Bayesian perspective, our aim is to estimate a posterior distribution q ( µ, σ 2 )
=
p ( µ, σ 2 | y 1 , . . . , y N ) .
To obtain an approximation for the true posterior, we assume that the estimated distribution can be factorized, (Parisi, 1988), such that q ( µ, σ 2 )
=
q ( µ ) q ( σ 2 ) .
In this case, the true distribution takes the form of a Normal Inverse-Gamma (N.I.G.) distribution:
[Equation]
A alternative, popular interpretation of the parameters of the conjugate prior distribution is in terms of “virtual-observations” (Jordan, 2009).
For example, the mean of a N.I.G. distribution can be interpreted as being estimated from λ virtual-observations with sample mean γ while its variance was estimated from 2 α virtual-observations with sample mean γ and sum of squared deviations 2 β .
Following from this interpretation, we deﬁne the total evidence, Φ , of our evidential distributions as the sum of all inferred virtual-observations counts ( Φ = λ + 2 α ).
Thus, we can interpret the estimated posterior q ( θ ) =
q ( µ, σ 2 ) as an evidential , higher-order probability distribution on top of the unknown lower-order likelihood distribution from which observations are drawn.
Drawing a single sample θ j from our evidential posterior yields ( µ j , σ 2 j ) representing a single instance of our likelihood function, namely N ( µ j , σ 2 j ) .
Thus, the parameters of the posterior, speciﬁcally ( γ, λ, α, β ), determine not only the location but also the dispersion concentrations, or uncertainty, associated with our inferred likelihood function.
For example, in Fig.
2A we visualize the posterior of different evidential N.I.G. distributions with varying model parameters.
We illustrate that by increasing the evidential parameters (i.e. λ, α ) of this distribution, the p.d.f.
becomes more tightly concentrated about its inferred likelihood function.
Considering a single parameter realization of this higher-order distribution, cf.
Fig.
2B, we can subsequently sample many lower-order realizations of our likelihood function, as shown in Fig.
2C.
In this work, we use neural networks to learn a higher-order evidential distribution that directly captures prediction uncertainties and evaluate our method on various regression tasks.
This approach presents several distinct advantages.
First, this method enables simultaneous learning of the desired regression task, along with uncertainty estimation, built in, due to our evidential priors.
Second, since the evidential prior is a higher-order N.I.G. distribution, the maximum likelihood Gaussian can be computed analytically from the expected values of the ( µ, σ 2 ) parameters, without the need for sampling.
Third, by explicitly modeling the evidence, we effectively capture the epistemic or model uncertainty associated with the network’s prediction.
This can be done by simply evaluating the variance of our inferred evidential distribution.
3. L EARNING THE EVIDENTIAL DISTRIBUTION
Having formalized the problem of using an evidential distribution to capture model uncertainty, we next describe our approach for actually learning this distribution.
Given a set of observations, variational inference methods aim to approximate a posterior distribution over unobserved variables or parameters by maximizing the evidence lower bound (ELBO) (Kingma & Welling, 2013).
Similarly, here we seek to estimate the posterior distribution q ( θ ) =
q ( µ, σ 2 ) governed by the higher-order distribution parameters m = ( γ, λ, α, β ) to maximize the likelihood of our observations.
Applying the principle of variational inference, we have:
[Equation]
where θ = ( µ, σ 2 ) .
Similar to the principle of variational inference, in the remainder of this section we will discuss how we learn evidential distributions for regression by maximizing the log-likelihood of model evidence and minimizing the distance to an uncertainty prior.
As we will see, maximizing the log-likelihood allows our model to ﬁt the data, while the regularization provides an “uncertainty” penalty so the model can express when it does not know the answer.
The “model evidence”, or marginal likelihood, is deﬁned as the likelihood of an observation, y i , given the evidential distribution parameters m and is computed using Bayesian probability theory by marginalizing over the likelihood parameters θ :
[Equation]
The model evidence is not, in general, straightforward to evaluate since computing it involves integrating out the dependence on model parameters.
However, by placing a N.I.G. prior on our
Gaussian likelihood function an analytical solution does exist.
[Equation]
For computational reasons it is common to instead minimize the negative log-likelihood of the model evidence ( L NLL i ( w ) ).
For a complete derivation please refer to Sec. 7.1.
L NLL i
[Equation]
Instead of modeling the loss using empirical Bayes, where the objective is to maximize model evidence, we can alternatively minimize the sum-of-squared (SOS) errors, between the evidential prior and the data that would be sampled from the associated likelihood.
Thus, we deﬁne L SOS i ( w ) as
[Equation]
SOS i
[Equation]
A step-by-step derivation is given in Sec. 7.1.
In our experiments, using L SOS i ( w ) resulted in greater training stability and increased performance, over the L NLL i ( w ) loss.
Therefore, L SOS i ( w ) is used in all presented results.
3. E XPRESSING L ACK OF E VIDENCE
In the previous subsection, we outlined a loss function for training a NN to output parameters of a
N.I.G. distribution which maximize the log-likelihood of our data.
In this subsection we describe how we regularize training against a prior where the model does not have any evidence (i.e. maximum uncertainty).
In variational inference this is done by minimizing the KL-divergence between the inferred posterior, q ( µ, σ 2 ) =
q ( θ ) , and a prior, p ( θ ) , cf.
Eq.
In the evidential setting, our prior is also a Normal Inverse-Gamma distribution, but with zero evidence (or inﬁnite uncertainty).
Therefore, during training we aim to minimize our evidence (or maximize our uncertainty) everywhere except where we have training data, as enforced by the negative log-likelihood loss term.
Unfortunately, the KL-divergence between an arbitrary N.I.G. distribution and another with inﬁnitely low evidence is not well deﬁned (Soch & Allefeld, 2016).
To address this, we formulate a custom evidence regularizer, L R i , based on the error of the i -th prediction,
[Equation]
where (cid:107) x (cid:107) p represents the L-p norm of x .
The value of p impacts the penalty imposed on the evidence when a wrong prediction is made.
For example, p = 2 , heavily over-penalizes the evidence on larger errors, whereas p = 1 and p = 0 .
5 saturate the evidence penalty for larger errors.
We found that p = 0 .
5 provided the optimal stability during training and use this value in all presented results.
This regularization loss imposes a penalty whenever there is an error in the prediction that scales with the total evidence of our inferred posterior.
Conversely, large amounts of predicted evidence will not be penalized as long as the prediction is close to the target observation.
The combined loss function employed during training consists of the two loss terms for maximizing model evidence and regularizing evidence,
[Equation]
3. E VALUATING ALEATORIC AND EPISTEMIC UNCERTAINTY
The aleatoric uncertainty, also referred to as statistical or data uncertainty, is representative of unknowns that differ each time we run the same experiment.
We evaluate the aleatoric uncertainty from E [ σ 2 ] = βα − 1 .
The epistemic, also known as the model uncertainty, describes the estimated uncertainty in the learned model and is deﬁned as Var[ µ ] =
β ( α − 1) λ .
Note that Var[ µ ] =
E [ σ 2 ] /λ , which is expected as λ is one of our two evidential virtual-observation counts.
E XPERIMENTS 4. P REDICTIVE ACCURACY AND UNCERTAINTY BENCHMARKING
[Figure]
We ﬁrst qualitatively compare the performance of our approach against a set of benchmarks on a one-dimensional toy regression dataset (Fig. 3).
For training and dataset details please refer to Sec.
We compare deterministic regression, as well as techniques using empirical variance of the networks’ predictions such as
MC-dropout, model-ensembles, and Bayes-byBackprop which underestimate the uncertainty outside the training distribution.
In contrast, evidential regression estimates uncertainty appropriately and grows the uncertainty estimate with increasing distance from the training data.
Additionally, we compare our approach to stateof-the-art methods for predictive uncertainty estimation using NNs on common real world datasets used in (Hernández-Lobato & Adams, 2015; Lakshminarayanan et al., 2017; Gal &
Ghahramani, 2016).
We evaluate our proposed evidential regression method against model-ensembles and BBB based on root mean squared error (RMSE), and negative log-likelihood (NLL).
We do not provide results for MC-dropout since it consistently performed inferior to the other baselines.
The
[Table]
results in Table 1 indicate that although the loss function for evidential regression is more complex than competing approaches, it is the top performer in RMSE and NLL in 8 out of 9 datasets.
Furthermore, we demonstrate that, on a synthetic dataset with a priori known noise, evidential models can additionally estimate and recover the underlying aleatoric uncertainty.
For more information please refer to Sec.
7.2.2 for results and experiment details.
4. D EPTH ESTIMATION
After establishing benchmark comparison results, in this subsection we demonstrate the scalability of our evidential learning by extending to the complex, high-dimensional task of depth estimation.
Monocular end-to-end depth estimation is a central problem in computer vision which aims to learn a representation of depth directly from an RGB image of the scene.
This is a challenging learning task since the output target y is very high-dimensional.
For every pixel in the image, we regress over the desired depth and simultaneously estimate the uncertainty associated to that individual pixel.
Our training data consists of over 27k RGB-to-depth pairs of indoor scenes (e.g. kitchen, bedroom, etc.)
from the NYU Depth v2 dataset (Nathan Silberman & Fergus, 2012).
We train a U-Net style
NN (Ronneberger et al., 2015) for inference.
The ﬁnal layer of our model outputs a single H × W activation map in the case of deterministic regression, dropout, ensembling and BBB.
Evidential models four ﬁnal activation maps corresponding to ( γ, λ, α, β ) .
Table 2 summarizes the size and speed of all models.
Evidential models contain signiﬁcantly fewer trainable parameters than ensembles (where the number of parameters scales linearly with the size of the ensemble).
BBB maintains a trainable mean and variance for every weight in the network, so its size is roughly 2 × larger as well.
Since evidential regression models do not require sampling in order to estimate their uncertainty, their forward-pass inference times are also signiﬁcantly more efﬁcient.
Finally, we demonstrate comparable predictive accuracy (through RMSE and NLL) to the other models.
For a more detailed breakdown of how the number of samples effects the baselines please refer to Tab. 3.
Note that the output size of the depth estimation problem presented signiﬁcant learning challenges for the BBB baseline, and it was unable to converge during training.
As a result, for the remainder of this analysis we compare against only spatial dropout and ensembles.
We evaluate these models in terms of their accuracy and their predictive uncertainty on unseen test data.
Fig.
4A-C visualizes the predicted depth, absolute error from ground truth, and predictive uncertainty across three randomly picked test images.
Ideally, a strong predictive uncertainty would capture any errors in the prediction (i.e., roughly correspond to where the model is making errors).
[Table]
[Figure]
PredictiveUncertainty
[Figure]
PredictedDepth AbsoluteError PredictiveUncertainty
Spatial Dropout EnsembleEvidential 0.
Specificity (FPR)
[Figure]
s i t i v i t y
[Figure]
T P
[Figure] [Figure]
Compared to dropout and ensembling, evidential uncertainty modeling captures the depth errors while providing clear and localized predictions of conﬁdence.
In general, dropout drastically underestimates the amount of uncertainty present, while ensembling occasionally overestimates the uncertainty
To evaluate predictive uncertainty calibration to the groundtruth errors, we ﬁt receiver operating characteristic (ROC) curves to normalized estimates of error and uncertainty.
Thus, we test the network’s ability to detect how likely it is to make an error at a given pixel using its predictive uncertainty.
ROC curves take into account sensitivity and speciﬁcity of the uncertainties towards error predictions and are stronger if they contain greater area under their curve (AUC).
Fig.
4D demonstrates that our evidential model provides uncertainty estimates concentrate to where the model is making the errors.
In addition to epistemic uncertainty, we also evaluate the aleatoric uncertainty estimates that are learned from our evidential models as well.
Fig.
5 compares the evidential aleatoric uncertainty to those obtained by Gaussian likelihood optimization in several domains with high data uncertainty (mirror reﬂections and poor illumination).
The results between both methods are in strong agreement, identifying mirror reﬂections and dark regions without visible geometry as sources of high uncertainty.
4. O UT - OF DISTRIBUTION TESTING
A key use of uncertainty estimation is to understand when a model is faced with test samples that fall out-of-distribution (OOD) or when the model’s output cannot be trusted.
In the previous subsection, we showed that our evidential uncertainties were well calibrated with the model’s errors.
In this subsection, we investigate the performance on out-of-distribution samples.
Fig.
6 illustrates predicted depth on various test input images (left) and outside (right) of the original distribution.
All images
[Figure] [Figure]
t
[Figure]
have not been seen by the model during training.
We qualitatively and quantitatively demonstrate that the epistemic uncertainty predicted by our evidential model consistently increases on the OOD samples.
4.3. R OBUSTNESS TO ADVERSARIAL SAMPLES
Next, we consider the extreme case of OOD detection where the inputs are adversarially perturbed to inﬂict maximum error on the model.
We compute adversarial perturbations to our test set using the fast gradient sign method (Goodfellow et al., 2014), with increasing scales, (cid:15) , of noise.
Fig.
7A conﬁrms that the absolute error of all methods increasing as adversarial noise is added.
We also observe a positive effect noise on our predictive uncertainty estimates in Fig.
7B.
An additional desirable property of evidential uncertainty modeling is that it presents a higher overall uncertainty when presented with adversarial inputs compared to dropout and ensembling methods.
Furthermore, we observe this strong overall uncertainty estimation despite the model losing calibration accuracy from the adversarial examples (Fig. 7C).
The robustness of evidential uncertainty against adversarial perturbations is visualized in greater detail in Fig.
7D, which illustrates the predicted depth, error, and estimated pixel-wise uncertainty as we perturb the input image with greater amounts of noise (left-to-right).
Note that the predictive uncertainty not only steadily increases as we increase the noise, but the spatial concentrations of uncertainty throughout the image maintain tight correspondence with the error.
D ISCUSSION AND R ELATED WORK
Uncertainty estimation has a long history in neural networks, from modeling probability distribution parameters over outputs (Bishop, 1994) to Bayesian deep learning (Kendall & Gal, 2017).
Our work builds on this foundation and presents a scalable representation for inferring the parameters of an evidential uncertainty distribution while simultaneously learning regression tasks via MLE.
In Bayesian deep learning, priors are placed over network weights and estimated using variational inference (Kingma et al., 2015).
Dropout (Gal & Ghahramani, 2016; Molchanov et al., 2017) and
BBB (Blundell et al., 2015) rely on multiple samples to estimate predictive variance.
Ensembles (Lakshminarayanan et al., 2017) provide a tangential approach where sampling occurs over multiple trained instances.
In contrast, we place uncertainty priors over the likelihood function and thus only need a single forward pass to evaluate both prediction and uncertainty.
Additionally, our approach of uncertainty estimation proved to be better calibrated and capable of predicting where the model fails.
A large topic of research in Bayesian inference focuses on placing prior distributions over hierarchical models to estimate uncertainty (Gelman et al., 2006; 2008).
Our methodology falls under the class of evidential deep learning which leverages the Theory of Evidence to model prior distributions over neural network predictions and interpret uncertainty.
Prior works in this ﬁeld (Sensoy et al., 2018;
Malinin & Gales, 2018) have focused exclusively on modeling uncertainty in the classiﬁcation domain with Dirichlet prior distributions.
Our work extends this ﬁeld into the broad range of regression learning tasks (e.g. depth estimation, forecasting, robotic control learning, etc) and demonstrates generalizability to out-of-distribution test samples.
C ONCLUSION
In this paper, we develop a novel method for training deterministic NNs that both estimates a desired target and evaluates the evidence in support of the target to generate robust metrics of model uncertainty.
We formalize this in terms of learning evidential distributions, and achieve stable training by penalizing our model for prediction errors that scale with the available evidence.
Our approach for evidential regression is validated on a benchmark regression task.
We further demonstrate that this method robustly scales to a key task in computer vision, depth estimation, and that the predictive uncertainty increases with increasing out-of-distribution adversarial perturbation.
This framework for evidential representation learning provides a means to achieve the precise uncertainty metrics required for robust neural network deployment in safety-critical domains.
A PPENDIX 7. M ODEL EVIDENCE DERIVATIONS
For convenience, deﬁne τ = 1 /σ 2 be the precision of a Gaussian distribution.
The change of variables transforms the Normal Inverse-Gamma distribution p ( µ, σ 2 | γ, λ, α, β ) to the equivalent
Normal Gamma distribution p ( µ, τ | γ, λ, α, β ) , parameterized by precision τ ∈ (0 , ∞ ) instead of variance σ 2 ,
[Equation]
7.1. T YPE II M AXIMUM L IKELIHOOD L OSS
Marginalizing out µ and τ gives the result of equation 7,
[Equation]
For computational reasons it is common to instead minimize the negative logarithm of the model evidence.
[Equation] i
[Equation]
7.1. S UM OF S QUARES L OSS
Similarly
[Equation]
we can marignalize out µ and τ to receive the result of equation 9,
L SOS
i
[Equation]
[Equation]
7. B ENCHMARK REGRESSION TASK EVALUATIONS 7.2. E PISTEMIC UNCERTAINTY ESTIMATION
The training set consists of training examples drawn from y = sin(
x ) / (3 x ) + (cid:15) , where (cid:15) ∼
N (0 , 0 . 02) in the region − 3 ≤ x ≤ 3 , whereas the test data is unbounded.
All models consisted of 100 neurons with 3 hidden layers and were trained to convergence.
The data presented in
Fig.
3 illustrates the estimated epistemic uncertainty and predicted mean accross the entire test set, − 3 ≤ x ≤ 3 .
7.2. A LEATORIC UNCERTAINTY ESTIMATION
The training set consists of training examples drawn from y = sin(
x ) / (3 x ) + (cid:15) ( x ) , where (cid:15) ( x ) ∼
N (0 , s ( x )) , and s ( x )
= 120 cos(3 .
3 x ) + 0 .
We evaluate against (Kendall & Gal, 2017) which presents an algorithm for heteroscedastic aleatoric uncertainty estimation by inferring the mean and variance of a Gaussian likelihood function.
As presented in the paper, training is done by minimizing the negative log-likelihood of the data given the inferred likelihood parameters.
Both our network and the baseline Gaussian NLL network consisted of 100 neurons with 3 hidden layers and were trained to convergence.
[Figure]
7. A DDITIONAL D EPTH E STIMATION P ERFORMANCE R ESULTS
[Table]