R EGRESSION P RIOR N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Prior Networks are a class of models which yield interpretable measures of uncertainty and have been shown to outperform state-of-the-art ensemble approaches on a range of tasks.
They can also be used to distill an ensemble of models via
Ensemble Distribution Distillation (EnD 2 ), such that its accuracy, calibration and uncertainty estimates are retained within a single model.
However, Prior Networks have so far been developed only for classiﬁcation tasks.
This work extends Prior
Networks and EnD 2 to regression tasks by considering the Normal-Wishart distribution.
The properties of Regression Prior Networks are demonstrated on synthetic data, selected UCI datasets and two monocular depth estimation tasks.
They yield performance competitive with ensemble approaches.
I NTRODUCTION
Neural Networks have become the standard approach to addressing a wide range of machine learning tasks (Girshick, 2015; Simonyan & Zisserman, 2015; Villegas et al., 2017; Mikolov et al., 2013b;a; 2010; Hinton et al., 2012; Hannun et al., 2014; Caruana et al., 2015; Alipanahi et al., 2015).
However, in order to improve the safety of AI systems (Amodei et al., 2016) and avoid costly mistakes in high-risk applications, such as self-driving cars, it is desirable for models to yield estimates of uncertainty in their predictions.
Ensemble methods are known to yield both improved predictive performance and robust uncertainty estimates (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017; Maddox et al., 2019).
Importantly, ensemble approaches allow interpretable measures of uncertainty to be derived via a mathematically consistent probabilistic framework.
Speciﬁcally, the overall total uncertainty can be decomposed into data uncertainty , or uncertainty due to inherent noise in the data, and knowledge uncertainty , which is due to the model having limited uncertainty of the test data (Malinin, 2019).
Uncertainty estimates derived from ensembles have been applied to the detection of misclassiﬁcations, out-of-domain inputs and adversarial attack detection (Carlini &
Wagner, 2017; Smith & Gal, 2018), and active learning (Kirsch et al., 2019).
Unfortunately, ensemble methods may be computationally expensive to train and are always expensive during inference.
A class of models called Prior Networks (Malinin & Gales, 2018; 2019; Malinin, 2019) was proposed as an approach to modelling uncertainty in classiﬁcation tasks by emulating an ensemble using a single model.
Prior Networks parameterize a higher order conditional distribution over output distributions, such as the Dirichlet distribution.
This enables Prior Networks to efﬁciently yield the same interpretable measures of total , data and knowledge uncertainty as an ensemble.
Unlike ensembles, the behaviour of Prior Networks’ higher-order distribution is speciﬁed via a loss function, such as reverse KL-divergence (Malinin & Gales, 2019), and training data.
However, such Prior
Networks yield predictive performance consistent with that of a single model trained via Maximum
Likelihood, which is typically worse than that of an ensemble.
This can be overcome via Ensemble
Distribution Distillation (EnD 2 ) (Malinin et al., 2020), which is an approach that allows distilling ensembles into Prior Networks such that measures of ensemble diversity are preserved.
This enables
Prior Networks to retain both the predictive performance and uncertainty estimates of an ensemble at low computational and memory cost.
While Prior Networks have many attractive properties, they have only been applied to classiﬁcation tasks.
In this work we develop Prior Networks for regression tasks by considering the Normal-Wishart distribution - a higher-order distribution over the parameters of multivariate normal distributions.
We derive all measures of uncertainty, the reverse KL-divergence training objective, and the Ensemble
Distribution Distillation objective in closed form.
Regression Prior Networks are then evaluated on synthetic data, selected UCI datasets and the NYUv2 and KITTI monocular depth estimation tasks, where they are shown to yield comparable or better performance to state-of-the-art ensemble approaches.
Crucially, they enable, via EnD 2 , to retain the predictive performance and uncertainty estimates of an ensemble within a single model .
R EGRESSION P RIOR N ETWORKS
In this section we develop Prior Network models for regression tasks.
While typical regression models yield point-estimate predictions, we consider probabilistic regression models which parameterizes a distribution p ( y | x , θ ) over the target y ∈ R K .
Typically, this is a normal distribution:
[Equation]
where µ is the mean, and Λ the precision matrix, a positive-deﬁnite symmetric matrix.
While normal distributions are usually deﬁned in terms of the covariance matrix Σ = Λ − 1 , parameterization using the precision tends to be more numerically stable during optimization (Bishop, 2006; Goodfellow et al., 2016).
While a range of distributions over continuous random variables can be considered, we will the normal as it makes the least assumptions about the nature of y and is mathematically simple.
As in the case for classiﬁcation, we can consider an ensemble of networks which parameterize multivariate normal distributions { p ( y | x , θ ( m ) )
} Mm =1 .
This ensemble can be interpreted as a set of draws from a higher-order implicit distribution over normal distributions.
A Prior Network for regression would, therefore, emulate this ensemble by explicitly parameterizing a higher-order distribution over the parameters µ and Λ of a normal distribution.
One sensible choice is the formidable Normal-Wishart distribution (Murphy, 2012; Bishop, 2006), which is a conjugate prior to the multivariate normal distribution.
This parallels how the Dirichlet distribution, the conjugate prior to the categorical, was used in classiﬁcation Prior Networks.
The Normal-Wishart distribution is deﬁned as follows:
[Equation]
where m and L are the prior mean and inverse of the positive-deﬁnite prior scatter matrix , while κ and ν are the strengths of belief in each prior, respectively.
The parameters κ and ν are conceptually similar to precision of the Dirichlet distribution
α 0 .
The Normal-Wishart is a compound distribution which decomposes into a product of a conditional normal distribution over the mean and an Wishart distribution over the precision.
Thus, a Regression Prior Network parameterizes the Normal-Wishart distribution over the mean and precision of normal output distributions as follows:
[Equation]
where Ω = { m , L , κ, ν } is the set of parameters of the Normal-Wishart predicted by neural network.
The posterior predictive of this model is the multivariate Student’s T distribution (Murphy, 2012), which is the heavy-tailed generalization of the multivariate normal distribution:
[Equation]
In the limit, as ν → ∞ , the T distribution converges to a normal distribution.
The predictive posterior of the Prior Network given in equation 4 only has a deﬁned mean and variance when ν > K + 1 .
Figure 1 depicts the desired behaviour of an ensemble of normal distributions sampled from a
Normal-Wishart distribution.
Speciﬁcally the ensemble should be consistent for in-domain inputs in regions of low/high data uncertainty , as in ﬁgures 1a-b, and highly diverse both in the location of the mean and in the structure of the covariance for out-of-distribution inputs, as in ﬁgure 1c.
Samples of continuous output distributions from a regression Prior Network should yield the same behaviour.
2. M EASURES OF U NCERTAINTY
Given a Normal-Wishart Prior Network which displays the behaviours above, it is possible to compute closed-form expression for all uncertainty measures previously discussed for ensembles and Dirichlet
Prior Networks (Malinin, 2019).
By considering the mutual information between the target y and the
[Figure]
parameters of the output distribution { µ , Λ } , we can obtain measures of knowledge , total and data uncertainty :
[Equation]
[Figure]
[Equation] [Equation]
This expression consists of the difference between the differential entropy of the posterior predictive and the expected differential entropy of draws from the Normal-Wishart prior.
We can also consider the expected pairwise KL-divergence (EPKL) between draws from the Normal-Wishart prior:
[Equation]
This is an upper bound on mutual information (Malinin, 2019).
Notably, estimates of data uncertainty are unchanged.
One practical use of EPKL is comparison with ensembles, as it is not possible to obtain a tractable expression for the mutual information of a regression ensemble (Malinin, 2019).
Alternatively, we can consider measures of uncertainty derived via the law of total variance:
[Equation]
This yields a similar decomposition to mutual information, but only ﬁrst and second moments are considered.
We provide closed-form expressions of (5)-(7) in appendix A. Note, however, that these variance-based measures are not scale-invariant, and are therefore sensitive to the scale of predictions which the model makes.
We omit the closed-form expressions for all terms here and instead provide them in appendix A.
2. T RAINING OBJECTIVES
Having discussed how to construct Prior Networks for regression, we now discuss how they can be trained.
Prior Networks are trained using a multi-task loss, where an in-domain loss L in and an out-of-distribution (OOD) loss L out are jointly minimized:
[Equation]
The OOD loss is necessary to teach the model the limit of its knowledge (Malinin, 2019).
Normal distributions sampled from a Prior Network should be consistent and reﬂect the correct level of data uncertainty in-domain, and diverse in both mean and precision out-of-domain.
Achieving the former is challenging, as the training data only consists of samples of inputs x and targets y , there is no access to the underlying distribution, and associated data uncertainty, represented by the precision
Λ .
Effectively, we are attempting to train a Normal-Wishart distribution from targets sampled from normal distribution that are sampled from the Normal-Wishart, rather than on the normal distribution samples themselves, which is challenging.
However, it was shown that for Dirichlet
Prior Networks minimizing the reverse KL-divergence between the model and an appropriate target
Dirichlet induces in expectation the correct estimate of data uncertainty (Malinin & Gales, 2019).
As the Normal-Wishart, like the Dirichlet, is a conjugate prior and exponential family member, the precision can be induced in expectation by considering the reverse KL-divergence between the model p ( µ , Λ | x , θ ) and a target Normal-Wishart p ( µ , Λ | ˆΩ ( i ) ) corresponding to each x ( i ) .
The appropriate
Normal-Wishart is speciﬁed via Bayes’s rule:
[Equation]
where Ω 0 = { m 0 , L 0 , κ 0 , ν 0 } are the parameters of the prior deﬁned as follows:
[Equation]
In other words, we consider a semi-informative prior which corresponds to the mean and scatter matrix of marginal distribution p ( y ) , and we see each sample of the training data ˆ β times.
The hyper-parameter ˆ β allows us to weigh the effect of the prior and the data.
(cid:15) is a small value, like 10 − 2 , so that κ 0 and ν 0 yield a maximally un-informative, but proper predictive posterior.
The reason to use a semi-informative prior is because in regression tasks, unlike classiﬁcation tasks, uninformative priors are improper and lead to inﬁnite differential entropy.
Furthermore, we do know something about the data purely based on the marginal distribution, and it is sensible to use that as the prior.
The reverse KL-divergence loss can then be expressed as:
[Equation]
where Z is a normalization constant.
For in-domain data ˆ β can be set to a large value, and for out-of-domain training data ˆ β = 0 , so that the model regresses to the prior.
In-domain the prior will add a degree of smoothing which may prevent over-ﬁtting and improve performance on small datasets.
The derivation and closed form expression for this loss is provided in appendix A.
2. E NSEMBLE D ISTRIBUTION D ISTILLATION
An interesting task which Prior Networks can solve is Ensemble Distribution Distillation (EnD 2 ) (Malinin et al., 2020), where the distribution of an ensemble’s predictions is distilled into a single model.
EnD 2 enables retaining an ensemble’s improved predictive performance and uncertainty estimates within a single model at low cost.
In contrast, standard Ensemble Distillation (EnD), which minimizes the KL-divergence between a model and the ensemble, loses information about ensemble diversity.
Consider an ensemble { p ( y | x , θ ( m ) )
} Mm =1 , where each model yields the mean and precision of a normal distribution.
We can deﬁne an empirical distribution over the mean and precision as follows:
[Equation]
EnD 2 can then be accomplished by minimizing the negative log-likelihood of the ensemble’s mean and precision under the Normal-Wishart prior:
[Equation]
This is equivalent to minimizing the KL-divergence between the model and the empirical distribution of the ensemble.
Note that here, unlike in the previous section, the parameters of a normal distribution are available for every input x , making forward KL-divergence the appropriate loss function.
However, while this is a theoretically sound approach, the optimization might be numerically challenging.
Similarly to (Malinin et al., 2020) we propose a temperature-annealing trick to make the optimization process easier.
First, the ensemble is reduced to it’s mean:
[Equation]
In the expression above we use inverses of the precision matrix Λ because we are actually interpolating the covariance matrices Σ .
Secondly, the predicted ˜ κ = Tκ and ˜ ν
= Tν are multiplied by T in order to make the Normal-Wishart also sharp around the mean.
The loss is divided by T in order to avoid scaling the gradients by T, yielding:
[Equation]
This splits learning into two phases.
First, when temperature is high, the model learns to match the ensemble’s mean (ﬁrst moment).
Second, as the temperature is annealed down to 1, the model will gradually focus on learning higher moments of the ensemble’s distribution.
This trick may be necessary, as the ensemble may have a highly non-Normal-Wishart distribution, which may be challenging to learn.
Note, that for EnD 2 it may be better to parameterize the Normal-inverseWishart distribution over the mean and covariance, due to numerical stability concerns.
However, for consistency, we describe EnD 2 in terms of the Normal-Wishart.
Finally, we emphasize that EnD ˆ does not require OOD training data, unlike the RKL objective above.
This eliminates the non-trivial challenge of ﬁnding appropriate OOD data.
E XPERIMENTS ON SYNTHETIC DATA
[Figure]
[Figure]
We ﬁrst examine Regression Prior Networks on a synthetic one-dimensional dataset with additive heteroscedastic noise.
We compare Regression Prior Network trained via RKL (NWPN) and distribution-distillation of an ensemble (EnD 2 ) to Deep Ensembles.
The ensemble consists ofmodels which yield a Normal output distribution and are trained via maximum likelihood.
All models use the same 2-layer architecture with 30 ReLU units.
Details of the setup are available in appendix B.
The NWPN was trained via reverse KL-divergence (11) and Ensemble Distribution Distillation done as described in section 2.3.
In-domain training data for all models is sampled between [-10,10] and
OOD training data for the NWPN model is sampled from [ − 25 , 20] ∪ [20 , 25] .
Measures of total , data and knowledge uncertainty are obtained via the law of total variance (7).
The results presented in Figure 2 show several trends.
Firstly, the total uncertainty of all models is high in the region of high heteroscedastic noise as well as out-of-domain.
Secondly, total uncertainty decomposes into data uncertainty and knowledge uncertainty .
The former is high in the region of high heteroscedastic noise and has undeﬁned behaviour out-of-domain, while the latter is low in-domain and large out-of-distribution.
Third, EnD 2 successfully replicates ensemble’s estimates of uncertainty, though they are consistently larger, especially estimates of data uncertainty out-of-domain.
This is a consequence of the ensemble being non-Normal-Wishart distributed when it is diverse, leading the
EnD 2 Prior Network to over-estimate support.
Thus, these results validate the principle claims that
Regression Prior Networks can emulate the behaviour of an ensemble via multi-task training using the RKL objective or via EnD 2 , and that they yield interpretable measures of uncertainty.
E XPERIMENTS ON UCI DATA
In this section, we evaluate Normal-Wishart Prior Networks trained via reverse-KL divergence (11) (NWPN) and Ensemble Distribution Distillation (EnD 2 ) relative to a Deep-Ensemble baseline on selected popular UCI datasets.
Other ensemble-methods are not considered, as Deep Ensembles have been shown to consistently outperform them using fewer ensemble members (Ashukha et al., 2020; Ovadia et al., 2019; Fort et al., 2019).
We follow the experimental setup of (Lakshminarayanan et al., 2017) with several changes, detailed in appendix C. Out-of-distribution training data for NWPN is generated using a factor analysis model. This model is a linear generative model that learns to approximate in-domain data with ˆ x ∼ N ( µ, WW T + Ψ) , where [ W, µ, Ψ] are model parameters.
The out-of-domain training examples are then sampled from N ( µ, 3 WW T + 3Ψ) , such that they are further from the in-domain region.
Table 1 shows a comparison of all models in terms of NLL and RMSE.
Single is a gaussian model, ENSM is an ensemble of gaussian models.
Unsurprisingly, ensembles yield the best RMSE, though both NWPN and EnD 2 give generally comparable NLL scores.
Furthermore, EnD 2 comes close to or matches the performance of the ensemble and outperforms NWPN.
Results on are available in appendix C
[Figure]
[Table]
In Table 2 we compare uncertainty measures derived from all models on the tasks of error detection and OOD detection. To evaluate error detection a Prediction Rejection Ratio (PRR) is used.
It shows what part of the best possible error-detection performance our algorithm covers and is deﬁned in C.
For the evaluation of OOD-detection performance, we took parts of other UCI datasets as OOD data. We made sure that the OOD-data comes from different domains and feature distributions are different.
Columns of each OOD dataset are normalized using statistics derived from the in-domain training dataset.
Details are available in appendix C.
The results show that all models achieve comparable error-detection using measures of total uncertainty .
In terms of OOD detection, EnD 2 generally reproduces the ensemble’s behavior, while NWPN usually performs worse.
However, on the MSD dataset NWPN yields the best performance.
This may be due to the nature of the OOD training datait may simply be better suited to MSD OOD detection.
Furthermore, the UCI datasets are generally small and have low input dimensionality - MSD, the largest, has 95 features.
It is therefore difﬁcult to assess the superiority of any particular model on these simple datasets - all we can say that they generally perform comparably.
In the next section, we validate Regression Prior Networks on a more complex, larger-scale task
[Table]
[Table]
M ONOCULAR D EPTH E STIMATION E XPERIMENTS
Having established that the proposed methods work on par with or better than ensemble methods on the UCI datasets, we now examine them on the large-scale NYU Depth v2 (Nathan Silberman & Fergus, 2012) and KITTI (Menze & Geiger, 2015) depth-estimation tasks.
We evaluate RPNs trained both via RKL and EnD 2 .
In this work the base model is DenseDepth (DD) which deﬁnes a
U-Net like architecture on top of DenseNet-169 features (Alhashim & Wonka, 2018).
The original approach trains it on inverted targets using a combination of L1, SSIM and Image-Gradient losses.
We replace this with NLL training using a model which yields mean and precision for each pixel (Single).
The data pre-processing, augmentation, optimization and evaluation protocol are kept unchanged.
On the challenging KITTI benchmark (Geiger et al., 2013) all models are evaluated on the split proposed by (Eigen et al., 2014a).
The baselines are a single Gaussian model and a
Deep-Ensemble of 5 Gaussian models (ENSM).
Other ensemble-methods are not considered, as
Deep Ensembles consistently outperform them (Ashukha et al., 2020; Ovadia et al., 2019).
We then distribution-distilled the ensemble into a Regression Prior Network (EnD 2 ) with a per-pixel NormalWishart distribution.
Additionally, we compare training RPN with the RKL loss function (NWPN).
On the NYU dataset we take KITTI as OOD traning data.
On KITTI, we use NYU as OOD training data.
Finally, we also consider standard ensemble distillation (EnD).
All experiments are repeatedtimes and we report mean.
Distribution-distillation is done with temperature annealing ( T = 1 . 0 vs. T = 10 . 0 ).
For T = 10 .
we train with initial temperature for 20% of epochs, linearly decay it to 1 .
during 60% of epochs, ﬁne-tune with T = 1 . 0 for the last 20% of epochs.
While not improving performance, we found it to greatly stabilizes training.
We report usual metrics for monocular depth estimation (Eigen et al., 2014b), detailed description of them can be found in appendix D. Results in table 3 show several trends.
First, probabilistic models outperform the original DenseDepth model in this setup.
Second, EnD 2 achieve predictive performance closest to the ensemble, both in terms of standard Depth-Estimation metrics, and in terms of test-set negative-log-likelihood (a metric of calibration).
Surprisingly, EnD 2 yields predictions which are better calibrated than ensembles.
This may be due to the ensemble being poorly modelled by a Normal-Wishart distribution, leading the Prior Network to over-estimate the support of the distribution.
As a consequence, the EnD 2 is less conﬁdent, and therefore better calibrated.
Finally, standard ensemble distillation (EnD) achieves very poor predictive and calibration performance.
This may be an artifact of trying to model a mixture of
Gaussians with a single Gaussian.
[Table]
2.98 NWPN 0.
1.52 EnD 0.
3.27 EnD0.
1.42 DER(Arxiv19) 0.
- - - - - - - DER(NeurIPS20) 0.
1.709 MDD 0.
In table 4 we asses all models on the task of out-of-domain input detection.
Two OOD test-datasets are considered: LSUN-church (LSN-C) and LSUN-bed (LSN-B) (Yu et al., 2015), which consist of images of churches and bedrooms.
The latter is most similar to NYU Depth-V2 and more challenging to detect.
Both datasets are very different from KITTI, which consists of outdoor images of roads.
OOD images are center-cropped and re-scaled to the in-domain data.
Results show several trends.
First, EnD 2 consistently outperforms the original ensemble using measures of knowledge uncertainty ( I , K and V [ µ ] ).
However, when considering measures of total uncertainty ( H [ E ] , V [ y ] ), the ensemble tends to yield superior performance on NYU.
This is likely due to over-estimation of the support of the ensemble’s distribution by EnD 2 .
Second, OOD detection is easier on moremismatched datasets.
Third, standard ensemble distillation (EnD) does much worse than even a single model, which was an effect that was also observed for classiﬁcation models in (Malinin et al., 2020).
Finally, variance-based measures outperform information-theoretic measures of total uncertainty , and sometimes knowledge uncertainty on NYU, but do exceptionally poorly on KITTI.
This is a result of their sensitivity to scale.
The models predict very low depth values for OOD data, which means that they have lower entropy and variance.
In contrast, MI and EPKL, which are scale invariant, are not affected.
[Figure]
Single 0.
- - - NWPN 0.
0.998 EnD
LSN-C 0.
- - - EnD0.
[Table]
DER(Arxiv19)
[Table]
[Table]
- - - - - DER(
[Table]
0.040 MDD0. 0.422 ENSM0. Training RPNs with RKL is challenging in this setting, as it is non-trivial to deﬁne what OOD is, especially for depth estimation.
In this paper we consider a different dataset to be OOD, however the distinction is too high-level.
An ablation study with varying OOD weight (appendix D), showed that there is a trade-off between predictive quality and OOD detection quality.
Results in the table above show that when trained appropriately, this yields the best OOD detection performance (KITTI), and if the balance is incorrect, the performance is poor (NYU).
Predictive performance also suffers.
This highlights the value of EnD 2 , which does not require OOD training data, and yields both good predictive and OOD detection performance.
Last, ﬁgure 3 shows the error and estimates of total and knowledge uncertainty yielded by an ensemble and an EnD 2 model for the same input image.
Both the ensemble and our model effectively decompose uncertainty.
Total uncertainty is large at object boundaries and distant points while knowledge uncertainty concentrates on the interior of unusual objects.
EnD 2 yields both error and measures of uncertainty which are very similar to that of the original ensemble.
This demonstrates that
EnD 2 is able to emulate not only the predictive performance of the ensemble, but also the behaviour of the ensemble’s measures of uncertainty.
Additional comparisons are provided in appendix D.
C ONCLUSION
This work extends Prior Networks to regression tasks, yielding a set of general, efﬁcient and interpretable uncertainty estimation approaches for regression.
A regression Prior Network (RPN) predicts the parameters of a Normal-Wishart distribution, which enables it to efﬁciently represent ensembles of regression models, allowing interpretable measures of uncertainty to be obtained at low computational cost.
In this work closed-form measures of total , data and knowledge uncertainty are obtained for Normal-Wishart RPNs.
Two RPN training approaches are proposed.
First, the reverse-KL divergence between the model and a target Normal-Wishart distribution, allowing the behaviour of an RPN to be explicitly controlled but requiring an OOD training dataset, is described.
Second, Ensemble Distribution Distillation (EnD 2 ) is used, where an ensemble of regression models is distilled into an RPN such that it retains the improved predictive performance and uncertainty estimates of the original ensemble.
This approach is particularly useful when it is challenging to deﬁne an appropriate out-of-domain training dataset, such as in depth-estimation.
The properties of
RPNs are evaluated on both a synthetic dataset and selected UCI datasets.
The usefulness of Ensemble
Distribution Distillation is also demonstrated on two large-scale monocular depth-estimation task, where an ensemble of probabilistic depth-estimation models is distribution-distilled into an RPN.
A D ERIVATIONS FOR N ORMAL -W ISHART P RIOR N ETWORKS
The current appendix provides mathematical details of the Normal-Wishart distribution and derivations of the reverse-KL divergence loss, ensemble distribution distillation and all uncertainty measures.
A. N ORMAL -W ISHART DISTRIBUTION
The Normal-Wishart distribution is a conjugate prior over the mean µ and precision Λ of a normal distribution, deﬁned as follows
[Equation]
where Ω = { m , L , κ, ν } are the parameters predicted by neural network, N is the density of the
Normal and W is the density of the Wishart distribution.
Here, m and L are the prior mean and inverse of the positive-deﬁnite prior scatter matrix , while κ and ν are the strengths of belief in each prior, respectively.
The parameters κ and ν are conceptually similar to precision of the Dirichlet distribution
α 0 .
The Normal-Wishart is a compound distribution which decomposes into a product of a conditional normal distribution over the mean and an Wishart distribution over the precision:
[Equation]
) where Γ K ( · ) is the multivariate gamma function and K is the dimensionality of y .
From (Murphy, 2012) the posterior predictive of this model is the multivariate T-distribution:
[Equation]
The T distribution is heavy-tailed generalization of the multivariate normal distribution deﬁned as:
[Equation]
where ν is the number of degrees of freedom.
However, the mean is only deﬁned when ν >
and the variance is deﬁned only when ν > 2 .
A. R EVERSE KL- DIVERGENCE T RAINING O BJECTIVE
Now let us consider in greater detail the reverse KL-divergence training objective (11):
[Equation]
) where Ω 0 =
[ m 0 , L 0 , κ 0 , ν 0 ] are prior parameters that we set manually as discussed in section 2.2.
It is necessary to show why the reverse KL-divergence objective will yield the correct level of data uncertainty.
Lets consider taking the expectation of the ﬁrst term in (11) with respect to the true distribution of targets p tr ( y | x ) .
Trivially, we can show that by exchanging the order of expectation, that we are optimizing the expected cross-entropy between samples from the Normal-Wishart and the true distribution:
[Equation]
This will yield an upper bound on the cross entropy between the predictive posterior and the true distribution.
However, if we were to consider the forward KL-divergence between Normal-Wishart distributions, we would not obtain such an expression and not correctly estimate data uncertainty.
Interestingly, the reverse KL-divergence training objective has the same form as an ELBO - the predictive term and a reverse KL-divergence to the prior.
Having established an important property of the RKL objective, we not derive it’s closed form expression.
Note, that in these derivations, we make extended use of properties for taking expectations of traces and log-determinants matrices with respect to the Wishart distribution detailed in (Gupta &
Srivastava, 2010).For the ﬁrst term in 20, we use the following property of the multivariate normal:
[Equation]
which allows us to get:
[Equation]
The second term in (20) may expressed as follows via the chain-rule of relative entropy(Cover &
Thomas, 2006):
[Equation]
The ﬁrst term in (24) can be computed as:
[Equation]
while the second term is:
[Equation]
A. U NCERTAINTY M EASURES
Given a Normal-Wishart Prior Network which displays the desired set of behaviours detailed in 2, it is possible to compute closed-form expression for all measures of uncertainty previously discussed for
Dirichlet Prior Networks (Malinin, 2019).
The current section details the derivations of uncertainty measures introduced in section 2.1 for the Normal-Wishart distribution.
We make extensive use of (Gupta & Srivastava, 2010) for taking expectation of log-determinants and traces of matrices.
A.3. D IFFERENTIAL E NTROPY OF P REDICTIVE P OSTERIOR
As discussed in section 2, the predictive posterior of a Prior Network which parameterizes a NormalWishart distribution is a multivariate T distribution:
[Equation]
The differential entropy of the predictive posterior will be a measure of total uncertainty .
The differential entropy of a standard multivariate student’s T distribution with an identity scatter matrix Σ
= I is given by:
[Equation] [Equation]
which is a result obtained from (ARELLANO-VALLE et al., 2013)
Using the property of differential entropy (Cover & Thomas, 2006), that if x ∼ p ( x ) and y = µ + Ax , then:
[Equation]
[Equation]
We can show that the differential entropy of a standard general multivariate student’s T distribution is given by:
[Equation]
Using this expression
[Equation]
we can show that the differential entropy of the predictive posterior of a
Normal-Wishart Prior Network is given by:
[Equation]
A.3. M UTUAL I NFORMATION
The mutual information between the target y and the parameters of the output distribution { µ , Λ } is measures of knowledge uncertainty
[Equation]
and it the difference between the (differential) entropy of the predictive posterior and the expected differential entropy of each normal distribution sampled from the Normal Wishart:
[Equation]
[Table]
[Equation]
The ﬁrst term, the differential entropy of the predictive posterior was derived above in (31).
We we derive the expected differential entropy as follows:
[Equation]
Thus
[Equation]
the ﬁnal expression for mutual information is:
[Equation] [Equation]
Note that this expression is no longer a function of L , which was important in representation data uncertainty .
A.3. E XPECTED P AIRWISE KL-D IVERGENCE
An alternative measures of knowledge uncertainty which can be considered is the expected pairwise kl-divergence (EPKL), which upper bounds mutual information (Malinin, 2019).
In this section we derive it’s closed form expression for the Normal-Wishart distribution.
[Equation]
Here p ( µ 0 , Λ 0 )
= p ( µ 1 , Λ 1 ) =
p ( µ , Λ | x ; θ ) .
In (35) the ﬁrst term is:
[Equation]
The second term in (35) is zero, and the third term is:
[Equation]
which in sum gives us:
[Equation]
Note that this is also not a function of L , just like mutual information.
Rather, it is only a function of the pseudo-counts κ and ν .
A.3. L AW OF T OTAL V ARIATION
Finally, in order to be able to compare with ensembles, we can also derive variance-based measures of total , data and knowledge uncertainty via the Law of total variance, as follows:
[Equation]
This has a similar decomposition as mutual information.
In this section we derive its closed form expression.
We can compute the expected variance by using the probabilistic change of variables:
[Equation]
and the variance of expected as:
[Equation]
Thus, the total variance is then expressed as:
[Equation] [Equation]
Note that this yields a measure which only considers ﬁrst and second moments.
In addition, in order to obtain a scalar estimate of uncertainty, it is necessary to consider the log-determinant of each measure.
B E XPERIMENT ON S YNTHETIC D ATA
The training data consists of 2048 inputs x uniformly sampled from [ − 10 , 10] with targets y ∼
N (sin x + x 10 ) .
We use a relu network with 2 hidden layers containing 30 units each to predict the parameters of either Gaussian or Normal-Wishart distribution on this data.
In all cases, we use
Adam optimizer with learning rate 10 − 2 and weight decay 10 − 4 for 800 epochs with batch size 128 .
Gaussian models in an ensemble are trained via negative log-likelihood starting from different random initialization.
To train a Regression Prior Network with reverse-KL divergence 512 points were uniformly sampled from [ − 25 , − 20] ∪ [20 , 25] as training-ood data.
We use objective (8) with coefﬁcient γ = 0 .
The prior belief is κ 0 = 10 − 2 and in-domain ˆ β is 10 2 .
For EnD 2 training we set
T = 1 and add gaussian noise to inputs with standard deviation 3 .
C UCI E XPERIMENTS
The current appendix provides additional details of experiments on the UCI regression datasets.
Note that we leave out the Yacht Hydrodynamics datasets, as it is the smallest with the fewest features.
The remaining datasets are described in the table below: [Table]
C
[Table]
T RAINING
Following (Lakshminarayanan et al., 2017) in all experiments except MSD we use 1 layer relu neural network with 50 hidden units, for MSD we use 100 hidden units.
We optimize weights with Adam for 100 epochs with batch size 32 .
All hyper-parameters, including learning rate, weight decay, RKL prior belief in train data κ 0 , RKL OOD coefﬁcient γ , EnD 2 initial temperature T and noise level ε are set based on grid search, where we use an equal computational budget in all models to ensure a fair comparison.
Additionally, we use 10 folds cross-validation and report standard deviation based on it.
C. C REATING OUT - OF - DOMAIN DATA
Here we detail how OOD training data for reverse KL-divergence trained Prior Networks is created and how evaluation OOD data is created.
As reverse KL-divergence training of Prior Networks requires out-of-domain training examples, we use a factor analysis model to generate samples for this in the same way as was done in (Malinin, 2019).
Speciﬁcally, we train a linear generative model that approximates inputs with ˆ x ∼ N ( µ, WW T +Ψ) , where [ W, µ, Ψ] are model parameters.
In-domain training data is used to train this model.
The out-of-domain training examples are then sampled from N ( µ, 3 WW T + 3Ψ) .
To estimate the quality of out-of-domain detection, we additionally create evaluation out-of-domain data from external UCI datasets: "Relative location of CT slices on axial axis Data Set" for MSD and "Condition Based Maintenance of Naval Propulsion Plants Data Set" for other datasets.
We drop all constant columns in them and leave ﬁrst K columns and ﬁrst N rows, where K is a number of features and N is a number of test examples in a corresponding dataset.
For each comparison, the out-of-domain datasets are normalized by the per-column mean and variance obtained on in-domain training data, in order to make out-of-domain detection task more difﬁcult.
C. P REDICTION R EJECTION R ATIO
Prediction Rejection Ratio (PRR) (Malinin et al., 2020) measures what part of the best possible error-detection performance our algorithm covers. In Figure 4 prediction-rejection curves are shown. They are built by iteratively rejecting test examples according to different orders and computing MSE error on each step.
Uncertainty curve corresponds to the order of decreasing uncertainty of some model.
Oracle curve corresponds to the best possible order. Prediction Rejection Ration is a ratio of the area between Uncertainty and Random curves AR uncertainty (orange in Figure 4) and the area between Oracle and Random curves AR oracle (blue in Figure 4): [Equation]
[Figure]
C. R ESULTS FOR ALL DATASETS
The current section provide a full set of predictive performance, error detection and OOD detection results on UCI datasets in tables 6-8, respectively.
Results in table 6 show that all models achieve comparable performance and that EnD 2 tends to come close to the ensemble.
Table 7 shows the error detection performance of all models in terms of prediction-rejection ratio (PRR).
The results clearly show that measures of total uncertainty are useful in detecting errors, though it is more challenging on some datasets.
At the same time, measures of knowledge uncertainty do signiﬁcantly worse.
Finally, table 8 shows the OOD detection performance in terms of % AUC-ROC.
Here measures of knowledge uncertainty do far better.
Notably, on the larger datasets, EnD 2 comes closer to the performance of the ensemble
[Table] [Table]
[Table]
[Table]
D D EPTH E STIMATION E XPERIMENTS D. M ETRICS DEFINITION
In this section we give a description for metrics used in Table 3. These metrics are common in prior work on monocular depth estimation. Assume that ( y, y ∗ ) is a pair consisting of a predicted depth map and target depth map, and y i corresponds for a prediction for i -th pixel, T is a set of all predictions y . [Equation]
D. C ALIBRATION
[Figure]
Table 9 shows that EnD also has signiﬁcantly worse calibration than even a single models, and the temperature annealing improves calibration, both in terms of NLL and C-AUC.
The latter is the area between the calibration curve and the diagonal (C-AUC).
A histogram of NLL and calibrations curves are provided in the ﬁgure below.
We can see that EnD 2 yields far more consistent NLL, and has the best predictive intervals in terms of C-AUC, which is the area between the diagonal and the calibration curve.
Note, that for all models we assume normally distributed predictive intervals, as was originally done in (Lakshminarayanan et al., 2017). [Table]
In RPN training with RKL objective, we observed the optimization trajectory to be very unstable and initialization-sensitive.
To combat this, we linearly increase γ from 0 to some predeﬁned value during the ﬁrst ﬁve epochs, which allows our models to concentrate initially on predictive performance, and then gradually capture the properties of “in-domain” samples.
Additionally, we performed an ablation study across different coefﬁcients γ , with the results provided in tables 11 and 12.
On NYU dataset, we see that models with lower γ improve performance at the cost of decreased quality of OOD detection.
This may be an indication that the task of accurate prediction may not align well with the model’s ability to detect unfamiliar samples.
Based on this, we decided to use the coefﬁcient of γ = 0 . 05 as achieving the best trade-off, and then ﬁne-tuned the respective model for 10 additional epochs until convergence. and vice-versa.
[Table]
[Table]
[Table]