P ARAMETRIC D ENSITY E STIMATION WITH U NCERTAINTY USING D EEP E NSEMBLES
Anonymous authors
Paper under double-blind review
A BSTRACT
In parametric density estimation, the parameters of a known probability densityare typically recovered from measurements by maximizing the log-likelihood.
Prior knowledge of measurement uncertainties is not included in this method, potentially producing degraded or even biased parameter estimates.
We proposean efﬁcient two-step, general-purpose approach for parametric density estimationusing deep ensembles.
Feature predictions and their uncertainties are returnedby a deep ensemble and then combined in an importance weighted maximumlikelihood estimation to recover parameters representing a known density alongwith their respective errors.
To compare the bias-variance tradeoff of differentapproaches, we deﬁne an appropriate ﬁgure of merit.
We illustrate a number ofuse cases for our method in the physical sciences and demonstrate state-of-the-artresults for X-ray polarimetry that outperform current classical and deep learningmethods.
I NTRODUCTION
The majority of state-of-the-art NN performances are single (high-dimensional) input, multipleoutput tasks, for instance classifying images (Krizhevsky et al., 2012), scene understanding (Redmon et al., 2015) and voice recognition (Graves et al., 2006).
These tasks typically involve one inputvector or image and a single output vector of predictions.
In parametric density estimation, there is a known probability density that the data (or latent featuresof the data) are expected to follow.
The goal is to ﬁnd representative distribution parameters for agiven dataset.
In simple cases where the likelihood is calculable, maximum likelihood estimationcan be used effectively.
In cases where latent features of the data follow a known distribution (e.g.,heights of people in a dataset of photographs), NNs can potentially be used to directly estimate thedistribution parameters.
For clarity, we deﬁne this direct/end-to-end approach as parametric featuredensity estimation (PFDE).
Such an approach requires employing entire datasets (with potentiallythousands to millions of high-dimensional examples) as inputs in order to output a vector of density parameters.
Furthermore, to be useful these NNs would need to generalize to arbitrarily sizeddataset-inputs.
One example of NNs making sense of large dataset-inputs is found in natural language processing.
Here large text corpora, converted to word vectors (Pennington et al., 2014; Devlin et al., 2019),can be input and summarized by single output vectors using recurrent neural networks (RNNs), forinstance in sentiment analysis (Can et al., 2018).
However, these problems and RNNs themselvescontain inductive bias – there is inherent structure in text.
Not all information need be given at onceand a concept of memory or attention is sufﬁcient (Vaswani et al., 2017).
The same can be saidabout time domain problems, such as audio processing or voice recognition.
Memory is inherentlyimperfect – for PFDE, one ideally wants to know all elements of the ensemble at once to makethe best prediction: sequential inductive bias is undesirable.
Ultimately, memory and architecturalconstraints make training NNs for direct PFDE computationally intractable.
On the other hand, density estimation on data directly (not on its latent features), is computationallytractable.
Density estimation lets us ﬁnd a complete statistical model of the data generating process.
Applying deep learning to density estimation has advanced the ﬁeld signiﬁcantly (Papamakarios,2019).
Most of the work so far focuses on density estimation where the density is unknown a priori .
This can be achieved with non-parametric methods such as neural density estimation (Papamakarios et al., 2018), or with parametric methods such as mixture density networks (Bishop, 1994).
In PFDE,however, we have a known probability density over some features of the whole dataset.
The featuresmay be more difﬁcult to predict accurately in some datapoints than others.
Typical parametric density estimation does not make use of data uncertainties where some elementsin the dataset may be more noisy than others.
Not including uncertainty information can lead tobiased or even degraded parameter estimates.
The simplest example of parametric density estimationusing uncertainties is a weighted mean.
This is the result of a maximum likelihood estimate for amulti-dimensional Gaussian.
For density estimation on predicted data features, PFDE, we wouldlike a way to quantify the predictive uncertainty.
A general solution is offered by deep ensembles(Lakshminarayanan et al., 2017).
While these are not strictly equivalent to a Bayesian approach,although they can be made such using appropriate regularization (Pearce et al., 2018), they offerpractical predictive uncertainties, and have been shown to generalize readily (Fort et al., 2019). Additionally Ovadia et al.
(2019) have shown deep ensembles perform the best across a numberof uncertainty metrics, including dataset shift, compared to competing methods such as stochasticvariational inference and Monte Carlo methods.
In this work, we propose a NN approach that circumvents large dataset-input training or recurrentarchitectures to predict known feature density parameters over large input datasets.
We use predictive uncertainties on features of individual dataset elements as importance weights in a maximumlikelihood estimation.
We will show that estimating known density parameters in a 2-step approachprovides greater interpretability and ﬂexibility.
We are able to predict uncertainties on our densityparameter estimates using bootstrap methods (Efron, 1979).
Our method is widely applicable to anumber of applied machine learning ﬁelds; § 3 showcases a few important examples.
Contributions: Our contributions in this paper are as follows: (1) We introduce a general, ﬂexible method for PFDE using NNs.
The method can be applied to any domain requiring PFDE.
Weillustrate a number of varied domain examples in the physical sciences in § 3.
In an in-depthevaluation we show that our method outperforms not only classical methods for density estimation,but also standard NN implementations in an application to X-ray polarimetry.
(3) We investigate thebias-variance tradeoff associated with our method and introduce a tuneable hyperparameter to control it.
Note:
In the following we focus on regression examples, (since unbinned density estimationis preferable to binned).
However, a similar method can be applied to prediction examples wheresoftmax class probabilities are used as heteroscedastic aleatoric uncertainty.
I MPORTANCE WEIGHTED ESTIMATION WITH DEEP ENSEMBLES2. P ROBLEM SETUP AND HIGH - LEVEL SUMMARY
We wish to estimate the feature density parameters of N high dimensional data points { x } :f
( { x n } N n =1 ) .
Here x ∈ R D can be any high dimensional data (e.g. images, time series). N isarbitrary, although usually large since otherwise density estimation is inaccurate.
For example, consider estimating the mean and variance of human heights from a dataset consisting of photographsof people.
A person’s height in each photograph is the image feature and we know this featureapproximately follows a Gaussian distribution.
We develop a method that can estimate the densityparameters (mean and variance) and generalize to any dataset of photographs.
In general, the function f mapping the high dimensional data points to the desired density parameters is unknown, since the high dimensional data is abstracted from its features.
Learning f directlyis typically infeasible because an entire ensemble of inputs { x n } Nn =1 must be processed simultaneously to estimate density parameters, and this approach would have to generalize to arbitrary N anddensity parameter values.
We discuss some special cases where this is possible in § 1. However, thefunction g mapping data features y n to the density parameters is known.
We cast this as a supervised learning problem where we have a dataset D consisting of N data points
D = { x n , y n } N train n =1 with labels y ∈ R K where x ∈ R D .
We want to estimate the density parametersψ 1 , ψ 2 , ... ψ k for an unseen test set g ( { y n } N test n =1 ) for arbitrary N test .
The basic recipe that comes to mind is training a single NN to predict output labels { y n } Nn =1 thenevaluate g directly.
This ignores the high variance in single NN predictions (dependent on train- ing/random initialization), that some individual examples may be more informative than others, andthat an objective to predict the most accurate output labels may not be the best for predicting gooddensity parameters (high bias may be introduced, for instance).
Our hybrid approach is as follows.
(i) Train a deep ensemble of M NNs 1 to predict { y n , σ n } Nn =1where σ n is the total uncertainty on each prediction y n , (ii) use the { σ n } Nn =1 as weights in animportance weighted maximum likelihood estimate.
The next section, § 2.2, describes procedure (i).
2. D EEP ENSEMBLES
Deep ensembles (Lakshminarayanan et al., 2017) return robust and accurate supervised learning predictions and predictive uncertainties, which enable the best density parameter predictions.
These usean ensemble of individual NNs (with different random initializations) trained to predict features andtheir aleatoric uncertainties.
Final predictions and their epistemic uncertainties are then recoveredby combining the estimates from each of the NNs in the ensemble.
In regression, deep ensembles model heteroscedastic aleatoric σ a uncertainty by modifying the typical mean-squared errors (MSE) objective to a negative log-likelihood (NLL) (Lakshminarayananet al., 2017),
[Equation]
Extensions using more complex distributions like mixture density networks or heavy tailed distributions may be more applicable to certain problems with prior knowledge about the error distribution.
In practice, the log-likelihood of any exponential family could be used; we ﬁnd this simple Gaussianapproach to be sufﬁcient and robust for regression problems.
Our results in § 3.4 for a compare
a Gaussian and Von Mises distribution. Epistemic uncertainty σ e is modelled using a uniformly weighted ensemble of M NNs eachtrained starting from a different random initialization.
The regression prediction and uncertaintyare approximated by the mean and standard deviation over the M NN ensemble predictions respectively (each NN in the ensemble contributes {+equally) i.e. ˆ y ( x ) = M − 1 (cid:80) Mm =
+}
ˆ y m ( x ) andσ 2 e ( x ) = Var( { ˆ y m ( x ) } Mm =1 ) . The epistemic uncertainty is then combined with the aleatoric inquadrature to arrive at the total uncertainty: σ 2 = σ 2 a + σ 2 e .
Typically M ∼ 5 − 15 .
In part (i) of our hybrid approach for PFDE, we train a deep ensemble to minimize the NLL (1) ondesired features y .
We follow the deep ensemble training procedure outlined in Lakshminarayananet al.
(2017) (with recast loss function from Kendall & Gal (2017)) without using adversial examples,using the full dataset for each NN.
Since the individual density parameters over predicted featuresare the ﬁnal desired values in PFDE, it is possible that an objective maximizing feature accuracy onthe validation set is not the true objective.
This is possible if the training dataset is biased or themodel (1) is highly misspeciﬁed for the particular problem. The Kitaguchi et al.
(2019) single CNNmethod in table 1, § 3.4, shows a clear case of training bias.
If de-biasing the training dataset or usinga more appropriate model is not possible, we have identiﬁed two potential ways of ameliorating thisissue for PFDE:
Include terms in the individual NN objectives to penalize known sources of bias.
Select the top M performing NNs, as measured by a criterion that includes density parameter prediction bias on a held out test set.
In practice both can be used simultaneously.
However, the former runs into batch size problems(since one needs a large sample size to accurately estimate bias), and the source of bias is not alwayswell understood.
The latter naturally arises from the use of deep ensembles, but could include itsown unwanted bias and risk underestimating the epistemic uncertainty.
We compare selecting thetop performing NNs for the ensemble by a domain speciﬁc criterion against randomly selecting NNsfor the ensemble in § 3.
2. I MPORTANCE WEIGHTED LOG - LIKELIHOOD
Provided a mapping between high dimensional inputs and interpretable features x n (cid:55)→ y n , we cancalculate the density parameters ψ 1 , ψ 2 , ...
ψ k by minimizing the appropriate negative log-likelihoodfunction p ( { y n }| ψ 1 , ψ 2 , ...ψ k ) .
Some feature predictions y n will have greater total predictive uncertainties, σ n .
We estimate feature density parameters by incorporating the total uncertainty into animportance weighted maximum likelihood estimate.
This makes up part (ii) of our hybrid method.
An importance weight quantiﬁes the relative importance of one example over another.
Importanceweighting an element should be the same as if that element were included multiple times in thedataset, proportional to its importance weight Karampatziakis & Langford (2011).
The deep ensemble, once trained, will act as mapping between high dimensional inputs x n and feature-uncertaintyoutput pairs y n , σ n .
For each input x n there will be M output pairs { ˆ y nm , ( σ a ) nm } Mm =1 , one foreach NN in the deep ensemble.
Both the features ˆ y nm and aleatoric uncertainty variances ( σ a ) 2 nmcan be combined by taking the appropriate mean over m ; this mean may depend on the distributionused in ?? , but for the simple Gaussian case the standard mean is sufﬁcient. Taking the mean resultsin a single output pair (ˆ y n , ( σ a ) n ) for each input. Epistemic uncertainties are included as in § 2.2,resulting in the ﬁnal output (ˆ y n , σ n ) .
In order to use all possible information when estimating the desired density parameters ψ 1 , ψ 2 , ...ψ k ,we deﬁne an importance weighted negative log-likelihood function
[Equation]
[Equation]
Each individual prediction y n has an associated importance weight w n .
The σ − λ n term weights eachy n by its predictive uncertainty.
The hyperparamter λ ≥ 0 controls the importance weighting distribution.
A high λ means the y n with the lowest (estimated) MSE will dominate the ﬁnal ensemblestatistic.
As always in estimation problems, there is a trade-off between lower variance predictionsand more bias.
This can be tuned for a speciﬁc application using λ ; we discuss the procedure indetail in our example application, § 3.
Final density parameters are found by minimizing (2) over thedomain of the density parameters ψ .
Typically, the weights in weighted likelihood estimation are determined heuristically (Hu & Zidek,2002).
In this example, we choose w = σ − λ since it approximates the simple functional form ofthe likelihood used in a weighted mean estimate ( λ = 2 ).
This weighting choice is also inspiredby the dispersion parameter used in generalized linear models (GLMs) (Nelder & Wedderburn,1972).
We expect that this weighting will retain similar robustness properties in terms of modelﬁtting, and will generalize well to many domains.
However, of course, any decreasing functionf : R + → R + may be used to determine weights, with the most suitable choice of function fwithin a given class of functions (in our case, parameterized by λ ) to be determined by either crossvalidation or performance on a holdout set. In some applications it is possible to ﬁnd the exactweighting function [ in prep.
, reference deleted to maintain integrity of review process ].
Furtherdiscussion of weight choice in our application is given in section § 3.4.
Conﬁdence intervals on the density parameters can be calculated using the non-parametric bootstrap Efron (1979): select N y n , σ n pairs with replacement and minimize (2).
In the limit of many trialswith different random subsamples, this will give the output distribution on the density parameters.
D ENSITY PARAMETER REGRESSION
For a special class of parameterized densities it is possible to ﬁnd the global minimizer or minimize(2) analytically (e.g. for a multivariate Gaussian).
In practice, the majority of parametric densitiesof interest for PFDE are likely to be convex (exponential families, our application example § 3, etc.),so will fall into this special class.
In the general case, minimization is performed numerically to ﬁndlocally optimal solutions.
In this work, we employ Ipopt (W¨achter & Biegler, 2006), an open-source interior-point solver forlarge-scale non-convex optimization problems, to minimize (2).
This method can be used for convex or non-convex parametric density estimates, but only convex ones are guaranteed to be global optimal.
Because Ipopt ﬁnds locally optimal solutions, which are highly dependent upon an initial guessof the parameters provided to the solver, in the non-convex case, we recommend nested sampling
Feroz et al.
(2009) to test many initial guesses and then select the best local solution.
Constraintson the density parameters, for instance if they have a ﬁnite domain, can be incorporated for both theconvex and non-convex case.
Of course, any optimizer appropriate for (2) can be used and this willdepend on the problem.
The overall training and evaluation procedure is summarized in Algorithm 1.
1: Identify output features y n relevant to the desired density parameter(s) (e
g., subject height in photographs).
2: Train a deep ensemble of NNs using loss function (1) to maximise accuracy on the desired output features 3: Evaluate the density parameter(s) using importance weights by minimizing (2).
4: Tune λ hyperparameter for the speciﬁc application.
E XPERIMENTS
3. X- RAY POLARIMETRY
Measuring X-ray polarization has been a major goal in astrophysics for the last 40 years.
X-ray polarization can provide essential measurements of magnetic ﬁelds very close to high energy sources,such as accreting black holes and astrophysical jets (Weisskopf, 2018).
The recent developmentof photoelectron tracking detectors (Bellazzini et al., 2003) has greatly improved the prospects ofdoing so.
X-ray polarization telescopes with photoelectron tracking detectors directly image electron tracks formed from photoelectrons scattered by the incoming X-ray photons.
We describe anapplication of our hybrid PFDE method to X-ray polarimetry using photoelectron tracking detectors.
We use data from the upcoming NASA Imaging X-ray Polarization explorer (IXPE) (Sgr`o &
IXPE Team, 2019) as a working example.
The problem of recovering polarization parameters froma dataset of (IXPE) electron track images has recently been announced as an open problem in themachine learning community (Moriakov et al., 2020).
The linear polarization of light can be fully described by two degrees of freedom: the polarizationfraction 0 ≤ Π ≤ 1 , (0% – 100%), and the electric vector position angle − π/ 2 ≤ φ ≤ π/ 2 .
These can be thought of as the magnitude and direction of a vector perpendicular to the directionof propagation of the light.
In imaging X-ray polarimetry, when the detector images an X-raysource, it measures individual 2D images of electron tracks excited by incoming X-ray photons.
The initial directions the electrons travel follow a known probability density that depend on thesource polarization, and the problem is to recover the polarization parameters Π and φ from thecollected dataset of 2D track images.
In the case of IXPE, charge tracks are imaged by hexagonal pixels.
Fig.
1 shows some examplephotoelectron tracks at different X-ray energies.
Each track represents the interaction of a singlephoton with a single gas molecule.
The initial track angle y follows the probability density [Equation]
[Equation]
where Π and φ are ﬁxed polarization parameters that depend on the source.
By estimating y for alarge number of tracks, we may recover the original polarization parameters Π and φ , using parametric density estimation.
Track morphologies vary greatly with energy (and even for the same energy); this affects how difﬁcult it is to recover an accurate intial photoelectron angle y .
Low energy tracks are typically lesselliptical and so more difﬁcult to estimate.
For this reason it is essential to incorporate some form ofquality control in the tracks used for polarization estimates.
Current IXPE methods estimate individual track y using a moment analysis (Sgro, 2017).
Thiscalculates the ﬁrst, second and third charge moments using the 2D coordinates of the hexagonal [Figure] detector pixels, combining them to extract y .
For each track, a single − π ≤ y ≤ π is output.
The polarization parameters are then estimated using a standard (unweighted) MLE.
The momentanalysis additionally outputs an estimate of the track ellipticity, which can be used as a proxy fory estimation accuracy.
The standard moment analysis uses a track cut to improve polarization recovery – 20% of the tracks are cut based on ellipticity.
NNs have also recently been applied tothis problem Kitaguchi et al.
This approach uses single CNNs for classiﬁcation on y , withbinned ﬁts to y histograms to extract polarization parameters and track quality cuts.
Our hybridmethod exhibits signiﬁcantly improved performance over both the standard IXPE method and thisbasic NN approach.
3. P ARAMETRIC FEATURE DENSITY ESTIMATION
Following § 2, we deﬁne CNNs that take single track images as input and (ˆ y, ˆ σ ) as output.
In thiscase the track angles y are the data features that follow the known density (4), the density parametersΠ ≡ ψ 1 , φ ≡ ψ 2 , and the CNNs will make up the deep ensemble.
To make the hexagonal track images admissable inputs to standard CNN architectures, we ﬁrstconvert the hexagonal images to square image arrays by shifting every other column and rescalingthe distance between points, as described in Steppa & Holch (2019).
Since there are two possibleshifts (odd and even rows), we apply both and stack the two shifted images, similar to color channelsin rgb images.
We do this to more closely approximate spatial equivariance of the CNN convolutionkernels in the hexagonal space.
At test time, we apply the deep ensemble to the same track 3 times,each time rotated by 120 ◦ in hexagonal space.
We ﬁnd this reduces all relevant prediction bias on ˆ y(and later Π , φ ) introduced when converting from hexagonal to square coordinates.
To recover Π , φ we need to predict 2 y , so we use the loss function (1) but parameterize the true angley as a 2D vector v = (cos2y , sin2y) to capture the periodicity.
The loss function is as follows:
[Equation]
The ﬁnal NN ensembles output the 3-vector ( ˆv , ˆ σ ) .
In this case the mean over ensemble predictionsis calculated using the circular mean of { ˆv m } Mm =1 .
Then ˆ y = 12 arctan ˆv 2 ˆv 1 .
To calculate the ﬁnalΠ , φ with an ensemble of M NNs for a given test dataset with N tracks we minimize the importanceweighted NLL (2) with likelihood
[Equation]
We can recast this as the convex optimization problemminimize
[Equation]
subject to
[Equation]
[Table] where v n = (cosˆ y n , sinˆ y n ) and x = (Πcos φ, Πsin φ ) .
By recasting (2) as a convex optimizationproblem, we have a guaranteed globally optimal solution for (Π , φ ) .
We can solve (7) quickly andefﬁciently using second order Newton methods.
In practice we use the robust open source software IpOpt, § 2.4.
We also consider a more domain speciﬁc, non-Gaussian likelihood function for our loss, (5).
Weuse the log-likelihood of the von Mises distribution for the NN loss:
[Equation]
where I 0 is the modiﬁed Bessel function of the ﬁrst kind.
This is a close approximation of thewrapped Gaussian on the circle.
It is more appropriate than the Gaussian (5) for angular estimatessince it can capture the π periodicity in ˆ y .
For very small ˆ σ this is equivalent to the Gaussian.
Wecompare the results from both losses in § 3.4 and table 1.
3.2. F IGURE OF MERIT
In polarization estimation, we want high recovered ˆΠ 100% (and accurate φ ) for a known 100%polarized source ( Π = 1 ), and low recovered ˆΠ 0% for an unpolarized source ( Π = 0 ).
Since thereis irreducible noise in the tracks, it is impossible for any method to achieve ˆΠ 100% ∼ 1 , so ˆΠ measestimates are calibrated to get the ﬁnal ˆΠ for an unknown source 2 : ˆΠ
= ˆΠ meas / ˆΠ 100% .
We deﬁne aﬁgure of merit for polarization estimation:
[Equation]
We use the FoM to evaluate model performance: a lower FoM means better polarization estimation.
This is effectively a measure of the signal to noise ratio, a simpliﬁed extension of the minimumdetectable polarization (MDP) typically deﬁned for X-ray polarization (Weisskopf et al., 2010) thatdoes not preclude biased estimators.
It is evaluated on unseen polarized and unpolarized datasets.
In estimating the FoM, we take the number of tracks N ∼ 360 ,
so we can compare directly to
Kitaguchi et al.
We average the FoM over 200 independent track dataset samples of size N .
We use the FoM as the criterion to select the hyperparameter λ in (2).
In this way we can tradeoffaccuracy and bias in our Π , φ estimates.
3. NN T RAINING AND SELECTION
Our training dataset consists of 3 million simulated tracks, examples of which are shown in ﬁg. 1.
The track energies uniformly span 1 .
− 9 . 0 keV, IXPE’s most sensitive range and are unpolarized(uniform track angle distribution).
Since we don’t know a priori what energy each track is, we want [Figure]
NNs that can make predictions for tracks of all energies.
This also makes for a more generalizablesystem, since some high energy tracks have similar characteristics to lower energy ones.
Each trackis labelled with its 2D angle vector v .
We use a ResNet-
(He et al., 2015) convolutional NN architecture as our base NN.
This particulararchitecture is large enough to overﬁt the training set, and trains in a reasonable amount of time.
Before training we preprocess the training data (square track images).
We apply pixelwise centeringand rescaling.
We use stochastic gradient descent with momentum and a decaying learning ratestarting at 1 e − 2 .
We choose batch sizes 512 , 1024 , 2048 (tracks per batch).
We trained for 150epochs, using early stopping to prevent overﬁtting.
We use L 2 -norm regularization 5 × 10 − 5 .
Wetrain 30 NNs and compare randomly selecting M = 10 NNs to selecting M = 10 NNs with the top MSEs on y for an unseen test dataset spanning all energies to make up our ﬁnal NN ensemble. Theresults for both methods are shown in table 1.
3. R ESULTS
Table 1 shows the results of our deep ensemble PFDE method alongside the current state of the artmethods.
The single CNN method with optimized cuts, developed in (Kitaguchi et al., 2019), provides signiﬁcant improvements in Π 100% over the moment analysis, but adds bias to the unpolarizedmeasurement Π 0% , increasing its FoM and making it a worse method for all energies.
We performan ablation study over our method, testing a single NN without using weighting when estimating(Π , φ ) (i.e. w n = 1 ∀ n , (3)), an ensemble of NNs without weighting, a randomly selected ensemblewith weighting, a top MSE selected ensemble with weighting and a von Mises loss weighted ensemble.
We ﬁnd a single NN without weighting beats the classical moments and moments with cutsbaselines. This result is visualized in the right panel of ﬁg. 3.3 for the 6.4keV dataset: the single NNshows improved ˆ y estimates and thus a density that more closely resembles the ground truth. Usingan ensemble of NNs improves this result slightly, but the real power of our method comes with theimportance weights. Our ﬁnal importance weighted ensemble method, with λ tuned accordingly foreach energy, signiﬁcantly outperforms the rest, especially in the power law datasets, where there isa reduction in FoM of almost a factor of 1 . 5 .
This shows the power of a simple weighted schemeover quality cuts in PFDE, it allows our method to take advantage of higher signal ( Π 100% ) at higherenergies in the power law datasets.
The λ tuning procedure is shown in the left panel of ﬁg.3.3.
Comparing a randomly selected ensemble with a top MSE selected ensemble we ﬁnd the resultsare almost identical. Random selection should yield more accurate approximations of the epistemicuncertainty and thus better weights, while selecting top performing NN on MSE should improve ˆ yaccuracy. Since the results are identical, but selecting NNs has the potential to bias density estimation, we recommend randomly selecting NNs. We note that, although not included in the table, asingle NN with importance weighting performs only slightly worse than than the weighted ensemble.
Since a single NN only produces aleatoric uncertainties, this suggests, as expected, that for acorrectly speciﬁed model aleatoric uncertainties dominate epistemic ones. Finally, the von Mises loss shows a small improvement over the simple Gaussian. This is expected, since characterizingthe predictive uncertainties by a periodic distribution is more appropriate for the polarimetry application, but the improvement is small, suggesting that the Gaussian is a robust starting point formany applications. We plan to release further results and more domain speciﬁc information for thisparticular application [ reference deleted to maintain integrity of review process ].
3. O THER APPLICATIONS
There are numerous application of PFDE with uncertainty in the physical sciences and engineering.
In high energy particle physics massive, short-lived particles can be detected by ﬁtting a Cauchydistribution to the frequencies of measured decay states.
Raw sensor data from hadronic particlecolliders like the LHC are very noisy with variable uncertainty, meaning our PFDE approach toestimate the Cauchy distribution parameters could be very fruitful.
This especially true with thewidespread current use of deep learning in particle physics (Guest et al., 2018).
Our approach isheuristically justiﬁed due to the asymptotic efﬁciency of the maximum likelihood estimator in a
Cauchy location model (Cohen Freue, 2007).
In manufacturing, GLMs ﬁt to binomial distributionsare commonly used to assess product quality, or the probability of a product being defunct.
Today,computer vision is used for much of the inspection (Rossol, 1983), making our hybrid PFDE methoda potential step forward.
These are just a few application examples – our method may be useful forany GLM based method with high dimensional data.
D ISCUSSION
We have proposed a supervised learning framework for parametric feature density estimation.
Ourmethod uses deep ensembles to predict high dimensional data features, their aleatoric and epistemicuncertainties.
We estimate feature density parameters by incorporating both of these uncertaintiesinto an importance weighted maximum likelihood estimate.
We include a tuneable weighting hyperparameter λ , allowing one to control the bias-variance tradeoff for density estimation.
Intuitively,in many real feature density estimation problems, some high dimensional data points may be muchmore informative than others due to complex noise or differing generative distributions.
Our methodmodels this explicitly, weighting datapoint features by their predictive uncertainty when estimatingdensity parameters.
This avoids throwing away valuable data with quality cuts, yielding improveddensity estimates.
Our method is scaleable to any feature dataset size and is completely ﬂexible forspeciﬁc domain applications; most NN architectures can be used.
We achieve state-of-the-art resultsover standard deep learning methods and classical algorithms in X-ray polarimetry - a recent openproblem in ML.
We expect our method would provide similar improvements to a number of PFDEapplication ﬁelds, including high energy particle physics and manufacturing.
We perform an ablation study comparing a single NN, a deep ensemble, and various importanceweighted deep ensembles.
A single NN approach or standard deep ensemble improves slightlyon the classical baselines, but importance weighting by predictive uncertainty provides the mainimprovements to our method.
Selecting NNs for the deep ensemble based on quality of densityestimation provides no additional gain in performance compared to random selection – since it ispossible performance-based NN selection can degrade epistemic uncertainty estimates, we recommend randomly selecting NNs for the ensemble.
Comparing the Gaussian and von Mises distributionfor feature prediction we ﬁnd the standard Gaussian likelihood (1) an effective and robust approximation, although results can potentially be improved for speciﬁc applications by choosing a moreappropriate distribution over the predictive uncertainties.
While our method works well for densities with convex log-likelihoods, non-convex ones will notnecessarily yield globally optimal solutions and may be very time consuming to evaluate.
Future
Work : Future additions to the method include more complex aleatoric uncertainty modelling.
Weassume a Gaussian distribution for our feature prediction (1), but for domain applications wherethere is an expected feature uncertainty, one could use an alternative distribution, or even a mixturedensity network (Bishop, 1994) for more ﬂexibility.
In that case the functional form of weightingwould have to be reconsidered.
Additionally, ﬁnding the optimal weighting function for speciﬁcproblem applications is likely to yield signiﬁcant improvements.