Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picture
Anonymous Author(s)
AffiliationAddress email
Abstract
Uncertainty estimation is essential to make neural networks trustworthy in realworld applications.
Extensive research efforts have been made to quantify andreduce predictive uncertainty.
However, most existing works are designed forunimodal data, whereas multi-view uncertainty estimation has not been sufficientlyinvestigated.
Therefore, we propose a new multi-view classification frameworkfor better uncertainty estimation and out-of-domain sample detection, where weassociate each view with an uncertainty-aware classifier and combine the predictions of all the views in a principled way.
The experimental results with real-worlddatasets demonstrate that our proposed approach is an accurate, reliable, and wellcalibrated classifier, which predominantly outperforms the multi-view baselinestested in terms of expected calibration error, robustness to noise, and accuracy forthe in-domain sample classification and the out-of-domain sample detection tasks.
Introduction
Reliable uncertainty estimation is critical for deploying deep learning models in a number of domainssuch as medical imaging diagnosis [34] or autonomous driving [8].
Even with accurate predictions,domain experts still raise questions of how trustworthy the models are [37].
For example, when amodel’s prediction contradicts a domain expert’s opinion, the quantification of the uncertainty of themodel’s predictions can help determine model’s reliability and justify model use.
Recently, uncertainty estimation of neural networks has been an active research area, where manymethods of quantifying uncertainty in predictions have been proposed [42, 9, 38, 48, 32, 24].
Themajority of existing work focuses on uncertainty estimation for unimodal data.
However, in manypractical problems, data can exhibit in multi-views or multi-modalities.
For example, LiDAR, radar,and RGB cameras can simultaneously capture complementary information about a scene [47], andcomputed tomography (CT) scans and x-ray images can be analyzed together to diagnose a disease[3].
Trustworthy uncertainty estimation with multi-view or multi-modalities data is important becausethe challenges it faces may differ from a unimodal setting (e.g., maintaining accurate predictionswith one of the input views’ domain shifted).
Despite the success of existing work on unimodality,modelling and estimating uncertainty for multi-view data remain a less explored question [11].
A way of solving this problem is to fuse multi-modalities into one modality and directly apply existingunimodal methods.
However, even the state-of-the-art unimodal model (e.g., SNGP [29]) can beprone to noise if one of the views in a multi-view dataset is noisy, as shown in Figure 1.
Withoutthe noisy view, unimodal models can produce accurate and confident predictions nearby the trainingdomain.
However, with the noisy view, the predictions become uncertain for samples even close to
[Figure]
ob
[Figure]
ilit [Figure] [Figure] P r ob a b ilit y the training domain (see Figure 3).
We also show that the existing multi-view classifiers (e.g., TMC[11]) have limited capacity to detect out-of-domain (OOD) samples in our experiments (see Table 4).
To this end, we propose the Multi-view Gaussian Process (MGP) that is a tailored frameworkproviding intrinsic uncertainty estimation for classification of multi-view/modal data.
Specifically,
MGP consists of a dedicated Gaussian process (GP) expert for each view whose predictions areaggregated by the product of expert (PoE).
In our proposed method, there is a natural way of capturinguncertainty by measuring the distance between training set and test samples in the reproducing kernel
Hilbert space (RKHS).
The contributions of our method can be summarized as follows:
We propose a new uncertainty estimation framework with GPs for multi-view data, which is an
under-explored yet increasingly important problem in safety-critical applications.2.
The framework provides better uncertainty estimation through a product of expert model, providing more robustness in dealing with noise and better capacity of detecting OOD data.3.
We develop an effective variational inference algorithm to approximate multi-view posterior
distributions in a principled way.
We conduct comprehensive and extensive experiments on both synthetic and real-world data, which show that our method achieves the state-of-the-art performance for uncertainty estimation
of multi-view/modal data.
Muti-view Gaussian Process
Given training data X
=
{ X 1 , X 2 , ..., X V } where V is the number of views, each view consists ofa training set of N samples
X v = { x v,i } Ni =1 and labels y =
{ y i } Ni =1 .
In other words, the i th datasample consists of V views { x v,i } Vv =1 (e.g., the CUB dataset consists of images as the first view andcaptions as the second view) and y i is the data sample’s ground-truth label shared across the views.
Without loss of generality, a multiview classification or regression problem can be formulated aspredicting y ∗ given testing samples { X ∗ ,v } Vv =1 .
In this paper, we propose the Multi-view Gaussian
Process (MGP), a novel framework for multi-view/modal data, where in a nutshell we first apply a
GP to each view of the data and then combine in a principled way the predictions from all the GPs asa unified prediction by using the product of expert (PoE)
2.1 GP for an Individual View
For each view, we consider a multiclass classification problem with C classes.
We set C independent
Gaussian priors over latent function f v ( · ) with zero-mean and N × N covariance matrix K NNwhose element is K ij = k
( x v,i , x v,j ) , where k ( · , · ) is a kernel function.
The radial basis function(RBF) which is commonly used in the GP literature [49, 33] is selected in this paper as the covariancefunction.
It is defined as k ( x , x ′ )
=
σ 2 v exp (cid:16) − ( x − x ′ )
2 2 l 2 v (cid:17) , where σ 2 v is the signal variance and l v isthe length-scale for each GP which are parameters to be optimized.
To bypass the limitations of standard GPs [28], namely high computational cost O ( N 3 ) and inconvenience of applying stochastic gradient descent (SGD), we propose to leverage the sparse variational
GP (SVGP) [28, 14, 15, 41], which is detailed as follows.
With SVGP, we introduce M ( M < N )inducing points Z v representing the training samples of view v with a smaller number of points, andthe inducing variable u v is the latent function evaluated at the inducing points (i.e., u v = f v ( Z v ) )where both Z v and u v are random variables to be optimized.
Similar to the Gaussian prior set for thelatent function, a joint prior can be set as:
[Equation]
where we use f v to indicate f v ( X v ) for notation convenience.
The use of inducing points can reducethe computational cost to O
( M 3 )
We outline the likelihood of GPs in Section 2.2 and theposterior in Section 2.3.
2.2 GPs for Multi-view Data with PoE
Product of Experts (PoE)
With one GP expert for each view, we propose to combine the GPexperts into a unified prediction by using the PoE mechanism [17, 28, 7, 6].
Specifically, we aggregateposterior distributions of individual views by:
[Equation]
For Gaussian posteriors with mean µ v and covariance Σ v , the aggregation using Equation (2) formsthe unified posterior
’s mean and covariance expressed as:
[Equation]
Dirichlet-based Likelihood In order to apply Equation (3) to a multi-view problem, the latentfunction f v in each view should refer to the same observable variable (i.e., N ( a | b, c ) cannot becombined with N ( c | d, e ) for a ̸ = c ).
However, in GP classification, the latent function is a nonobservable nuisance function that is squashed through sigmoid or softmax function to estimate labels[49], which is not necessarily the same for every independent view.
We alleviate this problem byreparameterizing the class labels to regression labels by:
[Equation]
where (cid:101)
y i is the transformed label and ˜
σ 2 i is the noise parameter fixed for all views.
Since (cid:101) y i
and
˜ σ 2 i areshared across the views, we ensure that f v ( x v,i ) refers to the same variable.
By using the log-normaldistribution, the Gaussian likelihood can be used in the log space as p ( (cid:101)
y i | f v ) =
N ( f v , ˜ σ 2 i ) .
To transform the class labels to regression labels, we propose to adopt representing the class probabilityπ i =
[ π i, 1 , π i, 2 , ..., π i,C ] over a Dirichlet distribution with the categorical likelihood [33]:
[Equation]
) , where π [Equation] ∼
[Equation]
α i )
[Equation]
i
[Equation]
g
[Equation]
g
[Equation]
, where g i,c ∼ Gamma( α i,c , 1) (4) where α i =
[ α i, 1 , α i, 2 , ..., α i,C ] is the concentration parameters, the shape parameter for Gammadistribution is α i,c , and the scale parameter for Gamma distribution is θ = 1 .
We approximate the
Gamma distribution in (4) with ˜ g
i,c ∼ Lognormal(˜ y i,c , ˜ σ 2 i,c ) by moment matching:
[Equation]
Thus, the transformed labels and the noisy parameter are expressed in terms of the concentrationparameters:
[Equation]
where α i,c = 1 + α ϵ if y i,c
= 1 and α i,c = α ϵ if y i,c = 0 with the one-hot label
y i,c .
α ϵ is aparameter to prevent the noise parameter from converging to infinity.
See Appendix for the impactsof α ϵ on the model performance.
Compared with other transforming methods such as Platt scaling[36], the used Dirichlet likelihood compromises classification accuracy less and requires no post-hoccalibrations after training.
2.3 Training of the Proposed Framework
Given the priors from Section 2.1 and the Gaussian likelihood from Section 2.2, the goal of trainingour framework is to estimate a posterior distribution via variational inference (VI) [14, 15, 4].
Byusing Equation (2), we propose an aggregated variational distribution for all the views as:
[Equation] [Equation]
where q ( f v ) is the variational distribution for each view that approximates the true posterior.
Wedefine q ( f v ) as:
[Equation]
(7) where p ( f v | u v ) is the conditional prior from Equation (1), and q ( u v ) is the marginal variationaldistribution of N ( m v , S v ) with optimizable model parameters m v and S v .
VI seeks to minimize thefollowing Kullback–Leibler divergence (KL) between the true posterior and variational distributions:
[Equation]
[Equation] [Equation]
where (cid:101)
y c
=
{ ˜ y
i,c } Ni =1 .
Lemma 1 (Additive Property of KL Divergence) .
If x = [ x 1 , · · · , x n ] ∈ X , p ( x ) = (cid:81) ni p ( x i ) andq ( x ) =
(cid:81) ni
q ( x i ) , we have:
[Equation]
Theorem 2 (KL Divergence with PoE) .
With Equations (2) and (6) , we have:
[Equation]
According to Theorem 2, the VI for the PoE splits to the VI of each expert/view.
For the v th view,the VI minimizes KL [ q ( f v ) || p ( f v | (cid:101) y c )] , which can be turned into the maximization of the evidencelower bound (ELBO):
[Equation]
where β is a parameter to control the KL term, similar to [16], which can be interpreted as aregularization term.
Proofs of Equation (9)-(11) are provided in Appendix.
In order to apply SGD, we match the expectation of stochastic gradient of the expected log likelihoodterm to the full gradient by multiplying the number of batches to the log likelihood term in Equation(11) [15].
The overall loss for all experts is:
[Equation]
The training steps are summarized in Algorithm 1.
Algorithm
Input: V views of training data
X =
{ X v } Vv =1 where each view has N samples of X v = { x v,i } Ni = and y =
{ y i } Ni =1 .
Transform: Reparameterize (cid:101)
y c by (5) 1 for minibatch do for v = 1 to V do
Compute q ( f v ) by (7)
Calculate ELBO v by (11) end for
Sum ELBOs by (12)
SGD update { l v , σ 2 v , Z v , m v , S v } Vv = 8 end for
Input: V views of testing data
X ∗ =
{ X ∗ ,v } Vv = 1 for v = 1 to V do
Compute q ( f ∗ ,v ) by (13)
Calculate γ ( X ∗ ,v ) by (17) 4 end for 5 Aggregate q PoE ( f ∗ ) by (16)
Output: E [ π i,c ] and V [ π i,c ] of class probability by (14)
2.4 Inference on Test Samples
Given test samples X ∗ =
{ X ∗ ,v } V v =1 , the predictive distribution p ( f ∗ ,v | (cid:101) y c ) is estimated by thevaritional distribution as:
[Equation]
(13) where p ( f ∗ ,v | u v ) can be formed by the joint prior distribution similar to Equation (1) (see Appendixfor a full derivation).
Similar to Equation (6), we aggregate the predictive distributions to formq PoE ( f ∗ ) that is sampled to approximate Gamma-distributed samples which in the end form theposterior of Dirichlet distribution as follows:
[Equation]
Equation (14) can be approximated with the Monte Carlo method.
The aggregated predictivedistribution can also be weighted by each expert’s predictive distribution by:
[Equation]
where γ ( X ∗ ,v ) is the weight controlling the influence of each expert to the aggregated prediction.
The mean and covariance of q PoE ( f ∗ ) with γ ( X ∗ ,v ) are:
[Equation]
In our experiments, we use negative entropy of predictive distribution:
[Equation]
Note that the original PoE [17] in Equation (6) is recovered if γ ( X ∗ ,v )
= 1 .
The intuition behindchoosing negative entropy is that the experts with lower posterior entropy, which means the loweruncertainty, gain more contribution to the aggregated predictions.
Please note that other choices ofγ ( X ∗ ,v ) can also be applied such as the difference in entropy from prior to posterior [6] and negativepredictive variance [7].
We obtain the better empirical results with negative entropy, but the choice offunction is flexible.
The inference steps are summarized in Algorithm 2.
Related Work
Uncertainty Estimation with GP GP has been one of the gold standards of uncertainty estimationbecause of GP’s high sensitivity to domain shift.
One of the common ways to implement GP with deep learning models is to place GP at the output layer on top of extracted features.
The features areoften extracted from deterministic deep neural networks [44, 29, 5], Bayesian neural networks [26],or graph data [30].
Similarly, MGP builds on these approaches and can be combined with variousfeature extractors.
However, our method differs from all of above studies because these studiesare designed for unimodal data.
However, MGP is a multi-view GP.
Other variants utilizing kernellearning for uncertainty estimation include deep GP [27] and RBF network [43].
Multi-view Learning Multi-view and multimodal learning aim for various downstream tasksby leveraging multiple data sources that describe the same event or object.
Canonical correlationanalysis (CCA) [18] holds a long history, which finds a common representation of multiple sources[1, 45].
Similarly, contrastive learning builds the common representation by forming positivepairs and negative pairs [40, 12].
Also, view-specific representations are learned to make modelsrobust to missing input view [50].
Recently, it has been theoretically and empirically shown thatvision-language models outperform unimodal models [19, 25, 21].
Other methods include gradientblending [46] and hierarchical metric learning [51].
Despite these extensive studies in multi-viewand multimodal learning, most of them are not mainly designed for uncertainty estimation.
Multi-view Uncertainty Estimation Few studies have been designed and evaluated for multi-viewand multimodal uncertainty estimation.
Multimodal regression with mixture of normal-inverse
Gamma distributions yields promising uncertainty estimation and predictions with the real-worlddata [31].
However, this method is designed for regression, which differs from our method forclassification.
The closest study to ours is the trusted multi-view classifier (TMC) which combinesevidence from different views by using Dempster’s combination rule [11].
However, the Dempsterd’scombination rule ignores predictions of conflicting views, which is an undesirable property especiallyin high-risk applications [11, 20].
In addition, our experiments show that TMC is overconfident aboutthe OOD samples.
Experiments
4.1 Synthetic Dataset
To illustrate predictive behaviours of baselines and MGP, we constructed a multi-view syntheticdataset with the Scikit-learn’s moon dataset 1 by scaling the data with three different scaling factors(the same dataset used in Figure 1).
Each view consists of 1,000 data points formed as two halfcircles: upper circle (class 1) and lower circle (class 2).
In each view, data points share the samerelative locations with the same labels.
Note that the points in the third view overlap each other,representing a noisy view.
Deep Ensemble [23] with late fusion [2] (DE(LF)), TMC, and MGP are comprised of dedicatedclassifiers for every view (view 1, 2, and 3 in Figure 2), and the predictions made in the views arecombined as single prediction represented as multi-view in Figure 2.
Since SNGP 2 is a single-viewclassifier, data points of all the views were concatenated as input.
The results of SNGP are shown in
Figure 1.
For experimental details, see Appendix.
One of the benefits of having a multi-view uncertainty estimator is that the prediction accuracy of anideal multi-view classifier remains high even if one of the input views does not provide meaningfulinformation.
This feature can be achieved by assigning lower weights to the views with highuncertainties when combining predictions.
High uncertainty here refers to unconfident predictionswith a uniform class probability across classes.
Figure 2 shows that the combined predictions of DE(LF) are moderately affected by the noise becausethe predictions are averaged across all the views.
TMC and MGP, on the other hand, are not affected
[Figure]
(b
[Figure]
View 2 (P
[Figure]
(c) View 3 (P) (d) Multi-view (P) 0.
P r ob a b ilit y (e) Multi-view (U) 0.
U n ce r t a i n t y (f) View 1 (P) (g) View 2 (P) (h) View 3 (P) (i) Multi-view (P) 0.
P r ob a b ilit y (j) Multi-view (U) 0. U n ce r t a i n t y
[Figure]
View
[Figure]
(P) (l
[Figure]
P [Figure] [Figure] View
[Figure]
[Figure] [Figure] Multi-
[Figure]
(P) 0. P r ob a b ilit y (o) Multi-view (U) 0.
U n ce r t a i n t y by the noisy view (View 3).
The majority of areas in the noisy view show high uncertainty (i.e.,class probability close to 0.5) which have minimal impact on combining predictions because TMCand MGP are aware of uncertainty of each view.
However, SNGP’s prediction is heavily impactedby the noisy view as shown in Figure 1c.
If SNGP were trained without any noisy view, it canproperly estimate class probability (Figure 1f).
This difference between Figure 1c and 1f illustratesthe degrading effect of noisy view to single-view classifiers.
Although TMC is robust to noise, it produces overconfident predictions at the regions far fromdecision boundaries (see Figure 2f-2i).
As a result, OOD samples are indistinguishable from indomain samples (see Figure 2j).
This is mainly caused by lack of distance awareness which MGP and
SNGP have in common due to GP [29].
Figure 2k-2o show that MGP is well aware of the distancebetween the training domain and OOD by allocating high uncertainty at the OOD samples whereuncertainty is calculated by the sum of variance across all classes.
4.2 Robustness to Noise
Experimental Settings We used six multi-view datasets [11] (Handwritten, CUB, PIE, Caltech101,
Scene15, and HMDB) with train-test split of 0.8:0.2.
For testing robustness to noise, we added
Gaussian noise of zero mean to half of the views for each dataset, following the experimental settingin [11].
In order to have the same impact of noise to all views, we normalized each view first andthen added the noise because the range of raw data of each view varies significantly.
We increasedthe noise standard deviation from 0.01 to 10.
To report test results invariant to selecting which viewsto add the noise, we ran all combinations of selecting noisy views (i.e., (cid:0) V V/ 2 (cid:1) configurations) andreport its average for each noise level.
[Figure]
Compared Methods We selected three single-view baselines and two multi-view baselines.
Forsingle-view baselines, we used early fusion (EF) technique [2] by concatenating multi-view featuresinto single feature.
The selected single-view baselines are as follows: MC Dropout [9] with dropoutrate of 0.2, Deep Ensemble (DE) [23] with 5 models, and SNGP ’s GP layer [29].
The multi-viewbaselines are following: Deep Ensemble (DE) with late fusion (LF) technique [2] where one modelwas trained for one view and predictions of all views were combined as single prediction by averagingthe predictions, and TMC [11].
For details of model settings, see Appendix.
Results For every metric, mean and standard deviation from five runs with different random seedsare reported.
The results of multi-view classifiers are generated from the combined predictions.
In Table 1 and 2, we provide test accuracy and test expected calibration error (ECE) [10] withoutadding noise (the definition of ECE is provided in Appendix).
In terms of test accuracy, MGPoutperforms DE(LF) over 4 out of 6 datasets and TMC over all the datasets.
Also, the test ECE of
MGP outperforms both DE(EF) and DE(LF) over all the datasets and TMC over 5 out of 6 datasets.
Figure 3 shows the test accuracy with respect to the standard deviation of Gaussian noise, representingthe domain-shift test accuracy.
This highlights that MGP is robust to noise while the accuracy ofothers degrades significantly.
We also report the average accuracy in Table 3.
4.3 OOD Samples Detection
Experimental Settings Similar to experimental settings in [29], we investigated OOD detectiontest with CIFAR10-C [13], SVHN [35], and CIFAR100 [22].
We used CIFAR10-C as a multi-viewdataset which is a corrupted version of CIFAR10 with 15 different types of corruption and 5 severitylevels.
The first three corruption types were selected as three multi-view inputs with severity levelsof 1, 3, and 5 respectively in order to have variety of noise levels.
Each view was trained with
CIFAR10-C and tested with SVHN and CIFAR100 as OOD detection tests.
Detecting SVHN samplesis an easier task since SVHN is distinct from CIFAR10-C, and detecting CIFAR100 samples is a moredifficult task since CIFAR100 looks similar to CIFAR10-C. For all methods, we used the [-Inceptionv
 [38]-]Inceptionv3 [39] pre-trained with ImageNet as the feature extractor without further training it.
Details ofexperimental settings are reported in Appendix.
Results Table 4 shows in-domain test accuracy, ECE for CIFAR10-C, and OOD results with
SVHN and CIFAR100.
MGP’s OOD results significantly outperform the others with comparable testaccuracy and ECE.
Especially, MGP outperforms TMC over all the metrics.
Note that the differencein ECE and OOD AUROC between TMC and MGP is considerably larger than the difference intest accuracy.
The similar pattern is observed when DE(LF) and MGP are compared, where test
[Table] [Table]
[Table] [Table]
accuracy of DE(LF) is slightly higher than MGP
[Table]
but MGP outperforms DE(LF) with ECE and OOD AUROC.
This highlights that although multi-view baselines can provide accurate predictions within-domain samples, their calibration and uncertainty estimation could be limited. We also provide
MGP’s predictive uncertainty density in Appendix
[Table]
Conclusion
In this work, we have proposed a new multi-view classification framework for better uncertaintyestimation and out-of-domain sample detection, where we associate each view with an uncertaintyaware GP classifier and combine the predictions of all the views with the PoE mechanism.
Withboth the synthetic and the real-world data, we empirically demonstrated that our method is robust todomain-shift and aware of OOD samples, outperforming other baselines in ECE and OOD detectionscores.
Our method is not limited to a particular feature extractor and can be attached on top ofexisting feature extractors.
A possible limitation of our work is that the weight term that balancespredictive distributions is introduced in a sub-optimal way.
We leave optimizing it as future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] Our contributions are supported by Section 2 andSection 4.(b)
Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix.(b)
Did you include complete proofs of all theoretical results?
See Appendix.
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Appendix.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4 and Appendix.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4 andAppendix.(b)
Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]