T RUSTED M ULTI -V IEW C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Multi-view classiﬁcation (MVC) generally focuses on improving classiﬁcation accuracy by using information from different views, typically integrating them into a uniﬁed comprehensive representation for downstream tasks.
However, it is also crucial to dynamically assess the quality of a view for different samples in order to provide reliable uncertainty estimations, which indicate whether predictions can be trusted.
To this end, we propose a novel multi-view classiﬁcation method, termed trusted multi-view classiﬁcation, which provides a new paradigm for multi-view learning by dynamically integrating different views at an evidence level.
The algorithm jointly utilizes multiple views to promote both classiﬁcation reliability and robustness by integrating evidence from each view.
To achieve this, the
Dirichlet distribution is used to model the distribution of the class probabilities, parameterized with evidence from different views and integrated with the DempsterShafer theory.
The uniﬁed learning framework induces accurate uncertainty and accordingly endows the model with both reliability and robustness for out-ofdistribution samples.
Extensive experimental results validate the effectiveness of the proposed model in accuracy, reliability and robustness.
I NTRODUCTION
Multi-view data, typically associated with multiple modalities or multiple types of features, often exists in real-world scenarios.
State-of-the-art multi-view learning methods achieve tremendous success across a wide range of real-world applications.
However, this success typically relies on complex models (Wang et al., 2015a;
Tian et al., 2019; Bachman et al., 2019; Zhang et al., 2019; Hassani & Khasahmadi, 2020), which tend to integrate multi-view information with deep neural networks.
Although these models can provide accurate classiﬁcation results, they are usually vulnerable to yielding unreliable predictions, particularly when presented with views that are not well-represented ( e.g. , information from abnormal sensors).
Consequently, their deployment in safety-critical applications ( e.g. , computer-aided diagnosis or autonomous driving) is limited.
This has inspired us to introduce a new paradigm for multi-view classiﬁcation to produce trusted decisions.
For multi-view learning, traditional algorithms generally assume an equal value for different views or assign/learn a ﬁxed weight for each view.
The underlying assumption is that the qualities or importance of these views are basically stable for all samples.
In practice, the quality of a view often varies for different samples which the designed models should be aware of for adaption.
For example, in multi-modal medical diagnosis (Perrin et al., 2009; Sui et al., 2018), a magnetic resonance (MR) image may be sufﬁcient for one subject, while a positron emission tomography (PET) image may be required for another.
Therefore, the decision should be well explained according to multi-view inputs.
Typically, we not only need to know the classiﬁcation result, but should also be able to answer “How conﬁdent is the decision?”
and “Why is the conﬁdence so high/low for the decision?”.
To this end, the model should provide in accurate uncertainty for the prediction of each sample, and even individual view of each sample.
Uncertainty-based algorithms can be roughly divided into two main categories, i.e. , Bayesian and non-Bayesian approaches.
Traditional Bayesian approaches estimate uncertainty by inferring a posterior distribution over the parameters (MacKay, 1992a; Bernardo & Smith, 2009; Neal, 2012).
A variety of Bayesian methods have been developed, including Laplace approximation (MacKay, 1992b), Markov Chain Monte Carlo (MCMC) (Neal, 2012) and variational techniques (Graves, 2011;
Ranganath et al., 2014; Blundell et al., 2015).
However, compared with ordinary neural networks, due to the doubling of model parameters and difﬁculty in convergence, these methods are computationally expensive.
Recent algorithm (Gal & Ghahramani, 2016) estimates the uncertainty by introducing dropout (Srivastava et al., 2014) in the testing phase, thereby reducing the computational cost.
Several non-Bayesian algorithms have been proposed, including deep ensemble (Lakshminarayanan et al., 2017), evidential deep learning (Sensoy et al., 2018) and deterministic uncertainty estimate (van
Amersfoort et al., 2020).
Unfortunately, all of these methods focus on estimating the uncertainty on single-view data, despite the fact that fusing multiple views through uncertainty can improve performance and reliability.
In this paper, we propose a new multi-view classiﬁcation algorithm aiming to elegantly integrate multiview information for trusted decision making (shown in Fig. 1(a)).
Our model combines different views at an evidence level instead of feature or output level as done previously, which produces a stable and reasonable uncertainty estimation and thus promotes both classiﬁcation reliability and robustness The Dirichlet distribution is used to model the distribution of the class probabilities, parameterized with evidence from different views and integrated with the Dempster-Shafer theory.
In summary, the speciﬁc contributions of this paper are:
(1) We propose a novel multi-view classiﬁcation model aiming to provide trusted and interpretable (according to the uncertainty of each view) decisions in an effective and efﬁcient way (without any additional computations and neural network changes), which introduces a new paradigm in multi-view classiﬁcation.
The proposed model is a uniﬁed framework for promising sample-adaptive multi-view integration, which integrates multi-view information at an evidence level with the DempsterShafer theory in an optimizable (learnable) way.
The uncertainty of each view is accurately estimated, enabling our model to improve classiﬁcation reliability and robustness.
(4) We conduct extensive experiments which validate the superior accuracy, robustness, and reliability of our model thanks to the promising uncertainty estimation and multi-view integration strategy.
R ELATED W ORK
Uncertainty-based Learning.
Deep neural networks have achieved great success in various tasks.
However since most deep models are essentially deterministic functions, the uncertainty of the model cannot be obtained.
Bayesian neural networks (BNNs) (Denker & LeCun, 1991; MacKay, 1992b;
Neal, 2012) endow deep models with uncertainty by replacing the deterministic weight parameters with distributions.
Since BNNs struggle in performing inference and usually come with prohibitive computational costs, a more scalable and practical approach, MC-dropout (Gal & Ghahramani, 2016), was proposed.
In this model, the inference is completed by performing dropout sampling from the weight during training and testing.
Ensemble based methods (Lakshminarayanan et al., 2017) train and integrate multiple deep networks and also achieve promising performance.
Instead of indirectly modeling uncertainty through network weights, the algorithm (Sensoy et al., 2018) introduces the subjective logic theory to directly model uncertainty without ensemble or Monte Carlo sampling.
Building upon RBF networks, the distance between test samples and prototypes can be used as the agency for deterministic uncertainty (van Amersfoort et al., 2020).
Beneﬁting from the learned weights of different tasks with homoscedastic uncertainty learning, (Kendall et al., 2018) achieves impressive performance in multi-task learning.
Multi-View Learning.
Learning on data with multiple views has proven effective in a variety of tasks.
CCA-based multi-view models (Hotelling, 1992; Akaho, 2006; Wang, 2007; Andrew et al., 2013; Wang et al., 2015a; 2016) are representative ones that have been widely used in multi-view representation learning.
These models essentially seek a common representation by maximizing the correlation between different views.
Considering common and exclusive information, hierarchical multi-modal metric learning (HM3L) (Zhang et al., 2017) explicitly learns shared multi-view and view-speciﬁc metrics, while AE 2 -Nets (Zhang et al., 2019) implicitly learn a complete (view-speciﬁc and shared multi-view) representation for classiﬁcation.
Recently, the methods (Tian et al., 2019; Bachman et al., 2019; Chen et al., 2020; Hassani & Khasahmadi, 2020) based on contrastive learning have also achieved good performance.
Due to its effectiveness, multi-view learning has been widely used in various applications (Kiela et al., 2018;
Bian et al., 2017; Kiela et al., 2019; Wang et al., 2020).
Evidence
[Figure]
Evidence ViewUncertainty
Uncertainty
Dirichlet Distribution Dirichlet Distribution Dirichlet Distribution Dirichlet Distribution
C o m b i n i n g b e li e f s
Uncertainty  33   ℳ 2 = { 𝑏 12 , 𝑏 22 , ⋯ , 𝑏 𝐾 2 , 𝑢 2 } ℳ = { 𝑏 1 , 𝑏 2 , ⋯ , 𝑏 𝐾 , 𝑢 } 𝐞 1 =
[ 𝑒 11 , 𝑒 21 , ⋯ , 𝑒 𝐾 1 ] 𝐞 2 =
[ 𝑒 12 , 𝑒 22 , ⋯ , 𝑒 𝐾 2 ] 𝛂 1 =
[ 𝛼 11 , 𝛼 21 , ⋯ , 𝛼 𝑘 1 ] 𝛂 2 =
[ 𝛼 12 , 𝛼 22 , ⋯ , 𝛼 𝑘 2 ] ℳ 1 = { 𝑏 11 , 𝑏 21 , ⋯ , 𝑏 𝐾 1 , 𝑢 1 }
[Figure]
trusted
[Figure]
𝑏𝑏 11 𝑏𝑏 21 𝑏𝑏 31 𝑏𝑏𝑏𝑏 𝐾𝐾 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 𝐾𝐾 𝑢𝑢𝑏𝑏 21 𝑏𝑏𝑏𝑏 31 𝑏𝑏𝑏𝑏 41 𝑏𝑏𝑏𝑏 𝐾𝐾1 𝑏𝑏 𝐾𝐾 𝑏𝑏 11 𝑏𝑏𝑏𝑏 11 𝑢𝑢 2 𝑏𝑏 21 𝑢𝑢 2 𝑏𝑏 31 𝑢𝑢 2 𝑏𝑏 41 𝑢𝑢𝑏𝑏 𝐾𝐾1 𝑢𝑢𝑢𝑢𝑏𝑏 12 𝑢𝑢𝑏𝑏 22 𝑢𝑢𝑏𝑏 32 𝑢𝑢𝑏𝑏 42 𝑢𝑢𝑏𝑏 𝐾𝐾2 𝑢𝑢𝑢𝑢 1 𝑢𝑢
[Figure]
Dempster-Shafer Evidence Theory (DST).
DST, which is a theory on belief functions, was ﬁrst proposed by Dempster (Dempster, 1967) and is a generalization of the Bayesian theory to subjective probabilities (Dempster, 1968).
Later, it was developed into a general framework to model epistemic uncertainty (Shafer, 1976).
In contrast to Bayesian neural networks, which indirectly obtain uncertainty through multiple stochastic samplings from weight parameters, DST directly models uncertainty.
DST allows beliefs from different sources to be combined with various fusion operators to obtain a new belief that considers all available evidence (Sentz et al., 2002; Jøsang & Hankin, 2012).
When faced with beliefs from different sources, Dempster’s rule of combination tries to fuse their shared parts, and ignores conﬂicting beliefs through normalization factors.
A more speciﬁc implementation will be discussed later.
T RUSTED M ULTI -V IEW C LASSIFICATION
It has been shown that using a softmax output as conﬁdence for predictions often leads to high conﬁdence values, even for erroneous predictions since the largest softmax output is used for the ﬁnal prediction (Moon et al., 2020; van Amersfoort et al., 2020).
Therefore, we introduce an evidencebased uncertainty estimation technique which can provide more accurate uncertainty and allow us to ﬂexibly integrate multiple views for trusted decision making.
3. U NCERTAINTY AND THE T HEORY OF E VIDENCE
In this subsection, we elaborate on evidential deep learning to quantify the classiﬁcation uncertainty for each of multiple views, which simultaneously models the probability of each class and overall uncertainty of the current prediction.
In the context of multi-class classiﬁcation, Subjective logic (SL) (Jøsang, 2018) associates the parameters of the Dirichlet distribution (Deﬁnition A.1 in the Appendix) with the belief distribution, where the Dirichlet distribution can be considered as the conjugate prior of the categorical distribution (Bishop, 2006).
Accordingly, we need to determine the concentration parameters, which are closely related to the uncertainty.
We elaborate on the Subjective logic (Jøsang, 2018), which deﬁnes a theoretical framework for obtaining the probabilities (belief masses) of different classes and overall uncertainty (uncertainty mass) of the multi-classiﬁcation problem based on the evidence collected from data.
Note that evidence refers to the metrics collected from the input to support the classiﬁcation (step 1 in
Fig. 1(a)) and is closely related to the concentration parameters of Dirichlet distribution.
Speciﬁcally, for the K classiﬁcation problems, subjective logic tries to assign a belief mass to each class label and an overall uncertainty mass to the whole frame based on the evidence .
Accordingly, for the v th view,
[Figure]
the K + 1 mass values are all non-negative and their sum is one:
[Equation]
where u v ≥ 0 and b vk ≥ 0 indicate the overall uncertainty and the probability for the k th class, respectively.
For the v th view, subjective logic connects the evidence e v =
[ e v 1 , · · · , e vK ] to the parameters of the
Dirichlet distribution α v =
[ α v 1 , · · · , α vK ] (step 2 in Fig. 1(a)).
Speciﬁcally, the parameter α vk of the Dirichlet distribution is induced from e vk , i.e. , α vk = e vk + 1 .
Then, the belief mass b vk and the uncertainty u v (step 3 in Fig. 1(a)) are computed as
[Equation]
(2) where S v = (cid:80) Ki
=
( e vi + 1) =
(cid:80)
Ki
=
α vi is the Dirichlet strength.
Eq.
2 actually describes the phenomenon where the more evidence observed for the k th category, the greater the probability assigned to the k th class.
Correspondingly, the less total evidence observed, the greater the total uncertainty.
The belief assignment can be considered as a subjective opinion.
Given an opinion, the mean of the corresponding Dirichlet distribution ˆ p v for the class probability ˆ p vk is computed as ˆ p vk =
α vk S v (Frigyik et al., 2010).
Differences from traditional deep-neural-network classiﬁers.
Firstly, the output of traditional neural network classiﬁers can be considered as a point on a simplex, while Dirichlet distribution parametrizes the density of each such probability assignment on a simplex.
Therefore, with the
Dirichlet distribution, SL models the second-order probability and uncertainty of the output.
Secondly, the softmax function is widely used in the last layer of traditional neural network classiﬁers.
However, using the softmax output as the conﬁdence often leads to over-conﬁdence.
In our model, the introduced SL can avoid this problem by adding overall uncertainty mass.
Existing methods (Gal &
Ghahramani, 2016; Lakshminarayanan et al., 2017) usually require additional computations during inference to output uncertainty.
Since the uncertainty is obtained during the inference stage, it is difﬁcult to seamlessly train a model with high accuracy, robustness and reasonable uncertainty in a uniﬁed framework.
Accordingly, the limitations underlying existing algorithms ( e.g. , inability to directly obtain uncertainty) also limits their extension to trusted multi-view classiﬁcation.
For clarity, we provide typical examples under a triple classiﬁcation task to illustrate the above formulation.
Let us assume that e = (cid:104) 40 , 1 , 1 (cid:105) and accordingly we have α = (cid:104) 41 , 2 , 2 (cid:105) .
The corresponding Dirichlet distribution, shown in Fig.
2(a), yields a sharp distribution centered on the top of the standard 2-simplex.
This indicates that sufﬁcient evidence has been observed to ensure accurate classiﬁcation.
In contrast, let us assume that we have the evidence e
= (cid:104) 0 . 0001 , 0 . 0001 , 0 . 0001 (cid:105) , which is little evidence for classiﬁcation.
Accordingly, we obtain the Dirichlet distribution parameter α = (cid:104) 1 . 0001 , 1 . 0001 , 1 .
(cid:105) and the uncertainty mass u ≈ 1 .
As shown in Fig.
2(b), in this case, the evidence induces quite a ﬂat distribution over the simplex.
Finally, when e = (cid:104) 5 , 5 , 5 (cid:105) , there is also a high uncertainty, as shown in Fig. 2(c), even though the overall uncertainty is reduced compared to the second case.
As shown in Fig. 2(d), we can convert a Dirichlet distribution into a standard 3-simplex (a regular tetrahedron with vertices (1,0,0,0), (0,1,0,0), (0,0,1,0) and (0,0,0,1) in R 4 ) based on the subjective logic theory (Eq. 1 and Eq. 2), where the point ( M ) in the simplex corresponding to (cid:8) { b k } 3 k =1 , u (cid:9) indicates an opinion.
Accordingly, the expectation value ˆ p of the
Dirichlet distribution is the projection of M on the bottom.
3. D EMPSTER ’ S R ULE OF C OMBINATION FOR M ULTI -V IEW C LASSIFICATION
Having introduced evidence and uncertainty for the single-view case, we now focus on their adaptation to data with multiple views.
The Dempster–Shafer theory of evidence allows evidence from different sources to be combined arriving at a degree of belief (represented by a mathematical object called the belief function) that takes into account all the available evidence (see Deﬁnition 3.1).
Speciﬁcally, we need to combine V independent sets of probability mass assignments {M v } V 1 , where M v = (cid:8) { b vk } Kk =1 , u v (cid:9) , to obtain a joint mass M = (cid:8) { b k } Kk =1 , u (cid:9) (step 4 in Fig. 1(a)).
Deﬁnition 3.1 ( Dempster’s combination rule for two independent sets of masses )
The combination (called the joint mass) M = (cid:8) { b k } Kk =1 , u (cid:9) is calculated from the two sets of masses M 1 = (cid:8) { b 1 k } Kk =1 , u 1 (cid:9) and M 2 = (cid:8) { b 2 k } Kk =1 , u 2 (cid:9) in the following manner:
[Equation]
The more speciﬁc calculation rule can be formulated as follows:
[Equation]
where C = (cid:80) i (cid:54) =
j b 1 i b 2 j is a measure of the amount of conﬂict between the two mass sets (the white blocks in Fig. 1(b)), and the scale factor 1 1 − C is used for normalization.
The joint opinion M is formed based on the fusion of opinions M 1 and M 2 .
The joint belief mass of class k ( b k ) and overall uncertainty ( u ) correspond to the brown blocks in Fig. 1(b).
Intuitively, the combination rule ensures: (1) when both views are of high uncertainty (large u 1 and u 2 ), the ﬁnal prediction must be of low conﬁdence (small b k ); (2) when both views are of low uncertainty (small u 1 and u 2 ), the ﬁnal prediction may be of high conﬁdence (large b k ); (3) when only one view is of low uncertainty (only u 1 or u 2 is large), the ﬁnal prediction only depends on the conﬁdent view.
Then, given data with V different views, we can obtain the above-mentioned mass for each view.
Afterwards, we can combine the beliefs from different views with Dempster’s rule of combination.
Speciﬁcally, we fuse the belief mass and uncertainty mass between different views with the following rule:
[Equation]
After obtaining the joint mass M = (cid:8) { b k } Kk =1 , u (cid:9) , according to Eq. 2, the corresponding joint evidence from multiple views and the parameters of the Dirichlet distribution are induced as
[Equation]
Based on the above combination rule, we can obtain the estimated multi-view joint evidence e and the corresponding parameters of joint Dirichlet distribution α to produce the ﬁnal probability of each class and the overall uncertainty.
Advantages of using subjective logic compared with softmax.
Compared with softmax output, using subjective uncertainty is more suitable for the fusion of multiple decisions.
Subjective logic provides an additional mass function ( u ) that allows the model distinguish between a lack of evidence.
In our model, subjective logic provides the degree of overall uncertainty of each view, which is important for trusted classiﬁcation and interepretability to some extent.
3. L EARNING TO F ORM O PINIONS
In this section, we will discuss how to train neural networks to obtain evidence for each view, which can then be used to obtain the corresponding masses {M v } V v =1 and M .
The neural networks can capture the evidence from input to induce a classiﬁcation opinion (Kiela et al., 2018), and the conventional neural-network-based classiﬁer can be naturally transformed into the evidence-based classiﬁer with minor changes.
Speciﬁcally, the softmax layer of a conventional neural-network-based classiﬁer is replaced with an activation function layer ( i.e. , RELU) to ensure that the network outputs non-negative values, which are considered as the evidence vector e .
Accordingly, the parameters of the Dirichlet distribution can be obtained.
For conventional neural-network-based classiﬁers, the cross-entropy loss is usually employed:
[Equation]
where p ij is the predicted probability of the i th sample for class j .
For our model, given the evidence of the i th sample obtained through the evidence network, we can get the parameter α i ( i.e. , α vi = e ii + 1 ) of the Dirichlet distribution and form the multinomial opinions D ( p i | α i ) , where p i is the class assignment probabilities on a simplex.
After a simple modiﬁcation on Eq.
7, we have the adjusted cross-entropy loss:
[Equation]
where ψ ( · ) is the digamma function.
Eq.
8 is the integral of the cross-entropy loss function on the simplex determined by α i .
The above loss function ensures that the correct label of each sample generates more evidence than other classes, however, it cannot guarantee that less evidence will be generated for incorrect labels.
That is to say, in our model, we expect the evidence for incorrect labels to shrink to 0.
To this end, the following KL divergence term is introduced:
[Equation]
where ˜ α i = y i
+ (1 − y i ) (cid:12) α
i is the adjusted parameter of the Dirichlet distribution which can avoid penalizing the evidence of the groundtruth class to 0, and Γ( · ) is the gamma function.
Therefore, given parameter α i of the Dirichlet distribution for each sample i , the sample-speciﬁc loss is
[Equation]
where λ t > 0 is the balance factor.
In practice, we can gradually increase the value of λ t so as to prevent the network from paying too much attention to the KL divergence in the initial stage of training, which may result in a lack of good exploration of the parameter space and cause the network to output a ﬂat uniform distribution.
To ensure that all views can simultaneously form reasonable opinions and thus improve the overall opinion, we use a multi-task strategy with following overall loss function:
[Equation]
The optimization process for the proposed model is summarized in Algorithm 1 (in the Appendix).
E XPERIMENTS 4. E XPERIMENTAL S ETUP
In this section, we conduct experiments on six real-world datasets: Handwritten 1 , CUB (Wah et al., 2011), Caltech101 (Fei-Fei et al., 2004), PIE 2 , Scene15 (Fei-Fei & Perona, 2005) and HMDB (Kuehne et al., 2011).
We ﬁrst compare our algorithm with single-view classiﬁers to validate the effectiveness of our algorithm in utilizing multiple views.
Then, we apply existing classiﬁers to multi-view features and conduct experiments under different levels of noise to investigate their ability in identifying multi-view OOD samples.
Details of these datasets and experimental setting can be found in the appendix. [Figure]
Compared methods .
We compare the proposed method with following models: (a) MCDO (monte carlo dropout) (Gal & Ghahramani, 2015) casts dropout network training as approximate inference in a Bayesian neural network; (b) DE (deep ensemble)
(Lakshminarayanan et al., 2017) is a simple, nonBayesian method which involves training multiple deep models; (c) UA (uncertainty-aware attention) (Heo et al., 2018) generates attention weights following a Gaussian distribution with a learned mean and variance, which allows heteroscedastic uncertainty to be captured and yields a more accurate calibration of prediction uncertainty; (d) EDL ( evidential deep
Learning) (Sensoy et al., 2018) designs a predictive distribution for classiﬁcation by placing a Dirichlet distribution on the class probabilities.
4. E XPERIMENTAL R ESULTS
Comparison with uncertainty-based algorithms using the best view.
We ﬁrst compare our algorithm with current uncertainty-based classiﬁcation methods.
The detailed experimental results are shown in Table 1.
Since most existing uncertainty-based classiﬁcation methods use single-view data, we report the results of each method with the best-performing view in terms of both accuracy and AUROC (Hand & Till, 2001) to comprehensively compare our method with others.
As shown in Table 1, our model outperforms other methods on all datasets.
Taking the results on PIE and Scene15 as examples, our method improves the accuracy by about 7.6% and 14.8% compared to the second-best models (EDL/MCDO) in terms of accuracy respectively.
Although our model is clearly more effective than single-view uncertainty-based models, it is natural to further ask - what happens if all algorithms utilize multiple views?
[Table]
Comparison with uncertainty-based algorithms using multiple views.
To further validate the effectiveness of our model in integrating different various views, we concatenate the original features of multiple views for all comparison methods.
We add Gaussian noise with different levels of standard deviations ( σ ) to half of the views.
The comparison results are shown in Fig.
As can be observed that when the data is free of noise, our method can achieve competitive results.
After introducing noise to the data, the accuracy of all the comparison methods signiﬁcantly decreases.
Fortunately, beneﬁting from the uncertainty-based fusion, the proposed method is aware of the view-speciﬁc noise and thus achieves impressive results on all datasets.
Therefore, the effectiveness for both clean and noisy multi-view data is well validated.
However, it will be more convincing to explicitly investigate the performance in uncertainty estimation.
Uncertainty estimation.
To evaluate the uncertainty estimation, we visualize the distribution of in-/out-of-distribution samples in terms of uncertainty.
We consider the original samples as indistribution data, while the samples with Gaussian noise are viewed as out-of-distribution data.
Speciﬁcally, we add Gaussian noise with the ﬁxed level of standard deviations ( σ = 10 ) to 50% [Figure] [Figure] of the test samples.
The experimental results are shown in Fig.
According to the results, the following observations are drawn: (1) Datasets with higher classiﬁcation accuracy ( e.g. , Handwritten) are usually associated with lower uncertainty for the in-distribution samples.
In contrast, datasets with lower accuracy are usually associated with higher uncertainty for the in-distribution samples.
Much higher uncertainties are usually estimated for out-of-distribution samples on all datasets.
These observations imply the reasonability of our model in estimating uncertainty, since it can facilitate discrimination between these classes.
Fig.
3 shows that our algorithm provides much more accurate predictions as the prediction uncertainty decreases.
This implies that trusted decisions are supported based on the output (classiﬁcation and its corresponding uncertainty) of our model.
C ONCLUSION
In this work we propose a novel trusted multi-view classiﬁcation (TMC) algorithm which, based on the Dempster-Shafer evidence theory, can produce trusted classiﬁcation decisions on multi-view data.
Our algorithm focuses on decision-making by fusing the uncertainty of multiple views, which is essential for making trusted decisions.
The TMC model can accurately identify the views which are risky for decision making and exploits informative views in the ﬁnal decision.
Furthermore, our model can produce the uncertainty of a current decision while making the ﬁnal classiﬁcation, providing intepretability.
The empirical results validate the effectiveness of the proposed algorithm in classiﬁcation accuracy and out-of-distribution identiﬁcation
A A PPENDIX A. D EFINITION OF D IRICHLET DISTRIBUTION
Deﬁnition A.1 ( Dirichlet distribution )
The Dirichlet distribution is parameterized by its concentration K parameters α
= [ α 1 , . .
. , α K ] .
The probability density function of the Dirichlet distribution is given by
D ( p | α ) = (cid:26) 1 B ( α ) (cid:81)
Ki =1 p α
i − 1 i for p ∈ S K , otherwise , (12) where S K is the K -dimensional unit simplex, deﬁned as
S K = (cid:40) p |
K (cid:88) i = p
i = 1 and 0 ≤ p 1 , . .
, p K ≤(cid:41) , (13) and B ( α ) is the K-dimensional multinomial beta function.
A. A LGORITHM FOR TRUST M ULTI -V IEW C LASSIFICATION
Algorithm 1: Algorithm for /*Training*/
Input: Multi-view dataset: D = {{ X vn } Vv =1 , y n } Nn =1 .
Initialize:
Initialize the parameters of the neural network.
while not converged do for v = 1 : V do e v ← evidential network output; α v ← e v + 1 ;
Obtain opinion M v with Eq. 2; end
Obtain joint opinion M with Eq. 5;
Obtain α with Eq. 6;
Obtain the overall loss by updating α and { α v } Vv =1 in Eq.
Update the networks with gradient descent according to Eq.
11; endOutput: networks parameters.
/*Test*/
Calculate the joint belief mass and the uncertainty mass.
A. D ETAILS OF THE DATASETS
To better evaluate our model, we conduct experiments on the following six real-world datasets.
Details of these datasets are as follows:
Handwritten consists of 2000 samples of 10 classes from digit ‘0’ to ‘9’ withsamples per class, where six different types of descriptors are used as multiple views.
• 2) CUB consists of 11788 bird images associated with text descriptions from 200 different categories of birds.
The ﬁrst 10 categories are used, where GoogleNet and doc2vec are used to extract image features and corresponding text features, respectively.
• 3) PIE consists of 680 facial images of 68 subjects.
Three types of features including intensity, LBP and Gabor are extracted.
• 4) HMDB is one of the largest human action recognition dataset.
There are 6718 samples of 51 categories of actions, where HOG and MBH features are extracted.
• 5) Caltech101 consists of 8677 images from 101 classes.
We extract two types of deep features with DECAF and VGG19 respectively.
• 6) Scene15 consists of 4485 images from 15 indoor and outdoor scene categories.
Three types features (GIST, PHOG and LBP) are extracted as multiple views.
A. C OMPARISON WITH CCA- BASED ALGORITHMS
We compared our method with the CCA-based algorithms. Speciﬁcally, we employ CCA-based methods to obtain the latent representations and then the linear SVM classiﬁer is used for classiﬁcation. The following CCA-based algorithms are used as baselines.
(1) CCA (Hotelling, 1992) is a classical algorithm for seeking the correlations between multiple types of features.
(2) DCCA (Andrew et al., 2013) obtains the correlations through deep neural networks.
(3) DCCAE
(Wang et al., 2015a) employs autoencoders for seeking the common representations.
(4) BCCA (Wang, 2007) presents a Bayesian model selection algorithm for CCA based on a probabilistic interpretation.
Due to randomness (e.g., training/testing partition and optimization) involved, for each method, we runtimes and report its mean and standard deviation in terms of classiﬁcation accuracy.
The experimental results are shown in Table A.3.
On all datasets, our method consistently achieves better performance compared with these CCA-based algorithms.
Note that our method is quite different from the CCAbased methods.
Ours is a classiﬁcation model while most CCA-based methods are unsupervised representation learning models.
Meanwhile, to the best of our knowledge, existing CCA-based algorithms are unable to provide trusted decisions.
[Table]
A. E XPERIMENTAL ANALYSIS OF REMOVING VIEW
[Table]
We conduct experiments by removing one view manually on two datasets (i.e., PIE and Scene15) associated with three views.
We run 30 times and report the mean and standard deviation in terms of classiﬁcation accuracy.
The experimental results are shown in Table 3.
It is observed that by using more information, the performance of our model tends to be improved.
A. D ETAILS OF THE IMPLEMENTATION
For our trusted multi-view classiﬁcation (TMC) algorithm, we employ the fully connected networks for all datasets, and l 2 -norm regularization is used with the value of the trade-off parameter being 0.0001.
The Adam (Kingma & Ba, 2014) optimizer is used to train the network.
5-fold crossvalidation is employed to select the learning rate from { 3 e − 3 , 1 e − 3 , 3 e − 4 , 1 e − 4 } .
For all datasets, 20% of samples are used as test set.
The hyperparameter λ t in Eq.
10 slowly increases from 0 to 1.
For all methods, we run 30 times to report the mean values and standard deviations.
A
[Equation] [Equation] ND
[Equation]
TO [Equation] END EXPERIMENT Although the effectiveness has been well validated on datasets with multiple types of features, we further conduct experiments in an end-to-end manner on multi-modal datasets.
The UMPC-FOOD dataset consists of 86,796 samples scraped from web pages, each sample is described with both image and text (Wang et al., 2015b).
Similarly to existing work (Kiela et al., 2019), there are 60, samples used as the training set, 5,000 samples used as the validation set, and the remaining 21, samples are used as the test set.
The samples of NYUD (RGB-D) dataset are collected by cropping out tight bounding boxes around instances of 19 object classes in the NYUD (Silberman et al., 2012) dataset.
There are 4,587 samples, and each sample is composed of both RGB and depth modalities.
We use 2,186, 1,200, and 1,201 samples as the training set, validation set, and test set respectively.
In this experiment, we used the original datasets which contain noise due to the collection process.
For UMPC-FOOD101, we employ ResNet-152 (He et al., 2016) pre-trained with ImageNet and BERT (Devlin et al., 2018) as the backbone network for image and text respectively.
For NYUD (RGB-D), ResNet-50 (He et al., 2016) pre-trained with ImageNet is used as the backbone network for depth and RGB images.
For comparison algorithms, we report the results of the best performing views, and furthermore, we concatenate the outputs of the two backbone networks as the input of the classiﬁer as shown in Fig.
For all algorithms, due to the randomness involved, we run 10 times to report the mean accuracy and standard deviation.
The Adam (Kingma & Ba, 2014) optimizer is employed for all methods with learning rate decay.
Quantitative experimental results.
The experimental results on the above datasets are shown in
Table 4.
We report the accuracy of each compared algorithm with the best-performing view (‘best view’ in Table 4) and combined views (‘feature fusion’ in Table 4).
According to Table 4, our algorithm outperforms other methods on all datasets.
For example, on the Food101 dataset, our method achieved 91 .
3% in terms of accuracy, compared with 90 . 5% from the second performer.
The performance of using the best single view is relatively low
[Figure]
[Table]
Qualitative experimental results.
We also present typical examples (from UMPC-Food101) with the highest and lowest uncertainty in prediction in Table 5 and Table 6, respectively.
Since the UMPCFood101 dataset is scraped from web pages, for a few samples the corresponding text descriptions are meaningless ( e.g. , the 1st one in Table 5).
At the same time, some images are also quite challenging to classify ( e.g. , the 2nd and the 4th in Table 5).
For these samples that can be considered as out-ofdistribution ones, it is difﬁcult to obtain evidence related to classiﬁcation.
As expected, these samples are assigned with high uncertainty (top 5) by our algorithm.
In contrast, Table 6 presents the topsamples with lowest uncertainty.
We can ﬁnd that for these samples the labels are usually explicitly mentioned in the text and the images also clearly show the characteristics of ‘macarons’.
[Table] [Table]