J OINT G AUSSIAN M IXTURE M ODEL FOR V ERSATILE D EEP V ISUAL M ODEL E XPLANATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Post-hoc explanations of deep neural networks improve human understanding on the learned representations, decision-making process and uncertainty of the model with faithfulness.
Explaining deep convolutional neural networks (DCNN) is especially challenging, due to the high dimensionality of deep features and the complexity of model inference.
Most post-hoc explaining methods serve a single form of explanation, restricting the diversity and consistency of the explanation.
This paper proposes joint Gaussian mixture model (JGMM), a probabilistic model jointly models inter-layer deep features and produces faithful and consistent post-hoc explanations.
JGMM explains deep features by Gaussian mixture model and inter-layer deep feature relations by posterior distribution on the latent component variables.
JGMM enables a versatile explaining framework that unifies interpretable proxy model, global or local explanatory example generation or mining.
Experiments are performed on various DCNN image classifiers in comparison with other explaining methods.
It shows that JGMM can efficiently produce versatile, consistent, faithful and understandable explanations.
I NTRODUCTION
Deep convolutional neural networks (DCNN) is a powerful type of machine learning model for visual recognition tasks.
The key reasons give rise to the power of DCNN include the expressive visual representations and the decision-making mechanism encoded in massive trainable convolution parameters.
However, increasing model complexity incurs heavier burden for human to understand learned representations and decision making of the model.
The high dimensionality and entanglement of deep features and the complexity of neural network inference are often considered the main hindrance to explaining black-box DCNN.
A recent proliferation of studies in post-hoc DCNN explainability show several effective and practical DCNN explaining methods.
Proxy models are interpretable models (e g. decision trees and linear models) that has approximate decision-making behavior as the black-box model inference.
Proxy models have inference process that can be intuitively understood by human, such as LIME (proposed byRibeiro et al.
(2016)), a linear classifier as a local proxy model.
To make sure the proxy model is an accurate surrogate, its faithfulness should be tested.
The proxy model’s predictions on unseen examples should be close to the black-box model’s prediction, even if different from the ground truth.
The intermediate representations of DCNN are usually explained globally (not associated with a specific data point) by explanatory examples and association with semantic concepts.
Prototypes, criticisms and influential examples are common types of global explanatory examples.
Prototypes are representative examples of a certain pattern of deep features, illustrating a learned visual concept of the model.
In contrast, criticisms (proposed by Kim et al.
(2016)) are examples not well-represented in the deep representations, i.e. outliers of the deep features, revealing the flaw of the learned representations.
Influential examples are hard examples for the model training, having more influence on the final decision boundary than others.
From a post-hoc view, influential examples lie close to the decision boundary.
An example can be both influential and representative (or unrepresentative).
Local explanations are based on a specific query example, showing how the change of the query example features will affect the model prediction.
Counterfactual examples offer an actionable re- course for the model decision.
Counterfactual examples answer the ’what if’ questions by minimal change of query example features and resulting different model decision against the query.
Counterfactual examples reveal the sensitive features for the query.
Semi-factual examples, in contrast, aim to answer the ’even if’ question.
Semi-factual examples have significant change on a certain feature(s) from the query, but both have the same model prediction.
Semi-factual examples reveal the insensitive features of the query.
For example, Kenny & Keane (2021) proposes a method to generate counterfactual and semi-factual examples from one system.
Most DCNN explaining methods are single-purpose systems.
Employing different explaining methods is possible to give diverse explanations, but it’s not guaranteed that different explanations are compatible and consistent with each other.
For example, a counterfactual example generation system may suggest that the model is sensitive to a certain feature; but a global explaining method, such as a proxy model, may have conflicts with the former explanation.
There is no hard rule to determine which explanation is correct or more understandable.
Thus a generic explaining framework that enables various and consistent explaining forms has important value for the explainability of
DCNN.
[Figure]
We propose a probabilistic framework for a versatile explaining method for DCNN.
Figure 1 demonstrates the pipeline of the framework and the enabled explanation forms.
The lower and the higher features are intermediate representations of DCNN from two different layers.
In the computational graph of DCNN, the higher feature is dependent on the lower feature and the black-box model (part of the DCNN) between them.
In an image classification setting, the higher feature can be the classification probabilistic prediction of the black-box model, and the lower features can be the input of the DCNN, i.e. the raw image.
The goal of the framework is to explain the learned representations of higher/lower features as well as the black-box model (either the whole DCNN or a part of DCNN) between them.
To serve this purpose, we propose joint Gaussian mixture model (JGMM) to jointly model the distribution of lower/higher features and produce a validatable proxy model and example-based global/local explanations.
JGMM is a probabilistic model based on GMM.
JGMM learns two Gaussian mixture model (GMM) respectively for the lower features X and higher features Y .
The latent categorical variables of both GMMs are connected by an estimated posterior probability matrix.
JGMM is introduced in section 3 in details.
Compared with single purpose explaining methods, the proposed JGMM-based versatile explaining method has two advantages: (1) various forms of model explanations are efficiently produced from one framework; (2) the consistency among different explanations is guaranteed, as they are computed from a common probabilistic model.
The proposed explaining method is evaluated with various DCNN models and benchmarks in comparison with other explaining methods in section 4.
The experiments show that JGMM can efficiently produce versatile, consistent, faithful and understandable explanations.
R ELATED W ORKS
Proxy models are widely leveraged method to produce model-agnostic explanations.
Local interpretable model-agnostic explanations (LIME) by Ribeiro et al.
(2016) is a typical method that learns a linear classifier by sampling from data points from a local region and the black-box classier.
For deep networks, Zilke et al.
(2016) propose to convert model parameters to a hard rule-set but keep it a computational approximation.
To explain the visual concepts encoded in DCNN and how the decision is made, Zhang et al.
(2018) and Zhang et al.
respectively design explanatory graphs and decision trees to explain the model inference.
Pixel attributing, or saliency mapping, is another useful technique to explain DCNN.
Compared with traditional feature attributing of machine learning models on tabular data, pixel attributing is specialized to the hierarchical structure of DCNN and visualizes the attention of the model.
Methods like CAM by Zhou et al.
Integrated Gradients by Sundararajan et al.
(2017), DeepLIFT ny
Shrikumar et al.
Grad-CAM ny Selvaraju et al.
(2017) utilize different heuristics to backpropagate the feature importance from deep layers to shallow layers.
Example-based explaining is a less explored field for deep visual model explanation.
Studies like
Goyal et al.
(2019), Kenny & Keane (2021) and Black et al.
(2021) generate counterfactual examples by simultaneous minimization of the feature distance after the query and maximization of model decision change.
Prototypes, the representative examples of a feature space cluster, are usually employed to explain a certain understandable visual concept learned by the model.
For example,
Nauta et al.
(2021) propose ProtoPNet to mine prototypical image patches and explain object parts.
Semi-factual examples (e.g. Kenny & Keane (2021)), criticisms (e.g. Kim et al.
and influential examples (e.g. Koh & Liang (2017)) can also help explaining the distribution of deep model features.
[Equation]
[Figure]
[Equation]
[Figure]
mixture model (JGMM) on lower feature X and higher feature Y .
We propose joint Gaussian mixture model, a probabilistic model base on GMM to estimate the posterior distribution connecting the latent variables behind the deep features.
Figure 2 illustrates the model in plate notation.
The model includes three parts: the GMM on observation Y , the conditioned GMM on observation X , and the probabilistic model between discrete latent variables z and w .
The model on observation Y resembles the classical GMM.
It includes the observed D y -dimension real-value higher features Y ∼ N ( ϕ w , Θ w ) .
Data point y n is the n -th of N sampled higher features.
The latent categorical variable w ∈ { 1 , ..., K y } decides which Gaussian component is Y generated from.
Hyper-parameter K y is the number of mixture components.
Prior probability π is a K y -D vector which represents the weight of each K y mixture component.
The model on observation X is also a Gaussian mixture model.
The observed D x -dimension lower features X is generated by the z -th Gaussian component: X ∼ N ( µ z , Σ z ) .
Latent categorical variable z ∈ { 1 , ..., K x } has hyper-parameter K x , which is the number of mixture components.
However, the key difference of our model from a traditional GMM is that the prior probability of z is dependent on posterior probability p ( z | w ) parameterized by Q .
The connection between latent categorical variables w and z is modeled by the posterior probability matrix Q ∈ R K x × K y .
The entry Q i,j is the conditional distribution of z , i.e. p ( z = i | w = j ) .
In figure 2, q k is the k -th column vector of Q , i.e. the distribution of z conditioned on w = k , whose sum is 1 .
By the chain rule of Bayesian inference, the likelihood function of JGMM is( G is Gaussian distribution density function):
[Equation]
The maximum likelihood estimation (MLE) problem of JGMM parameters can be solved by expectation-maximization (EM) algorithm.
The optimization algorithm can be straightforwardly derived from the EM algorithm framework, whose details are described in the appendix section A.
The complexity of the optimization is similar to separately learning two GMM for X and Y .
Equation 1 shows that the trained JGMM is still a GMM with ( K x K y ) components, preserving the desired properties of GMM.
Discussions below use the following notations for simplicity.
For a given observation x and y , g x ∈ R ( K x × 1) and g y ∈ R ( K y × 1) are probability density on each component, and
Ψ ∈ R ( K x × K y ) is the conditional distribution of w :
[Equation]
where ˆ Q , ˆ µ , ˆ Σ , ˆ ϕ , ˆ Θ and ˆ π are parameters estimated by MLE.
ˆ Q i, ∗ is the i -th row vector of ˆ Q .
Trained JGMM can work as a bi-directional Bayesian classifier to predict latent variable z and w .
Conditioned on a known x or y , the distribution of the other feature’s latent component variable w or z is:
[Equation]
Posterior probability in equation 3 is naturally a Bayesian classifier to predict the component category of the given example.
This enables a traditional training-test evaluation manner on the proxy model to evaluate the faithfulness of modelling.
JGMM also makes a handy example generator and miner to improve human understanding on the model by examples.
When the conditional distribution of w or z is computed by equation 3, JGMM on y or x degrades to a normal GMM, which is easy for sampling and interpretation.
3. M INING AND G ENERATING E XPLANATORY E XAMPLES
This section introduces how to mine or generate explanatory examples (prototypes, criticisms, influential examples, counterfactuals and semi-factuals) by a trained JGMM.
The example mining problem is to search for the best explanatory examples in the dataset, which requires a scoring function f ( x ) to evaluate the quality.
Generation problem is the sampling of data that maximize the scoring function.
We respectively define the scoring function for each type of explanatory examples and introduce the way to mine or generate examples.
Prototypes.
From a probabilistic aspect, a representative example x given w = j is at the center of the cluster, whose neighboring region has high cumulative probability on p ( x | w = j ) , which is a GMM.
So we define the prototype scoring function of the j -th higher feature component as equation 4.
[Equation]
Sampling prototypes for the j -th higher feature component is trivially sampling from multivariate
Gaussian distribution p ( x | w = j ) .
Criticisms.
Criticisms can be considered as outliers that is hardly represented by any prototypes.
So we search for examples that are least likely sampled from any w , which is the opposite of the prototype score, as equation 5.
[Equation]
Sampling criticisms is not supported in our method, because the realistic data distribution is often very sparse compared to the feature space.
Most sampled outliers are not understandable examples.
We only focus on criticisms found in realistic data.
Influential examples.
Influential examples are close to the decision boundary and can lead to model decision with low confidence.
From JGMM, we can measure the influence of an example by the entropy of the posterior probability of w conditioned on x .
Higher the entropy, higher the uncertainty and influence.
The scoring function is defined as equation 6.
[Equation]
Influential examples can be generated by sampling from each components of X and select those with largest f influ ( x ) .
Sampled influential examples, though not actually in the training data of the black-box, can be useful for exploring uncertain examples for the black-box model.
Counterfactuals.
A counterfactual example is close to the query example x query in the feature space but have very different posterior distribution on w .
Therefore, the scoring function for counterfactuals is defined as the probability that x and x query are sampled from the same z but different w :
[Equation]
To maximize the counterfactual score, there actually exists an optimal distribution of p ( z | x ) .
Let this distribution be α , the maximization of equation 7 is:
[Equation]
Note that all terms except α are constant, which makes it a linear programming problem with sum constraint.
Though it’s still hard to derive optimal x , the solution still provides an efficient sampling approach.
The approach to generate counterfactuals is to sample from the GMM under this optimal distribution α and select the best counterfactuals.
Semi-factuals.
A semi-factual example is likely sampled from a different z from the query example x query but have probably the same posterior distribution on w , which is the reverse of counterfactuals.
Therefore, the scoring function for counterfactuals is defined as equation 9.
[Equation]
Similar as the counterfactuals, the maximization of semi-factual score leads to an optimal distribution of p ( z | x ) .
We sample from the GMM of this optimal distribution and select the best semifactuals.
3. E XTENSION FOR DCNN
The deep feature maps of DCNN usually has additional spatial dimensions apart from the feature dimension, such as the time dimension of signal data and the vertical and the horizontal dimensions of 2D images.
The feature maps can be formulated with spatial encoding s ∈ S , i.e. X ( s ) .
For example, for a feature map of width W and height H , the spatial encoding is S
= { ( i, j ) | 1 ≤ i ≤ W, 1 ≤ j ≤ H } .
In later discussions, S and T respectively denote the spatial dimensions of X and Y .
The addition of spatial dimensions makes the latent variable relation changes from one-to-one to many-to-many, i.e. modelling each pair of { ( s, t ) | s ∈ S, t ∈ T } .
Theoretically, JGMM can be extended to such scenario by estimating each of Q ( s,t ) , and make GMM parameters µ , Σ , ϕ , Θ , π shared by every s ∈ S .
However, both E-step and M-step in the EM algorithm of JGMM requires enumeration at all possible latent variable assignment (see equation 13), where the combinatorial space ( K | S | x K | T | y elements) is so large that enumeration is infeasible.
However, most modern DCNN architectures have translation-invariant mapping of features, which means translation of lower features over spatial dimensions will cause the same translation (relative) of higher features.
Also, though the relative receptive field (the region on the lower features that affects a specific position of higher feature) of T on S can be very large, research by Luo et al.
(2016) shows that the central area (the effective receptive field) has main influence on the higher feature.
Therefore we can extend the proposed probabilistic model to a one-to-many relation.
Let this relative effective receptive field be R . R is a region cast the lower feature and connected to a point on higher feature. We train | R | posterior distribution matrix Q ( r ) between S and T .
The size of R is a hyper-parameter.
Also, the parameters µ and Σ are shared by all positions.
Figureshows the graphical model of JGMM extended for deep features with spatial dimensions.
The hyper-parameter | R | decides how many lower feature positions will affect the higher feature.
Like the setting of convolution hyper-parameters, the size of the effective receptive field should satisfy | S | + 1 − | T | ≥ | R | .
[Equation]
An important property of the extended JGMM is that X ( r ) is conditionally independent from other r ′ ∈ R on w :
[Equation]
So the inference of the extended JGMM can be decomposed to the inference on each of the | R |
JGMMs:
[Equation]
Therefore, the inference and sampling methods of JGMM are directly applicable to the extended
JGMM.
E XPERIMENTS 4. I MPLEMENTATION D ETAILS
JGMM implementation.
For a vanilla GMM, the covariance matrix contributes the most parameters to be estimated.
To reduce the complexity of JGMM training, we whiten the lower and higher features by PCA (principal component analysis).
All PCA components are kept to make the transformation invertible and all dimensions are uncorrelated with each other empirically.
Thus, training
JGMM can be less complex by estimating two diagonal covariance matrices, i.e. only the diagonal entries of Σ and Θ are estimated and others are constantly zero.
Data splitting.
To fairly test the faithfulness of JGMM, we only use the training data of the blackbox model to train JGMM and use the test data to evaluate JGMM’s accuracy in predicting the black-box model’s decision.
The effectiveness test of counter/semi-factual example generation is also performed with the test data.
But the explaining step (the generation of all kinds of explanatory examples) makes use of all data.
Features.
For a DCNN image classifier, the classification output (without softmax) is always selected as a higher feature to explain.
Because it directly links the understandable image classes with feature dimensions.
For models with large input image resolution, the feature map is very large and, because JGMM sample from each position, makes the training of JGMM difficult.
To balance the data size and training feasibility, the feature maps are average-pooled with 2 × 2 or 3 × 3 kernels to shrink the feature map.
Following JGMM training and explaining on the pooled features.
Hyper-parameters.
The training of JGMM requires two hyper-parameters: the number of components of the lower and the higher features.
The component number of the lower feature is chosen by validation performance on the validation dataset (split from the training data).
When the higher feature is the classification output of the model, the number of components is set to the number of classes; otherwise, the higher feature has already been modelled by a JGMM where it’s the lower feature, and the number of component is inherited from the last JGMM.
The effective receptive field size is as large as the computation resource allows to capture as many positional information as possible.
But when | R | is too large, the training of JGMM will be unacceptably slowed down.
We set the effective receptive field size from 2 × 2 to 5 × 5 .
4. E XPERIMENTAL S ETTINGS
Networks and data.
The experiments are performed on VGG-16/19(Simonyan & Zisserman (2015)), ResNet-18/50(He et al.
and MobileNet-V1/V2(Sandler et al.
(2018)), respectively trained on MNIST (LeCun et al.
(1998)), CIFAR-10(Krizhevsky (2009)) and CUB-200(Wah et al.
The dataset split is the same as the official split.
We choose three feature maps from each network, named as ’high’, ’middle’ and ’low’ in the tables and graphics below.
In experiments, we train JGMM with 32 components on middle features and 128 components on low features for
MNIST; 64 components on middle features and 256 components on low features for CIFAR-10;components on middle features and 1024 components on low features for CUB-200.
Faithfulness evaluation.
We quantitatively evaluate the faithfulness of JGMM and other interpretable proxy models by classification accuracy on the test dataset (note that the ground truth label for this evaluation is the prediction of the black-box model).
As equation 3 shows, JGMM can predict higher feature latent variable distribution given observation of the lower feature.
The GMM of the higher feature is dependent on JGMM, which is a biased prediction target, so we only choose the ’high’ feature map (which is associated with black-box model classification output) for faithfulness evaluation.
By associating each component of the ’high’ feature neuron with a dataset class, the
JGMM between the ’middle’ and the ’high’ layers can directly predict class label with uncertainty.
The counterpart interpretable proxy models include decision tree (DT), logistic regression (LR), k-nearest neighbor (kNN) and Gaussian process classifier (GP).
Counterfactuals quality.
The quality of counterfacutal examples is evaluated by the ratio of prediction change from the query input (higher is better) and the distance between the query input and the generated data (lower is better).
To mitigate the conflict between the two metrics, we set different constraints on the l 2 distance between query feature and the generated counterfacutals.
For any tested methods, a generated counterfacutal is rejected if it breaks the constraint.
The examples in the test dataset are queried, and we randomly sample 1000 data points by the local explanatory example sampling method in section 3.2.
JGMM is compared with MADWachter et al.
(2017), DiCEMothilal et al.
(2020), Proto-GuidedLooveren & Klaise (2021) and
K-d TreeGoyal et al.
The evaluation metric is the counterfactual effectiveness, i.e. the ratio of generated counterfactual examples that have different black-box prediction from the queried example, under a specific distance constraint.
4. G LOBAL E XPLANATIONS
5w=
j       z = i
P(z=i|w=
j) 5w=
j       z
= i
P(w= j|z =i)
[Figure]
Ψ z= z= z= z= z=
Prototypes
Criticisms 7vs 0vs 5vs
9vs 7vs 9vs 4vs 1vs
Influentials
[Figure]
[Table]
z= z= z= z=
Influential
[Table]
[Figure]
Table 1 presents proxy model accuracy of JGMM and other interpretable proxy models on various datasets and networks.
It shows that JGMM is highly faithful to the black-box, which increases the credibility of the explanations.
We also find that, if the black-box classifier is tasked with more classes, the accuracy of proxy models is lower.
This is mainly because the black-box model tends to encode more complex deep features, but the fitting efficiency of interpretable models is often not comparable with the deep network. Figure 4 presents the estimated posterior Q and the derived Ψ of JGMM and global explanatory examples mined from the test dataset. The visualization of Q suggests that each latent component z of the lower feature is mainly mapped to a single higher feature component (though noisy). The visualization of Ψ illustrates that each higher feature component (linked to digit class label) has multiple patterns of examples.
As observed from Ψ , we can find that components z ∈ { 2 , 18 , 19 } are learned patterns for digit ”4”, and z ∈ { 4 , 14 } are learned patterns for digit ”1”.
The prototypes of those components are visualized.
It can be found that each latent component z is encoded with a certain style of writing.
The mined criticisms in figure 4 can hardly be represented by any latent component, which is intuitively true by the examples and technically evaluated by equation 5.
The influentials are presented with two most likely p ( w = j | x ) in equation 6.
For example, the first influential example can probably be classified as ’7’ and ’9’ with similar posterior probability of w .
Figure 5 presents global explanatory examples mined from ResNet-18 trained on CUB-200 with bird species classification task.
The ’middle’ feature map has encoded different visual concepts into the deep representations, which is revealed by the prototypes, such as z = 7 ’long pointed beak’, z = 16 ’yellow body part’, z = 26 ’branches texture’ and z = 82 ’red body part’.
An influential example x has close posterior probability on w = 5 and w = 48 .
The bird in the influential example has red feather at the top of the head and pointed beak, but the shapes of the red feather and the beak are not typical in either w = 5 or w = 48 , which makes it near the decision boundary.
Criticisms of CUB-200 are often rarely observed sub-species or whose image features are not sufficient for discrimination.
4. L OCAL E XPLANATIONS
[Table]
[Figure]
[Figure]
In table 2, we test the counterfactual generation effectiveness of JGMM and other methods on
MobileNet-v2 trained on CUB-200, with l 2 constraint of feature distance varying from 0 .
1 to 3 . 0 .
JGMM’s sampling method shows higher counterfactual generation effectiveness than the counterparts, especially in low distance-constraint settings.
An advantage of JGMM on counterfactual effectiveness is that we can estimate the distance between the generated example and the queried example and reject too distant examples by equation 7, which significantly reduce the number of distant examples.
Figure 6 presents query images for CIFAR-10 and MNIST as well as some mined counterfacutals and semi-factuals.
For example, the first CIFAR-10 image query is a side view of a horse, but many deer images have very similar visual features as the query.
So most mined counterfacutals are deers viewed from one side, which has different label prediction as the query.
The third CIFAR-10 query is a flying bird viewed from the top, which can be easily confused (similar deep features) with a plane, such as the counterfacutals.
The semi-factuals provide examples of the same prediction but have very different deep features from the query.
The second query of MNIST also demonstrates how the perturbation of deep features will change the prediction.
The query image (the black-box model and JGMM predict ’6’) can be confused with ’4’ counterfacutals if the last draw is slightly longer.
In contrast, other ’6’ semi-factuals have very different latent encoding.
C ONCLUSIONS
In this paper joint Gaussian mixture model is proposed to explain the deep features and model inference of black-box DCNN.
As a probabilistic model that links two deep features, JGMM enable consistent and diverse explanation forms, including proxy models and example-based explaining.
Experiments show that JGMM can faithfully interpret model inference and efficiently mine or generate explanatory examples.
The computational efficiency of JGMM worth improvement, especially with large dataset and higher feature dimensionality, as training with EM algorithm in such scenarios is still very time and memory costly.
A EM A LGORITHM FOR JGMM
The maximum likelihood estimation (MLE) problem of JGMM is the optimization on the loglikelihood of N observations { x 1 , y 1 , ..., x N , y N } : [Equation] [Equation]
[Equation]
Let the parameters at the t -th iteration step be:
[Equation]
The expectation step (E-step) involves the estimation of the latent variables based on θ ( t ) .
Let g ( t ) n,i and h ( t ) n,j respectively be the expectation of the latent variable p ( z n = i ) and p ( w n = j ) at the t -th iteration.
They computed by:
[Equation]
The maximization step (M-step) maximize the likelihood function by updating the estimation θ ( t ) to θ ( t +1) .
The update of Gaussian parameters is:
[Equation]
[Equation]
The optimization of latent distribution parameters Q and π are:
[Equation]
As above, the E-step and the M-step of the EM algorithm for JGMM are defined.
We first initialize
Q and π by uniform distribution parameters.
Then, we cluster the data by K-means ( K x clusters for
X and K y clusters for Y ) and initialize Gaussian parameters by cluster sample mean and covariance.
Last, the algorithm execute E-step and M-step until convergence.