L EARNING TO E STIMATE S HAPLEY V ALUES WITH V ISION T RANSFORMERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Transformers have become a default architecture in computer vision, but understanding what drives their predictions remains a challenging problem.
Current explanation approaches rely on attention values or input gradients, but these provide a limited view of a model’s dependencies.
Shapley values offer a theoretically sound alternative, but their computational cost makes them impractical for large, high-dimensional models.
In this work, we aim to make Shapley values practical for vision transformers (ViTs).
To do so, we first leverage an attention masking approach to evaluate ViTs with partial information, and we then develop a procedure for generating Shapley value explanations via a separate, learned explainer model.
Our experiments compare Shapley values to many baseline methods (e.g., attention rollout, GradCAM, LRP), and we find that our approach provides more accurate explanations than existing methods for ViTs.
I NTRODUCTION
Transformers (Vaswani et al., 2017) were originally introduced for NLP, but in recent years they have been successfully adapted to a variety of other domains (Wang et al., 2020; Jumper et al., 2021).
In computer vision, transformer-based models are now used for problems including image classification, object detection and semantic segmentation (Dosovitskiy et al., 2020; Touvron et al., 2021; Liu et al., 2021), and they achieve state-of-the-art performance in many tasks (Wortsman et al., 2022).
The growing use of transformers in computer vision motivates the question of what drives their predictions: understanding a complex model’s dependencies is an important problem in many applications, but the field has not settled on a solution for the transformer architecture.
Transformers are composed of alternating self-attention and fully-connected layers, where the selfattention operation associates attention values with every pair of tokens.
In vision transformers (ViTs) (Dosovitskiy et al., 2020), the tokens represent non-overlapping image patches, typically a total of 14 × 14 = 196 patches each of size 16 × 16 .
It is intuitive to view attention values as indicators of feature importance (Abnar and Zuidema, 2020; Ethayarajh and Jurafsky, 2021), but interpreting transformer attention in this way is potentially misleading.
Recent work has raised questions about the validity of attention as explanation (Serrano and Smith, 2019; Jain and Wallace, 2019; Chefer et al., 2021), arguing that it provides an incomplete picture of a model’s dependence on each token.
If attention is not a reliable indicator of feature importance, then what is?
We consider the perspective that transformers are no different from any other architecture, and that we can explain their predictions using model-agnostic approaches that are currently used for other architectures.
Among these methods, Shapley values are a theoretically compelling approach with feature importance scores that are designed to satisfy many desirable properties (Shapley, 1953; Lundberg and Lee, 2017).
The main challenge for Shapley values in the transformer context is calculating them efficiently, because a naive calculation has exponential running time in the number of patches.
If Shapley values are poorly approximated, they are unlikely to reflect a model’s true dependencies, but calculating them with high accuracy is currently too slow to be practical.
Thus, our work aims to make Shapley values practical for transformers, and for ViTs in particular.
Our contributions include:
We investigate several approaches for withholding input features from vision transformers, which is a key operation for computing Shapley values.
We find that ViTs can accommodate missing image patches by masking attention values for held-out tokens, and that training with random masking is important for models to properly handle partial information.
We develop a learning-based approach to estimate Shapley values efficiently and accurately.
Our approach involves fine-tuning an existing ViT using a loss function designed specifically for Shapley values (Jethani et al., 2021b), and we prove that our loss bounds the estimation error without requiring ground truth values during training.
Once trained, our explainer model provides a significant speedup over methods like KernelSHAP (Lundberg and Lee, 2017).
Our experiments compare the Shapley value-based approach to several groups of competing methods: attention-based, gradient-based and removal-based explanations.
We find that our approach provides the best overall performance, correctly identifying influential and noninfluential features for both target and non-target classes.
We verify this using one natural image dataset and one medical image dataset, and the results are consistent across multiple metrics.
Our work shows that Shapley values can be made practical for vision transformers, and that they provide a compelling alternative to current attention- and gradient-based approaches.
R ELATED WORK
Understanding neural network predictions is a challenging problem that has been actively researched for the last decade (Simonyan et al., 2013; Zeiler and Fergus, 2014; Ribeiro et al., 2016).
We focus on feature attribution , or identifying the specific input features that influence a prediction, but prior work has also considered other problems (Olah et al., 2017; Kim et al., 2018).
The various techniques that have been developed can be grouped into several categories, which we describe below.
Attention-based explanations Transformers use self-attention to associate weights with each pair of tokens (Vaswani et al., 2017), and a natural idea is to assess which tokens receive the most attention (Clark et al., 2019; Rogers et al., 2020; Vig et al., 2020).
There are several versions of this approach, including attention rollout and attention flow (Abnar and Zuidema, 2020), which analyze attention across multiple layers.
Attention is a popular interpretation tool, but it is only one component in a sequence of nonlinear operations that provides an incomplete picture of a model’s dependencies (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), and direct usage of attention weights has not been shown to perform well in vision tasks (Chefer et al., 2021).
Gradient-based methods For other deep learning models such as CNNs, gradient-based explanations are a popular family of approaches.
There are many variations on the idea of calculating input gradients, including methods that modify the input (e.g., SmoothGrad, IntGrad) (Smilkov et al., 2017; Sundararajan et al., 2017; Xu et al., 2020), operate at intermediate network layers (GradCAM)
(Selvaraju et al., 2017), or design modified backpropagation rules (e.g., LRP, DeepLift) (Bach et al., 2015; Shrikumar et al., 2016; Chefer et al., 2021).
Although they are efficient to compute for arbitrary network architectures, gradient-based explanations achieve mixed results in quantitative benchmarks, including for object localization and the removal of influential features (Petsiuk et al., 2018;
Hooker et al., 2019; Saporta et al., 2021; Jethani et al., 2021b), and they have been shown to be insensitive to the randomization of model parameters (Adebayo et al., 2018).
Removal-based explanations Finally, removal-based explanations are those that quantify feature importance by explicitly withholding inputs from the model (Covert et al., 2021).
For models that require all features to make predictions, several options for removing features include setting them to default values (Zeiler and Fergus, 2014), sampling replacement values (Agarwal and Nguyen, 2020) and blurring images (Fong and Vedaldi, 2017).
These methods work with any model type, but they tend to be slow because they require making many predictions.
The simplest approach of removing individual features (also known as leave-one-out , Ethayarajh and Jurafsky 2021) is relatively fast, but the computational cost increases as we examine more feature subsets.
Shapley values (Shapley, 1953) are an influential approach within the removal-based explanation framework.
By examining all feature subsets, they provide a nuanced view of each feature’s influence and satisfy many desirable properties (Lundberg and Lee, 2017).
They are approximated in practice using methods like TreeSHAP and KernelSHAP (Lundberg et al., 2020; Covert and Lee, 2021), but current approaches either are not applicable or do not scale to large ViTs.
Recent work highlighted a connection between attention flow and Shapley values (Ethayarajh and Jurafsky, 2021), but this approach is fundamentally different from SHAP (Lundberg and Lee, 2017): attention flow treats each feature’s influence on the model as strictly additive, which is computationally convenient but fails to represent feature interactions.
Our work instead focuses on the original formulation (Lundberg and
Lee, 2017) and aims to make Shapley values based on feature removal practical for ViTs.
B ACKGROUND
Here, we define notation used throughout the paper and briefly introduce Shapley values.
3. N OTATION
Our focus is vision transformers trained for classification tasks, where x ∈ R 224 × 224 × 3 denotes an image and y ∈ { 1 , . .
, K } denotes the class.
We write the image patches as x = ( x 1 , . . .
, x d ) , where ViTs typically have x i ∈ R 16 × 16 × 3 and d = 196 .
The model is given by f ( x ; η ) ∈ [0 , 1] K and f y ( x ; η ) ∈ [0 , 1] represents the probability for the y th class.
Shapley values involve feature subsets, so we use s ∈ { 0 , 1 } d to denote a subset of indices and x s = { x i : s i = 1 } a subset of image patches.
We also use 0 and 1 ∈ R d to denote vectors of zeros and ones, and e i ∈
R
d is a vector with a one in the i th position and zeros elsewhere.
Finally, bold symbols x , y are random variables, x, y are possible values, and p ( x , y ) denotes the data distribution.
3. S HAPLEY VALUES
Shapley values were developed in game theory for allocating credit in coalitional games (Shapley, 1953).
A coalitional game is represented by a set function, and the value for each subset indicates the profit achieved when the corresponding players participate.
Given a game with d players, or a set function v : { 0 , 1 } d (cid:55)→ R , the Shapley values are denoted by ϕ 1 ( v ) , . . .
, ϕ d ( v ) ∈ R for each player, and the value ϕ
i ( v ) for the i th player is defined as follows:
[Equation]
Intuitively, eq.
(1) represents the change in profit from introducing the i th player, averaged across all possible subsets to which i can be added.
Shapley values are defined in this way to satisfy many reasonable properties: for example, the credits sum to the value when all players participate, players with equivalent contributions receive equal credit, and players with no contribution receive zero credit (Shapley, 1953).
These properties make Shapley values attractive in many settings: they have been applied with coalitional games that represent a model’s prediction given a subset of features (SHAP) (Štrumbelj and Kononenko, 2010; Lundberg and Lee, 2017), as well as several other use-cases in machine learning (Ghorbani and Zou, 2019; 2020; Covert et al., 2020).
There are two main challenges when using Shapley values to explain individual predictions (Chen et al., 2022).
The first is properly withholding feature information, and we explore how to address this challenge in the ViT context (Section 4).
The second is calculating Shapley values efficiently, because their computation scales exponentially with the number of inputs d .
Traditionally, they are approximated using sampling-based estimators like KernelSHAP (Castro et al., 2009; Štrumbelj and Kononenko, 2010; Lundberg and Lee, 2017), but we build on a more efficient learning-based approach (FastSHAP) recently introduced by Jethani et al.
(2021b)
(Section 5).
E VALUATING VISION TRANSFORMERS WITH PARTIAL INFORMATION
The basic idea behind Shapley values, as well as other removal-based explanations (Covert et al., 2021), is to evaluate the model with partial feature information and analyze how a prediction changes.
Most models need values for all the features to make predictions, so in practice we require a mechanism to represent feature removal.
For example, we can set held-out image regions to zero, or we can average the prediction across randomly sampled replacement values.
With vision transformers, the options for removing features are slightly different.
Recent work has demonstrated the robustness of ViTs to randomly zeroed pixel values (Naseer et al., 2021), but the self-attention operation enables a more elegant approach: we can simply ignore tokens for image patches we wish to remove (Jain et al., 2021).
We achieve this by masking attention values at each self-attention layer, or setting them to a large negative value before applying the softmax operation (see Appendix A).
This resembles causal attention masking in transformer language models such as
GPT-3 (Brown et al., 2020), but we use masking for a different purpose.
Alternatively, we could use a unique token value as in masked language models such as BERT (Devlin et al., 2018), which would involve simply setting held-out tokens to the mask value.
Using our attention masking approach, we can evaluate a ViT model f
( x ; η ) given subsets of image patches, denoted by x s .
However, the predictions with partial information may not behave as desired for models that were not trained with held-out tokens, because these partial inputs represent offmanifold examples.
We have two options to correct this: 1) we can ensure that the original model is trained with random masking, or 2) we can fine-tune the model to preserve its predictions with full images while encouraging sensible behavior with missing patches.
The first option is more direct, but it does not allow us to explain models trained without masking.
For the latter option, we can create an updated model denoted by g ( x s ; β ) that we fine-tune using the following loss,
[Equation]
where p ( s ) is a distribution over subsets.
In practice, we sample the cardinality m = 1 ⊤ s from m ∼ Unif (0 , d ) and then sample m patches uniformly at random.
Intuitively, eq.
(2) encourages g ( x s ; β ) to preserve the original model’s predictions even with missing features.
We use this loss because it satisfies the desirable property that the optimal model g ( x s ; β ∗ ) outputs the expected prediction given the available information (Covert et al., 2021), or g
[Equation]
s ;
[Equation] [Equation]
[Equation]
Note that this represents a best-effort prediction, because if f ( x ; η ) = p ( y | x ) then we have g ( x s ; β ∗ )
= p ( y | x s ) .
Similarly, in the case where f ( x s ; η ) is trained directly with random masking, the training process estimates f ( x s ; η ) ≈ p
( y | x s ) (see Appendix B).
We refer to the fine-tuned model g ( x s ; β ) as a surrogate , following the naming in prior work (Frye et al., 2020).
Whether we use the original model or a version fine-tuned with random masking, our attention masking approach enables us to probe how individual predictions change as we remove groups of image patches.
L EARNING TO ESTIMATE S HAPLEY VALUES
Given our approach for evaluating ViTs with partial information, we can use Shapley values to identify influential image patches for an input x and class y .
This involves evaluating the model with many feature subsets x s , so we define a coalitional game v xy ( s ) =
g y ( x s ; β ) .
Alternatively, if we use a model trained with masking, we can define the coalitional game as v xy ( s ) =
f y ( x s ; η ) .
Common Shapley value approximations are based on sampling feature permutations (Castro et al., 2009; Štrumbelj and Kononenko, 2010) or fitting a weighted least squares model (Lundberg and Lee, 2017; Covert and Lee, 2021), but these can require hundreds or thousands of model evaluations to explain a single prediction.
1 Instead, we develop a learning-based estimation approach.
Our goal is to obtain an explainer model that estimates Shapley values directly.
To do so, we train a new vision transformer ϕ ViT ( x , y ; θ ) ∈ R d that outputs approximate Shapley values for an inputoutput pair ( x, y ) in a single forward pass.
Crucially, rather than training the model using a dataset of ground truth Shapley value explanations, we train it by minimizing the following objective,
[Equation]
where p Sh ( s ) is a distribution defined as p Sh ( s ) ∝ ( 1 ⊤ s − 1)!
( d − 1 ⊤ s − 1)!
for 0 < 1 ⊤ s < d and p
Sh ( 1 ) = p
Sh ( 0 ) = 0 .
Intuitively, eq.
(4) encourages the explainer model to output feature scores that provide an additive approximation for the predictions with partial information, where the predictions are represented by v xy ( s ) and the additive approximation by v xy ( 0 ) + s ⊤ ϕ ViT ( x , y ; θ ) .
The loss in eq.
(4) was introduced by Jethani et al.
(2021b)
and is derived from an optimization-based characterization of the Shapley value (Charnes et al., 1988).
To rigorously justify this training approach, we derive new results that show how this objective controls the Shapley value estimation error.
Proofs are in Appendix D. First, we show that the explainer’s loss for a single input is strongly convex in the prediction, a result that implies the existence of unique optimal predictions.
Lemma 1.
For a single input-output pair ( x, y ) , the expected loss under eq.
(4) for the prediction ϕ ViT ( x, y ; θ ) is µ -strongly convex with µ
= H − 1 d − 1 , where H d − 1 is the ( d − 1) th harmonic number.
[Figure]
Next, we utilize the strong convexity property from Lemma 1 to prove our main result: that the explainer model’s loss function upper bounds the distance between the exact and approximated
Shapley values.
This is notable because we do not utilize ground truth values during training.
Theorem 1.
For a model ϕ ViT ( x , y ; θ ) whose predictions satisfy the constraint in eq.
(4) , the objective value L ( θ ) upper bounds the Shapley value estimation error as follows,
[Equation]
where L ∗ represents the loss achieved by the exact Shapley values.
This shows that our objective is a viable approach for training without exact Shapley values, because optimizing eq.
(4) minimizes an upper bound on the estimation error.
In other words, if we can iteratively optimize the explainer model so that its loss approaches the optimum obtained by the exact
Shapley values ( L ( θ ) → L ∗ ), our estimation error will go to zero.
In practice, we train the explainer model ϕ ViT ( x , y ; θ ) using stochastic gradient descent, and several other steps are important during training.
First, we normalize the explainer’s unconstrained predictions in order to satisfy the objective’s constraint in eq.
(4); this ensures that the Shapley value’s efficiency property holds (Shapley, 1953).
Next, rather than training the explainer from scratch, we fine-tune an existing model that can be either the original classifier or a ViT pre-trained on a different supervised or self-supervised learning task (Touvron et al., 2021; He et al., 2021); ViTs are more difficult to train than convolutional networks, and we find that fine-tuning is important to train effectively (Table 3).
Finally, we simplify the architecture by estimating Shapley values for all classes simultaneously.
Our training approach is described in more detail in Appendix C.
By using a ViT to estimate Shapley values, we leverage the model’s internal representation to produce similar explanations for similar inputs.
And by fine-tuning an existing model, we allow the explainer to re-use visual features that were informative for other challenging tasks.
Ultimately, the explainer cannot guarantee exact Shapley values, but no approximation algorithm can; instead, it offers a favorable trade-off between accuracy and efficiency, and we find empirically that this approach offers a powerful alternative to the methods currently used for ViTs.
[Figure]
i
[Figure]
E XPERIMENTS
We now demonstrate the effectiveness of our approach, termed ViT Shapley .
First, we evaluate attention masking for handling held-out patches in ViTs (Section 6.1).
Next, we compare explanations from ViT Shapley to several existing methods.
Our baselines include attention-, gradient- and removal-based explanations, and we compare these methods via several metrics for explanation quality, including insertion/deletion of important features (Petsiuk et al., 2018), sensitivity-n (Ancona et al., 2018), faithfulness (Bhatt et al., 2021) and ROAR (Hooker et al., 2019)
(Section 6.2).
Our experiments are based on two image datasets.
We use ImageNette, a natural image dataset consisting of ten ImageNet classes (Howard and Gugger, 2020; Deng et al., 2009), and MURA, a medical image dataset of musculoskeletal radiographs classified as normal or abnormal (Rajpurkar et al., 2017).
ImageNette contains 13,394 examples and ten classes, and MURA contains 40, examples and two classes; see Figure 1 for examples from each dataset.
We use ViT-Base models (Wightman, 2019) as classifiers for both datasets.
6. E VALUATING IMAGE PATCH REMOVAL
Our initial experiments test whether attention masking is effective for handling held-out image patches.
We fine-tuned the classifiers for both datasets following the procedure described in Section 4, and we also tested several approaches without performing any fine-tuning: attention masking, attention masking applied after the softmax operation (how dropout is often implemented for ViTs, Wightman 2019), setting input patches to zero (Naseer et al., 2021), setting token embeddings to zero, and replacing with random patches from the dataset.
Finally, we performed identical fine-tuning while replacing input patches with zeros, which is equivalent to introducing a fixed mask token.
As a measure of how well missing patches are handled, we calculated the KL divergence relative to the full-image predictions as random patches are removed; this can be interpreted as a divergence measure between the masked predictions and the predictions with patches marginalized out (see Appendix B), or how close we are to correctly removing patch information.
The metric is calculated on the test set with randomly generated patch subsets, and it represents whether the model can make reasonable predictions given partial inputs.
Similar results using top-1 accuracy are in Appendix H.
Figure 2 shows the results.
Most methods perform well with < 25% of patches missing, leading to only small increases in KL divergence.
This is especially true for ImageNette, where large objects make the model more robust to missing patches.
However, the methods with no fine-tuning begin to diverge as larger numbers of patches are removed and the partial inputs become increasingly off-manifold.
Fine-tuning becomes necessary to avoid poor performance as more patches are removed.
For both datasets, we find that fine-tuning with either attention masking or input patches set to zero provide comparable performance, and that these perform best across all numbers of patches.
This means that fine-tuning makes attention masking significantly more effective for marginalizing out missing patches, and these results suggest that training ViTs with held-out tokens may be necessary to enable robustness to partial information.
As prior work suggests, properly handling held-out information is crucial for generating informative explanations (Frye et al., 2020; Covert et al., 2021), so the remainder of our experiments proceed with the fine-tuned attention masking approach.
[Table]
6. E VALUATING EXPLANATION ACCURACY
Next, we implemented ViT Shapley by training explainer models for both datasets.
We used the finetuned classifiers from Section 6.1 to handle partial information
[Table]
and we used the ViT-Base architecture with extra output layers to output Shapley values for all patches.
The explainer models were trained by optimizing eq.
(4) using using stochastic gradient descent (see details in Appendix C), and once trained, the explainer outputs approximate Shapley values in a single forward pass (Figure 1).
As comparisons for ViT Shapley, we considered a large number of baselines.
For attention-based methods, we use attention rollout and the last layer’s attention directed to the class token
(Abnar and Zuidema, 2020).
Similar to prior work (Chefer et al., 2021), we did not use attention flow due to the computational cost.
Next, for gradient-based methods, we use Vanilla Gradients (Simonyan et al., 2013),
Integrated Gradients (Sundararajan et al., 2017), SmoothGrad (Smilkov et al., 2017),
VarGrad (Hooker et al., 2019), LRP (Chefer et al., 2021) and GradCAM (Selvaraju et al., 2017).
We used existing LRP and GradCAM implementations for ViTs (Chefer et al., 2021), and the remaining gradient-based methods were run using the Captum package (Kokhlikyan et al., 2020).
Finally, for removal-based methods, we use the leave-one-out approach (Zeiler and Fergus, 2014) and RISE (Petsiuk et al., 2018), which we implemented from scratch.
Appendix F describes the baselines in more detail, including how several methods were modified to provide patch-level explanations, and
Appendix H summarizes the running time for each method.
Given our set of baselines, we used several metrics to evaluate ViT Shapley.
Evaluating explanation accuracy is difficult when the true importance is not known a priori, so we rely on metrics that test how removing (un)important features affects a model’s predictions.
Intuitively, removing influential features for a particular class should reduce the class probability, and removing non-influential features should not affect or even increase the class probability.
Removal-based explanations are implicitly related to such metrics, but attention- and gradient-based methods may be hoped to provide strong performance with lower computational cost.
First, we implemented the widely used insertion and deletion metrics (Petsiuk et al., 2018).
For these, we generate predictions while inserting/removing features in order of most to least important, and we then evaluate the area under the curve of prediction probabilities (see Figure 1).
Here, we average the results across 1,000 images for their true class.
We use random test set images for ImageNette, and for MURA we use test examples that were classified as abnormal because these are more important in practice.
When removing information, we use the fine-tuned classifier because this represents the closest approximation to truly removing information from the model (Section 4).
Table 1 displays the results, and we find that ViT Shapley offers the best performance on both datasets.
RISE and LRP tend to be the most competitive baselines, and perhaps surprisingly, certain other methods fail to outperform a random baseline (GradCAM, SmoothGrad, VarGrad).
The baselines are sometimes competitive with ViT Shapley on insertion, but the gap for the deletion metric is larger.
[Figure]
C o rr e l a t i o n
ImageNette
[Figure]
Practically, this means that ViT Shapley identifies important features that quickly drive the prediction towards a given class, and that quickly reduce the prediction probability when deleted
[Figure]
[Table]
Next, we modified these metrics to address a common issue with model explanations: that their results are not specific to each class (Rudin, 2019).
ViT Shapley produces separate explanations for each class, so it can identify relevant patches even for nontarget classes (see Figure 1).
Tableshows insertion/deletion results averaged across all non-target classes for ImageNette.
The attention-based methods do not produce class-specific explanations, and the remaining baselines generally provide poor results. Empirically, this is because the explanations are often similar across classes (see Appendix I).
ViT Shapley performs best, particularly on insertion, and RISE is the best-performing baseline.
Appendix H shows results for MURA, as well as the curves used to calculate these results.
The insertion and deletion metrics only test importance rankings, so we require other metrics to test the specific attribution values.
Sensitivity-n (Ancona et al., 2018) was proposed for this purpose, and it measures whether attributions correlate with the impact on a model’s prediction when a feature is removed.
The correlation is typically calculated across subsets of a fixed size, and then averaged across many predictions.
Faithfulness (Bhatt et al., 2021) is a similar metric where the correlation is calculated across subsets of all sizes.
Table 1 and Table 2 show faithfulness results.
Among the baselines, RISE and LRP remain most competitive, but ViT Shapley again performs best for both datasets.
Figure 3 shows sensitivity-n results calculated across a range of subset sizes.
Leave-one-out naturally performs best for large subset sizes, but ViT Shapley performs the best overall, particularly with smaller subsets.
The sensitivity-n results focus on the target class, but Appendix H shows results for non-target classes where ViT Shapley’s advantage over many baselines (including LRP) is even larger.
Finally, we performed an evaluation inspired by ROAR (Hooker et al., 2019), which tests how a model’s accuracy degrades as important features are removed.
ROAR suggests retraining with masked inputs, but this is unnecessary here because the fine-tuned classifier is designed to handle held-out patches.
We therefore generated multiple versions of the metric.
First, we evaluated accuracy while using the fine-tuned classifier to handle masked patches.
Second, we repeated the evaluation using a separate evaluator model trained directly with held-out patches, similar to EVAL-X
(Jethani et al., [Figure] 2021a).
Third, we performed masked retraining as described by ROAR.
The first version represents the original classifier’s best-effort prediction [Figure] and the second is a best-effort prediction disconnected from the original model; masked retraining is similar, but the retrained model can exploit information communicated by the masking, such as the shape and position of the removed object.
Figure 4 shows the results when removing important patches.
ViT Shapley consistently outperforms the baselines across the first two versions of the metric, yielding faster degradation when important patches are removed.
ViT Shapley also performs best when inserting important patches, yielding a faster increase in accuracy (Appendix H).
It is outperformed by several baselines with masked retraining in the deletion direction (Figure 4 bottom left), but we suspect this is due to spatial information leaked by ViT Shapley’s deleted patches; indeed, when we retrained without positional embeddings , we found that ViT Shapley achieved the fastest degradation with a small number of deleted patches (Figure 4 bottom right).
Interestingly, positional embeddings in ViTs offer a unique approach to alleviate ROAR’s known information leakage issue (Jethani et al., 2021a).
In addition to these experiments, we observed consistent results when replicating all metrics using
ViT-Small and a ViT classifier trained directly with random masking (Appendix H).
We also generated metrics comparing ViT Shapley’s approximation quality with KernelSHAP (Lundberg and Lee, 2017), and we found that ViT Shapley’s accuracy is equivalent to running KernelSHAP for roughly 120, model evaluations (Appendix H).
Lastly, we provide qualitative examples in Appendix I, including comparisons with the baselines and with KernelSHAP.
Overall, these results show that ViT Shapley is a practical and compelling approach for explaining ViT predictions.
C ONCLUSION
In this work, we developed a learning-based approach to generate Shapley value explanations for vision transformers.
Our approach involves training a separate explainer model using a loss function that circumvents the need for ground truth Shapley values, and ViT Shapley outperforms a variety of attention- and gradient-based methods across a range of accuracy metrics.
Our experiments provide empirical evidence for the potential for Shapley values as an alternative to attention-based approaches, which remain popular for transformer models.
Future directions involve extending ViT Shapley to transformer models in NLP, operating with arbitrary token groups or superpixels, and accelerating or otherwise improving the explainer model’s training.
A A TTENTION MASKING
This section describes our attention masking approach in detail.
First, recall that ViTs use query-keyvalue self-attention (Vaswani et al., 2017; Dosovitskiy et al., 2020), which accepts a set of input tokens and produces a weighted sum of learned token values.
Given an input z ∈ R d × h and parameters
U qkv ∈
R h × 3 h
′ , we compute the self attention output SA( z ) for a single head as follows:
[Equation]
In multihead self-attention, we perform this operation in parallel over k attention heads and project the concatenated outputs.
Denoting each head’s output as SA i ( z ) and the projection matrix as
U msa ∈ R k · h ′ × h , the multihead self-attention output MSA( z ) is
[Equation]
Multihead self-attention can operate with any number of tokens, so given a subset s ∈ { 0 , 1 } d and an input x , we can evaluate a ViT using only tokens for the patches x s = { x i : s i = 1 } .
However, for implementation purposes it is preferable to maintain the same number of tokens within a minibatch.
We therefore provide all tokens to the model and achieve the same effect using attention masking.
Our exact approach is described below.
Let z ∈ R
d
× h represent the full token set for an input x and let s be a subset.
At each selfattention layer, we construct a mask matrix S =
[ s , . . .
, s ] ⊤ ∈ { 0 , 1 } d × d and calculate the masked self-attention output SA( z , s ) as follows:
[Equation]
The masked multihead self-attention output is then calculated similarly to the original version:
[Equation]
Due to the masking in eq.
(9), each output token in MSA( z , s ) is guaranteed not to attend to tokens from x 1 − s = { x i : s i = 0 } .
We use masked self-attention in all layers of the network, so that the tokens for x s remain invariant to those for x 1 − s throughout the entire model, including after the layer norm and fully-connected layers.
When the final prediction is calculated using the class token, the output is equivalent to using only the tokens for x s .
If the final prediction is instead produced using global average pooling (Beyer et al., 2022), we can modify the average to account only for tokens we wish to include.
B M ASKED TRAINING
In this section, we provide proofs to justify training a ViT classifier with held-out tokens, either as part of the original training or as part of a post-hoc fine-tuning procedure (the surrogate model training described in Section 4).
Our proofs are similar to those in prior work that discusses marginalizing out features using their conditional distribution (Covert et al., 2021).
First, consider a model trained directly with masking.
Given a subset distribution p ( s ) and the data distribution p ( x , y ) , we can train a model f ( x s ; η ) with cross-entropy loss and random masking by minimizing the following:
[Equation]
To understand the global optimizer for this loss function, consider the expected loss for the prediction given a fixed model input x s :
[Equation]
The expression in eq.
(13) is equal to the KL divergence D KL ( p ( y | x s ) ||
f ( x s ; η ))
up to a constant value, so the prediction that minimizes this loss is p ( y | x s ) .
For any subset s ∈ { 0 , 1 } d where p ( s ) > 0 , we then have the following result for the model f
( x s ; η ∗ ) that minimizes eq.
[Equation]
Intuitively, this means that training the original model with masking estimates f ( x s ; η ) ≈ p
( y | x s ) .
In practice, we use a subset distribution p ( s ) where p ( s ) > 0 for all s ∈ { 0 , 1 } d : we set p ( s ) by sampling the cardinality uniformly at random and then sampling the members, which is equivalent to defining p ( s ) as
[Equation]
Alternatively, we can use a model f ( x ; η ) trained without masking and fine-tune it to better handle held-out features.
In our case, this yields a surrogate model (Frye et al., 2020) denoted as g ( x s ; β ) that we fine-tune by minimizing the following loss:
[Equation]
To understand the global optimizer for the above loss, we can again consider the expected loss given a fixed input
[Equation]
The distribution that minimizes this loss is the expected output given the available features, or E [ f ( x ; η ) | x s ] .
By the same argument presented above, we then have the following result for the optimal surrogate g ( x s ; β ∗ ) that minimizes eq.
[Equation]
Notice that if the initial model is optimal, or f ( x ; η ) = p ( y | x ) , then the optimal surrogate satisfies g ( x s ; β ∗ )
= p ( y | x s ) .
C E XPLAINER TRAINING APPROACH
In this section, we summarize our approach for training the explainer model and describe several design choices.
Recall that the explainer is a vision transformer ϕ ViT ( x , y
; θ ) ∈ R
d that we train by minimizing the following loss:
[Equation]
Additive efficient normalization The constraint on the explainer predictions is necessary to ensure that the global optimizer outputs the exact Shapley values, and we use the same approach as prior work to enforce this constraint (Jethani et al., 2021b).
We allow the model to make unconstrained predictions that we then modify using the following transformation:
[Equation]
This operation is known as the additive efficient normalization (Ruiz et al., 1998), and it can be interpreted as projecting the predictions onto the hyperplane where the constraint holds (Jethani et al., 2021b).
We implement it as an output activation function, similar to how softmax is used to ensure valid probabilistic predictions for classification models.
Subset distribution The specific distribution p
Sh ( s ) in our loss function is motivated by the Shapley value’s weighted least squares characterization (Charnes et al., 1988; Lundberg and Lee, 2017).
This result states that the Shapley values for a game v : { 0 , 1 } (cid:55)→ R are the solution to the following optimization problem:
[Equation]
We obtain p
Sh ( s ) by normalizing the weighting term in the summation, and doing so yields a distribution p
Sh ( s ) ∝ ( 1 ⊤ s − 1)!
( d − 1 ⊤ s − 1)!
for 0 < 1 ⊤ s < d and p
Sh ( 1 ) = p
Sh ( 0 ) = 0 .
To sample from p
Sh ( s ) , we calculate the probability mass on each cardinality, sample a cardinality m from this multinomial distribution, and then select m indices uniformly at random.
Stochastic gradient descent As is common in deep learning, we optimize our objective using stochastic gradients rather than exact gradients.
To estimate our objective, we require a set of tuples ( x , y , s ) that we obtain as follows.
First, we sample an input x ∼ p ( x ) .
Next, we sample multiple subsets s ∼ p
Sh ( s ) .
To reduce gradient variance, we use the paired sampling trick (Covert and Lee, 2021) and pair each subset s with its complement 1 − s .
Then, we use our explainer to output Shapley values simultaneously for all classes y ∈ { 1 , . . .
, K } .
Finally, we minibatch this procedure across multiple inputs x and calculate our loss across the resulting set of tuples ( x , y , s ) .
Fine-tuning Rather than training the ViT explainer from scratch, we find that fine-tuning an existing model leads to better performance.
This is consistent with recent work that finds ViTs challenging to train from scratch (Dosovitskiy et al., 2020).
We have several options for initializing the explainer: we can use 1) the original classifier f ( x ; η ) , 2) the fine-tuned classifier g ( x s ; β ) , or 3) a ViT pre-trained on another task.
We treat this choice as a hyperparameter, selecting the initialization that yields the best performance.
We also experiment with freezing certain layers in the model, but we find that training all the parameters leads to the best performance.
Explainer architecture We use standard ViT architectures for the explainer.
These typically append a class token to the set of image tokens (Dosovitskiy et al., 2020), and we find it beneficial to preserve this token in pre-trained architectures even though it is unnecessary for Shapley value estimation.
We require a separate output head from the pre-trained architecture, and our explainer head consists of one additional self-attention block followed by three fully-connected layers.
Each image patch yields one Shapley value estimate per class, and we discard the results for the class token.
Hyperparameter tuning To select hyperparaters related to the learning rate, initialization and architecture, we use a pre-computed set of tuples ( x , y , s ) to calculate a validation loss.
These are generated using inputs x that were not used for training, so our validation loss can be interpreted as an unbiased estimator of the objective function.
This approach serves as an inexpensive alternative to comparing with ground truth Shapley values for a large number of samples.
Training pseudocode Algorithm 1 shows a simplified version of our training algorithm, without minibatching, sampling multiple subsets s , or parallelizing across the classes y .
Algorithm 1: Explainer training
Input: Coalitional game v xy ( s ) , learning rate α
Output: Explainer ϕ ViT ( x , y ; θ ) initialize ϕ ViT ( x , y ; θ ) while not converged do sample ( x, y ) ∼ p ( x , y ) , s ∼ p ( s ) predict ϕ ← ϕ ViT ( x, y ; θ ) set
ϕ ← ϕ + d − 1 (cid:0) v xy ( 1 ) − v xy ( 0 ) − 1 ⊤ ϕ (cid:1) calculate L ← (cid:0) v xy ( s ) − v xy ( 0 ) − s ⊤ ϕ
(cid:1)update θ ← θ − α ∇ θ L end
C. H YPERPARAMETER CHOICES
When training the original classifier and fine-tuned classifier models, we used a learning rate of 10 −and trained for 25 epochs and 50 epochs, respectively.
The MURA classifier was trained with an up-weighted loss for negative examples to account for class imbalance.
The best model was selected based on the validation criterion, where we used 0-1 accuracy for ImageNette and Cohen Kappa for
MURA.
When training the explainer model, we used the same ViT-Base architecture as the original classifier and initialized using the fine-tuned classifier, as this gave the best results.
We used the AdamW (Loshchilov and Hutter, 2018) optimizer with a cosine learning rate schedule and a maximum learning rate of 10 − 4 , and we trained the model for 100 epochs, selecting the best model based on the validation loss.
We used standard data augmentation steps: random resized crops, vertical flips, horizontal flips, and color jittering including brightness, contrast, saturation, and hue.
We used minibatches of sizewith 32 subset samples s per x sample, and we found that using a tanh nonlinearity on the explainer predictions was helpful to stabilize training.
Finally, we modified the ViT architecture to output Shapley values for each token and each class by removing the classification head and adding an extra attention layer followed by three fully-connected layers with width 4 times the embedding dimension, and we fine-tuned the entire ViT backbone.
These choices were determined by an ablation study with different model configurations, and we also compared with training training the ViT from scratch and training a separate U-Net explainer model (Ronneberger et al., 2015) (see Table 3).
[Table]
We used a machine with 2 GeForce RTX 2080Ti GPUs to train the explainer model, and due to GPU memory constraints we loaded the classifier and explainer to separate GPUs and trained with mixed precision using PyTorch Lightning.
3 Training the explainer model required 19 hours and 20 minutes for the ImageNette dataset and 2 days, 13 hours and 15 minutes for the MURA dataset.
D P ROOFS
Here, we re-state and prove our main results from Section 5.
Lemma 1.
For a single input-output pair ( x, y ) , the expected loss under eq.
(4) for the prediction ϕ ViT ( x, y ; θ ) is µ -strongly convex with µ
= H − 1 d − 1 , where H d − 1 is the ( d − 1) th harmonic number.
Proof.
For an input-output pair ( x, y ) , the expected loss for the prediction ϕ
= ϕ ViT ( x, y ; θ ) under our objective is given by
[Equation]
This is a quadratic function of ϕ with its Hessian given by
[Equation]
The convexity of h xy ( ϕ ) is determined by the Hessian’s eigenvalues, and its entries can be derived from the subset distribution p
Sh ( s ) ; see similar results in Simon and Vincent (2020) and Covert and
Lee (2021).
The distribution assigns equal probability to subsets of equal cardinality, so we define the shorthand notation p k ≡ p
Sh ( s ) for s such that 1 ⊤ s = k .
Specifically, we have
[Equation]
and
[Equation]
We can then write A ≡ E p
Sh ( s ) [ ss ⊤ ] and derive its entries as follows:
[Equation]
Based on this, we can see that A has the structure
A =
( b − c ) I d + c 11 ⊤ for b ≡ A ii − A ij and c ≡
A ij .
Following the argument by Simon and Vincent (2020), the minimum eigenvalue is then given by λ min ( A ) =
b − c .
Deriving the specific value shows that it depends on the ( d − 1) th harmonic number, H d − 1 :
[Equation]
The minimum eigenvalue is therefore strictly positive, and this implies that h xy ( ϕ ) is µ -strongly convex with µ given by
[Equation]
Note that the strong convexity constant µ does not depend on ( x, y ) and is determined solely by the number of features d .
Theorem 1.
For a model ϕ ViT ( x , y ; θ ) whose predictions satisfy the constraint in eq.
(4) , the objective value L ( θ ) upper bounds the Shapley value estimation error as follows,
[Equation]
where L ∗ represents the loss achieved by the exact Shapley values.
Proof.
We first consider a single input-output pair ( x, y ) with prediction given by ϕ =
ϕ ViT ( x, y ; θ ) .
Rather than writing the expected loss h xy ( ϕ ) , we now write the Lagrangian L xy ( ϕ, γ ) to account for the linear constraint in our objective, see eq.
[Equation]
Regardless of the Lagrange multiplier value γ ∈ R , the Lagrangian L xy ( ϕ, γ ) is a µ -strongly convex quadratic with the same Hessian as h xy ( ϕ ) :
[Equation]
Strong convexity enables us to bound ϕ ’s distance to the global minimizer via the Lagrangian’s value.
First, we denote the Lagrangian’s optimizer as ( ϕ ∗ , γ ∗ ) , where ϕ ∗ is given by the exact Shapley values (Charnes et al., 1988):
[Equation]
Next, we use the first-order strong convexity condition to write the following inequality:
[Equation]
According to the Lagrangian’s KKT conditions (Boyd et al., 2004), we have the property that ∇ ϕ L xy ( ϕ ∗ , γ ∗ )
= 0 .
The inequality therefore simplifies to
[Equation]
or equivalently,
[Equation]
If we constrain ϕ to be a feasible solution (i.e., it satisfies our objective’s linear constraint), the KKT primal feasibility condition implies that the inequality further simplifies to
[Equation]
Now, we can consider this bound in expectation over p ( x , y ) .
First, we denote our full training loss as L ( θ ) , which is equal to
[Equation]
Next, we denote L ∗ as the training loss achieved by the exact Shapley values, or
[Equation]
Given a network ϕ ViT ( x , y ; θ ) whose predictions are constrained to satisfy the linear constraint, taking the bound from eq.
(17) in expectation yields the following bound on the distance between the predicted and exact Shapley values:
[Equation]
Applying Jensen’s inequality to the left side
[Equation]
we can rewrite the bound as follows:
[Equation]
Substituting in the strong convexity parameter µ from Lemma
[Equation]
we arrive at the final bound:
[Equation]
Finally
[Equation]
we also present a corollary to Theorem 1.
This result formalizes the intuition that if we can iteratively optimize the explainer such that its loss approaches the optimum, our Shapley value estimation error will go to zero.
Corollary 1.
Given a sequence of models ϕ ViT ( x , y ; θ 1 ) , ϕ ViT ( x , y ; θ 2 )
[Equation]
whose predictions satisfy the constraint in eq.
(4) and where L ( θ n ) → L ∗ , the Shapley value estimation error goes to zero:
[Equation]
[Table]
Proof.
Fix ϵ > 0 .
By assumption, there exists a value n ′ such that L ( θ n ) − L ∗ < µϵ 2 2 for n > n ′ .
Following the result in Theorem 1, we have E p ( x , y ) (cid:104)(cid:12)(cid:12)(cid:12)(cid:12) ϕ ViT ( x , y ; θ n ) − ϕ
( v xy ) (cid:12)(cid:12)(cid:12)(cid:12) 2 (cid:105) <
ϵ for n > n ′ .
E D ATASETS
The ImageNette dataset contains 9,469 training examples and 3,925 validation examples, and we split the validation samples at random to obtain a validation set with 1,962 examples and a test set with 1,963 examples.
The MURA dataset contains 36,808 training examples and 3,197 validation examples.
We use the validation examples as a test set, and we split the training examples to obtain train and validation sets containing 33,071 and 3,737 examples, respectively, ensuring that images from the same patient belong to a single split.
For both datasets, the training and validation data were used to train the original classifiers, fine-tuned classifiers and explainer models, and the test data was used only when calculating performance metrics.
F B ASELINE METHODS
This section provides implementation details for the baseline explanation methods.
We used a variety of attention-, gradient- and removal-based methods as comparisons for ViT Shapley, and we modified several approaches to arrive at patch-level feature attribution scores.
Attention last This approach calculates the attention directed from each image token into the class token in the final self-attention layer, summed across attention heads (Abnar and Zuidema, 2020;
Chefer et al., 2021).
The results are provided at the patch-level, but they are not generated separately for each output class.
Attention rollout This approach accounts for the flow of attention between tokens by summing across attention heads and multiplying the resulting attention matrices at each layer (Abnar and
Zuidema, 2020).
Like the previous method, results are not generated separately for each output class.
We used an implementation provided by prior work (Chefer et al., 2021).
Common gradient-based methods Several methods that operate via input gradients are Vanilla gradients (Simonyan et al., 2013), SmoothGrad (Smilkov et al., 2017),
VarGrad (Hooker et al., 2019), and IntGrad (Sundararajan et al., 2017).
These methods were run using the Captum package (Kokhlikyan et al., 2020), and we used 10 samples per image for SmoothGrad, VarGrad and IntGrad.
We tried applying these at the level of pixels and patch embeddings, and in both cases we created class-specific, patch-level attributions by summing across the unnecessary dimensions.
We calculated the absolute value before summing for Vanilla and SmoothGrad, VarGrad automatically produces non-negative values, and we preserved the sign for IntGrad because it should be meaningful.
GradCAM Originally designed for intermediate convolutional layers (Selvaraju et al., 2017),
GradCAM has since been generalized to the ViT context.
The main operations remain the same, only the representation being analyzed is the layer-normed input to the final self-attention layer, and the aggregation is across the embedding dimension rather than convolutional channels (GradCAM LN)
(Gildenblat and contributors, 2021).
We also experimented with using a different internal layer for generating explanations (the attention weights computed in the final self-attention layer, denoted as
GradCAM Attn.
(Chefer et al., 2021)).
Layer-wise relevance propagation (LRP)
Originally described as a set of constraints for a modified backpropagation routine (Bach et al., 2015), LRP has since been implemented for a variety of network layers and architectures, and it was recently adapted to ViTs (Chefer et al., 2021).
We used an implementation provided by prior work (Chefer et al., 2021).
Leave-one-out The importance scores in this approach are the difference in prediction probability for the full-image and the iamge with a single patch removed.
We removed patches by setting pixels to zero, similar to the original version for CNNs (Zeiler and Fergus, 2014).
RISE This approach involves sampling many occlusion masks and reporting the mean prediction when each patch is included.
The original version for CNNs (Petsiuk et al., 2018) used a complex approach to generate masks, but we simply sampled subsets of patches.
As in the original work, we
[Table]
sample from all subsets with equal probability, and we use 2,000 mask samples per sample to be explained.
We occlude patches by setting pixel values to zero, similar to the original work.
Random Finally, we included a random baseline as a comparison for the insertion, deletion and
ROAR metrics.
These metrics only require a ranking of important patches, so we generated ten random orderings and averaged the results across these orderings.
Table 4 shows the same metrics as Table 1 with additional results for alternative implementations of several baselines.
For the methods based on input gradients, we experimented with generating explanations at both the pixel level and embedding level; the preferred approach depends on the method and metric, but both versions tend to underperform ViT Shapley, with the exception of faithfulness on ImageNette where the 95% confidence intervals overlap for many methods.
We also experimented with two versions of GradCAM (described above) and find that the GradCAM
LN implementation generally performs slightly better.
In the main text, we present results only for
GradCAM LN and the remaining gradient-based methods generated at the embedding level
[Table]
G M ETRICS DETAILS
This section provides additional details about the performance metrics used in the main text experiments (Section 6).
Insertion/deletion These metrics involve repeatedly making predictions while either inserting or deleting features in order of most to least important (Petsiuk et al., 2018).
While the original work removed features by setting them to zero, we use the fine-tuned classifier that was trained to handle partial information.
We calculated the area under the curve for individual predictions and then averaged the results across 1,000 test set examples; we used random examples for ImageNette, and only examples that were predicted to be abnormal for MURA.
Table 1 presents results for the true class only, and Table 2 presents results averaged across all the remaining classes.
Sensitivity-n This metric samples feature subsets at random and calculates the correlation between the prediction with each subset and the sum of the corresponding features’ attribution scores (Ancona et al., 2018).
It typically considers subsets of a fixed size n , which means sampling from the following subset distribution p n ( s ) :
[Equation]
Mathematically, the metric is defined for a model f ( x ) , an individual sample x and label y , feature attributions ϕ ∈ R d and subset size n as follows:
[Equation]
Similar to insertion/deletion, we use the fine-tuned classifier to handle held-out patches and calculate the metric across 1,000 test set images.
We use subset sizes ranging from 14 to 182 patches with step size 14, and we estimate the correlation for each example and subset size using 1,000 subset samples.
Faithfulness This metric is nearly identical to sensitivity-n, only it calculates the correlation across subsets of all sizes (Bhatt et al., 2021).
Mathematically, it is defined as
[Equation]
and we sample from a distribution with equal probability mass on all cardinalities, or
[Equation]
We use the fine-tuned classifier to handle held-out patches, and we compute faithfulness across 1, test set images and with 1,000 subset samples per image.
ROAR Finally, ROAR evaluates the model’s accuracy after removing features in order from most to least important (Hooker et al., 2019).
We also experimented with inserting features in order of most to least important.
Crucially, the ROAR authors propose handling held-out features by retraining the model with masked inputs.
We performed masked retraining by performing test-time augmentations for all training, validation and test set images, generating explanations to identify the most important patches for the true class, and setting the corresponding pixels to zero.
Because masked retraining leaks information through the masking, we also replicated this metric using the fine-tuned classifier model, and with a separate evaluator model trained directly with random masking; the evaluator model trained with random masking has been used in prior work (Jethani et al., 2021a;b).
We generated results for each number of inserted/deleted patches (1, 3, 7, 14, 28, 56, 84, 112, 140, 168, and 182) with the final accuracy computed across the entire test set.
Ground truth metrics Previous work has considered evaluations involving comparison with ground truth importance, where the ground truth is either identified by humans (Chefer et al., 2021) or introduced via synthetic dataset modifications (Zhou et al., 2022).
An important issue with such methods is that they test explanations against what a model should depend on rather than what it does depend on, so the results do not reflect the explanation’s accuracy for the specific model (Petsiuk et al., 2018).
We thus decided against including such metrics.
H A DDITIONAL RESULTS
This section provides additional experimental results.
We first show results involving similar baselines and metrics as in the main text, and we then show results comparing ViT Shapley to KernelSHAP.
H. M AIN BASELINES AND METRICS
Figure 5 shows our evaluation of attention masking for handling held-out image patches using two separate metrics: 1) KL divergence relative to the full-image predictions (also shown in the main text), and 2) top-1 accuracy relative to the true labels.
The former can be understood as a divergence measure between the predictions with masked inputs and the predictions with patches marginalized out using their conditional distribution (see Appendix B).
The latter is a more intuitive measure of how much the performance degrades.
The results are similar between the two metrics, showing that the predictions diverge more quickly if the model is not fine-tuned with random masking.
Table 5 shows insertion, deletion and faithfulness results for the MURA dataset with examples that were predicted to be normal, but while evaluating explanations for the abnormal class.
ViT Shapley outperforms the baseline methods, reflecting that our explanations correctly identify patches that influence the prediction towards and against the abnormal class even for normal examples.
Table 6 and Table 7 show insertion, deletion, and faithfulness metrics for ImageNette when using the
ViT-Small architecture (Wightman, 2019; Dosovitskiy et al., 2020) for the classifier and explainer.
They show results for target-class explanations and non-target-class explanations, respectively.
The results are consistent with those obtained for ViT-Base, and ViT Shapley outperforms the baseline methods across all three metrics.
This shows that our explainer model can be trained successfully even with a relatively small architecture and number of parameters.
Table 8 shows insertion, deletion and faithfulness results for a ViT classifier trained directly with random masking.
Whereas our Section 6 experiments utilize a fine-tuned classifier to handle missing patches, a classifier trained with random masking allows us to bypass the fine-tuning stage and train the explainer directly.
The results are similar to Table 1, and we find that ViT Shapley consistently achieves the best performance.
Figure 6 and Figure 7 show the average curves used to generate the insertion/deletion AUC results.
Both sets of plots reflect that explanations from ViT Shapley identify relevant patches that quickly move the prediction towards or away from a given class.
In the case of ImageNette, we observe that this holds for both target and non-target classes.
Figure 8 shows the sensitivity-n metric evaluated for non-target classes on the ImageNette dataset.
Similarly, these results show that ViT Shapley generates attribution scores that represent the impact of withholding features from a model, even for non-target classes.
In this case, RISE and leave-one-out are more competitive with ViT Shapley, but their performance is less competitive when the correlation is calculated for subsets of all sizes (see faithfulness in Table 2).
Next, Figure 9 shows ROAR results generated in both the insertion and deletion directions, using the four patch removal approaches: 1) the fine-tuned classifier, 2) the separate evaluator model trained directly with random masking, 3) masked retraining, and 4) masked retraining without positional embeddings.
The results show that in addition to strong performance in the deletion direction, ViT
Shapley consistently achieves the best results in the insertion direction, even in the case of masked retraining with positional embeddings.
Figure 10 shows ROAR results for all settings with the ViT-Small architecture.
We observe the same results obtained with ViT-Base.
Except for the deletion direction with masked retraining and positional embeddings enabled, ViT Shapley achieves the best performance among all methods.
Finally, Table 9 shows the time required to generate explanations using each approach.
Because
ViT Shapley requires a single forward pass through the explainer model, it is among the fastest approaches and is paralleled only by the attention-based methods.
The gradient-based methods require forward and backward passes for all classes, and sometimes for many altered inputs (e.g., with noise injected).
RISE is the slowest of all the approaches tested because it requires making several thousand predictions to explain each sample.
Our evaluation was conducted on a GeForce RTX
Ti GPU, with minibatches of 16 samples for attention last, attention rollout and ViT Shapley; batch size of 1 for Vanilla Gradients, GradCAM, LRP, leave-one-out and RISE; and internal minibatching for SmoothGrad, IntGrad and VarGrad (implemented via Captum (Kokhlikyan et al., 2020)).
ViT Shapley is the only method considered here to require training time, and as described in Appendix C, training the explainer models required roughly 0.8 days for ImageNette and 2.5 days for MURA.
The training time is not insignificant, but investing time in training the explainer is worthwhile if 1) high-quality explanations are required, 2) there are many examples
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
K L d i v e r g e n c e
ImageNette
[Figure] [Figure]
[Figure] [Figure]
K L d i v e r g e n c e
MURA
[Figure]
r a c y
[Figure]
MURA
[Table] [Table]
[Table] [Table]
[Figure] [Figure]
ili
[Figure]
y (
[Figure]
l )
[Figure]
[Figure] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
196 # of Inserted Patches 0.
A cc u r a c y
Fine-tuned Classifier (Insertion)       196 # of Deleted Patches 0.
A cc u r a c y
Fine-tuned Classifier (Deletion)       196 # of Inserted Patches 0.
A cc u r a c y
Separate Evaluator (Insertion)       196 # of Deleted Patches 0.
A cc u r a c y
Separate Evaluator (Deletion)       196 # of Inserted Patches 0.
A cc u r a c y
Masked Retraining w/ Positional Embed.
(Insertion)       196 # of Deleted Patches 0.
A cc u r a c y
Masked Retraining w/ Positional Embed.
(Deletion)       196 # of Inserted Patches 0.
A cc u r a c y
Masked Retraining w/o Positional Embed.
(Insertion)       196 # of Deleted Patches 0.
[Figure]
196 # of Inserted Patches 0.
A cc u r a c y
Fine-tuned Classifier (Insertion)       196 # of Deleted Patches 0.
A cc u r a c y
Fine-tuned Classifier (Deletion)       196 # of Inserted Patches 0.
[Figure]
A cc u r a c y
Separate Evaluator (Insertion)       196 # of Deleted Patches
[Figure]
[Figure] [Figure]
[Figure] [Figure]
A cc
[Figure]
a c y
Separate Evaluator (Deletion)       196 # of Inserted Patches 0.
A cc u r a c y
Masked Retraining w/ Positional Embed.
(Insertion)       196 # of Deleted Patches 0.
A cc u r a c y
Masked Retraining w/ Positional Embed.
(Deletion)       196 # of Inserted Patches 0.
[Figure] [Figure]
cc
[Figure]
r
[Figure]
y
Masked Retraining w/o Positional Embed.
(Insertion)       196 # of Deleted Patches 0.
[Figure]
[Table] [Table]
K ERNEL SHAP COMPARISONS
Here, we provide two results comparing ViT Shapley with KernelSHAP.
First, Figure 11 compares the approximation quality of Shapley value estimates produced by ViT
Shapley and KernelSHAP.
The estimates are evaluated in terms of L2 distance, Pearson correlation and Spearman (rank) correlation, and our ground truth is generated by running KernelSHAP for a large number of iterations.
Specifically, we use the convergence detection approach described by
Covert and Lee (2021) with a threshold of t = 0 .
The results are computed using just 100 randomly selected ImageNette images due to the significant computational cost.
Based on Figure 11, we observe that the original version of KernelSHAP takes roughly 120, model evaluations to reach the accuracy that ViT Shapley reaches with a single model evaluation.
KernelSHAP with paired sampling (Covert and Lee, 2021) converges faster, and it requires roughly 40,000 model evaluations on average.
ViT Shapley’s estimates are not perfect, but they reach nearly 0.8 correlation with the ground truth for the target class, and nearly 0.7 correlation for non-target classes.
Next, Table 10 compares the ViT Shapley estimates and the fully converged estimates from KernelSHAP via the insertion and deletion metrics.
The fully converged KernelSHAP estimates performed better than ViT Shapley on both metrics, and the gap is largest for deletion with the target class.
The results were also computed using only 100 ImageNette examples due to the computational cost.
These results reflect that there is room for further improvement if ViT Shapley’s estimates can be made more accurate.
KernelSHAP itself is not a viable option in practice, as we found that its estimates took between 30 minutes and 2 hours to converge when using paired sampling (Covert and
Lee, 2021) (equivalent to roughly 300k and 1,200k model evaluations), but it represents an upper bound on how well ViT Shapley could perform with near-perfect estimation quality.
[Figure]
I Q UALITATIVE EXAMPLES
This section provides qualitative examples for ViT Shapley and the baseline methods.
When visualizing explanations from each method, we used the icefire color palette, a diverging colormap implemented in the Python Seaborn package (Waskom, 2021).
Negative influence is an important feature for ViT Shapley, and a diverging colormap allows us to highlight both positive and negative contributions.
To generate the plots shown in this paper, we first calculated the maximum absolute value of an explanation and then rescaled the values to each end of the color map; next, we plotted the original image with an alpha of 0.85, and finally we performed bilinear upsampling on the explanation and overlaid the color-mapped result with an alpha of 0.9.
The alpha values can be tuned to control the visibility of the original image.
Figure 12 shows a comparison between ViT Shapley and KernelSHAP explanations for several examples from the ImageNette dataset.
The results are nearly identical, but ViT Shapley produces explanations in a single forward pass while KernelSHAP is considerably slower.
Determining the number of samples required for KernelSHAP to converge is challenging, and we used the approach proposed by prior work with a convergence threshold of t = 0 .
2 (Covert and Lee, 2021).
With this setting and with acceleration using paired sampling, the KernelSHAP explanations required between 30 minutes to 1 hour to generate per image, versus a single forward pass for ViT Shapley.
Figure 13, Figure 14 and Figure 15 show comparisons between ViT Shapley and the baselines on
ImageNette samples.
We only show results for attention last, attention rollout, Vanilla Gradients,
Integrated Gradients, SmoothGrad, LRP, leave-one-out, and ViT Shapley; we excluded VarGrad,
GradCAM and RISE because their results were less visually appealing.
The explanations are shown only for the target class, and we observe that ViT Shapley often highlights the main object of interest.
We also find that the model is prone to confounders, as ViT Shapley often highlights parts of the background that are correlated with the true class (e.g., the face of a man holding a tench, the clothes of a man holding a chainsaw, the sky in a parachute image).
Similarly, Figure 16, Figure 17 and Figure 18 compare ViT Shapley to the baselines on example images from the MURA dataset.
The ViT Shapley explanations almost always highlight clear signs of abnormality, which are only sometimes highlighted by the baselines.
Among those shown here,
LRP is most similar to ViT Shapley, but they disagree in several cases.
Finally, Figure 19 shows several examples of non-target class explanations.
These results show that ViT Shapley highlights patches that can push the model’s prediction towards certain non-target classes, which is not the case for other methods.
These results corroborate those in Table 2 and
Table 5, which show that ViT Shapley offers the most accurate class-specific explanations.
[Figure]
K e r n e l S H
A
[Figure]
V i T S
h
a p l e y
K e r n e l S H
A P
V i T S
h
a p l e y
K e r n e l S H
A P
V i T S
h
a p l e y
K e r n e l S H
A P
V i T S
h
a p l e y
K e r n e l S H
A P
V i T S
h
a p l e y
[Figure]
Cassette player
Garbage truck
Tench
English springer
Church
Parachute
French horn
Chain saw
Golf ball
Gas pump
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout Attention rollout
Vanilla Vanilla
Vanilla Vanilla
Vanilla Vanilla
Vanilla Vanilla
Vanilla Vanilla
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
LRP LRP
LRP LRP
LRP LRP
LRP LRP
LRP LRP
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
Cassette player
Garbage truck
Tench
English springer
Church
Parachute
French horn
Chain saw
Golf ball
Gas pump
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention rollout Attention rollout
Attention rollout Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout
Vanilla
Vanilla
Vanilla Vanilla Vanilla
Vanilla Vanilla
Vanilla Vanilla
Vanilla
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
LRP LRP
LRP LRP
LRP LRP
LRP LRP
LRP LRP
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
Cassette player
Garbage truck
Tench
English springer
Church
Parachute
French horn
Chain saw
Golf ball
Gas pump
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout Attention rollout
Vanilla Vanilla
Vanilla Vanilla
Vanilla Vanilla
Vanilla Vanilla
Vanilla Vanilla
IntGrad
IntGrad IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
LRP LRP
LRP LRP
LRP LRP
LRP LRP
LRP LRP
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
Abnormal
Abnormal Abnormal
Abnormal Abnormal
Abnormal Abnormal
Abnormal Abnormal
Abnormal
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout Attention rollout
Attention rollout Attention rollout
Vanilla Vanilla
Vanilla Vanilla
Vanilla Vanilla
Vanilla Vanilla
Vanilla Vanilla
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
LRP LRP
LRP LRP
LRP LRP
LRP LRP
LRP LRP
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
ViT Shapley ViT Shapley
ViT Shapley
ViT Shapley
ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley
ViT Shapley
ViT Shapley
Abnormal
Abnormal
Abnormal
Abnormal
Abnormal
Abnormal
Abnormal
Abnormal
Abnormal
Abnormal
Attention last Attention last
Attention last Attention last
Attention last Attention last
Attention last
Attention last Attention last
Attention last Attention rollout Attention rollout Attention rollout
Attention rollout Attention rollout Attention rollout Attention rollout
Attention rollout Attention rollout Attention rollout Vanilla
Vanilla
Vanilla Vanilla Vanilla Vanilla
Vanilla Vanilla Vanilla Vanilla
IntGrad IntGrad IntGrad IntGrad
IntGrad IntGrad IntGrad IntGrad
IntGrad IntGrad SmoothGrad SmoothGrad
SmoothGrad SmoothGrad SmoothGrad SmoothGrad
SmoothGrad SmoothGrad SmoothGrad SmoothGrad
LRP LRP LRP LRP
LRP
LRP LRP LRP LRP
LRP
Leave-one-out Leave-one-out Leave-one-out Leave-one-out
Leave-one-out Leave-one-out Leave-one-out Leave-one-out
Leave-one-out Leave-one-out ViT Shapley ViT Shapley
ViT Shapley ViT Shapley ViT Shapley ViT Shapley
ViT Shapley ViT Shapley ViT Shapley ViT Shapley
Abnormal Abnormal Abnormal Abnormal
Abnormal Abnormal Abnormal Abnormal
Abnormal Abnormal Attention last Attention last
Attention last Attention last Attention last Attention last
Attention last Attention last Attention last Attention last
Attention rollout
Attention rollout Attention rollout Attention rollout Attention rollout
Attention rollout Attention rollout Attention rollout Attention rollout
Attention rollout
Vanilla Vanilla Vanilla Vanilla
Vanilla
Vanilla Vanilla Vanilla Vanilla
Vanilla
IntGrad
IntGrad IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
IntGrad IntGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
SmoothGrad SmoothGrad
LRP LRP
LRP LRP
LRP LRP
LRP LRP
LRP LRP
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
Leave-one-out Leave-one-out
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
ViT Shapley ViT Shapley
[Table]
I
[Table]
L R
[Table]
L
[Table]
a
[Table] [Table]
n e -
[Table]
u
[Table]
[Table] [Table]
[Figure]
A tt e n t i o n l a s t
[Figure]
French horn
A tt e n t i o n r o ll o u t
V a n ill a
I n t G r a d
S m oo t h
G r a d
L R P
L e a v e - o n e - o u t
V i T S
h
a p l e y 0.
[Figure] [Figure]