A S AMPLE -B ASED M ETHOD FOR S EMANTIC U NDERSTANDING OF N EURAL N ETWORK D ECISIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Interpretability in deep learning is one of the largest obstacles to its more widespread adoption in critical applications.
A variety of methods have been introduced to understand and explain decisions made by Deep Models.
A class of these methods highlights which features are most influential to model predictions.
These methods have some key weaknesses.
First, most of these methods are applicable only to the atomic elements that make up raw inputs to the model (e.g. pixels or words).
Second, these methods generally do not distinguish between the importance of features individually and their importance due to interactions with other features.
As a result, it is difficult to explore high-level questions about how models use features during decision-making.
We tackle these issues by proposing
Sample-Based Semantic Analysis (SBSA).
We use Sobol variance decomposition as our sample-based method which allows us to quantify the importance of semantic combinations of raw inputs and highlight the extent to which these features are important individually as opposed to due to interactions with other features.
We demonstrate the ability of Sobol-SBSA to answer a richer class of questions about the behavior of Deep Learning models by exploring how CNN models from
AlexNet to DenseNet use regions when classifying images.
We present three key findings.
The architectural improvements from AlexNet to DenseNet manifested themselves in CNN models utilizing greater levels of region interactions for predictions.
These same architectural improvements increased the importance that CNN models placed on the background of images 3) Adversarially robust CNNs reduce the reliance of modern CNNs on both of the descrived trends.
Our proposed method is generalizable to a wide variety of network and input types and can help provide greater clarity about model decisions.
I NTRODUCTION
Deep learning models are becoming endemic in various applications.
As models are increasingly used for critical applications in medicine such as detecting lung nodules (Schultheiss et al., 2021) or autonomous driving (Li et al., 2021), it is important to either create interpretable models or to make opaque models human interpretable.
This paper focuses on the latter.
Existing methods developed over the last decade for doing this can be broken down into model agnostic vs model dependent.
Model agnostic methods, such as Shapley values (Kononenko et al., 2013) and Integrated Gradients (Sundararajan et al., 2017) weigh the importance of input features without relying on the structure of the model.
In contrast, methods such as GradCam (Selvaraju et al., 2017) and GradCam++ (Chattopadhay et al., 2018) are heavily dependent on model architecture.
While both of these methods yield valuable information about models, they share two common gaps.
First, neither of them reports metrics that distinguish between the features in input space that are individually important and features that are important because of their interaction with other features.
Second, the above methods are generally applied to inputs at their most granular level (pixels, words, etc..).
The combination of these two gaps limits the conclusions that Machine Learning practitioners can make about individual model predictions, as well the behavior of models as a whole.
We address these limitations in two key ways.
First, we introduce a two-part framework called
Sample-Based Semantic Analysis (SBSA).
The first part of the framework is a function that generates semantic representations of inputs and is able to map those semantic combinations to real numbers.
The second part of the framework is a black-box sample-based sensitivity method.
In this case, the Sobol method which reports the importance of individual features and their interactions.
Second, we demonstrate the ability of Sobol-SBSA to answer a richer set of questions than standard interpretability methods by applying it to CNN models in the context of ImageNet.
We ask two particular questions.
1) How did CNN model’s use of regions evolve as architectures were developed from AlexNet to DenseNet? 2)
How do adversarially training robust versions of these models change the manner in which these architectures utilize regions such that they are less likely to exploit spurious correlations?
The key results and contributions of this paper are as follows:
We present a general-purpose framework for using sample-based sensitivity methods to analyze the importance of semantic representations of inputs and test it using a variety of black-box methods.
We demonstrate that the Sobol method outperforms other popular black box methods, Integrated Gradients, Shapley (Kernel Shap), and LIME, for selecting both the most and least important regions to CNN predictions.
We show, through direct measurement , that the main impacts of the evolution of CNN architectures were increasing the extent to which they used region interactions and to which they relied on background information in images. Similarly, we show that adversarially robust versions of CNNs reduce both of these effects for modern CNNs.
R ELATED W ORK
In recent years, interest has grown in interpretability methods that can quantify the importance of feature interactions.
Janizek et al.
(2020) introduced an extension to Integrated Gradients, called Integrated Hessians, that quantified the importance of pairwise interactions of features, while Zhang et al.
(2020) built on this work by presenting a similar quantification through Shapley Interactions.
While both methods are powerful, they are expensive.
These methods require the separate calculation of individual feature importance and interactions, as well as pairwise comparisons in order to quantify interactions.
The work most closely related to ours was done by Fel et al.
(2021), who addressed the issue of additional computational expense by using the Sobol method to calculate the importance of regions in ImageNet models.
They demonstrated that Sobol outperformed white box methods like GradCAM and another black box method, RISE (Petsiuk et al., 2018).
However, they did not perform comparisons against other popular black box attribution methods like Integrated Gradients (Sundararajan et al., 2017), Shapley (Kononenko et al., 2013), and Local Interpretable Model-Agnostic
Explanations, LIME (Ribeiro et al., 2016).
Additionally, they did not demonstrate how the Sobol measurement of both interactive and primary feature importance could be used to answer larger overarching questions about models.
Finally, they did not explore how the assumption that input features to a model are uncorrelated limits the circumstances under which Sobol can be used.
We address all of these issues in our work.
M ETHODOLOGY
In this section, we describe the two components of SBSA and specify how we use it to analyze the importance of image regions in ImageNet.
In particular, we describe how we semantically map image regions to quantities that can be analyzed with a sampling-based method, and the specifics of
Sobol as a sampling-based sensitivity method.
3. S AMPLE -B ASED S EMANTIC A NALYSIS (SBSA)
Let us define the following variables.
x ∈ R
d is an input to a model, f : x → y ∈ R k is a model that takes x as an argument and produces y , ˜ x i ∈ R
d is a sample of x , and N ∈ Z is a prescribed integer that helps to determine the number of ˜ x
i samples generated.
Most sample-based sensitivity methods operate by generating a number of samples that is some function of N and d .
For example, ˜ x 1 , ˜ x 2 , . .
˜ x Nd .
The model is then evaluated on these samples yielding f (˜ x 1 ) , f (˜ x 2 ) , . . .
, f (˜ x Nd ) .
Sample-based sensitivity methods, such as Sobol, use these outputs to determine the importance of components of x to the model output, y .
One thing that immediately becomes clear is that for deep learning applications with highdimensional inputs, such as images, videos, and long documents, applying this process to raw inputs is prohibitively expensive.
This issue can be greatly minimized if one turns to semantic representations of inputs instead.
In this paper, a semantic representation of an input is defined as follows.
A semantic representation of an input, x , is some combination of the raw components of that input which yields a human recognizable higher order feature, such as the colors or regions of an image or grammatical parts of sentences .
We define this semantic representation as S ∈ R m , where m < d .
Recalling that most sample-based sensitivity methods operate on real numbers, we define two mapping objects.
[Equation]
G maps the raw input, x , to some lower dimensional semantic representation S , and H maps the semantic representation to some lower dimension vector of real numbers, r .
Both mappings are invertible.
Given these mappings, SBSA generates samples of r , ˜ r 1 , ˜ r 2 , . . .
, ˜ r ld .
From these samples, samples of the original input are generated by first mapping these samples of real numbers to samples of the semantic representations of x , ˜ S
i , and then mapping these samples of semantic representations to samples of the raw inputs, ˜ x i .
[Equation]
The model is then evaluated on the samples of the raw input, ˜ x i .
The key here is that since the sampling was done in r , the sensitivity analysis reports the importance of different components of the semantic representations of the input, S .
The author’s note that, despite the notation, G and H need not be explicit functions.
Any mechanism that that provides a mapping between the original inputs, x , semantic components of these inputs, S , and a reduced representation of the semantic components, r , is can be used.
Despite the multitude of choices, there are two properties that make a mapping ideal in this context.
Property 1 (Sensitivity)
Any change in the reduced representation of semantic components of x should result in a change x. Sample based sensitivity methods observe how different samples of inputs change model outputs in order to determine the importance of different components of the inputs.
Property 1 : insures that all of these samples contribute information to the sample based sensitivity analysis method. Property 2 (Approximate Reconstruction) : When using the reduced representation produced from the original input, x , the inverse functions should closely approximate the original input. Sample based sensitivity analysis methods construct samples that are uniformly distributed between 0 and 1, U ≈ [0 , 1] m .
These samples are scaled to the values of the inputs of interest and the value of the original input of interest, r is used as the mean of this uniform distribution (Herman & Usher, 2017).
Property 2 insures that samples of the reduced representation of x will be able to amplify and mask different components of x with approximately equal probability.
In the following section we describe the use of SBSA via image regions.
3. S AMPLE -B ASED S EMANTIC A NALYSIS (SBSA) A PPLIED TO I MAGE N ET AND R EGIONS
In this section, we illustrate how we perform sensitivity analysis of different regions in ImageNet models based on the SBSA framework.
We will first discuss our mapping functions.
Then we will detail the Sobol method for sensitivity analysis and the measures of importance that it produces.
Given an image x ∈ R d where d = W × L , our input to semantic mapping function, G , extracts regions of x , R k .
The mapping, H , from the semantic representations to a reduced real number vector, r , is simply the sum of the value of pixels in the region.
[Equation]
Finally, the inverse mapping, H − 1 involves normalizing each region, { x o : p,r : s } j by the sum of the values of the pixel in that region, (cid:80) { t,p }∈ R k x tp , and multiplying it by r j .
Then assigning the relevant indices in x the new values in the region.
When r j is equal to the summation of pixels in that region from the original image, this simply recovers the original image.
When sampled, this reconstructs samples, ˜ x i .
Mathematically, the inverse mapping is as follows.
[Equation]
The above mapping, while simple, satisfies the properties of sensitivity and approximate reconstruction.
As a result the samples of ˜ r , which represents semantic components of x , S , produces image samples, ˜ x , that amplify and mask regions of the image with relatively equal probability and uniformly.
The Saltelli method, (Saltelli et al., 2010), which chooses optimal points in [0 , 1] for the
Sobol method are used to construct these samples and the model outputs of images are fed to the
Sobol method. Given these mappings, we apply the Sobol Sensitivy method to the low dimensional real vectors, r j .
Here we will briefly give an overview of the Sobol method.
For more details about the implementation of Sobol, see Saltelli et al.
and Herman & Usher (2017).
The Sobol method is a variance-based sensitivity method.
Given a model, f : ( x 1 , . . .
, x d ) ∈ R d
→ y , the variance based first order effect of a component of the input, x , is:
[Equation]
where y j is a component of the model output, x i is a component of the input, x , x ∼ i are samples of the input where all components of the input except for i are varied, and V x i and E x ∼ i are the variance and expectation over x i and x ∼ i respectively.
The Sobol first-order sensitivity is then the ratio of V i to the variance of the model output, y j .
[Equation]
S i quantifies the individual importance of the i th component of the input, x , to the j th component of the output, y .
This equation essentially states that Sobol first order index, S i , is the fraction of the variance of the output, y j that is accounted for by x i .
What makes Sobol unique is that it simultaneously calculates the importance of the interaction between a component of the input, x i , and all other components of the input.
This is called the total effect index and is calculated as:
[Equation]
This equation essentially states that the total effect index for x i , S Ti , is the difference betweenand the fraction of the variance of the output, y j , that is accounted for by all samples of the input in which every component of the input varies, except for i .
Dividing both sides of 11 by S T i yields
[Equation]
The first term in equation 12 reports the extent to which a feature, x i , is important to the model output due to the feature by itself.
The larger the term, the greater the importance of the region by itself. Contrarily, the smaller the term larger the importance of the interaction of the feature to other features. We will refer to as the Primary Index Ratio (PIR) for the rest of the paper.
[Equation]
We implement the Sobol method using the Sensitivity Analysis Library (SALib) (Herman & Usher, 2017), which is an open source implementation of the Saltelli method Saltelli et al. (2010).
We note that the number of samples used for the Sobol method is N ( d + 2) .
For all of the experiments that follow, N = 50 .
3.2. C HOICE OF R EGIONS
In order to demonstrate the ability of our method to address high level questions about CNNs, and to explore these questions at different levels of semantic representations of an image, we apply our pipeline to three types of regions. 1) equally sized image patches 2) machine annotated semantic segmentations of the ImageNet training set from Salient ImageNet Singla & Feizi (2021) and 3) human annotated semantic segmentations from the ImageNet-S 919 validation set (Gao et al., 2021). Salient ImageNet segments images into core regions (regions that should be important for predictions) and spurious ones.
ImageNet-S is a dataset that was created for use in judging image segmentation task.
A key difference between image patches and Salient ImageNet, and ImageNet-S is that ImageNet-S strictly respects boundaries in images where the other two don’t.
Figure 1 shows our pipeline when applied to example images from the ImageNet validation set and ImageNet-S 919.
We demonstrate the validity of our pipeline by showing that Sobol-SBSA is able to accurately rank the importance of regions. We additionally demonstrate that Sobol-SBSA outperforms other standard black box models. Both of these are measured based on multiple measures of the effectiveness of interpretability methods.
With this established, we demonstrate the ability of Sobol-SBSA to answer high level questions about Deep models using CNNs as a case study. We explore three questions. 1) How did the use of region interactions evolve with CNNs? 2) How did the use of objects vs backgrounds evolve with
CNNS? What impact did training CNNs robustly have on the firs two questions? We provide quantitative results to these questions by applying Sobol-SBSA to region patches, Salient ImageNet segments, and ImageNet-S segments.
4. C OMPARISON OF S OBOL -SBSA TO O THER B LACK B OX M ODELS
Our SBSA model can be used with a variety of black box models.
Thus, we compare the following two methods against Sobol-SBSA
• Shapley (Shap-SBSA) (Kononenko et al., 2013):
Used with SBSA in a similar manner as Sobol-SBSA.
• LIME (LIME-SBSA) (Ribeiro et al., 2016):
Used with SBSA in a similar manner as Sobol-SBSA.
In addition we also compare against the following models.
• Integrated Gradients (Sundararajan et al., 2017): Cannot be used with SBSA since it requires gradients with respect to pixels.
Instead, we aggregate pixel importance in regions and rank region importance based on this aggregation.
• Random:
A control baseline where regions are randomly selected.
We use two standard metrics for comparing interpretability methods.
The first is measuring the change in the ground truth label score of a model when regions, as specified by the sensitivity method, are masked.
We mask the top and bottom 20% of regions in an image.
The second measurement is sensitivity- n correlation.
Sensitivity- n correlation is a quantity proposed by Ancona et al.
(2017) for comparing the effectiveness of attribution methods when determining which features are important to a model.
Quantitatively, it is a measurement of the Pearson correlation between is the associated with the n input that masked
S is the difference in the score that the model produces when n input features are masked versus when none are.
For each value of i , i random features are selected to be masked 100 times and the correlation is averaged over the examined data.
A value closer to one means a more effective method.
For all methods, we evaluate the importance of regions to ResNet50 on 1000 randomly selected
ImageNet validation images.
To explore whether or not region size would impact our results, we performed our experiments for 4 × 4 , 8 × 8 , and 16 × 16 regions.
The number of samples is selected as a function of the number of regions, as detailed in section 3.2.
For 4 × 4 , 8 × 8 , and 16 ×regions, the number of samples used for all methods are 900 , 3300 , and 12900 respectively.
Figures 2 and 3 show the results of the masking and sensitivity- n correlations respectively for all regions.
Figure 2 plots the change in the model score when the most important regions are masked versus when the least.
A steeper curve is more ideal since it means that the sensitivity method significantly decreases the model score when masking the most important regions, but has little impact when masking the least important regions.
The Sobol-SBSA is the most effective by both the masking and sensitivity- n measures .
The key takeaways from the plot are 1) Both Sobol measurements,
S T i and S i , are more effective at selecting the most important regions in an image, 2) S T
i performs the best overall in picking the most and least important regions in an image, as well as more properly ranking the regions in between (as measured by sensitivity- n ), and 3) S T i is the most robust to changes in region size, followed closely by Integrated Gradients.
[Equation] EYOND R EGION I MPORTANCE ?
We have shown Sobol-SBSA’s efficacy at ranking the importance of regions to CNN predictions.
While Sobol-SBSA outperforms other black box methods in standard interpretability measures, the real advantage offered by Sobol-SBSA over other black box methods is the ability to answer large overarching questions about the behavior of Deep Learning models.
We demonstrate this by using Sobol-SBSA to quantify how the use of regions evolves in CNN models from AlexNet to
DenseNet161, and to understand the mechanism that causes adversarially robust CNN models to be less susceptible to spurious correlations in the data.
[Figure]
To answer the question as to how the use of regions has changed as CNNs have evolved, we use three quantities derived from Sobol-SBSA.
The PIR , defined in equation 13, which states the extent to which a model relies on the interaction of features The smaller this ratio the larger the interactive effect.
We additionally calculate the difference in individual and total importance between the segmented and background portions of an image in ImageNet-S 919 and Salient ImageNet ( S T segmentation − S T background andS i segmentation − S i background ).
We apply Sobol-SBSA to AlexNet, VGG16, VGG16 BN, InceptionV3, ResNet50, and DenseNet161.
For all models with the exception of InceptionV3, the number of image patches chosen are 4 × 4 , 8 × 8 , and 14 × 14 .
The pre-trained InceptionV3 operates on an image that is 299 × 299 rather than 224 × 224 .
In order to compare similar region sizes, the number of regions for which Sobol-SBSA was applied to InceptionV3 were 5 × 5 , 11 × 11 , and 18 × 18 .
We note that we do not test our pipeline on InceptionV3 for ImageNet-S 919 or Salient ImageNet.
The Salient ImageNet dataset masks are 224 × 224 which is not a valid input to the pre-trained InceptionV3. We therefore omit it for both segmentations in order to keep comparisons between the segmentations consistent.
All of the results displayed in this section are based on 10000 random samples from the ImageNet validation set.
Figure 4a shows the average PIR for AlexNet, VGG16, VGG16 BN, InceptionV3, ResNet50, and
DenseNet161 for different region sizes and the segmentations of ImageNet-S and Salient ImageNet.
For the region patches, the average is calculated based top 20% of regions as measured by S T i .
The key takeaway from this figure is that one of results of the architectural changes that yielded modern CNNs was to increase the extent to which models used region interactions when making predictions .
This change was most noticeable for region patches and Salient ImageNet segmentations, both of which partition the images into areas that don’t strictly respect boundaries.
For ImageNet-S there is a slight decrease in the relative importance of interaction from AlexNet to VGG16, VGG16 BN, and ResNet50 before a large increase in DenseNet161.
Figure 4b displays the difference between the importance of segmented parts of the image and the background for both S T and S i . For both ImageNet-S and Salient ImageNet, it is seen that this difference decreases as CNNs grow more modern. This figure shows that another key result of the changes that yielded modern CNNs was to increase the importance of the background of images to model predictions .
To the authors’ knowledge, Sobol-SBSA has enabled the first direct measurement of two quantities. The importance of interactions between semantically relevant features and the relative importance between segment objects and background for CNN outputs. The work of which the authors are aware, indirectly quantify the importance of segmented and background regions based on noise applied to these regions (Singla et al., 2022) or quantify the interactions of CNNs exclusively at the image patch level (Brendel & Bethge, 2019)
4.2. W HY ARE A DVERSARIALY R OBUST CNN S R ESISTANT TO S PURIOUS F EATURES ?
In this section, we explore the mechanism by which Adversarialy robust CNNs are less susceptible to spurious correlations in the data.
To do this, we apply Sobol-SBSA to robust versions of
VGG16 BN, ResNet50, and DenseNet161 and compare their PIR and difference in the segmentation and background importance to their standard versions.
We use pre-trained robust weights from Salman et al.
(2020) where the l 2 threat radius was 3.
In this case, we
Figure 5a shows the PIR for the normal and robust versions of VGG16 BN, ResNet50, and
DenseNet161. We see that when the segmented regions of interest are areas of the image that contain relevant objects to classification, but that do not strictly respect boundaries the impact of Robust models is to greatly decrease the interaction between core and background areas for all models.
This is not the case for segmentations defined by ImageNet-S, which strictly respect object boundaries.
We see that the robust versions of VGG16 BN and ResNet50 do not greatly impact the extent to which these models rely on interactions between objects and the background, though the robust version of DenseNet161 does.
Figure 5b show the average difference in importance between segmented objects and the background for 10000 images in the ImageNet-S and Salient ImageNet datasets.
We see that for all models, and both forms of segmentation, the robust version of these CNN models placed larger relative importance on the segmented objects than their vanilla counterparts.
Figureshows an example of this trend.
We have shown through direct measurement that CNN models trained to be adversarially robust are resistant to exploiting spurious correlations in the data both because they reduce the extent to which models rely on image backgrounds, and because they reduce the extent to which models rely on interactions between core and background regions of an image. [Figure]
4.2. T HE L IMIT OF S OBOL -SBSA
While the Sobol-SBSA is a powerful method, it has a similar weakness to other sample-based black box methods.
It assumes that the input features that it is analyzing are uncorrelated.
When this is not the case, the ability of Sobol to decompose importance into independent subsets of the feature space is weakened (Li et al., 2010).
This manifests itself in the Sobol Indices, which are supposed to satisfy the property that PIR ≤ 1 , having at least some input features for which PIR > 1 .
In the case of regions the assumption of uncorrelated input features breaks down as regions become smaller in size since the regions will tend more to belong to the same object on an image (the wing of a bird).
We observe this in two ways.
First we examine figure 4a and note that PIR increases as the region sizes increase as region size decreases.
This is intuitively an incorrect result since interactive effects should be greater for smaller regions, as demonstrated by Brendel & Bethge (2019).
To confirm that this result is due to a breakdown in Sobol assumptions we calculate the average percent of regions for which PIR > 1 across all models for a given region size.
For 4 × 4 , 8 × 8 , and 14 × 14 regions the percent of regions for which PIR > 1 are 22%, 33%, and 39% respectively.
The key takeaway here is that Sobol-SBSA is should be used to compare how models use identically correlated features .
A key direction for future work is to implement a version of Sobol-SBSA that accounts for correlations in the data.
D ISCUSSION AND C ONCLUSION
We proposed Sobol-SBSA, a general method for understanding how Deep Learning Models use semantic features when making decisions.
We demonstrated the ability of this method to answer richer questions about models by applying it to how CNN models use regions of ImageNet models.
We found (1) that the primary impact of the evolution of CNN models was to make greater use of region interactions and to increase the importance of image background to model predictions, and (2) that adversarially robust CNN models are less susceptible to spurious correlations in the data because they force CNN architectures to rely less on region interactions and on image backgrounds.
The Sobol-SBSA method has a variety of potential applications beyond the image/region-based analysis that we presented here.
Many different types of input partitioning and input modalities can be analyzed using the SBSA method.
For images, the partitioning into image regions can be done via external metadata, such as object detection results.
For natural language processing (NLP), the use of SBSA is even more straightforward since words/tokens provide a natural way to partition the input sequence.
Hence, our model can be utilized to understand how parts of speech are utilized for different iterations of various NLP models, including translation, classification, and generative text models.
Beyond unimodal content, SBSA can also be easily extended to multimodal setups.
For example, Sobol-SBSA can be used to measure the strength of bias in Image Captioning systems by quantifying the extent to which parts of an image correspond to which parts of the generated caption.
In addition to these applications, there are multiple avenues for future work.
These include exploring the combination of Sobol-SBSA with automatic feature detectors, implementing a version of Sobol-SBSA that accounts for input correlations, and exploring whether PIR can be used as a proxy for the robustness of non-CNN Deep Learning models.
Through SBSA and Sobol-SBSA we have proposed a strong foundation for obtaining a richer understanding of how models use semantic representations of inputs, regardless of whether these representations are generated automatically or by end users.
We hope that it can be used to both provide clarity about the mechanisms by which Deep Learning makes decisions, and to influence how we further develop these models