Distilled Gradient Aggregation: Purify Features for Input Attribution in the Deep Neural Network
s
AffiliationAddress
Abstract
Measuring the attribution of input features toward the model output is one of thepopular post-hoc explanations on the Deep Neural Networks (DNNs).
Amongvarious approaches to compute the attribution, the gradient-based methods arewidely used to generate attributions, because of its ease of implementation and themodel-agnostic characteristic.
However, existing gradient integration methods suchas Integrated Gradients (IG) suffer from (1) the noisy attributions which cause theunreliability of the explanation, and (2) the selection for the integration path whichdetermines the quality of explanations.
FullGrad (FG) is an another approachto construct the reliable attributions by focusing the locality of piece-wise linearnetwork with the bias gradient.
Although FG has shown reasonable performance forthe given input, as the shortage of the global property, FG is vulnerable to the smallperturbation, while IG which includes the exploration over the input space is robust.
In this work, we design a new input attribution method which adopt the strengthsof both local and global attributions.
In particular, we propose a novel approachto distill input features using weak and extremely positive contributor masks.
Weaggregate the intermediate local attributions obtained from the distillation sequenceto provide reliable attribution.
We perform the quantitative evaluation comparedto various attribution methods and show that our method outperforms others.
Wealso provide the qualitative result that our method obtains object-aligned and sharpattribution heatmap.
Introduction
Deep Neural Networks (DNNs) are increasingly applied to many fields in human-life such as selfdriving, medical predictions and time-series forecasts.
Along with these improvements, the recentmodels get bigger and more complicated that humans cannot investigate and understand the internaldecision mechanism of them.
Identifying and analyzing the reasons for the model predictions areimportant because the malfunction or the groundless decision of the model can cause the criticalproblems.
As an effort to provide the evidences on the decisions, the input attribution has beenwell-studied, especially in visual tasks.
Input attribution method aims to measure how much eachinput feature contributes to the model prediction.
Because the output of this method takes the form ofheatmap to provide relative importance among input features, it helps to locate the appearance ofhuman-level semantics in the input.
Previous work also applies the attribution to debug the model bywiping out Clever Hans of DNNs [Lapuschkin et al. , 2019].
However, obtaining the trustful inputattribution is still challenging because (1) the highly nonlinear structure of modern DNNs makes itdifficult to correctly track the relationship between the input and the output, and (2) quantifying thereliability of attribution methods is non-trivial because the ground-truth is not available.
[Figure]
Local Attribution
Post Processing
WC: Weak Contributor EPC: Extreme Positive Contributor
Dingo Cat Car ∇ 𝑏 𝑓 𝑥;𝑏 𝑇 𝑏∈ℝ +
Normalization
WC
EPC
Local Attribution 0 - + Local Attribution 0 - +
Local Attribution
Final Mask
[Figure]
Distillation
Attribution Aggregation
Attribution Σ
Mask Extraction
Mask Extraction
Mask Extraction
[Figure]
Gradient-based input attribution is one of the main techniques to derive the relationship betweenthe model decision and the input features.
The partial derivative of the output with respect to theinput provides the measure of sensitivity, which is easily computed in DNNs.
Integrated Gradients(IG)
[Sundararajan et al. , 2017] is a commonly used gradient-based method, which provides theaxiomatic properties to support the reliability of attributions.
However, IG inheres the problem ofnoisy attribution, which originates from the gradient integrating path, and several variants of IG havebeen proposed to alleviate this issue [Smilkov et al. , 2017; Kapishnikov et al. , 2019, 2021; Pan et al. ,2021].
FullGrad (FG)
[Srinivas and Fleuret, 2019] also raises the counter-intuitive behaviors of IG.
FG avoids this problem by considering only the local gradients instead of the path integration andproposes to use the bias gradient.
But FG is vulnerable to the small perturbation in the inputs due toits locality.
In this work, we provide the analysis on the weakness of (1) FG method which is unavoidable if itconsiders only single anchor point, and (2) IG method which the continuous path based gradientintegration may fail to quantify the intuitive attribution.
To complement the shortcomings of the twomethods, we propose to aggregate the attribution from the multiple anchors.
For the selection ofanchor points, we devise an algorithm to sequentially distill the irrelevant features to generate thereliable attribution.
The main contributions of our work are,
• Propose a novel feature distillation algorithm based on the intermediate local attribution to
generate the sequence of meaningful anchor points.
• Devise Distilled Gradient Aggregation (DGA), an attribution method by aggregating the
intermediate local attributions from the distilled input sequence for the reliable attributions.
• Qualitative and quantitative evaluations to validate the proposed method outperforms existing
gradient-based attribution methods.
Related Work
Input attribution is one of the post-hoc explanation methods, which aims to identify the influenceof each input feature to the model output on the trained model.
There exists a considerable varietyof techniques to derive the input attribution.
With the property that the feature map obtained by theconvolutional layers includes the spatial information, Class Activation Mapping (CAM) methodscompute the attribution by the weighted sum of the feature maps [Zhou et al. , 2016; Selvaraju et al. ,2017].
Layer-wise Relevance Propagation (LRP) method propagates the model output backward tothe input [Bach et al. , 2015; Nam et al. , 2020]. LRP extends the Taylor decomposition to the DNNsand distributes the relevance in layer-wise sense. 𝜎 =𝜎 = 0.
𝜎 = 0.
𝜎 = 0. 𝜎 = 0.
𝜎 =
I npu t I m a g e
F G
I G
O u r s
MoRF ( ↓ is better) ■ FG ■ IG ■ Ours
LeRF ( ↑ is better) 𝜎 =
𝜎 = 0.2 𝜎 = 0.
4 𝜎 = 0.
6 𝜎 = 0.
8 𝜎 =0.
+ - + - + -
There are another approach by measuring the behavior of the model by perturbing the input features.
Optimizing the input by gradient ascent gives an example which would maximally activates the targetneuron [Erhan et al. , 2010; Nguyen et al. , 2016; Olah et al. , 2018].
Instead of maximizing the targetneuron activation, Extremal Perturbation optimizes the mask which removes or reveals the part of theinput to localize the attributed part of input [Fong et al. , 2019].
It is extended by using Integrated
Gradients [Sundararajan et al. , 2017] for the optimization to make the optimization more stable [Qiet al. , 2019].
By collecting the pair of partially removed inputs and corresponding model ouptuts,training a linear model to resembles the mapping would give the feature importance in terms oflinear weights [Ribeiro et al. , 2016].
Rather than training a new model, Randomized Input Samplingfor Explanation (RISE) computes the attribution by aggregating multiple randomly masked inputs,weighted by the model outputs [Petsiuk et al. , 2018].
Based on Aumann-Shapley value [Aumann and Shapley, 2015], which is one solution of the fairdistribution in the cooperative game theory, Integrated Gradients (IG) has been proposed [Sundararajan et al. , 2017].
IG is equipped with axiomatic properties which are desirable for the attributionmethods.
IG is computed by integrating gradients over the straight path from the predefined baselineto the input.
As the attribution is corrupted by the noisy information along the path, alternativesfor the different paths have been proposed [Smilkov et al. , 2017; Kapishnikov et al. , 2019, 2021].
FullGrad (FG)
[Srinivas and Fleuret, 2019], which utilizes the bias-gradient, is proposed to suppressthe counter-intuitive behavior of IG which is caused by the weak-dependency between the local linearregions.
Distilled Gradient Aggregation Method
In this section, we propose our gradient aggregation method, Distilled Gradient Aggregation (DGA).
We first provide an example and analysis about the inconsistency observed by FullGrad, which usesa single anchor point to compute attributions.
We also provide the counter-intuitive behavior of IGcaused by the continuous gradient integration path.
To complement both shortcomings, we proposean aggregation method, which ensembles the local attribution from the sequence of inputs.
To reducecomputational cost and reinforce the features that are in charge of the model decision, we suggest asequential feature distillation algorithm, which distill irrelevant features from the input.
3.1 Analyzing FG and IG on Simple Models
Assume we have the input vector x ∈ R 2 and a simple neural network f equipped with partial linearactivation (e.g., ReLU).
This network f can be regarded as the combination of piece-wise linearfunctions [Montufar et al. , 2014].
Each piece-wise linear function is only defined and feasible in
0 - + 𝑥𝑥(a) (b)
A Baseline
B (c)
IG
FG 𝑥 2 A tt r
i bu ti on 𝑥 1 Attribution
A
B
A
B (d) corresponding linear region R ( k ) , where ∪ k R ( k ) =
R 2 and R ( k 1 )
∩ R ( k 2 ) = ∅ for any k 1 and k 2 .
Such piece-wise linear function is formulated as follow,
[Equation]
where w ( k ) ∈ R 2 and b ( k ) ∈ R denote weight and bias of k -th linear region respectively.
An illustrativeexample of the this function f is depicted in Figure 3 1 .
Vulnerability of FullGrad FullGrad [Srinivas and Fleuret, 2019] suggests that the attribution shouldbe same inside the same linear region R ( k ) , and this reduces the dependency between the attributionand the input x .
This property is introduced as weak dependency .
However, such weak dependencyderives the attribution to be vulnerable to the model perturbation.
For example, let we have twoinputs, x and x ′ =
x
+ ϵ , where ϵ be any small enough random perturbation.
If we find any x and x ′ ,such that the model output is same, f ( x ) =
f
( x ′ ) , but the region is different, then the attribution oneach input should be different.
This can be visualized by the simple experiment by generating noiseperturbed image x + ϵ and measuring the attribution, where ϵ ∼ N (0 , σ ) .
Figure 2 shows an examplewhere the FullGrad generates inconsistent attribution along with the simple Gaussian noise is added.
Counter-intuitive behavior of IG To visualize the counter-intuitive behavior of IG, we select twolinear regions ( A , B ) in Figure 3c and calculate the attribution in each region.
In particular, weselect a sequence of data from a (white dot) to b (green dot), which is only shifted in x 1 dimension.
Figure 3d illustrates corresponding attribution for two selected linear regions.
We observe that onlyattribution of x 1 changes for the sequence of region A in both IG and FG methods.
However, forthe sequence of region B , the IG attribution of both x 1 and x 2 changes at the same time, while FGattribution shows attribution change only in x 1 .
We conjecture this counter-intuitive behavior of IG iscaused by the baseline selection.
with the zero-baseline ( ¯ x = 0 ), the integration paths of samplesin region A traverse only a single region to compute IG.
On the other hand, paths of samples inregion B traverse through multiple regions. When traversing multiple regions, the counter-intuitivebehavior can be induced due to passing the undesirable linear regions. From this observation, wecan identify that the selection of baseline determines (1) which linear regions are traversed by thepath, and (2) how much portion of the path is included in each selected linear region. Although theproper selection of baseline can be an one option to adjust (1) and (2) for the reliable attributions, itis still non-trivial to control the sequence of the meaningful linear regions and each weight by onlychanging the baseline.
Based on these insights, we raise the weakness of the local and global attribution methods: (1)vulnerability and (2) counter-intuitive behavior. To alleviate these issues, we desire an attribution
D i s t ill a t
i
o n
A tt r i bu t i o n
[Figure]
D i s t ill a t
i
o n
A tt r i
[Figure]
t i o n
[Figure]
method to combine the strengths of each type of attribution methods. To resolve desire step by step,we start from the local attribution which is free from the baseline selection. Then, the remainedproblem is how to select the meaningful linear regions to generate the reliable attributions.
3.2 Sequential Feature Distillation
For the strategy to select linear regions, RISE [Petsiuk et al. , 2018] suggests the random perturbationbased approach.
RISE explores the multiple linear regions with randomly ablated masks to measurethe importance of each ablated features.
However, the randomized ablation includes the stochasticprocess which requires expensive computational cost to achieve reliable attributions.
We are alsoinspired from the adaptive exploration for the perturbed inputs in Guided IG (GIG)
Kapishnikovet al. [2021], which can improve the final attribution. Thus, we hypothesize that the adaptiveexploration of linear regions based on the intermediate local attribution can reduce the cost ofrandomized exploration and follow the spirit of the adaptive exploration. Finally, we propose thesequential feature distillation algorithm to obtain a sequence of ablated inputs, the sequence of inputs˜ X =
[˜ x (0) , ˜ x (1) , ˜ x (2) , · · · ] , where the irrelevant features are distilled.
Motivated by previous workthat masking out the irrelevant features using IG [Fong et al. , 2019; Qi et al. , 2019], we propose todistill the impurities by using the intermediate local attribution obtained along the sequence.
For reliable local attribution, FG proposes the bias gradient with post processing Ψ( · ) which includesthe normalization and upsampling.
As Ψ( · ) suggested by FG is usually over-estimated by the biasgradient in deeper layers [Grabska-Barwinska et al. , 2021], we redefine Ψ( · ) as uniformly distributingfunction for the bias gradient to alleviate the over-estimation problem.
[Equation]
where 1 d denotes a d -dimensional all-ones vector.
We call FG with redefined post-processing Ψ( · ) as
Uniform FullGrad (UFG), ϕ UFG ( · ) .
Then we use UFG as the intermediate local attribution methodthroughout the remained paper.
In the sequence ˜ X , the relation between n and n +1 -th ablated input is formalized as,
[Equation]
where ˜ x (0) = x and M ( · ) is a mask extractor.
To distill off the uninformative features, we build amask to zero out the features with low magnitude of local attribution.
We define this mask extractor asthe Weak Contributor (WC) mask, M WC .
The level of the WC mask increases along the distillationsequence X with the pre-defined number of steps N and finally the entire pixels become zero (i.e.,˜ x ( N ) = 0 ).
We define M WCj ( · ) for each feature j as,
[Equation]
1 otherwise
[Equation] [Equation] Output
) 1: Let
Φ = ∅ 2: for n
N } do 3: Φ =
Φ ∪ { ϕ UFG (˜ x ( n )) } 4:
M = nN M WC (˜ x ( n ) , n ; N )+(1 − nN )
M EPC (˜ x ( n ); q ) 5: ˜ x ( n ˜
0) ⊙ M 6: for 7: DGA Φ max ϕ,
+ β · min ( ϕ, 0)) 8: return DGA ( x ) where S WCj ( x ) is a set of feature indices that the magnitude of corresponding local attribution issmaller than | ϕ UFGj ( x ) | . Practically, M WCj ( x, n ; N ) can be equivalently derived by thresholdingwith n/N quantile of absolute local attributions.
To implement the smooth change of features, wegradually apply the mask with the scale factor proportional to the current step n .
The sequentialrelation with WC mask is defined as,
[Equation]
Figure 4 (a) depicts the sequence of distilled inputs ˜ X with WC mask.
We can identify that thedistillation by WC mask can remove the uninformative information (e.g., human body) to predict theobject class, French horn .
However, we observe that considering only WC mask can reassign the same attributions to the samepixels, which makes the overall attribution to be saturated.
In Figure 4 (a), the human face (red box)remains until the end of the distillation with strong attribution.
We identify that when the strong localattribution is assigned temporarily to irrelevant features, WC mask is hard to distill these featuresin remaining steps.
The saturated distillation sequence ˜ X disturbs the strength of multiple ablatedinputs to build reliable attribution.
We conjecture that this phenomenon is caused when the noiseexists in the gradient or the value of feature itself is too large, the pixels can have extremely highcontribution although the pixels do not have relevant information to predict the target class.
We define additional mask to reduce the saturation by filtering out features with extremely strongattribution.
We call this mask as Extreme Positive Contributor (EPC) mask M EPC ( · ) and it isformulated as,
[Equation]
where q is EPC threshold to control the ratio of ablation.
Finally, we combine two masks for ourdistillation algorithm with relative weights w.r.t.
the current step n as,
[Equation]
We note that in early distillation step, EPC mask takes high weight to reduce saturation at too highlyattributed feature.
In the late stage, WC mask gains high weight to remain the relevant features.
Thedistillation sequence ˜ X with WC and EPC masks is shown in Figure 4 (b).
We identify that usingboth WC and EPC masks removes irrelevant features (red box) and iteratively assign the attributionto relevant features (blue box).
3.3 Attribution Aggregation
With N distillation steps, we obtain N local attributions.
The remaining question is how to aggregatethese local attributions to acquire the final attribution.
Likewise previous studies [Selvaraju et al. ,2017; Kindermans et al. , 2018; Bach et al. , 2015], we desire to take the positive contribution fromeach local attribution.
Thus, we take the ReLU before the aggregation.
[Equation]
Finally, we call the unification of preceding modules as Distilled Gradient Aggregation (DGA)method.
DGA method consist of the distillation algorithm with WC and EPC mask to generate theablated inputs to achieve the local attribution, and the aggregation process considering features withthe positive and the negative attributions.
The illustration of the overall structure is depicted in Figure1 and pseudo code is provided in Algorithm 1.
Experiments
In this section, we validate the effectiveness of DGA by both quantitative and qualitative comparison.
We first provide the quantitative comparison using two metrics: (1) pixel perturbation [Samek et al. ,2017], and (2) RemOve-And-Retrain (ROAR)
[Hooker et al. , 2019] to verify the proposed methodcan assign more model-relevant attribution.
Then we provide the qualitative comparison amongdifferent attribution methods.
In the following experiments, we set the hyperparameters for DGAas N =30 , q =0 .
9 and β =0 with simple grid search.
The details for the hyperparameter explorationis available in Appendix.
We select various gradient-based attribution methods as the baselines:
Gradient*Input (G*I), Guided BackPropagation (GBP), Integrated Gradients (IG), FullGrad (FG),and GuidedIG (GIG).
4.1 Pixel Perturbation
Pixel perturbation is widely used method to benchmark the attribution methods if they correctlycapture the relevance between the input features and the model output.
To quantify the relevancebetween the input features and the model output, pixel perturbation method removes the pixel valuesin order of relevance obtained by attribution methods.
Then it measures the change of softmax outputfor the target class with the perturbation.
There is two orders of removal, Most-Relevant-First (MoRF)to remove the pixels with top k % relevance and Least-Relevant-First (LeRF) to remove the pixelswith bottom k % relevance.
If input feature is actually highly related to the model prediction, thesoftmax output should decrease steeply when it is removed.
Thus, MoRF is better if it is lower.
In thesame manner, higher LeRF is better.
We use 50k images of the validation set provided by ImageNet [Russakovsky et al. , 2015].
We usethree publicly available pre-trained models: VGG-16 [Simonyan and Zisserman, 2015], Inception-v3[Szegedy et al. , 2016], ResNet-18 [He et al. , 2016].
Table 1 indicates MoRF and LeRF resultsfor the various attribution methods and model architectures.
We identify that DGA shows the bestperformance in both MoRF and LeRF measure on entire architectures.
4.2 RemOve-And-Retrain (ROAR)
ROAR is another metric to evaluate how well the attribution method captures the relevance of featurein the perspective of the model training.
ROAR is performed by measuring the performance of there-trained model with inputs modified according to relative ordering of the attribution.
Each inputin the dataset is modified by removing pixels with top k % attribution and replacing them with theaverage pixel value of the input.
We perform ROAR experiment with simple CNN (6 Conv + 3 Linear)trained on 50k images of training set provided by CIFAR-10 dataset [Krizhevsky et al. , 2009] using
Adam optimizer with learning rate 3e-4 and 100 epochs.
After training, the performance of the modelis quantified using the standard test dataset with 10k images.
We note that the attribution methodcaptures more relevant features if the test accuracy is lower.
We provide the average performanceover 10 trials for each attribution method, where the parameters are random initialized at each trialand fixed between attribution methods.
Figure 6 shows the test accuracy measure in the ROAR [Figure] GIG FG
DGA (Ours) [Figure] experiment for each attribution method. The result indicates that the model trained on the modifieddataset with DGA steeply decreases the test accuracy even with 10% removed.
We conclude that
DGA can extract the features which are relevant to training procedure in DNNs.
4.3 Qualitative comparison
We qualitatively compare the various attribution methods by visualizing the attribution heatmap andtop 10% most relevant feature at the same time.
In Figure 5, we provide the result of randomlyselected images from the validation set of ImageNet with the pre-trained VGG-16.
We can identifythat the attributions are more aligned the object comparing other methods.
For example, in theright-top row, DGA focuses the person who grabs a paddle while almost methods distribute the [Table] [Figure] relevant pixels to sky and ocean.
Although FG concentrates to the person, the relevant patch has lesssharp than patch of DGA.
We provide more examples and results for different models in Appendix.
Discussion
In this paper, we propose a novel gradient-based attribution method, Distilled Gradient Aggregation(DGA).
We provide the vulnerability of FG against the input perturbation and the counter-intuitivebehavior of IG due to the continuous integration path.
To complement the weakness of both methods,we propose the gradient aggregation method along the distillation sequence that generates the impuritydistilled inputs.
Our method obtains high quality attributions with its sharpness and object-alignment,and we verify the method through pixel perturbation and ROAR evaluation metrics.
We believe thatour DGA method can be broadly applied to explain a decision of various DNNs.
Broader Impact Transparency of deep models is a matter of the highest priority for the applicationof such models in the real world, e.g., medical diagnosis [Caruana et al. , 2015] and autonomousdriving [Yurtsever et al. , 2020].
We believe that providing the evidence which is well-aligned withthe model decision would help the users of such applications to place great trust and the developersto improve or fix the model for better performance.
Discovering unintended biases in the model isanother issue [Stock and Cisse, 2018].
Such biases may occur from the dataset [Kim et al.
, 2018]or the model itself.
Identifying the root cause and removing such biases would be another expectedfuture work, beyond the explanation on the input features.
Limitation Although our method has empirically outperformed in qualitative comparison and variousquantitative experiments compared to previous work, the notion of better input attribution methodis still vague.
In this work, we adaptively find the sequence of inputs by using local attribution, butthere would exist better justification of the sequence or the set of inputs that are essential clues foridentifying the core features in the input.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] See Appendix(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Figure 6 includes the error range obtained by multipleruns
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]