Integrated Directional Gradients: Feature Interaction Attribution for Neural NLP Models (2021)

Abstract: In this paper, we introduce Integrated Directional Gradients (IDG), a method for attributing importance scores to groups of features, indicating their relevance to the output of a neural network model for a given input. The success of Deep Neural Networks has been attributed to their ability to capture higher level feature interactions. Hence, in the last few years capturing the importance of these feature interactions has received increased prominence in ML interpretability literature. In this paper, we formally define the feature group attribution problem and outline a set of axioms that any intuitive feature group attribution method should satisfy. Earlier, cooperative game theory inspired axiomatic methods only borrowed axioms from solution concepts (such as Shapley value) for individual feature attributions and introduced their own extensions to model interactions. In contrast, our formulation is inspired by axioms satisfied by characteristic functions as well as solution concepts in cooperative game theory literature. We believe that characteristic functions are much better suited to model importance of groups compared to just solution concepts. We demonstrate that our proposed method, IDG, satisfies all the axioms. Using IDG we analyze two state-of-the-art text classifiers on three benchmark datasets for sentiment analysis. Our experiments show that IDG is able to effectively capture semantic interactions in linguistic models via negations and conjunctions.

## introduction in the last decade deep neural networks (dnn) have been immensely successful. much of this success can be attributed to their ability to learn from complex higher order interactions from raw features @xcite . this success of dnns has led to them being increasingly adopted * equal contribution for algorithmic decision making. this in turn has led to increasing concerns over explainability and interpretability of these models, given the important role they are beginning to take in society @xcite . one area of work that has emerged in recent years is that of black box model explanation strategies that "explain" the output of a dnn for a given input using feature attribution scores or saliency maps @xcite . numerous studies have been published in recent years proposing different strategies to answer the question "which features in the input were most important in deciding the output of the dnn?" however, modern dnns take as input raw data as features, and learn from higher order interaction of those features. thus in the past year a number of studies have instead focused on explaining feature interactions rather than explaining individual features @xcite @xcite @xcite . one issue that remains, however, is that given two methods for attributing importance scores, it is not entirely straight forward to objectively compare them. as has been noted by earlier studies @xcite , if the output of an attribution method seems non-intuitive it is not easy to answer if that is caused by (i) limitations of the attribution method, (ii) limitations of the dnn model being explained, or (iii) limitation of the data on which the dnn model was trained. like multiple previous studies @xcite @xcite we take an axiomatic approach to this problem, whereby we first define the set of properties/axioms that a "good" solution must satisfy, followed by development of a solution that satisfies those axioms. the method for computing feature group attribution (interchangeably referred to as feature inter-action attribution) presented in this study is called integrated directional gradients or idg. like multiple earlier methods in this area, idg is a cooperative game theory inspired method. however, unlike earlier cooperative game theory inspired axiomatic methods which only borrowed axioms from solution concepts (such as shapley value) for individual feature attributions and introduced their own extensions to model interactions, our formulation is inspired by axioms satisfied by well behaved characteristic functions as well as solution concepts in cooperative game theory literature. we find that well behaved characteristic functions provide a much simpler and intuitive framework for defining axioms for group attributions. we apply idg on state-of-the-art models on the nlp domain. as part of its input idg requires a set of meaningful feature sets, that have a hierarchical structure (section 2.1). in this paper we use parse tree of sentences to construct the meaningful feature structures. figure 1 shows an illustrative example of the nature of explanations and attributions computed using idg. the major contributions of the current work are as follows: ## results to precisely visualize the interactions between phrases, we search over the test examples for instances of negations. we follow the methodology proposed in @xcite . in specific, we look into the parse tree for each review and check if the left child consists of a negation phrase (e.g., lacks, never etc.) in the first two words and the right child has a positive or a negative sentiment. since for sst, each phrase is also annotated with their corresponding sentiment labels in the form of a constituency parse tree, this can be easily ob- for figure 2 (a) the classification model is xlnet-base and the ground truth as well as the inferred class is negative. the first part (though everything might be literate and smart) has a positive sense. but when appended with the second part (it never took off and always seemed static), a negative sense is manifested. this is captured by the classification model as demonstrated by our framework. for the example in figure 2(b ), the classifier model is bertitpt and the inferred as well as the ground-truth class is negative. this example consists of two sentences while the first one nice atmosphere has a positive sense, when combined with the second sentence cheeseburger was not at all that, the overall sense turns negative. this is again conveniently manifested in the scores assigned by our framework. we also report the results on imdb reviews @xcite in appendix. ## related work feature attribution based method. these methods essentially assign importance scores to individual features thereby explaining the decisions of the classifier model. the scores are mostly calculated by either backpropagating a custom relevance score @xcite or directly using the gradients. the gradient based methods aim to calculate the sensitivity of the inference function with respect to the input features and thereby measuring its importance. the method was first introduced in @xcite and further investigated in @xcite . @xcite adopts an axiomatic approach and deem it to be more suitable as the feature attribution methods are hard to evaluate empirically. the other set of methods usually backpropagates their custom relevance scores down to the input to identify relevance of an input feature @xcite @xcite . unlike the gradient based methods, these are not implementation invariant (i.e., the back propagation process is architecture specific). game theoretic aspect. (lundberg and lee, 2017) adopts results (shapely values in specific) from coalition game theory to obtain feature attribution scores. the key idea is to consider the features as individual players involved in a coalition game of prediction which is considered the payout. the payout then can be fairly distributed among the players (features) to measure their importance. this has been further explored in @xcite @xcite . quantifying feature interactions. the methods mentioned above fail to properly capture the importance of feature interaction. @xcite proposes to capture pair-wise interaction by building upon integrated gradients framework. @xcite learns global pair-wise interactions in bayesian neural networks. @xcite introduces contextual decomposition to capture interaction among words in a text for a lstm-based classifier. @xcite further extends the method to other architectures. more recent research endeavors in this direction include @xcite @xcite . we elaborate more on the methods closest to our work in section 3. ## conclusion in this paper we investigated the problem of feature group attribution and proposed a set of axioms that any framework for feature group attribution should fulfill. we then introduced idg, a novel method, as a solution to the problem and demonstrated that it satisfies all the axioms. through experiments on real-world datasets with state-ofthe-art dnn based classifiers we demonstrated the effectiveness of idg in capturing the importance of feature groups as deemed by the classifier.