Asymmetric feature interaction for interpreting model predictions (2023)

Abstract: In natural language processing (NLP), deep neural networks (DNNs) could model complex interactions between context and have achieved impressive results on a range of NLP tasks. Prior works on feature interaction attribution mainly focus on studying symmetric interaction that only explains the additional influence of a set of words in combination, which fails to capture asymmetric influence that contributes to model prediction. In this work, we propose an asymmetric feature interaction attribution explanation model that aims to explore asymmetric higher-order feature interactions in the inference of deep neural NLP models. By representing our explanation with an directed interaction graph, we experimentally demonstrate interpretability of the graph to discover asymmetric feature interactions. Experimental results on two sentiment classification datasets show the superiority of our model against the state-of-the-art feature interaction attribution methods in identifying influential features for model predictions.

## introduction deep neural networks (dnns) have demonstrated impressive results on a range of natural language processing (nlp) tasks. unlike traditional models (e.g. crfs and hmms) that optimize weights on human interpretable features, deep neural models operate like a black box by applying multiple layers of non-linear transformation on the vector representations of text data, which fails to provide insights to understand the inference process of deep neural models over the features (e.g. words and phrases) involved in modeling. interpreting the prediction of a black box model could help understand model inference behaviors and increase user trust in applying the model to realworld applications. prior efforts in nlp mainly focus on quantifying the contributions of individual * corresponding author. word or word interactions to the prediction. fig. 1 demonstrates word-level and pairwise word interaction explanations for a sentiment classification task, where the word "not" and the interaction between "not" and "funny" contribute positively to the prediction negative. figure 1 : explanations for a negative movie review (computed by shapley value and shapley interaction index), where the color indicates contribution of the corresponding word/pairwise word interaction to the model prediction. figure 2 : symmetric versus asymmetric pairwise interaction (computed by our method) where the directed edge @xmath0. the presence of "very" does not influence "funny" much while "funny" further modifies "very" and thus the interaction influence of "funny" → "very" is stronger than that of "very" → "funny". studying word interaction could help identify to what extent a set of words exert influence in combination as opposed to independently. however, most interaction attribution methods assume symmetric interaction, which may fail to capture asymmetric influence that contributes to model prediction. fig. 2 presents some symmetric pairwise interactions with graph representation for the instance in fig. 1 , where words becomes nodes and edges between words represent interaction. in individuallevel explanation "funny" has negative influence while the symmetric interaction between "funny" and "not" produces positive influence to model prediction. therefore the influence of the presence of "not" to "funny" is not the same as that of the presence of "funny" to "not". "funny" has weak positive contribution in the presence of "not" and "funny" further modifies "not", the interaction influence of "funny" → "not" could be stronger than the interaction of "not" → "funny". for the ideal asymmetric interaction, the presence of other features should not negate the positive influence of the important feature and other features would lose their influence when important features are present. constructing asymmetric interaction graph for the predicted instance could help human have a nuanced understanding toward the inference of deep nlp models. in this paper, our work aims to provide the explanation that incorporates asymmetric feature interaction foot_0 . the contributions are summarized as follows: ## asymmetric shapley interaction value this section first revisits shapley value for explaining model prediction, then describes the definition of asymmetric shapley interaction value and the corresponding approximating computation. the following notations will be used throughout the paper. for a classification task, given a text sequence @xmath1, ..., x n ) and a trained model f , ŷ is the prediction label and f (•) denotes the model output probability on ŷ. ## conclusion in this paper we propose an asymmetric feature interaction attribution method to explain asymmetric higher-order feature interactions in the inference of deep neural nlp models. we extend shapley value to asymmetric shapley interaction value and investigate three different sampling strategies in computing marginal contribution of value function in nlp field. by evaluating our proposed model and five model-agnostic feature interaction attribution methods on two sentiment datasets with bert and roberta, our model achieves the best performance in identifying the influential words for model prediction. also, for the three different sampling strategies we empirically show that none is generally better than the others or more broadly applicable, which could provide guidelines for the selection of reference distribution in nlp field. for example, random sampling strategy is effective for the short text sequence and for the long text sequence in-domain sampling could produce more smooth and domain-dependent context. in the future, we consider generating sparse and differential causal structure for explaining model prediction.