Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection (2020)

Abstract: Generating explanations for neural networks has become crucial for their applications in real-world with respect to reliability and trustworthiness. In natural language processing, existing methods usually provide important features which are words or phrases selected from an input text as an explanation, but ignore the interactions between them. It poses challenges for humans to interpret an explanation and connect it to model prediction. In this work, we build hierarchical explanations by detecting feature interactions. Such explanations visualize how words and phrases are combined at different levels of the hierarchy, which can help users understand the decision-making of black-box models. The proposed method is evaluated with three neural text classifiers (LSTM, CNN, and BERT) on two benchmark datasets, via both automatic and human evaluations. Experiments show the effectiveness of the proposed method in providing explanations that are both faithful to models and interpretable to humans.

## introduction deep neural networks have achieved remarkable performance in natural language processing (nlp) @xcite @xcite , but the lack of understanding on their decision making leads them to be characterized as blackbox models and increases the risk of applying them in real-world applications @xcite @xcite . understanding model prediction behaviors has been a critical factor in whether people will trust and use these blackbox models @xcite . a typical work on understanding decisionmaking is to generate prediction explanations for each input example, called local explanation generation. in nlp, most of existing work on local explanation generation focuses on producing wordlevel or phrase-level explanations by quantifying contributions of individual words or phrases to a model prediction @xcite @xcite . figure 1 : different explanations for a negative movie review a waste of good performance, where the color of each block represents the contribution of the corresponding word/phrase/clause (feature) to the model prediction. from the hierarchical explanation, we obtain a set of features in each timestep (t), where the most important one is waste of good. figure @xcite and (b) present a word-level and a phrase-level explanation generated by the lime @xcite and the contextual decomposition (cd) @xcite ) respectively for explaining sentiment classification. both explanations provide scores to quantify how a word or a phrase contributes to the final prediction. for example, the explanation generated by lime captures a keyword waste and the explanation from cd identifies an important phrase waste of. however, neither of them is able to explain the model decision-making in terms of how words and phrases are interacted with each other and composed together for the final prediction. in this example, since the final prediction is negative, one question that we could ask is that how the word good or a phrase related to the word good contributes to the model prediction. an explanation being able to answer this question will give users a better understanding on the model decision-making and also more confidence to trust the prediction. the goal of this work is to reveal prediction behaviors of a text classifier by detecting feature (e.g., words or phrases) interactions with respect to model predictions. for a given text, we propose a model-agnostic approach, called hedge (for hierarchical explanation via divisive generation), to build hierarchical explanations by recursively detecting the weakest interactions and then dividing large text spans into smaller ones based on the interactions. as shown in figure 1 (c ), the hierarchical structure produced by hedge provides a comprehensive picture of how different granularity of features interacting with each other within the model. for example, it shows how the word good is dominated by others in the model prediction, which eventually leads to the correct prediction. furthermore, the scores of text spans across the whole hierarchy also help identify the most important feature waste of good, which can be served as a phrase-level explanation for the model prediction. the contribution of this work is three-fold: (1) we design a top-down model-agnostic method of constructing hierarchical explanations via feature interaction detection; (2) we propose a simple and effective scoring function to quantify feature contributions with respect to model predictions; and our implementation is available at https:// github.com/uva-nlp/hedge. ## related work over the past years, many approaches have been explored to interpret neural networks, such as contextual decomposition (cd) for lstm @xcite or cnn model @xcite , gradient-based interpretation methods @xcite , and attention-based methods @xcite @xcite . however, these methods have limited capacity in realworld applications, as they require deep understanding of neural network architectures @xcite or only work with specific models @xcite . on the other hand, model-agnostic methods @xcite generate explanations solely based on model predictions and are applicable for any black-box models. in this work, we mainly focus on model-agnostic explanations. ## method this section explains the proposed algorithm on building hierarchical explanations (subsection 3.1) and two critical components of this algorithm: detecting feature interaction (subsection 3.2) and quantifying feature importance (subsection 3.3). ## experiments the proposed method is evaluated on text classification tasks with three typical neural network models, a long short-term memories @xcite , a convolutional neural network @xcite , and bert @xcite , on the sst @xcite and imdb @xcite datasets, via both automatic and human evaluations. ## conclusion in this paper, we proposed an effective method, hedge, building model-agnostic hierarchical interpretations via detecting feature interactions. in