Contextual explanation rules for neural clinical classifiers (2021)

Abstract: Several previous studies on explanation for recurrent neural networks focus on approaches that find the most important input segments for a network as its explanations. In that case, the manner in which these input segments combine with each other to form an explanatory pattern remains unknown. To overcome this, some previous work tries to find patterns (called rules) in the data that explain neural outputs. However, their explanations are often insensitive to model parameters, which limits the scalability of text explanations. To overcome these limitations, we propose a pipeline to explain RNNs by means of decision lists (also called rules) over skipgrams. For evaluation of explanations, we create a synthetic sepsis-identification dataset, as well as apply our technique on additional clinical and sentiment analysis datasets. We find that our technique persistently achieves high explanation fidelity and qualitatively interpretable rules.

## introduction understanding and explaining decisions of complex models such as neural networks has recently gained a lot of attention for engendering trust in these models, improving them, and understanding them better @xcite @xcite . several previous studies developing interpretability techniques provide a set of input features or segments that are the most salient for the model output. approaches such as input perturbation and gradient computation are popular for this purpose @xcite . a drawback of these approaches is the lack of information about interaction between different features. while heatmaps @xcite and partial dependence plots (lundberg and lee, 2017) are popularly used, they only provide a qualitative view which quickly * research conducted while at clips. gets complex as the number of features increases. to overcome this limitation, rule induction for model interpretability has become popular, which accounts for interactions between multiple features and output classes @xcite @xcite @xcite @xcite . most of these work treat the explained models as black boxes, and fit a separate interpretable model on the original input to find rules that mimic the output of the explained model. however, because the interpretable model does not have information about the parameters of the complex model, global explanation is expensive, and the explaining and explained models could fit different curves to arrive to the same output. @xcite incorporates model gradients in the explanation process to overcome these challenges, but this technique cannot be used with current stateof-the-art models that use word embeddings due to their reliance on interpretable model input in the form of bag-of-words. @xcite explain long short term memory networks (lstms) @xcite ) by means of ngram rules, but their rules are limited to presence of single ngrams and do not capture interaction between ngrams in text. to learn explanation rules for rnns while overcoming the limitations of the previous approaches, we have the following contributions in the paper: ## explanation pipeline we propose a method to find decision lists as explanation rules for rnns with word embedding input. we quantify word importance in an rnn by comparing multiple pooling operations (qualitatively and quantitatively). after establishing a desired pooling technique, we move to finding importance of skipgrams, which provides larger context around words in explanations. we then find decision lists that associate the relative importance of multiple skipgrams in the rnn to an output class. this is an extension of our prior work @xcite where we find if-then-else rules for feedforward neural networks. however, the previous approach relies on using interpretable inputs independent of word order and is not scalable to the current stateof-the-art approaches that use word embeddings instead. moreover, explanation of binary classifiers is not supported by that pipeline, and the explanation rules are not generalized to unseen examples. furthermore, the previous explanation rules are hierarchical, and hence cannot be understood independently without parsing the entire rule hierarchy. in the proposed research, we address all these limitations and extend the explanations to binary cases, unseen data, and to sequential neural networks with word embedding input. additionally, these explanation rules can be understood as an independent decision path. we present the complete pipeline for our approach, which we name unravel, in figure 1 . code for the paper is available on https: //github.com/clips/rnn_expl_rules . ## conclusions and future work we have successfully developed a pipeline to obtain transferable, accurate gradient-informed explanation rules from rnns. we have constructed a synthetic dataset to qualitatively and quantitatively evaluate the results, and we obtain informative explanations with high fidelity scores. we obtain similar results on clinical datasets and sentiment analysis. our approach is transferable to all similar neural models. in future, it would be interesting to extend the capabilities of this approach to obtain more accurate, less complex and scalable explanations for classifiers with more complex patterns.