Interpretation of NLP models through input marginalization (2020)

Abstract: To demystify the “black box” property of deep neural networks for natural language processing (NLP), several methods have been proposed to interpret their predictions by measuring the change in prediction probability after erasing each token of an input. Since existing methods replace each token with a predefined value (i.e., zero), the resulting sentence lies out of the training data distribution, yielding misleading interpretations. In this study, we raise the out-of-distribution problem induced by the existing interpretation methods and present a remedy; we propose to marginalize each token out. We interpret various NLP models trained for sentiment analysis and natural language inference using the proposed method.

## introduction the advent of deep learning has greatly improved the performances of natural language processing (nlp) models. consequently, the models are becoming more complex @xcite , rendering it difficult to understand the rationale behind their predictions. to use deep neural networks (dnns) for making high-stakes decisions, the interpretability must be guaranteed to instill the trust in the public. hence, various attempts have been undertaken to provide an interpretation along with a prediction @xcite . research in computer vision aims to interpret a target model by measuring attribution scores, i.e., how much each pixel in an input image contributes to the final prediction @xcite @xcite . since a pixel of an image corresponds to a token in a sentence, the attribution score of each token can provide an insight into the nlp model's internal reasoning process. a straightforward approach is to ask, "how would the model reaction change if each token was not there?" and *correspondence to: sungroh yoon (sryoon@snu.ac.kr) in this study, we ask instead; "how would the model react differently if there were other tokens instead of each token?", as proposed by @xcite ; @xcite . we propose to marginalize each token out to mitigate the ood problem of the existing erasure scheme. during the marginalization, our method measures the contribution of all probable candidate tokens considering their likelihoods. to calculate the likelihoods, we use the masked language modeling (mlm) of bidirectional encoder representations from transformers (bert) @xcite . our contributions are as follows: ## methods we propose input marginalization to mitigate the ood issue. in the following subsections, we measure the attribution score using the weight of evidence and marginalize over all possible candidate tokens using the mlm of bert. we extend the method to multi-token cases and introduce adaptively truncated marginalization for an efficient computation. finally, we propose a new metric, auc rep , to evaluate the proposed method faithfully. the overall algorithm is provided in algorithm 1. ## conclusion interpretability is becoming more important owing to the increase in deep learning in nlp. hence, several interpretation methods have been proposed, and we reviewed their limitations throughout the paper. among them, we focused on the ood problem arising from the widely used zero erasure scheme, which results in misleading interpretation. to the best of our knowledge, neither the ood problem has been raised in interpreting nlp models nor the attempt to resolve it has been undertaken. our proposed input marginalization, which can mitigate the ood problem, can result in a faithful interpretation, thereby enabling better understanding of "black box" dnns. the scope of this study was primarily focused on interpreting dnns for sentiment analysis and natural language inference. regarding the model-agnostic and task-agnostic properties of our method, they are applicable to any types of nlp model for various tasks, such as neural machine translation and visual question answering. it will be meaningful to interpret the state-of-the-art models such as xlnet @xcite and elec-tra @xcite . in addition, as experimentally analyzed, the interpretation result of our method is affected by the likelihood modeling performance. we can expect even more faithful interpretation if the modeling performance improves.