Considering Likelihood in NLP Classification Explanations with Occlusion and Language Modeling (2020)

Abstract: Recently, state-of-the-art NLP models gained an increasing syntactic and semantic understanding of language, and explanation methods are crucial to understand their decisions. Occlusion is a well established method that provides explanations on discrete language data, e.g. by removing a language unit from an input and measuring the impact on a model’s decision. We argue that current occlusion-based methods often produce invalid or syntactically incorrect language data, neglecting the improved abilities of recent NLP models. Furthermore, gradient-based explanation methods disregard the discrete distribution of data in NLP. Thus, we propose OLM: a novel explanation method that combines occlusion and language models to sample valid and syntactically correct replacements with high likelihood, given the context of the original input. We lay out a theoretical foundation that alleviates these weaknesses of other explanation methods in NLP and provide results that underline the importance of considering data likelihood in occlusion-based explanation.

## introduction explanation methods are a useful tool to analyze and understand the decisions made by complex nonlinear models, e.g. neural networks. for example, they can attribute relevance scores to input features (e.g. word or sub-word units in nlp). nevertheless, explanation methods can be misleading @xcite and they need to be analyzed for their well-foundedness. gradient-based methods provide explanations by analyzing local infinitesimal changes to determine the shape of a network's function. the implicit assumption is that the local shape of a function is there are discrete inputs, i.e. combination of tokens, with a data likelihood greater than zero. all other inputs in the embedding space have likelihood zero because they have no corresponding tokens. occlusion methods (green) create unlikely input. gradient-based explanation methods (red arrow) consider infinitesimal changes to the input and thus data with no likelihood. in nlp, however, the input consists of natural language, which is discrete, i.e., the data that has positive likelihood is a discrete distribution (see figure 1 ). this means that local neighborhoods need not be indicative of the model's prediction behaviour and a model's prediction function at points with zero likelihood need not be relevant to the model's decision. thus, we argue that black-box method relevances max. value olm (ours) forced , familiar and thoroughly condescending . 0.76 olm-s (ours) forced , familiar and thoroughly condescending . 0.47 delete forced , familiar and thoroughly condescending . 1 unk forced , familiar and thoroughly condescending . 0.35 sensitivity analysis forced , familiar and thoroughly condescending . 0.025 gradient*input forced , familiar and thoroughly condescending . 0.00011 integrated gradients forced , familiar and thoroughly condescending . 0.68 occlusion is a well suited method due to its ability to produce explanations on data with discrete likelihood. for example, by replacing or deleting a language unit in the original input and measuring the impact on the model's prediction. however, the likelihood of the replacement data is usually low. consider, for example, a sentiment classification task and assume a model that assigns syntactically incorrect inputs a negative sentiment. it correctly predicts "it 's a masterpiece ." as positive, but assigns negative sentiment to syntactically incorrect inputs produced by occlusion, e.g. "it 's a ." or "it 's @xmath0.", which have low data likelihood (see figure 1 ). this may result in a large prediction difference for many tokens in a positive sentiment example and no prediction difference for many tokens in a negative sentiment example (see we argue that current nlp state-of-the-art models have increasing syntactic @xcite and hierarchical @xcite understanding. therefore, methods that explain these models should consider syntactically correct replacement that is likely given the unit's context, e.g. in figure 1 "classic" or "failure" as replacements for "masterpiece" in "it 's a masterpiece ." our experiments show that presenting these models with perturbed ungrammatical input changes the explanations. ## methods in this section, we introduce our novel explanation method that combines occlusion with language modeling. instead of deleting or replacing a linguistic unit in the input with an unlikely replacement, olm substitutes it with one generated by a language model. this produces a contextualized distribution of valid and syntactically likely reference inputs and allows a more faithful analysis of models with increasing syntactic capabilities. this is followed by an axiomatic analysis of olm's properties. finally, we introduce olm-s, an extension that measures sensitivity of a model at a feature's position. for our approach we employ the difference of probabilities formula from @xcite . let x i be an attribute of input x and x \i the incomplete input without this attribute. then the relevance r given the prediction function f and class c is note that f c (x \i ) is not accurately defined and needs to be approximated, as x \i is an incomplete input. for vision, @xcite approximate f c (x \i ) by using the input data distribution p data to sample xi independently of x or use a gaussian distribution for xi conditioned on surrounding pixels. we argue sampling should be conditioned on the whole input and depend on the probability of the data distribution. we argue that in nlp a language model p lm generates input that is as natural as possible for the model and thus approximate in general, x i should be units of interest such as phrases, words or subword tokens. thus, olm's relevance for a language unit is the difference in prediction between the original input and inputs with the unit resampled by conditioning on information in its context. the relevance of every language unit is in the interval [-1, 1], with the sign indicating contradiction or support, and can be interpreted as the value of information added by the unit for the model. ## experiments in our experiments, we aim to answer the following question: do relevances produced by our method differ from those that either ignore the discrete structure of language data or produce syntactically incorrect input, and if so, how? we first train a state-of-the-art nlp model (roberta, @xcite on three sentence classification tasks (section 3.2). we then compare the explanations produced by olm and olm-s to five occlusion and gradient-based methods (section 3.1). to this end, we calculate the relevances of words over a whole input regarding the true label. we calculate the pearson correlation coefficients of these relevances for every sentence and average this over the whole development set of each task. in our experiments we use bert base @xcite for olm resampling. ## conclusion we argue that current black-box and gradient-based explanation methods do not yet consider the likelihood of data and present olm, a novel explanation method, which uses a language model to resample occluded words. it is especially suited for word-level relevance of sentence classification with state-of-the-art nlp models. we also introduce the class zero-sum axiom for explanation methods, compare it with an existing axiom. furthermore, we show other axioms that olm satisfies. we argue that with this more solid theoretical foundation olm can be regarded as an improvement over existing nlp classification explanation methods. in our experiments, we compare our methods to other occlusion and gradient explanation methods. we do not consider these experiments to be exhaustive. unfortunately, there is no general evaluation for explanation methods. we show that our method adds value by showing distinctive results and better founded theory. a practical difficulty of olm is the approximation with a language model. first, a language model can create syntactically correct data, that does not make sense for the task. second, even state-ofthe-art language models do not always produce syntactically correct data. however, we argue that using a language model is a suitable way for finding reference inputs. in the future, we want to extend this method to language features other than words. nlp tasks with longer input are probably not very sensitive to single word occlusion, which could be measured with olm-s.