Discretized Integrated Gradients for Explaining Language Models (2021)

Abstract: As a prominent attribution-based explanation algorithm, Integrated Gradients (IG) is widely adopted due to its desirable explanation axioms and the ease of gradient computation. It measures feature importance by averaging the model’s output gradient interpolated along a straight-line path in the input data space. However, such straight-line interpolated points are not representative of text data due to the inherent discreteness of the word embedding space. This questions the faithfulness of the gradients computed at the interpolated points and consequently, the quality of the generated explanations. Here we propose Discretized Integrated Gradients (DIG), which allows effective attribution along non-linear interpolation paths. We develop two interpolation strategies for the discrete word embedding space that generates interpolation points that lie close to actual words in the embedding space, yielding more faithful gradient computation. We demonstrate the effectiveness of DIG over IG through experimental and human evaluations on multiple sentiment classification datasets. We provide the source code of DIG to encourage reproducible research.

## introduction in the past few years, natural language processing has seen tremendous progress, largely due to strong performances yielded by pre-trained language models @xcite @xcite . but even with this impressive performance, it can still be difficult to understand the underlying reasoning for the preferred predictions leading to distrust among end-users @xcite . hence, improving model interpretability has become a central focus in the community with an increasing effort in developing methods that can explain model behaviors @xcite @xcite 2017; @xcite @xcite . explanations in nlp are typically represented at a word-level or phrase-level by quantifying the contributions of the words or phrases to the model's prediction by a scalar score. these explanation methods are commonly referred as attributionbased methods @xcite . integrated gradients (ig) @xcite is a prominent attribution-based explanation method used due to the many desirable explanation axioms and ease of gradient computation. it computes the partial derivatives of the model output with respect to each input feature as the features are interpolated along a straight-line path from the given input to a baseline value. for example, say we want to compute the attribution for the word "good" in the sentence "the movie was good!" using ig. the straight-line interpolation path used by ig is depicted in green in figure 1 . here, the baseline word is defined as the "<@xmath0. while this method can be used for attributing inputs in both continuous (e.g., image, audio, etc.) and discrete (e.g., text, molecules, etc.) domains @xcite , their usage in the dis-crete domain has some limitations. since the interpolation is done along a straight-line path joining the input word embedding and the baseline embedding ("<@xmath1, the interpolated points are not necessarily representative of the discrete word embedding distribution. specifically, let a dummy word embedding space be defined by the words represented by black dots in figure 1 . then we can see that some of the green squares can be very far-off from any original word in the embedding space. since the underlying language model is trained to effectively work with the specific word embedding space as input, using these out-of-distribution green interpolated samples as intermediate inputs to calculate gradients can lead to sub-optimal attributions. to mitigate these limitations, we propose a discretized integrated gradients (dig) formulation by relaxing the constraints of searching for interpolation points along a straight-line path. relaxing this linear-path constraint leads to a new constraint on the interpolation paths in dig that points along the path should be monotonically situated between the input word embedding and the baseline embedding. hence, in dig, our main objective is to monotonically interpolate between the input word embedding and baseline such that the intermediate points are close to real data samples. this would ensure that the interpolated points are more representative of the word embedding distribution, enabling more faithful model gradient computations. to this end, we propose two interpolation strategies that search for an optimal anchor word embedding in the real data space and then modify it such that it lies monotonically between the input word and baseline (see fig. 1 for an illustration). we apply dig using our proposed interpolation algorithms to generate attributions for three pre-trained language models -bert @xcite , distilbert @xcite , and roberta @xcite , each fine-tuned separately on three sentiment classification datasets -sst2 @xcite , imdb @xcite , and rotten tomatoes @xcite . we find that our proposed interpolation strategies achieve a superior performance compared to integrated gradients and other gradient-based baselines on eight out of the nine settings across different metrics. further, we also observe that on average, end-users find explanations provided by dig to be more plausible justifications of model behavior than the explanations from other baselines. ## method in this section, we first describe our proposed discretized integrated gradients (dig) and the desirable explanation axioms satisfied by it. then we describe an interpolation algorithm that leverages our dig in discrete textual domains. please refer to appendix a for a brief introduction of the attribution-based explanation setup and the integrated gradients method. ## experimental setup in this section, we describe the datasets and models used for evaluating our proposed algorithm. datasets. the sst2 @xcite dataset has 6920/872/1821 example sentences in the train/dev/test sets. the task is binary classification into positive/negative sentiment. the imdb @xcite dataset has 25000/25000 example reviews in the train/test sets with similar binary labels for positive and negative sentiment. similarly, the rotten tomatoes (rt) @xcite dataset has 5331 positive and 5331 negative review sentences. we use the processed dataset made available by huggingface dataset library foot_1 @xcite . language models. we use pre-trained bert @xcite , distilbert @xcite , and roberta @xcite text classification models individually fine-tuned for sst2, imdb, and rt datasets. 4 the fine-tuned checkpoints used are provided by the huggingface library @xcite . ## related works there has been an increasing effort in developing interpretability algorithms that can help understand a neural network model's behavior by explaining their predictions @xcite . attributions are a post-hoc explanation class where input features are quantified by scalar scores indicating the magnitude of contribution of the features toward the predicted label. explanation algorithms that generate attributions can be broadly classified into two categories -model-agnostic algorithms, like @xcite , input occlusion @xcite , integrated gradients foot_3 @xcite , shap (lundberg and lee, 2017), etc. and model-dependent algorithms, like lrp @xcite , deeplift @xcite , cd @xcite , acd @xcite , soc @xcite , etc. while the model-agnostic algorithms can be used as blackbox explanation tools that can work for any neural network architecture, for the latter, one needs to understand the network's architectural details to implement the explanation algorithm. typically, model-dependent algorithms require specific layer decomposition rules @xcite which needs to be defined for all the components in the model. model-agnostic methods usually work directly with the model outputs and gradients which are universally available. due to the many desirable explanation axioms and ease of gradient computation, there has been several extensions of integrated gradients. for example, @xcite study the effect of saturation in the saliency maps generated by integrated gradients. @xcite extend integrated gradients to certain classes of discontinuous functions in financial domains. further, @xcite use knns and auto-encoders to learn latent paths for rnas. different from prior work, our focus here is to improve integrated gradients specifically for the discrete textual domain. while the idea of learning latent paths for text data is quite interesting, it brings a significant amount of challenge in successfully modeling such a complex latent space and hence, we leave this for future work. ## conclusion in this paper, we proposed discretized integrated gradients (dig) which is effective in explaining models working with discrete text data. further, we proposed two interpolation strategies -dig-greedy and dig-maxcount that generate non-linear interpolation paths for word embedding space. finally, we established the effectiveness of dig over integrated gradients and other gradientbased baselines through experiments on multiple language models and datasets. we also conduct human evaluations and find that dig enhances human trust on model predictions.