Sequential Integrated Gradients: a simple but effective method for explaining language models (2023)

Abstract: Several explanation methods such as Integrated Gradients (IG) can be characterised as path-based methods, as they rely on a straight line between the data and an uninformative baseline. However, when applied to language models, these methods produce a path for each word of a sentence simultaneously, which could lead to creating sentences from interpolated words either having no clear meaning, or having a significantly different meaning compared to the original sentence. In order to keep the meaning of these sentences as close as possible to the original one, we propose Sequential Integrated Gradients (SIG), which computes the importance of each word in a sentence by keeping fixed every other words, only creating interpolations between the baseline and the word of interest. Moreover, inspired by the training procedure of language models, we also propose to replace the baseline token “pad” with the trained token “mask”. While being a simple improvement over the original IG method, we show on various models and datasets that SIG proves to be a very effective method for explaining language models.

## introduction language models such as bert @xcite have demonstrated to be effective on various tasks, for instance on sentiment analysis @xcite , machine translation @xcite ), text summarization @xcite or intent classification @xcite . however, with the increased performance and usage of such models, there has been a parallel drive to develop methods to explain predictions made by these models. indeed, bert and its variations are complex models which do not allow a user to easily understand why a certain prediction has been produced. on the other hand, it is important to be able to explain a 1 an implementation of this work can be found at https: //github.com/josephenguehard/time_interpret figure 1 : comparison between ig, dig, and our method: sig. while dig improves on ig by creating discretized paths between the data and the baseline, it can produce sentences with a different meaning compared to the original one. our method tackles this issue by fixing every word to their true value except one, and moving the remaining word along a straight path (sig) model's predictions, especially when this model is used to make high-stake decisions, or when there is a risk of a discriminating bias, for instance when detecting hate speech on social media @xcite . as a result, developing effective methods to explain not only language models, but also machine learning models in general, has recently gained significant attention. many different methods have therefore been proposed such as: lime @xcite , grad*inp @xcite , integrated gradients (ig) @xcite , deeplift @xcite or @xcite . among these methods, some can be characterised as path-based, which means that they rely on a straight line between the data and an uninformative baseline. for instance, ig computes gradients on interpolated points along such a path, while deeplift and gradientshap can be seen as approximations of ig @xcite . while these methods aim to be used on any type of models and data, some have been tailored to the specificity of language models. for instance, @xcite challenge the use of continuous paths on a word embedding space which is inherently discrete. they propose as a result discretized integrated gradient (dig), which replaces the continuous straight path with a discretized one, where interpolated points are words. in our work, we suggest another potential issue when applying path-based explanation methods on language models. these models are usually designed to be used on individual or multiple sentences, in order to perform for instance sentiment analysis or question answering. however, a path-based method applied on such models creates straight lines between each word and a baseline simultaneously. when interpolated points are grouped together to form a sentence, this sentence could have a very different meaning compared with the original one. as a result, we propose a simple method to alleviate this potential issue: computing the importance of each word in a sentence or a text by keeping fixed every other word and only creating interpolations between the baseline and the word of interest. after computing the importance of each word in this way, we normalise these attributions across the sentence or text we aim to explain. we call this method sequential integrated gradients (sig), as, although we focus in this work on language models, such a method could be used on any sequential modelling. we also propose to use the token "mask" as a baseline, when possible, as its embedding has been trained to replace part of sentences when training language models. as a result, our method follows closely the training procedure of these models. in this setting, we keep the baseline as similar to the original sentence as possible, only changing the word of interest. this method of explaining a word is also kept similar to the way these language models are usually pre-trained, by randomly masking part of sentences. let's now define our sequential integrated gradients (sig) method. for a word x i and a feature j, sig is defined as: similar to the original ig @xcite , we compute the gradient of f along a straight line between x i and x for each word x i , the main difference being that the baseline differs for each word. also similar to the original ig, we approximate in practice the integral with riemann summation. finally, we compute the overall attribution of a word by computing the sum over the feature dimension j, and normalising the result: ## conclusion in this work, we have defined an attribution method specific to text data: sequential integrated gradients (sig). we have shown that sig yields significantly better results than the original integrated gradients (ig), as well as other methods specific to language models, such as discretized integrated gradients (dig). this suggests that keeping the meaning of interpolated sentences close to the original one is key to producing good explanations. we have also shown that, although sig can be computationally intensive, reducing the number of interpolations still yields better results than ig with a greater number of interpolations. we have also highlighted in this work the benefit of using the token "mask" as a baseline, instead of "pad". although sig seems to be robust across both tokens, this is especially important when using ig, as it significantly improves the quality of explanations. using the trainable token "mask" is indeed closer to the training procedure of language models, and should yield better interpolations as a result. we recommend therefore using this token as a baseline, when possible, when explaining predictions made by a language model. moreover, while this study was conducted on bidirectional language models such as bert, sig could also be used on auto-regressive models such as gpt-2 @xcite , by iteratively computing the attribution of a token, while keeping previous tokens fixed, and masking future tokens if any has been already computed.