Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining (2022)

Abstract: To explain NLP models a popular approach is to use importance measures, such as attention, which inform input tokens are important for making a prediction. However, an open question is how well these explanations accurately reflect a model’s logic, a property called faithfulness. To answer this question, we propose Recursive ROAR, a new faithfulness metric. This works by recursively masking allegedly important tokens and then retraining the model. The principle is that this should result in worse model performance compared to masking random tokens. The result is a performance curve given a masking-ratio. Furthermore, we propose a summarizing metric using area-between-curves (ABC), which allows for easy comparison across papers, models, and tasks. We evaluate 4 different importance measures on 8 different datasets, using both LSTM-attention models and RoBERTa models. We find that the faithfulness of importance measures is both model-dependent and task-dependent. This conclusion contradicts previous evaluations in both computer vision and faithfulness of attention literature.

## introduction the ability to explain neural networks benefits both accountability and ethics when deploying models @xcite and helps develop a scientific understanding of what models do @xcite . particularly, in nlp, attention @xcite is often used as an explanation to provide insight into the logical process of a model @xcite . attention, among other methods such as gradient @xcite and integrated gradient @xcite , explain which input tokens * equal contribution. a major challenge in the field of interpretability is ensuring that an explanation is faithful: "a faithful interpretation is one that accurately represents the reasoning process behind the model's prediction" @xcite . unfortunately, importance measures that are claimed to have strong theoretical foundations and are widely used in practice @xcite often later turn out to be questionable @xcite @xcite @xcite . accurately measuring if an explanation is faithful is therefore paramount. such faithfulness metrics are difficult to develop as the models are too complex to know what the correct explanation is. @xcite says a faithfulness metric should use "some formal definition of interpretability as a proxy for explanation quality." in this work, we use the definition of faithfulness by @xcite and @xcite : if information (input tokens) is truly important, then removing it should result in a worse model performance compared to removing random information (tokens). we build upon the roar metric by @xcite , which adds that it is necessary to retrain the model after information is removed, to avoid out-of-distribution issues. finally, the model performance is compared with removing random information. a limitation of roar is that it is theoretically impossible to measure the faithfulness of an importance measure when dataset redundancies exist. for example, if two tokens are equally relevant but only one of them is identified as important, roar fails to remove the second token. we propose recursive roar which solves this limitation. in addition to the recursive roar metric, we introduce a summarizing metric (racu) which aggregates the results into a scalar metric. we hope that such a metric will make it more feasible to compare importance measures across papers. using the proposed faithfulness metrics, we perform a comprehensive comparative study of 4 different importance measures and two popular architectures: bilstm-attention and roberta @xcite . we use 8 different datasets which are commonly used in the faithfulness of attention literature @xcite . our comparative study reveals that no importance measure is consistently better than others. instead, we find that faithfulness is both task and model dependent. this is valuable knowledge, as although each importance measure might be equal in faithfulness, they are not equal in computational requirements or understandability to humans. in particular, we find that attention generally provides more sparse explanations than gradient or integrated gradient. although their faithfulness may be the same, a sparser explanation is often easier for humans to understand @xcite . computationally speaking, integrated gradient is approximately 50 times more expensive than the gradient method. this additional complexity is usually justified by being considered more faithful than gradient. however, our results indicate that this is rarely a worthwhile trade-off. ## related work much recent work in nlp has been devoted to investigating the faithfulness of importance measures, particularly attention. in this section, we categorize these faithfulness metrics according to their underlying principle and discuss their drawbacks. roar @xcite and our recursive roar metrics differ significantly from these approaches. the works on attention are all based on the bilstm-attention models and datasets from @xcite , they are therefore highly comparable. we use the same models and datasets, while also analyzing roberta. ## roar: remove and retrain to address the shortcomings of the current faithfulness measures as described in section 2, we base our metric on roar @xcite . roar has been used in computer vision to evaluate the faithfulness of importance measures and to a limited extent in nlp @xcite . the central idea is that if information is truly important, then removing it from the dataset and retraining a model on this reduced dataset should worsen model performance. this can then be compared with an uninformative baseline, where information is removed randomly. for example, at a step size of 10%, one can remove the top-{10%, 20%, • • • 90%} allegedly important tokens, evaluate the model performance, and compare this with removing {10%, 20%, • • • 90%} random tokens. if the importance measures is faithful, the former should result in a worse model performance than the latter. this section covers how roar is adapted to an nlp context. furthermore, we explain the dataset redundancy issue which is solved by our proposed recursive roar metric. finally, we show that recursive roar is an improvement on roar using a synthetic task. ## importance measures in this section, we describe the importance measures that will be evaluated. we choose these explanations as they are common and computationally feasible to evaluate on every observation. as attention does not attend to the begin-ofsequence token, end-of-sequence token, and auxiliary sequence in paired-sequence problems, these tokens are also not considered for other importance measures. this is to ensure a fair comparison. attention these are the attention weights of a bilstm-attention model. we repeat the definitions in appendix c.1. while we also look at a transformer-based model which also have internal attention mechanisms, these models do not provide one specific way to convert attention scores into an importance measure. there are proposals to turn the many attention heads into an importance measure @xcite . however, these are computationally expensive and requires knowing which layer to select. performing this analysis is a standalone research topic which we will not answer. gradient let the logits be denoted as f (x). then the gradient explanation is ∇ x f (x), where x is a one-hot-encoding of the input @xcite . to reduce away the vocabulary dimension, we use an l 2 -norm. input times gradient this explanation is x ∇ x f (x). note that because x is a one-hot encoding, only one element per token will be non-zero. this non-zero element is considered as the explanation. integrated gradient (ig) @xcite argue this to be more faithful, via axiomatic proofs, compared to previous gradient-based methods. a disadvantage is that it is significantly more computationally intensive as it requires computing k gradients. we use @xmath0, and use @xmath1: the datasets, performance metrics, and the bilstm-attention model are identical to those used in @xcite and most other literature evaluating the faithfulness of attention. in addition, we use the roberta-base model with the standard fine-tuning procedure @xcite . details are in appendix c foot_0 . we report model performance on the 8 studied datasets in ## important findings based on the results in figure 4 and faithfulness is model-dependent. in particular, the faithfulness with snli is highly modeldependent as seen in faithfulness is task-dependent. for bilstm-attention, in attention can be faithful. in because attention is computationally free and attention is more sparse (appendix e), which is important for human understanding @xcite , attention can be an attractive explanation. integrated gradient is not necessarily more faithful than gradient or input times gradient. for bilstm-attention, in importance measures often work best for the top-20% most important tokens. in figure 4 , we observe that the largest drop tends to happen at about 10% or 20% tokens masked. this indicates that importance measures are best at ranking the most important tokens, while for less important tokens, they become noisy. this is particularly observed in babi for both models and diabetes with the bilstm-attention model. class leakage can cause the model performance to increase. because the importance measures explain predictions of the target label, they can leak the target label when allegedly important tokens are masked. consider a sentiment classification task. if an importance measure indicates that the word bad is a strong indicator of negative sentiment, then in the next iteration bad would be masked in negative sentences. this means the presence of bad now leaks the true label (positive sentiment) which may increase the performance. this issue is particularly observed with babi-3 using roberta in figure 4 , where the performance increases slightly at 60% tokens masked. this issue affects both roar and recursive roar (appendix g). in fact, it likely affects most faithfulness metrics. however, recursive roar can mitigate this issue to some extent. we discuss this more in appendix a. ## conclusion we show that recursive roar is an improvement on roar. in a synthetic setting, recursive roar matches the ground truth, while roar does not. additionally, we argue why other faithfulness metrics may be either invalid or limited in scope. we then use recursive roar to measure the faithfulness of the most common importance measures, including attention. this is done on both recurrent and transformer-based neural models. in general, we find that the faithfulness of importance measures is both model-dependent and taskdependent. this means that no general recommendation can be made for nlp practitioners considering the current importance measures. instead, it is necessary to measure the faithfulness of different importance measures given a task and a model. because recursive roar works on real-world datasets and not just synthetic problems, we hope it can serve as a standardized benchmark for the faithfulness of importance measures in nlp.