Faithfulness and the Notion of Adversarial Sensitivity in NLP Explanations (2024)

Abstract: Faithfulness is arguably the most critical metric to assess the reliability of explainable AI. In NLP, current methods for faithfulness evaluation are fraught with discrepancies and biases, often failing to capture the true reasoning of models. We introduce Adversarial Sensitivity as a novel approach to faithfulness evaluation, focusing on the explainer’s response when the model is under adversarial attack. Our method accounts for the faithfulness of explainers by capturing sensitivity to adversarial input changes. This work addresses significant limitations in existing evaluation techniques, and furthermore, quantifies faithfulness from a crucial yet underexplored paradigm.

## introduction deep learning-based language models (lms) are increasingly used in high-stakes natural language processing (nlp) tasks @xcite . however, these models are extremely opaque. to build user trust in these models' decisions, various post-hoc explanation methods @xcite have been proposed @xcite . despite their popularity, these explainers are frequently criticized for their 'faithfulness', which is loosely defined as how well the explainer reflects the underlying reasoning of the model @xcite . in the context of nlp, explainers assign weights to each token indicating their importance in prediction, and faithfulness is measured by how consistent these assignments are with the model's reasoning. however, since the explainer is not the model itself @xcite , practitioners have developed several heuristics to measure the quality of these assignments @xcite @xcite @xcite . a common assumption behind many of these heuristics is the linearity assumption, which posits that the importance of each token is independent of the others @xcite . based on this, a group of practitioners hypothesised that removing important tokens indicated by a faithful explainer should change the prediction, whereas removing the least important ones should not. jacovi et al. @xcite ) addressed these as erasure. deyoung et al. @xcite generalize the same with comprehensiveness and sufficiency. however, it has been exhaustively shown that the removal of features can produce counterfactual inputs foot_0 that are out of distribution @xcite @xcite @xcite , socially misaligned @xcite , and often severely pathogenic @xcite . furthermore, evaluation metrics such as area under the perturbation curve (aupc) @xcite are suspected to be severely misinformative @xcite . instead of evaluating faithfulness, these methods primarily compute the similarity between the evaluation metric and explanation techniques, assuming the evaluation metric itself to be the ground truth @xcite . another line of work, known as adversarial robustness @xcite , assumes that similar inputs with similar outputs should yield similar explanations. however, ju et al. @xcite has empirically shown that the change in attribution scores may be because the model's reasoning process has genuinely changed, rather than because the attribution method is unreliable. moreover, this assumption is mainly valid when the model is 'astute' @xcite and doesn't necessarily apply to explainers that don't perform local function approximation for feature importance estimation @xcite . as a result, this assumption is practically restrictive and vague, leading practitioners to hesitate in endorsing this approach for assessing faithfulness @xcite . across almost all popular lines of thought, the settings in which faithfulness is quantified are linear @xcite , restrictive @xcite , misinformative @xcite , and thus the judgements on explainer quality based on such quantification could be arguable. since, understanding the model's reasoning is challenging, and aforementioned assumptions are often deceptive, in this work, we take a fundamental approach. previous research has demonstrated that deep models are not only opaque but also severely fragile @xcite . as explainers are primarily to facilitate trust on these complex models, we argue that a faithful explainer is obligated to uncover such vulnerabilities and anomalous behaviour of the model to the end user. in this context, we introduce the notion of 'adversarial sensitivity' for the explainers. we seek the most similar (semantically and/or visually) counterpart(s) from the entire input space (subjected to certain constraints) that produces a different output, aka 'adversarial examples' @xcite . these pairs of inputs are always bounded by a certain distance, ensuring they are sufficiently comparable. consequently, unlike counterfactuals, these pairs are much less likely affected by abrupt semantic shifts @xcite that often lead to out-of-distribution scenarios @xcite @xcite , making our comparisons more nuanced and robust. however, since these pairs yield different outputs, their underlying reasoning in the model is bound to differ @xcite . faithful explanations should reflect these changes, highlighting the difference in the model's inherent reasoning. we formally define the same as 'adversarial sensitivity' of the explainers. our contributions in this paper are summarised as: this paper is organised as follows: we introduce the notion of adversarial sensitivity, exploring its significance and relation with faithfulness in section 2. in section 3, we details our methodology, outlining the framework used to conduct our investigations. in section 5, we present our findings, offering in-depth analysis and interpretations of the data. we contextualize our work within the broader research landscape in section 6, highlighting how our study contributes to and extends existing knowledge. finally, in section 7, we conclude by summarizing our key findings and proposing directions for future research, emphasizing the potential avenues for further exploration. ## adversarial sensitivity in this section, we introduce the notion of adversarial sensitivity, exploring its significance and relation with faithfulness. thereafter, we propose the guideline for evaluating faithfulness with adversarial sensitivity. definition 1. adversarial example (ae): given a model f : @xmath0, where x is the space of textual inputs and y is the set of classes, if there exists x ′ for a given input @xmath1: adversarial sensitivity and faithfulness: given x ′ is an ae of x for f, if i is 'faithful' to f, then w x,f and w x ′ ,f should be dissimilar. in our setup, we report the mean distance over all obtained pairs of (x, x ′ ). this is a necessary but not sufficient condition for faithfulness. currently (at the time of writing this paper) there is no necessary and sufficient condition for faithfulness @xcite . however, following the argument of lyu et al. @xcite , as these metrics are primarily (meta)heuristic based evaluations, accessing faithfulness with several necessary tests is much more practical than attempting to formulate an exhaustive list of necessary and sufficient conditions and then evaluating against all of them. adversarial sensitivity is one of such necessary tests to evaluate the faithfulness of explainers. adversarial machine learning research has extensively demonstrated that even minimal perturbations in the input space can deceive well-trained models @xcite @xcite @xcite @xcite @xcite @xcite . given the discrete and combinatorially large nature of the input space, finding all possible adversarial examples (aes) under all possible constraints is often impractical, especially in a black-box setting. therefore, we advocate for greedily searching for aes within a well-tested set of constraints to avoid obfuscating and low-quality examples. in this study, we select extensively used word-level, character-level, and behavioural invariance constraints. whether methods like back-translation, paraphrasing, or hybrid attacks etc @xcite ) maintain semantic and structural similarity while generating aes, and suitability for faithfulness evaluation are kept for further study. obtaining aes is conducted in two ways: assuming the model to be either white-box or blackbox. in a white-box setting, gradients are primarily used first to identify the importance of tokens and then perturb them to create an adversarial input if the output changes. for our setting, this approach has two distinct problems. firstly, gradient-based feature importance can be untrustworthy and manipulative @xcite . secondly, a class of post-hoc explainers (e.g., gradient, integrated gradient) also uses the gradient to retrieve the importance of tokens. comparing these with explainers that do not use gradient informa-tion, such as lime or shap, may lead to biased comparisons. lastly, popular gradient-based attacks such as hotflip @xcite are often less likely to adhere to perturbation constraints while crafting adversarial examples @xcite . therefore, we do not consider investigation on white-box attacks for adversarial sensitivity and adhere to a more practical, model-agnostic, and transferable black-box attacking framework. however, even in the black-box settings we employ some ad-hoc heuristics for greedily perturbing the words based on its relative importance @xcite , but modern explainers do not use such ad-hoc methods for calculating feature importance @xcite . therefore, our faithfulness test is unbiased towards the underlying mechanisms of (almost) all types of modern post-hoc explainers. ## results & discussion from under erasure methods across all datasets, gradient is a moderately well-performing explainer, whereas gradient × input performs much worse. however, according to adversarial sensitivity, gradient × input is one of the best performers, with gradient being the worst among all. like gradient × input, integrated gradient also largely performs worse than gradient in erasure, but it remains consistently moderate according to adv. sens. both lime and shap not only perform very well in both adv. sens. and erasure metrics but also the difference b/w their magnitudes for both erasure metrics and adv. sens. are (considerably) nominal. integrated gradient × input is substantially similar to lime, shap in adv sens., but we observe a considerable drop in comprehensiveness for sst-2 and ag news for both the models, unlike adv. sens. to demonstrate, how to evaluate the explainers based on the consensus ranking, we are considering the case of sst-2 for the bert model. we use the kemeny-young method here; as this has been extensively used for condorcet ranking @xcite ; it also satisfies highly desirable social choice properties for fair voting @xcite . kemeny-young aggregation also have been used in biology and social science extensively @xcite @xcite . we first convert the columns of a1, a2, a3 into ranking vectors using a ranking function. in our case, we used the traditional ranking: the higher the score (here the score is average distance obtained), the lower the ranking. we obtained the consensus ranking vector as [2, 3, 6, 5, 4, 1]. here, the indices of the vector denote the respective position of explainers (starting from 1 onwards) in the 'explainer' column. deyoung et al. @xcite advocated for both high comprehensiveness and low sufficiency for adequate explanations but unlike us; they did not propose any consensus evaluation for explainers with these two parameters taken together. according to the definition, both metrics measure two different aspects of explanations. this makes the evaluation of explainers even confusing @xcite reported the same observation previously. as javoci et al. said, "lacking a standard definition, different works evaluate their methods by introducing tests to measure properties that they believe good interpretations should satisfy. some of these tests measure aspects of faithfulness. these ad-hoc definitions are often unique to each paper and inconsistent with each other, making it hard to find commonalities." @xcite . although evaluation metrics are inherently different from one another, for the sake of demonstrating an inter-comparison between erasures and adv. sens. 7 , we rank the explainers based on the scores they obtain in individual erasure methods in the case of sst-2 for the bert model in nevertheless, erasure has been used in several novel affairs and benchmarkings @xcite @xcite due to its easy-to-implement and seemingly reasonable assumption. however, we observe in our experimentation that erasure methods are inconsistent except perturbation based explainers with our proposed metric. unlike erasure, which makes simplistic assumptions about the independence of the token's importance and absence of non-sensical ood results while removing tokens @xcite , adversarial sensitivity is founded on the assumption that faithful explainers should capture the intrinsic dissimilarity of model reasoning when fooled. we, therefore, advocate for the adoption of adversarial sensitivity as a foundational metric for a necessary test of faithfulness for assessing explainers. following the hypothesis of adversarial robustness, which comes under the robustness evaluation category, successful adversarial attack on explainer aims to perturb the input such that an explainer generates dissimilar (non-robust) explanations subject to 'similar' input and 'similar' (bounded by a certain distance) output @xcite ) (advxai). however, rather than any ad-hoc distance to compare the similarity of explainer alvarez et al. @xcite emphasize on the (local) lipschitz continuity measurement in this setting. @xcite has recently analysed the theoretical bounds of (dis)similarity under this setting when the explainer and classifier (bhattacharjee and chaudhuri, 2020) are astute. anyways, ad-vxai is not a formally accepted measure of faithfulness @xcite @xcite , as the model may yield different reasoning rather than the explanation is non-robust. anyhow, in this work we conduct attacks to deceive the model, not the explainer following the aforesaid hypothesis. for a broad overview on faithfulness evaluation we suggest the reader to refer to @xcite . adversarial examples (advai) can be crafted at several levels: word level, character level, phrase level, paraphrasing, back translation, invariance testing, etc. in white-box and black-box settings primarily @xcite . we employed word level, char level and invarience testing attacks. noppel et al. @xcite systematised the underlying relations of advai and ad-vxai. for a broder overview, we refer the reader to @xcite . adversarial attacks on nlp systems have been carried out primarily in 2 types: white box and black box @xcite , we didn't go with white box ones as they primarily leverage gradient information also, as several explainers such as integrated gradient or gradient access the same information which could constitute a biased evaluation @xcite as the attacking mechanism and the explanation method are similar and both leverage gradient information. counterfactual explanations (mothilal et al., 2020), which demonstrate the changes would produce a distinct outcome, differ fundamentally from adversarial examples @xcite , which aim to deceive models with minimal input changes. counterfactuals should be semantically and/or visually different @xcite . thus, it is not intended to deceive the underlying model. in the context of natural language inference (nli), atanasova et al. @xcite experimented with counterfactuals to investigate faithfulness. camburu et al. @xcite explored inconsistencies in explanations for nli but did not adhere to the constraints necessary for generating adversarial inputs required in our setting. moreover, counterfactual explanations can potentially highlight necessary features but may miss sufficient ones for prediction @xcite . similarity measures in previous works, especially in advxai @xcite @xcite , have used mainly correlations, distance measures, and top 'k'% intersection in tokens. burger et al. @xcite comprehended the common issues with such metrics due to tokenization discrepancies and employed rbo @xcite . we did not select rbo having the free parameter user persistence (p), as we argue that faithfulness should not be based on the unnecessary human evaluations. we rather select the distance invented by moreno et al. @xcite that satisfies all the axioms for non-strict, incomplete rank-ings and also satisfies the desirable social choice properties of the kemeny-snell distance @xcite for fair and conclusive rankings. ## conclusion and further work in this work, we explored the shortcomings of widely used faithfulness measures in nlp and proposed a test to evaluate explainers based on their sensitivity to adversarial inputs. through extensive experiments on six post-hoc explainers, we found that gradient & integrated gradient aren't (sufficiently) sensitive, while lime, shap, and gradient × input, and integrated gradient × input show better sensitivity. we also observed notable differences between our evaluation and traditional erasure-based faithfulness measures. future work will explore adversarial sensitivity for multilingual datasets, low-resource languages, and advanced lms.