“Are Your Explanations Reliable?” Investigating the Stability of LIME in Explaining Text Classifiers by Marrying XAI and Adversarial Attack (2023)

Abstract: LIME has emerged as one of the most commonly referenced tools in explainable AI (XAI) frameworks that is integrated into critical machine learning applications (e.g., healthcare and finance). However, its stability remains little explored, especially in the context of text data, due to the unique text-space constraints. To address these challenges, in this paper, we first evaluate the inherent instability of LIME on text data to establish a baseline, and then propose a novel algorithm XAIFooler to perturb text inputs and manipulate explanations that casts investigation on the stability of LIME as a text perturbation optimization problem. XAIFooler conforms to the constraints to preserve text semantics and original prediction with small perturbations, and introduces Rank-biased Overlap (RBO) as a key part to guide the optimization of XAIFooler that satisfies all the requirements for explanation similarity measure. Extensive experiments on real-world text datasets demonstrate that XAIFooler significantly outperforms all baselines by large margins in its ability to manipulate LIME’s explanations with high semantic preservability.

## introduction machine learning has witnessed extensive research, leading to its enhanced capability in predicting a wide variety of phenomena @xcite . unfortunately, this increased effectiveness has come at the cost of comprehending the inner workings of the resulting models. to address this challenge, explainable ai (xai), also known as interpretable ai @xcite , has emerged as a discipline focused on understanding why a model makes the predictions it does. xai continues to grow in importance due to both legal and societal demands for elucidating the factors contributing to specific model predictions. while explainability in ai as a general concept has yet to fully coalesce to an accepted set of definitions @xcite , the concept of stability starts to occur throughout the discussions and remains prominent @xcite @xcite . a stable explanation refers to one where small changes to the input should have a corresponding small effect on the output. in other words, similar inputs to a model should produce similar explanations. lack of stability in an explanatory method undermines its trustworthiness @xcite , and renders all subsequent explanations suspect. this lack of trustworthiness is one reason why the adoption of ai in disciplines like healthcare has progressed slower than in other disciplines @xcite . previous work on xai stability has focused on models where function-based continuity criteria naturally apply to tabular and image-based data @xcite @xcite . however, text data in natural language is not so amenable to such direct quantification. therefore, the process of generating appropriate perturbations to test stability in text explanations remains little explored @xcite . unlike perturbations in tabular or imagebased data, text perturbations pose unique chal-lenges to satisfy necessary constraints of semantic similarity and syntactic consistency. violation of these constraints can create a fundamentally different meaning conveyed by the document. under the assumption the explanatory algorithm works correctly, these perturbations may alter the resulting explanation, where quantifying difference between explanations also needs to characterize their properties, such as feature ordering and weights within explanations presented in fig. 1 . in this paper, we explore the stability of explanations generated on text data via lime @xcite , which is a widely used explanatory algorithm in xai frameworks. we first examine the inherent stability of lime by altering the number of samples generated to train the surrogate model. using this baseline, we then propose xaifooler, a novel algorithm that perturbs text inputs and manipulates explanations to investigate in depth the stability of lime in explaining text classifiers. given a document, xaifooler proceeds with iterative word replacements conforming to the specified constraints, which are guided by rank-biased overlap (rbo) that satisfies all the desired characteristics for explanation similarity measure. as such, xaifooler yields advantages that only small perturbations on the least important words are needed, while the semantics and original prediction of the input are preserved yet top-ranked explanation features are significantly shifted. fig. 1 shows an example that xaifooler only performs two perturbations (i.e., "gets" → "becomes" and "regularly" → "consistently"), but effectively demotes the top-3 features-e.g., "joint" (3rd-rank)→(11thrank), that explains the "joint pain" symptom without changing the prediction of peptic ulcers disease. our contributions are summarized as follows. ## preliminary analysis inherent instability of lime. this section first assesses the inherent instability of lime. our aim is to determine if lime produces inconsistent results even when no changes are made to the target model's inputs or parameters. let d be a document whose prediction under a target model f (•) is to be explained. a simplified process of lime's explanation generation for f (d) is as follows. dataset mean median min gb 82.0% (↓ ∆18%) 82.0% 75.5% s2d 84.0% (↓ ∆16%) 84.4% 72.9% imdb 71.5% (↓ ∆28.5%) 70.4% 67.3% to test the inherent instability of lime, we first select an arbitrary number of documents and generate explanations at different sampling rates n from 1k-7k for three state-of-the-art classifiers, including bert @xcite , roberta @xcite , distilbert @xcite , trained on three corpus varying in lengths. then, we compare them against the base explanation generated with the default sampling rate (@xmath1. overall, lime itself is inherently unstable, especially when n is small, even without any modifications to the target model or its inputs. contrast to small sampling rates-e.g., @xmath3, adopted in existing analytical works, our analysis implies that n should be sufficiently large according to each dataset to maximize the trade-off between computational cost and explanation stability. however, the source of the instability within lime is yet undetermined. we emphasize the importance of this question but leave its investigation for future work. motivation this inherent instability is significant in practice due to the fact that many important ai systems-e.g., ml model debugging @xcite or human-in-the-loop system (nguyen and choo, 2021), use surrogate methods such as lime as a core component. this instability also has implications on software security where malicious actors can exploit this instability and force ai models to produce unfaithful explanations. given that there exists some level of instability within the explanations by default, we now ask if alterations to the input text can demonstrate instability beyond that of the baseline levels already seen. that is, can perturbations to the input that are carefully chosen to retain the underlying meaning of the original text result in appreciably different explanations. if so, the concerns about instability mentioned above become that much more pressing. to do this, we propose to investigate the robustness of lime by formulating this perturbation process as an adversarial text attack optimization problem below. ## problem formulation our goal is to determine how much a malicious actor can manipulate explanations generated by lime via text perturbations-i.e., minimally perturbing a document such that its explanation significantly changes. specifically, a perturbed document d p is generated from a base document d b , such that for their respective explanations e dp and e d b we minimize their explanation similarity: trivially, changing words chosen arbitrarily and with equally arbitrary substitutions would, eventually, produce an explanation different from the original. however, this will also likely produce a perturbed document d p whose semantic meaning drastically differs from d b . thus, we impose a constraint on the semantic similarity between d b and d p to ensure that the perturbed document d p does not alter the fundamental meaning of d b : even with the semantic constraint in eq. ( 3 ), there can still be some shift regarding the context and semantics of d b and d p that might not be impeccable to humans yet cannot algorithmically captured by the computer either. ideally, the malicious actor wants the perturbed document d p to closely resemble its original d b . however, as the number of perturbations grows larger, the document retains less of its original context and meaning. to address this issue, we impose a maximum number of perturbations allowed through the use of a hyperparameter ϵ as follows: we can now generate perturbations in a way that maintains the intrinsic meaning of the original document. if we are to manipulate the explanation (eq. ( 1 )) while maintaining both eq. ( 2), eq. (3) and eq. (4 ), it is trivial to just replace some of the most important features of f . however, in practice, changing the most important features will likely result in a violation to constraint in eq. ( 2 ). moreover, this will not provide meaningful insight to the analysis on stability in that we want to measure how many changes in the perturbed explanation that correspond to small (and not large) alterations to the document. without the following constraint, highly weighted features will often be perturbed, even with a search strategy focused towards features of minimal impact on the base explanation. this happens due to the strength of the previous constraints focused on maintaining the quality of the document. thus, the set of top k feature(s) belonging to the base explanation e d b must appear in the perturbed explanation e dp : objective function: given a document d b , a target model f and hyper-parameter δ, ϵ, k, our goal is to find a perturbed document d p by optimizing the objective function: ## xaifooler to solve the aforementioned objective function (eq. ( 6 )), we propose xaifooler, a novel greedybased algorithm to manipulate the explanations of a text-based xai method whose explanations are in the form of a list of words ordered by importance to the surrogate model. we then demonstrate that xaifooler can effectively alter the explanations of lime beyond the intrinsic instability (sec. 3) via carefully selected text perturbations. ## discussion case study: xai and content moderation. xai systems are often used in conjunction with human-in-the-loop tasks such as online content moderation-e.g., detecting hate speech or fake news, where quality assurance is paramount. thus, manipulations on such xai systems will lead to human experts receiving incorrect signals and hence resulting in sub-optimal results or even serious consequences. evaluation of xaifooler on 300 positive examples from a twitter hate speech detection dataset with k←2 shows that xaifooler often demotes the 1st-rank feature out of top-2, reducing rc to 0.4 and ins to 0.78, while still maintaining the original semantics with @xmath4.9. this means xaifooler can force the ai system to output correct predictions but for wrong reasons. instability distribution. fig. 4 illustrates the distributions of explanation shifts after perturbations on imdb (s2d, gb results are included in the appendix). this shows that lime's instability expresses differently among documents, with some are easier to manipulate than others. fig. 4 also shows that if one to calculate attack success rate using a threshold, which is often customized to specific applications, xaifooler will still consistently outperform the baselines. document length v.s. sampling rate v.s. runtime. the majority of computation time is spent generating explanations using lime. ## conclusion this paper affirms that lime is inherently unstable and its explanations can be exploited via text perturbations. we re-purpose and apply rbo as a key part of our algorithm xaifooler, which we have shown to provide competitive results against other adversarial explanation generation methods, and do so in a way that satisfies our criteria from earlier when determining just how best to compare an explanation. future works include understanding the interactions between the target model and lime that lead to such instability.