Investigating the Impact of Model Instability on Explanations and Uncertainty (2024)

Abstract: Explainable AI methods facilitate the understanding of model behaviour, yet, small, imperceptible perturbations to inputs can vastly distort explanations. As these explanations are typically evaluated holistically, before model deployment, it is difficult to assess when a particular explanation is trustworthy. Some studies have tried to create confidence estimators for explanations, but none have investigated an existing link between uncertainty and explanation quality. We artificially simulate epistemic uncertainty in text input by introducing noise at inference time. In this large-scale empirical study, we insert different levels of noise perturbations and measure the effect on the output of pre-trained language models and different uncertainty metrics. Realistic perturbations have minimal effect on performance and explanations, yet masking has a drastic effect. We find that high uncertainty doesn’t necessarily imply low explanation plausibility; the correlation between the two metrics can be moderately positive when noise is exposed during the training process. This suggests that noise-augmented models may be better at identifying salient tokens when uncertain. Furthermore, when predictive and epistemic uncertainty measures are over-confident, the robustness of a saliency map to perturbation can indicate model stability issues. Integrated Gradients shows the overall greatest robustness to perturbation, while still showing model-specific patterns in performance; however, this phenomenon is limited to smaller Transformer-based language models.

## introduction though language models have become increasingly popular for personal and industrial use, these blackbox models have been prone to perpetuate discrimination and output hallucinations @xcite @xcite . to use these models safely, it is important to instil a level of trust in their output. some methods of instilling trust in a model output include uncertainty estimation and explainable ai (xai). uncertainty is a reflection of a model's confidence in its output, given, for example, ambiguous or unfamiliar data. while uncertainty can be estimated at inference time in an unsupervised manner, xai is typically holistically evaluated for a model and task @xcite . however, xai techniques give unstable explanations given small changes in input data @xcite @xcite . while these studies have been critiqued for inserting unnatural noise into the input data, even relatively realistic perturbations to images can disrupt most gradient-based saliency map techniques @xcite . due to this instability, it is difficult to know when we can trust a specific explanation. ideally, we would like to use xai to understand both why a model succeeds and fails to identify points of failure in a model pipeline-these failures could arise from mistakes in the model training or ambiguity within the data. it is also vital to understand when explanations are trustworthy, as the inclusion of xai can cause an over-reliance on models @xcite , give users the false impression of global task understanding @xcite , and lead to overall poorer performance than if no human-ai collaboration @xcite . therefore, we would like to assess if the uncertainty of a model's output can give any indication of an explanation's quality and if the instability of an explanation can provide insight into the model's performance. we expect noise at inference time, especially for text: words can be accidentally ablated, mispelled or otherwise mutated. different authors have distinct linguistic styles, and new words emerge or change in mean-noise type example text (unperturbed) "an artful intelligent film that stays within the confines of a well-established genre" token-mask "an @xcite film that stays within the confines of a [mask] genre" token-unk "an [unk] [unk] film that stays within the confines of a [unk] genre" charinsert "an artfuvl intdelligent film that stays within the confines of a well-mestablished genre" charswap "an artfjl intellhgent film that stays within the confines of a pell-established genre" butterfingers "an artdul intelligegt film that stays within the confines of a well-esfablished genre" l33t "an @r7fu1 1n7311193n7 film that stays within the confines of a w311-357@611543d genre" synonym "an disingenous sound film that stays within the confines of a good-established genre" in this paper, we conduct a large-scale empirical investigation into the effect of noise on pre-trained language models via a controlled experiment by artificially injecting varying degrees and types of realistic noise (see here, we provide the following contributions: we find that high uncertainty does not imply low explanation plausibility; models trained with noisy data can still generate coherent explanations despite high uncertainty amid noise. furthermore, we argue that explanation instability can give some insight into model performance and can show patterns in saliency attribution: common, realistic perturbations (like synonym replacement) have smaller effects on model performance and saliency maps, yet l33t speak and token replacement have a larger impact. this pattern is seen typically strongest in integrated gradients, which also shows the greatest robustness for smaller language models. ## related work assessing trustworthiness there are many ways to assess a model's trustworthiness for a task or inference. the confidence in an output can be quantified via its uncertainty, and the reasonability of an output can be assessed via xai. furthermore, the overall quality of an xai method can be evaluated, either via the similarity to human annotations or via other metrics like robustness to noise or conciseness @xcite @xcite . there is some controversy within these measures: models that output explanations with high similarity to human-annotations may result in unfaithful explanations, as models may not actually rely on this information to compute their output @xcite . moreover, these explanations can also be unstable and prone to large changes in output given small changes in input data @xcite @xcite @xcite . however, these (often image) studies do not investigate the causes of the instabilities or how they relate to other measures, like uncertainty. noise on language model performance several other studies have looked specifically at the effect of noise on the performance and confidence of bert-related models. surprisingly, there are contrasting effects of noise on machine and human ability to perform natural language understanding tasks. perturbations that do not affect a human's ability to understand text significantly perturb bert performance @xcite , yet perturbations that worsen human performance do not affect model performance @xcite @xcite . the impact of different kinds of noise differs across model types @xcite , and the more "learnable" a kind of noise is for a model, the less performance decays given noise-augmented data @xcite . however, as these studies focus on bert-related models, there is limited focus on other model types, like gpt, and they also do not evaluate explanations. ## results we present the motivation and results of each investigation. we show the results for sst-2 but summarize and interpret the results for all investigated datasets; the data for all datasets are in the appendix. ## discussion in this section, we discuss the causes behind patterns seen across the experiments in §4, which are further supplemented with extra analyses in our appendix. we investigate over-arching patterns as well as patterns across perturbation types and datasets. overarching patterns while noise consistently deteriorates model performance and explanation plausibility, the impact of increasing noise on model confidence varies across model and task. unlike previous studies, we do not typically see an increase in confidence after perturbation @xcite ; though the observed decrease is not at the same rate of performance decline. however, both cited studies perturb at the word and sentence structure level, unlike our study. furthermore, we see a similar pattern between the predictive and epistemic uncertainty measures, suggesting that over-confidence after perturbation stems from the training process. overall, human-based perturbations have the strongest effect on task performance and uncertainty measures, and gradient-based perturbation is only more effective than random perturbation at low levels of noise (α = .05). this suggests that the human-generated annotations of each dataset are faithful indicators of true saliency, as their perturbation degrades model performance more than gradient-based approaches, further justifying our use of plausibility as a quality metric. perturbation-level patterns across all models, realistic perturbations, such as butterfingers or synonym have the smallest impact on task performance and explanation plausibility, yet masking has the greatest impact. furthermore, mask has the greatest effect on both measures of uncertainty. we expect that the embedding layer of the plms is better equipped to handle synonym-level perturbations, allowing the hidden representations of the input to change minimally, so that model performance and explanations are minimally impacted. furthermore, we explore model-level differences in appendix c.2 and c.3 and find that the perturbations that have the maximal impact on model performance (e.g. mask for electra in semeval) also uniquely impact saliency maps at low levels of perturbation. this suggests that the instability we separately show the effect on bert, roberta, electra, gpt2 and opt. ## conclusion we provide an empirical investigation across language models, noise perturbations, and saliency maps to investigate a relationship between uncertainty and explanation plausibility. following an array of perturbation techniques, we show that noise injection simultaneously affects model performance, uncertainty, and explanation plausibility. we do not find a strong negative relationship between uncertainty and explanation plausibility; model fine-tuned with noisy data typically show a moderately positive correlation between plausibility and uncertainty, which suggests that these models may even be better at identifying salient tokens when uncertain. we also show that the instability of a saliency map to noise can also provide insights into a model's performance, and suggest integrated gradients for future work in human-xai collaboration, due to its robustness to noise for smaller language models.