Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting (2024)

Abstract: The growing capabilities of AI models are leading to their wider use, including in safety-critical domains. Explainable AI (XAI) aims to make these models safer to use by making their inference process more transparent. However, current explainability methods are seldom evaluated in the way they are intended to be used: by real-world end users. To address this, we conducted a large-scale user study with 85 healthcare practitioners in the context of human-AI collaborative chest X-ray analysis. We evaluated three types of explanations: visual explanations (saliency maps), natural language explanations, and a combination of both modalities. We specifically examined how different explanation types influence users depending on whether the AI advice and explanations are factually correct. We find that text-based explanations lead to significant over-reliance, which is alleviated by combining them with saliency maps. We also observe that the quality of explanations, that is, how much factually correct information they entail, and how much this aligns with AI correctness, significantly impacts the usefulness of the different explanation types.

## introduction ai models have progressed rapidly in recent years and are being used increasingly across various domains, including medical applications @xcite . the communication interface of generative ai is often language-based @xcite , which offers a human-like mode of interaction. some research suggests that this linguistic interface "humanizes" these ai systems and thereby increases reliance on them @xcite . at the same time, a remaining significant barrier to the adoption and regulatory approval of deep learning models in medical imaging is the limited transparency of the reasoning processes underlying these models @xcite . insufficient model robustness @xcite , bias (algorithms are prone to amplifying inequalities that exist in the world) @xcite , and the high stakes in clinical applications @xcite are all obstacles to their wider use. the practical utility of ai explainability methods that aim to address this remains poorly understood, as evaluating them is a challenging task. there can often be several correct ways to explain a decision and the criteria for judging their quality are diverse (e.g., plausibility, faithfulness, clarity @xcite ). since one of the primary benefits of explanations is their utility to end-users, evaluating them with human subjects is crucial. as explanations can lead to confirmation bias and user preference frequently does not align with desired quality requirements (e.g., a user might prefer an explanation type even if they are more likely to misinterpret the ai), explanation usefulness needs to be evaluated via proxy measures @xcite @xcite . only a few studies attempt this, with some suggesting that these methods may not work as well as anticipated @xcite @xcite . we address this by carrying out a large-scale human subject study to evaluate the usefulness of natural language explanations (nles), saliency maps, and a combination of both, in the setting of imperfect ai and imperfect xai. saliency maps, which attribute importance weights to regions in an image, are the prevailing mode of interpretability in medical imaging @xcite . nles, on the other hand, are becoming more widespread with recent advances in large language models @xcite and have been advocated for deployment in clinical practice @xcite . we also study the combination of both explanation modalities, to understand if they can complement each other. we consider imperfect ai and xai to reflect real-world applications, where both the ai predictions and the ai explanations can contain errors. specifically, we investigate how different types of explanations, taking into account both ai and xai correctness, affect users in a clinical decision-support system (cdss) environment. as we focus on ai that enhances medical practitioners @xcite , rather than replaces them, our proxy for the usefulness of explanations is how much they improve human performance in human-ai collaborative chest x-ray analysis. in our study, 85 doctors and medical students analyse 80 unique images each, distributed across four different cdss set-ups: either of the three explanation types, or the "no explanation" control condition. our study design is illustrated in fig. 1 . our results highlight the pitfalls of languagebased explanations, which lead to overreliance. interestingly, however, saliency maps and nles complement each other, and their combination is the most useful explanation type. we also find that explanation correctness, and how it aligns to ai correctness, is an important factor in determining whether explanations are helpful or harmful to users. when they misalign (e.g., the ai is correct but the explanation contains a lot of incorrect information), they are detrimental to human performance, but equally, when the ai is incorrect, correct explanations mislead users into agreeing with the ai. we find that the alignment between explanation correctness and ai correctness is critical in determining whether explanations are helpful or harmful to users. when they misalign-such as when the ai is correct but the explanation contains many inaccuracies-this negatively impacts human perfor-mance. conversely, when they align, explanations improve our participants' task performance. ## methods we designed a study to evaluate the usefulness of nles, saliency maps, and their combination in a clinical decision-support context. we also control for ai advice correctness c @xmath0, 1} and explanation correctness c χ ∈ @xcite . our main research question is how different explanation types, in the context of advice correctness c ai and explanation correctness c χ , affect human performance on the task of classifying chest x-rays, where explanation usefulness equates by human performance. definition of explanation correctness c χ : c χ captures to what extent the information provided in an explanation is clinically, factually correct. an explanation can be incorrect (i.e., contain a lot of incorrect information) even when the ai prediction was correct, and vice versa, similar to definitions from @xcite and @xcite . note that this is different from other explanation criteria such as faithfulness (i.e., how "accurately it reflects the true reasoning process of the model") and plausibility (i.e., how convincing the explanation is to humans) @xcite . we obtain the ground-truth for both advice and explanation correctness from annotations by three expert radiologists. for each of the three explanation scenarios, c χ is rated on a 7-point likert scale. the evaluation interface given to the annotators is shown in figure 12 in the appendix. ## summary and outlook in this work, we conducted a large-scale user study simulating a real clinical decision support set-up and included in-domain, clinical experts. we juxtaposed textual (nles) and visual (saliency maps) explanations, and found that nles lead to severe overreliance, but can be helpful when combined with visual explanations. we also show that alignment between explanation and advice correctness is a strong predictor for explanation usefulness. this study sheds light on the pitfalls of convincing-sounding language-based explanations and we hope it enables future research on optimizing such explanations to lead to safe use of ai.