The Illusion of Competence: Evaluating the Effect of Explanations on Users’ Mental Models of Visual Question Answering Systems (2024)

Abstract: We examine how users perceive the limitations of an AI system when it encounters a task that it cannot perform perfectly and whether providing explanations alongside its answers aids users in constructing an appropriate mental model of the system’s capabilities and limitations. We employ a visual question answer and explanation task where we control the AI system’s limitations by manipulating the visual inputs: during inference, the system either processes full-color or grayscale images. Our goal is to determine whether participants can perceive the limitations of the system. We hypothesize that explanations will make limited AI capabilities more transparent to users. However, our results show that explanations do not have this effect. Instead of allowing users to more accurately assess the limitations of the AI system, explanations generally increase users’ perceptions of the system’s competence – regardless of its actual performance.

## introduction machine learning-based technologies (often called 'artificial intelligence', ai) are now commonly being deployed and used in real-world applications, influencing human decision-making (or automating decision-making altogether) with implications for societies, organizations, and individuals. despite continuous advances and impressive performance on many tasks, these technologies are not always accurate and will likely never be. machine learning models depend on curation of the data they are trained on, they are optimized according to criteria that may not do justice to the complexity of reality, and the context in which they are used cannot be fully modeled, to name a few reasons for their limitations. in addition, the underlying algorithms themselves have inherent weaknesses. large code and data of study are available at https://doi.org/ 10.17605/osf.io/4kdb5 or https://github.com/clausebielefeld/illusionofcompetence-vqa-explanations . language models (llms), e.g., are well known to hallucinate, i.e., to make predictions that are inconsistent with facts or themselves @xcite , or to be highly sensitive to spurious variations in their inputs/prompts @xcite . many machine learning models also suffer from their own complexity: consisting of millions, billions, or even trillions of parameters, they are blackboxes, opaque to human understanding. however, in order to reliably use machine learning models and ai systems based on such models, human users must be able to assess their limitations and deficiencies, and to understand the decisions that such systems make and why (codified, for example, as the right "to obtain an explanation of the decision reached" in the legal framework of the general data protection regulation of the european union; gdpr, 2016, recital 71). research in explainable ai (xai) addresses this need, and recent years have seen an explosion of explainability methods that aim to make the internal knowledge and reasoning of ai systems transparent and explicit, and thus interpretable and accessible to users. explainability of model predictions is thus seen as a solution, and it is assumed that they enable users to construct functional 'mental models' @xcite of ai systems, i.e., models that closely correspond to the actual capabilities of the systems. whether this is the case is an active research question and there is evidence that explainability comes with new challenges. important questions in xai are what actually makes a good explanation, which criteria it needs to satisfy, and how the quality of explanations can be measured @xcite . furthermore, recent perspectives emphasize that explanations should be social @xcite and constructed interactively, taking into account the user's explanation needs @xcite . @xcite argue that evaluations of explanations should carefully distinguish plausibility (does it seem plausible to users) and faithfulness (does it reflect the model's internal reasoning) and that non-faithful, but plausible, explanations can be dangerous in that they let users construct faulty, and eventually dysfunctional, mental models that can lead to unwarranted trust @xcite . in this paper, we investigate the effects of providing natural language explanations on users' mental models of an ai system in terms of its capabilities, and whether these explanations allow them to diagnose system limitations. we present the results of a study in the visual question answering and explanation (vqa/x) domain, artificially inducing a simple limitation by providing two vqa/x systems with images stripped of color information, i.e., in grayscale (see figure 1 ). participants, unaware of the manipulation, see the unmanipulated full color image, the question, the system's answer, and its explanation for the answer, and have to judge various system capabilities (including its ability to recognize colors) and its competence. this visual domain does not require participants to understand the internal processes of the system but should still enable them to estimate what it can and cannot do. the comparison of judgments to responses to non-manipulated system input and judgments of responses without explanations sheds light on participants' difficulties in using (natural language) xai explanations to build accurate mental models, even for such a simple case. this raises the question of how effective explanations can be in real-world applications of xai technology that involve more complex reasoning and problems. ## background our work is related to previous studies that have examined whether explanations enhance users' trust in ai systems. @xcite , for example, compared trust in personal (human) versus impersonal (recommender system) recommendation sources and examined the impact of explanation quality on trust. their results showed that users rated human explanations higher than system-generated ones and that the quality of explanations significantly influenced trust in the recommendation source. @xcite investigated whether explanations help humans anticipate when an ai system is potentially incorrect. they used scenarios where an ai system helps participants to solve a task (text classification or question answering), providing visual explanations (highlighted words) under certain conditions. their findings revealed that explanations increased question: what season is it? nlx-gpt (color): answer: summer expl: because the grass is green and the trees are casting shadows nlx-gpt (greyscale): answer fall expl: because the trees are green and the grass is dry question: are there more large yellow shiny objects than metal things? nlx-gpt (color): answer: no expl: because there are no large yellow shiny things and there is a metal cylinder nlx-gpt (greyscale): answer yes expl: because there is a large yellow shiny cylinder and there are two metal spheres and a metal cube the likelihood of the participants to accept the ai system's recommendations, irrespective of their accuracy. thus, rather than fostering appropriate reliance on ai systems, explanations tended to foster blind trust. similarly, @xcite conducted a large-scale user study for visual explanations, showing that these do not allow users to distinguish correct from incorrect predictions. @xcite investigated how users develop and regain trust in ai systems in human-ai collaborations. they found that nlp systems that confidently make incorrect predictions harm user trust, and that even a few incorrect instances can damage trust, with slow recovery. while these studies evaluate the influence of system explanations on users' trust in the system's output (a proxy for its perceived competence), they do not investigate users' understanding of the systems' reasoning processes and capabilities. in our study, we specifically address this issue and investigate the users' mental model of the systems' capabilities and limitations. while the studies above found that nonverbal explanations can be misleading to users, natural language explanations are assumed to be more transparent or less difficult to interpret @xcite . verbal explanations also offer the advantage that they can be collected from humans, which has led to the development of explanation benchmarks, particularly in multimodal domains @xcite . thus, the dominant approach to verbal explanation generation currently is to leverage human explana-tions during model training @xcite @xcite @xcite . @xcite discuss potential faithfulness issues related to supervising explanation generation with human explanations, we are not aware of work that explicitly tests these supervised models in a user-centered setting similar to ours. ## approach we conduct a study to investigate how users of an ai system perceive its limitations when it encounters tasks that it cannot perform perfectly. we aim to investigate whether providing explanations alongside model responses helps users build an appropriate mental model of the ai system's capabilities and limitations. in the following, we detail the rationale of our approach (section 3.1-3.3) and the hypotheses of our experiments (section 3.4). ## results we organize the discussion of results based on the hypotheses outlined in section 3. since we ask whether explanations help participants determine that the systems could not recognize color, the following discussion concentrates on the grayscale condition and the differences between the grayscale and color conditions (see appendix a.3 for detailed results of the color condition). all systems received high ratings in all competency and capability dimensions when tested in the color condition of exp.a and x, on both datasets (see ## discussion of implications it is still not well understood how xai can bridge the gap between highly complex black-box models with largely opaque internal reasoning processes and users' intuitive understanding of these. generally, our study provides evidence that explanations generated by state-of-the-art systems do not always lead to the expected effects of xai and that explanations may even further obstruct ais' reasoning processes and trick users into believing that the ai is more competent than it actually is. this result is particularly noteworthy in light of the fact that the manipulation employed in our study introduced an obvious error that should be easy to spot for users (defects in systems' color recognition). ## conclusion this paper investigates the effects of providing natural language explanations on users' ability to construct accurate mental models of ai systems' capabilities, and whether these explanations allow them to diagnose system limitations. results from two experiments show that natural language explanations generated by state-of-the-art vqa-x systems may actually hinder users from accurately reflecting capabilities and limitations of ai systems. participants who received natural language explanations projected more competence onto the system and rated its limited capabilities higher than those who did not receive explanations.