A Diachronic Perspective on User Trust in AI under Uncertainty (2023)

Abstract: In human-AI collaboration, users typically form a mental model of the AI system, which captures the user’s beliefs about when the system performs well and when it does not. The construction of this mental model is guided by both the system’s veracity as well as the system output presented to the user e.g., the system’s confidence and an explanation for the prediction. However, modern NLP systems are seldom calibrated and are often confidently incorrect about their predictions, which violates users’ mental model and erodes their trust. In this work, we design a study where users bet on the correctness of an NLP system, and use it to study the evolution of user trust as a response to these trust-eroding events and how the user trust is rebuilt as a function of time after these events. We find that even a few highly inaccurate confidence estimation instances are enough to damage users’ trust in the system and performance, which does not easily recover over time. We further find that users are more forgiving to the NLP system if it is unconfidently correct rather than confidently incorrect, even though, from a game-theoretic perspective, their payoff is equivalent. Finally, we find that each user can entertain multiple mental models of the system based on the type of the question. These results highlight the importance of confidence calibration in developing user-centered NLP applications to avoid damaging user trust and compromising the collaboration performance.

## introduction ai systems are increasingly being touted for use in high-stakes decision-making. for example, a doctor might use an ai system for cancer detection from lymph node images @xcite , a teacher may be assisted by an ai system when teaching students @xcite , or individuals may rely on ai systems to fulfill their information requirements @xcite . ai systems are integrated across diverse domains, with an expanding presence in user-centric applications. despite their growing performance, today's ai systems are still sometimes inaccurate, reinforcing the need for human involvement and oversight. an effective approach for facilitating decisionmaking in collaborative settings is for the ai system to offer its confidence alongside its predictions. this is shown in figure 1 , where the ai system provides an additional message that enables the user to either accept or reject the system's answer based on the additional message, such as the confidence score. this makes a strong case for the ai's confidence being calibrated @xcite -when the confidence score aligns with the probability of the prediction being correct. when a user interacts with an ai system, they develop a mental model @xcite of how the system's confidence relates to the integrity of its prediction. the issue of trust has been extensively studied in psychology and cognitive science with @xcite ; @xcite finding that incongruence (mismatch between mental model and user experience) creates distrust. given the ever-increasing reliance on ai systems, it is crucial that users possess a well-defined mental model that guides their trust in these systems. nevertheless, our current understanding regarding the evolution of user trust over time, its vulnerability to trust-depleting incidents, and the methods to re-store trust following such events remain unclear. addressing these inquiries holds great significance in the advancement of reliable ai systems. in this paper, our objective is to investigate user interactions with an ai system, with a specific focus on how the system's confidence impacts these interactions. through a series of carefully designed user studies, we explore the implications of miscalibrated confidences on user's perception of the system and how this, in turn, influences their trust in the system. our experiments shed light on how users respond to various types of miscalibrations. we find that users are especially sensitive to confidently incorrect miscalibration (section 4.1) and that the trust does not recover even after a long sequence of calibrated examples. subsequently, we delve into an analysis of how trust degradation corresponds to the extent of miscalibration in the examples provided (section 4.2). then, we assess whether diminished trust in an ai system for a specific task can extend to affect a user's trust in other tasks (section 4.3). we also explore different methodologies for modeling a user's trust in an ai system (section 5). our results show how reduced trust can lower the performance of the human-ai team thus highlighting the importance of holistic and user-centric calibration of ai systems when they are deployed in high-stakes settings. trust and confidence calibration. a common method ai systems use to convey their uncertainty to the user is by its confidence @xcite . for the system's confidence to reflect the probability of the system being correct, the confidence needs to be calibrated, which is a long-standing task @xcite . this can be any metric, such as quality estimation @xcite that makes it easier for the user to decide on the ai system's correctness. related to calibration is selective prediction where the model can abstain from predicting. the latter has been studied in the context of machine learning @xcite and its various applications @xcite @xcite . trust calibration is the relation between the user's trust in the system and the system's abilities @xcite @xcite @xcite @xcite . specifically, @xcite explore jointly optimization of calibration (transformation of the ai system reported confidence) with human feedback. they conclude that uncalibrated models improve human-ai collaboration. however, apart from their experimental design being different from ours, they also admit to not studying the temporal effect of miscalibrations. because of this, our results are not in contradiction. ajenaghughrure et al. (2019); zhou et al. (2019 ) predictively model the user trust in the ai system. while successful, they use psychological signals, such as eeg or gsr, for their predictions, which is usually inaccessible in the traditional desktop interface setting. @xcite use combination of demographic information together with interaction history to predict whether the user is going to accept or reject ai system's suggestion. the field has otherwise focused on theoretical frameworks to explain factors that affect trust in mostly human-robot interaction scenarios @xcite @xcite . ## human ai interaction over time we begin by providing a preliminary formalism for a human-ai interaction over time. it comprises of two interlocutors, an ai system and a user. at time t, the user provides the ai system with an input or a question q t and the ai system responds with an answer y t along with a message comprising of its confidence in the answer m t . the user has two options, either they accept the ai's answer or reject it and try to find an answer themselves. the ai is either correct (a @xmath0. the combination of correctness a t and confidence m t results in four different possibilities each with a different reward, or risk, shown in figure 2 . for example, confidently incorrect may lead to the user disastrously accepting a false answer while unconfidently correct will make the user spend more time finding the answer themselves. ## experiments we perform three types of experiments. in section 4.1, we establish the different effects of confidently incorrect and unconfidently correct stimuli. then, in section 4.2 we see how the size of confidently incorrect intervention affects the users interaction with the ai system and in section 4.3 explore if miscalibration is transferable between question types. lastly, we predict the user interaction in section 5. ## modeling user trust in human-ai collaboration systems, it is the collaboration performance that is more important than the accuracy of the ai system itself @xcite . in such cases, an ai system that can understand and adapt to how its output is used is more better. an important challenge in understanding the user's behavior is estimating how likely the user is to trust the system. this would also allow the system to adapt when user trust in the system is low by perhaps performing a positive intervention that increases user trust. we apply our learnings from the previous section and show that systems that explicitly model the user's past interactions with the system are able to better predict and estimate the user's trust in the system. we now develop increasingly complex predictive statistical models of user behavior, which will reveal what contributes to the user process and affects trust. for evaluation, we use f 1 and accuracy (agreement) and mean absolute error (bet value) for interpretability. ## discussion we now contextualize our findings to real-world applications and discuss the differences and their implications. miscalibration impacts user trust. even a small (5) number of miscalibrated examples affects how users trust the system in the future. in our controlled setting we consider a symmetric risk-reward setup. however, past work has shown that trust is linked to risk. in real applications, the reward and cost of trusting the system might not be the same. for example in an ai system detecting cancer, having a doctor manualy do the screening has lower cost than a misdiagnosis. confidently incorrect examples lower trust more than confidently correct examples. standard methods of evaluating model calibration, such as the expected calibration error (ece), do not take this into account. a holistic calibration metric should take these user-centric aspects into account, particularly, how users interpret these confidence scores and how it affects their trust in the system. miscalibration effects persist and affect user behavior over long time spans. in our setup, users interact with the system continuously over a ses-sion. after the intervention, their trust decreases over several interactions. real-life user interactions with ai systems might not always follow this pattern. for example, a user might use a search engine in bursts when they have an information need. the larger time intervals between interactions might dampen the strong feelings of trust or mistrust. mistrust transfers between input types. our experiments reveal that the model's miscalibration on a certain type of input also reduces the user's trust in the model on other types of inputs. in real-world applications, ai systems are generally presented to users as an abstraction and the user may or may not be aware of the underlying workings of the system. for example, recent user-facing llms often employ techniques such as a mixture-of-experts or smaller specialized models that perform different tasks. in such cases, the transfer of miscalibration can be erroneous. rnn outperforms linear models in modeling user trust. this is indicative that modeling user trust is complex and requires more sophisticated non-linear models. like most deep learning models, a recurrent network requires more data for accurate prediction. however, user-facing applications can collect several features and with more data deep learning models might generalize better and help us dynamically track and predict user trust. ## conclusion when interacting with ai systems, users create mental models of the ai's prediction and identify regions of the system's output they can trust. our research highlights the impact of miscalibrations, especially in confidently incorrect predictions, which leads to a notable decline in user trust in the ai system. this loss of trust persists over multiple interactions, even with just a small number of miscalibrations (as few as five), affecting how users trust the system in the future. the lower trust in the system then hinders the effectiveness of human-ai collaboration. our experiments also show that user mental models adapt to consider different input types. when the system is miscalibrated for a specific input type, user trust is reduced for that type of input. finally, our examination of various trust modeling approaches reveals that models capable of effectively capturing past interactions, like recurrent networks, provide better predictions of user trust over multiple interactions.