REL - A . I .: An Interaction-Centered Approach To Measuring Human- LM Reliance (2025)

Abstract: The ability to communicate uncertainty and knowledge limitations is crucial for the safety of large language models (LLMs). Current evaluations of these abilities typically examine the correspondence between model accuracy and its internal probabilities or linguistic outputs. However, evaluation of the uncertainty of LLM communication should also focus on the behaviors of their human interlocutors: how much do users rely on what the LLM says? We introduce an interaction-centered evaluation approach called Rel-A.I. (pronounced “rely”) that quantifies whether and how humans rely on LLMs’ responses, complementing existing calibration evaluations. Through nine user studies with 450 participants, we investigate three crucial aspects that influence user reliance. We show that emphatic expressions of politeness (e.g., “I’m happy to help!”) that precede LLM answers will cause participants to perceive these models as more competent, and in turn, rely 30% more on their generations. Additionally, the context of the interaction, such as the knowledge domain and nature of previous interactions with the LLM, substantially influences user reliance (e.g., users will rely 10% more on LLMs when responding to questions involving calculations). Our results show that calibration and language quality alone are insufficient in informing which LLMs are safely calibrated, and illustrate the need to consider features of the interactional context.

## introduction with the rise in capabilities of large language models (llms), we see an explosion of use cases and tasks for human-ai interactions. 1 the nature of human-lm interactions is also more complex and nuanced than ever before; with users tasking llms with complex requests, over repeated interactions, across domains. with new contexts for human-lm interactions and the known overconfidence of 1 we use the term llms except in hyphenated cases where -lm is used to refer to llms. figure 1: we introduce rel-a.i., an interaction-centered approach to evaluating llm risks based on human reliance behaviors. we study the effects of interactional contexts on reliance and find that reliance isn't solely contingent on the quality of model answer, and that contextual cues such as warmth heavily weigh on human perception of model competence and reliability. prior work has tried to accurately communicate model uncertainties via linguistic @xcite or numerical calibration @xcite @xcite @xcite @xcite . in both cases, if the confidence of the verbalized generation matches the probability of the response being correct, the model is considered verbally calibrated. however, our work contends verbal calibration is insufficient to properly measure the safety of human-lm interactions as calibrated generations might still result in unexpected and risky behaviors. even as models provide risks and hesitations-will humans react with the correct behavior? instead, we must consider the human behaviors triggered as a result of the generation and evaluate if the generated language triggers the appropriate downstream behaviors. to address this gap, we introduce an evaluation framework that measures in-situ human reliance on lm-generated responses with epistemic markers called rel-a.i. (pronounced "rely") the framework allows us to measure risks of human over-reliance and understand how interactional contexts might influence human decisions. for example, motivated by the rise of humanlike attributes in language models @xcite , how might the generation of greetings or other social cues influence human reliance (rq1)? next, if a model is typically uncertain (as verbalized) but occasionally expresses moderate confidence, would those statements of moderate confidence be relied on more frequently than if they appeared among a highly confident model (rq2)? lastly, as these language models are being deployed across a variety of domains @xcite , how might human reliance vary across subject matter (rq3) even if models express the same degree of confidence? across our experiments, we discover that varying contexts of human-lm interactions can significantly impact human reliance behaviors. prior interactions, subject matter, and greetings of the model all significantly impact human reliance. for example, statements of moderate confidence appearing in a highly certain model (as verbalized) will be relied on significantly less than the same statement appearing from a typically not confident model. generations in computational domains like math are relied on more often than noncomputational subjects like law and philosophy. greetings like, "i'm happy to help! i believe the answer is b" will trigger significantly higher reliance than the same expression without the greeting. our findings consistently show that the exact same expressions of (un)certainty lead to differences in human behavior depending on the context of the interaction. at a broader level, our findings demonstrate that models that appear identical in verbal calibration and can trigger drastically human behaviors of reliance. canonical metrics focus on verbal calibration can miss key factors that influence the safety of human-lm interactions. by shifting our measurements toward human reliance, we not only get closer measurements of real-world harms but also evaluate the risks introduced by novel interactional contexts. we urge the research and industry communities to use this approach to evaluate human-lm reliance on their unique language technologies prior to public deployment. ## background machine learning as focused largely on improving model calibration @xcite @xcite @xcite )-aligning the model probabilities with accuracy. other recent work has examined how pretraining @xcite and scaling @xcite impact lm calibration. works in linguistic calibration focused on finetuning llms to be calibrated between what is internally represented and what is verbalized either numerically @xcite @xcite @xcite , ordinally @xcite , or verbally @xcite . recent work also explored calibration based on a model's internal representational states @xcite . the field of human-computer interaction is rich with studies on how and when humans choose to rely on ais and their explanations @xcite @xcite @xcite , identifying the pitfalls of human over-reliance. related to our work are studies in decision-making and human overrides @xcite , over-reliance and cognitive load @xcite , and perception of warmth and competence @xcite . we ground our work in llms' expressions of (un)certainty (e.g., "i'm certain the answer is..."). these linguistic cues, or epistemic markers, mark speaker stance and commitment and contribute to human sense-making and decision-making @xcite @xcite . in this work, we are interested in how these expressions of (un)certainty influence humans' decisions on whether or not to rely on an lm-generated response. 2 closest to our work is zhou et al.'s (2024) work on how humans rely on expressions of (un)certainty, dhuliawala et al.'s (2023) studies on how humans interpret numerical confidences in (mis)calibrated settings, and kim et al.'s (2024) work on how humans rely on expressions in a medical setting. ## varying the presentation of the model (rq1) the first interactional context we consider is the presentation and anthropomorphism of large language technologies @xcite @xcite . system attributes like names @xcite as well as human-like language such as greetings and expressions of warmth @xcite are coveted features as they enable more engaging and enriching human-model interaction. however, these changes in presentation hold the potential for encouraging users to unquestioningly rely on system outputs that are prone to errors and hallucinations @xcite . as publicly deployed models have been tuned to incorporate helpfulness @xcite , we are interested in the impact of greetings on human reliance of epistemic markers. experimental set-up users interact with a control agent and an experimental one (a control and a greet ). the two agents are identical in response and use of epistemic markers however a greet randomly begin half of its generations with a greeting (see similarly to the epistemic markers, these greetings are frequently generated phrases as elicited by ...sounded like something a friend or i would say." 8 if you had another round of questions, would you like to answer the trivia questions by yourself or with the agent? @xcite from nine publicly deployed models. here, we are interested in how humans behaviors change with the introduction of a simple greeting. since users have different baseline reliance rates and perceptions, we use a control to normalize values from a greet (i.e., taking the difference between the two). ## influence of prior interactions (rq2) the next shift in human-lm interactions is the frequency in which llms are being used in everyday human tasks, with recent work showing that over 40% of survey participants were using chat models on a daily basis @xcite . humans are known to develop mental models of systems @xcite and the ai agents @xcite they interact with and update these mental models as interactions progress @xcite . prior work @xcite @xcite @xcite @xcite on calibration has made the implicit assumption that human-lm interactions can be evaluated independently from one another. as long as each individual expression is calibrated with the right human behavior, then the model is considered correctly calibrated. however, recent studies @xcite show that users form mental models early and these mental models have repercussions on future decisions. specifically, both studies showed that an overconfident system (whether using numerical or verbal uncertainties) will lead to incorrect long-term reliance decisions, even if the model later becomes calibrated. experimental set-up our experiment asks how differences in prior interactions, specifically differences in an agent's verbalized confidence level, might influence human reliance decisions. each user interacts with two agents, b conf and b unconf , to answer 30 identical geography trivia questions. b conf uses only statements of high confidence (strengtheners) and moderate confidence (weakenened strengtheners). b unconf uses only statements of moderate confidence (weakenened strengtheners) and low confidence (weakeners). as with rq1, all the epistemic markers are displayed in randomized order. we measure if the reliance rate on these otherwise identical weakened strengtheners (e.g., "i'm pretty sure it's...") for these two systems. we hypothesize that weakened strengtheners, which are ambiguously interpreted by users @xcite and most commonly generated by llms @xcite , are more susceptible to changes in interpretation. ## domain specific reliance (rq3) the last interactional context we consider is how the domain of interaction might influence reliance. language models are being deployed wildly across a spectrum of tasks. here we ask, does the domain of human-lm interaction influence reliance behaviors? experimental set-up in this experimental setting, we measure human reliance on expressions of (un)certainty across five subject domains. participants interact with five agents and are asked to rely on their generations for questions from college mathematics, abstract algebra, philosophy, world religion, and law ( ## discussion and conclusion in this work, we introduce rel-a.i., an interaction-centered approach for evaluating human reliance on lm-generated responses. with this methodology, we tackle three emergent properties of human-lm interactions and ask how these new characteristics might influence human decision-making. we find that prior interactions, the warmth of the model, and the subject domain of the interaction can all impact how and when humans choose to rely on the exact same expressions of (un)certainty. these findings illustrate the limitations of prior in vitro evaluations which exclusively focused on the calibration between internal model probabilities and the verbalized confidence of llms. we now discuss the implications and aspirations of our contributions.