Relying on the Unreliable: The Impact of Language Models’ Reluctance to Express Uncertainty (2024)

Abstract: As natural language becomes the default interface for human-AI interaction, there is a need for LMs to appropriately communicate uncertainties in downstream applications. In this work, we investigate how LMs incorporate confidence in responses via natural language and how downstream users behave in response to LM-articulated uncertainties. We examine publicly deployed models and find that LMs are reluctant to express uncertainties when answering questions even when they produce incorrect responses. LMs can be explicitly prompted to express confidences, but tend to be overconfident, resulting in high error rates (an average of 47%) among confident responses. We test the risks of LM overconfidence by conducting human experiments and show that users rely heavily on LM generations, whether or not they are marked by certainty. Lastly, we investigate the preference-annotated datasets used in post training alignment and find that humans are biased against texts with uncertainty. Our work highlights new safety harms facing human-LM interactions and proposes design recommendations and mitigating strategies moving forward.

## introduction natural language is becoming the default interface for humans to engage with artificial intelligence systems whether it be information seeking, summarization, or image captioning @xcite @xcite . as input, natural language serves as a rich and versatile medium, enabling users to articulate intricate tasks and inquiries effectively. as for output, natural language provides an opportunity for language models (lms) to generate not only informative, but nuanced responses that better support the collaboration between humans and ai. a pivotal aspect of fostering reliable human-ai interactions lies in the apt communication of model question: what is the capital of mauritania? answer: nouakchott ## epistemic markers in language models our work focuses on the alignment between lm accuracy and lm-articulated epistemic markers as perceived by users. this is referred to as linguistic calibration @xcite which builds off of work in both linguistics and machine learning. linguists has extensively studied epistemic markers as ubiquitous linguistic features that signal speaker commitment and stance. these markers broadly fall into two categories: weakenersexpressions of uncertainty, and strengthenersexpressions of certainty @xcite @xcite . in machine learning, work has focused on improving model calibration @xcite @xcite @xcite ) by calibrating the confidence value assigned by a model and model accuracy through a measure called ece @xcite . recent work has focused explicitly on how pretraining @xcite and scaling @xcite impacts the calibration of language models. most relevant to our work is @xcite 's studies on how humans interpret numerical confidences in calibrated and miscalibrated settings. a key issue remains, as numeric confidence values are known to be challenging for users to interpret @xcite . our work, in contrast, aims for a more comprehensive understanding of how humans interpret lm-generated verbal epistemic markers. we begin by eliciting open-ended generations, a departure from prior methods which prompts models to produce a predefined set of confidence expressions either numerically @xcite @xcite @xcite or using an ordinal scale @xcite . instead, our strategy enables a qualitative bottom-up (charmaz, 2006) approach towards understanding how lms generate epistemic markers, mimicking how real-world users might engage with lms. ## how do lms use epistemic markers? to answer our first motivating question, we investigate how lms such as gpt, llama-2, and claude express uncertainties within a broad and challenging, question-answering context. we find that lms prefer to respond with answers free of epistemic markers and when lms do use epistemic modifiers, they rely too much on strengtheners, leading to overconfident but incorrect generations. ## human interpretations of uncertainty with more robust understanding of how lms use epistemic markers, we shift our focus to the second inquiry: how do humans interpret lm-generated epistemic markers? using a subset of expressions generated by lms, we set up a task to evaluate the effect of these markers on user reliance on ai. we find that users by default are highly reliant on lm-generated responses and that even minor miscalibrations in systems can have long-term consequences in human-lm collaborations. ## origin of model overconfidence our work shows that lms are overconfident and that humans are highly reliant on plain statements and statements with strengtheners; all of which is exacerbated by models often being incorrect when expressing certainty. here, we turn to our last motivating question: what is the origin of model confidence and what are potential mitigations? in this section, we pinpoint lm overconfidence to an artifact of the post-training alignment process, specifically a bias from human annotators against uncertainty. ## desired criteria and mitigating solutions moving forward our research highlights the overconfidence of language models, the reliance of humans on generations, and the long-term consequences of miscalibration in human-ai interactions. here, we propose three criteria and potential implementations to mitigate overconfidence in lms. criteria: unsolicited epistemic markers language models should autonomously emit expressions of uncertainty without prompting, akin to human behavior, in order to appropriately convey levels of confidence. this is critical as the lack of epistemic marker emittance is perceived by humans as tacit certainty. concretely, this could be addressed in various stages of the rlhf pipeline, including data augmentation, annotator training, or dataset evaluation. rlhf datasets could be augmented to include additional and more representative epistemic markers. similar studies have been successful in improving safety @xcite , modeling different perspectives @xcite , and performing moral self-correction @xcite . annotators could also receive training @xcite and priming @xcite to be cognizant of potential biases (e.g., biases against uncertainty). lastly, automatic sensibility checks and data filtering using our heuristics could be used to evaluate the effectiveness of the proceeding interventions, building transparency towards the levels of overconfidence in rlhf datasets. criteria #2: comprehensive coverage language models ought to have the capacity to generate and interpret the full range of epistemic markers. however, current pre-training data is often limited to written internet data, leading to numerous limitations and biases @xcite @xcite . concretely, incorporating more diverse data sources would allow models to learn a larger range of epistemic markers. for example, hedges have long been studied in speech in formal and informal settings @xcite such as presidential debates and speeches (al-rashady, 2012; mansour and alghazo, 2021), ted talks @xcite , and student presentations @xcite . leveraging alternative training sources such as podcasts, news transcripts, and peer conversations could broaden the use of expressions of uncertainty by lms. ## conclusion our work set out to explore how users interpret epistemic markers as generated by lms in an effort to better understand the shortcomings of human-lm communications. we find that lms are overconfident in their generations and that users are highly reliant on lm responses whether there is implicit or explicit confidence. we trace the origin of model overconfidence to the rlhf process and find that annotators have a bias against uncertainty in text.