Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models (2024)

Abstract: As the use of Large Language Models (LLMs) becomes more widespread, understanding their self-evaluation of confidence in generated responses becomes increasingly important as it is integral to the reliability of the output of these models. We introduce the concept of Confidence-Probability Alignment, that connects an LLM’s internal confidence, quantified by token probabilities, to the confidence conveyed in the model’s response when explicitly asked about its certainty. Using various datasets and prompting techniques that encourage model introspection, we probe the alignment between models’ internal and expressed confidence. These techniques encompass using structured evaluation scales to rate confidence, including answer options when prompting, and eliciting the model’s confidence level for outputs it does not recognize as its own. Notably, among the models analyzed, OpenAI’s GPT-4 showed the strongest confidence-probability alignment, with an average Spearman’s \hat{\rho} of 0.42, across a wide range of tasks. Our work contributes to the ongoing efforts to facilitate risk assessment in the application of LLMs and to further our understanding of model trustworthiness.

## introduction in recent years, we have witnessed the rapid development and deployment of large language models (llms) across various disciplines. llms such as gpt @xcite @xcite , palm @xcite , chincilla @xcite , and llama @xcite , have showcased remarkable performance across a diverse range of nlp tasks, and their capabilities in empowering chatbots have ignited a surge of interest among the general public. with the ongoing integration of these models into high-stakes areas such as healthcare, law, and education, a critical evaluation of their behavior and trustworthiness is becoming increasingly essential. many contemporary prompting techniques, such as self-consistency @xcite , tree of thoughts @xcite , and multi-agent debating @xcite , rely heavily on a model's self-evaluation of its reasoning process. recent works like @xcite use llms' selfconfidence for tasks like question selection based on uncertainty. however, if a misalignment exists between the model's expressed self-reasoning and its true internal confidence, these techniques could yield misleading results, undermining their practical utility. trust and decision-making by users of these models in real-world applications are directly impacted by understanding this alignment. this is further complicated by llm hallucination @xcite @xcite , where llms produce outputs that seem plausible but are factually incorrect or fabricated. these hallucinations often come with high expressions of confidence, challenging non-expert users to distinguish them from reliable outputs. notable examples of these are reported in @xcite , where chatgpt confidently cited non-existent references for specific medical claims. amid these challenges, our work aims to enhance trustworthiness in llms through a detailed understanding of model confidence. we define verbalized certainty as an llm's explicit expression of its confidence level in response to a question and investigate its relationship with the model's actual internal confidence, quantified by token probabilities. our exploration of the correlation between these two measures, termed confidence-probability alignment, posits that this alignment is crucial for the reliability of a model's output. drawing upon a range of question datasets, our investigation explores the nuances of internal and verbalized confidence across an extensive array of models, including openai's gpt-3, its variants such as instructgpt and the rlhf version, and the more recent gpt-4, in addition to open-source models, namely microsoft's phi-2-2.7b and hug-gingface's zephyr-7b. we probe the models with questions from a variety of tasks, gather response token probabilities, and solicit confidence in the answers (visually summarized in figure 1 ). we also explore model response variability under different parameters and prompts and examine the connection between a model's confidence and its actual performance. figure 2 provides a visual overview of how gpt-4 responds to different questions and its associated expressions of confidence. our contributions are three-fold: ## confidence-probability alignment confidence-probability alignment refers to the correlation between a model's verbalized certainty and its internal confidence, each quantified in specific ways. we define verbalized certainty as a model's explicit expression (i.e., in the generated text) of its confidence level regarding its own answer to a given question. the internal confidence, on the other hand, is quantified using answer token probabilities. the alignment of these two aspects can be determined using the following procedures. ## related work estimation of confidence in llms numerous studies have explored estimating confidence in llms. some emphasize model sensitivity to input changes @xcite or employ hints in neural machine translation for confidence @xcite . others use prompt engineering for verbalized probabilities @xcite , fine-tune models on question-answer accuracy probabilities @xcite , or modify prompts for uncertainty expressions @xcite . additional research quantifies uncertainty using metrics such as semantic entropy @xcite or assesses overconfidence with atypical inputs @xcite . with many llms being proprietary, some have probed confidence using both transparent and opaque methods @xcite verbalized certainty and token probabilities. self-evaluation in prompt-based techniques prompt-based techniques have been a major focus in the research community, with a particular emphasis on the accuracy of llms' self-assessments for effective functionality. newer prompting techniques, such as tree of thoughts (tot) @xcite and self-consistency (sc) @xcite , have embraced this emphasis, utilizing llms' self-evaluation to enhance their effectiveness. this represents a departure from previous approaches like chain-of-thought prompting @xcite , which did not involve the use of language models as evaluators. in our work, we too leverage the self-evaluation paradigm, but extend this by posing multiple-choice questions and providing the original answer options in the prompt, followed by a likert scale as new answer options. ## conclusion we introduced the concept of confidence-probability alignment to critically assess the transparency and reliability of llms. our findings show gpt-4 exhibiting moderate alignment, high-lighting both advancements and the pressing need for further improvements. the next steps involve formulating strategies to enhance this alignment, establishing a concrete metric for gauging model trustworthiness. innovating in this direction is crucial for advancing the development of llms that are both dependable and open.