Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (2023)

Abstract: A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions. Recent studies have shown that unsupervised pre-training produces large language models (LMs) whose conditional probabilities are remarkably well-calibrated. However, the most widely-used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typically better-calibrated than the model’s conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%.

## introduction real-world prediction systems invariably make errors. however, some mitigation of these errors is possible if the system produces well-calibrated 1 confidence estimates. in this case, the system's least confident predictions correspond to those that are most likely to be incorrect, potentially allowing these predictions to be skipped or overridden by a human. in the context of language models, one consequence of poor calibration may be hallucination, where a language model confidently asserts incorrect facts or reasoning. while the ability of very large lms to absorb and synthesize knowledge about the outside world has gained significant attention @xcite @xcite , relatively little attention has been given to their well-calibratedness @xcite . further, most existing analyses of the calibratedness of llms focus on models trained with maximum likelihood, while in practice, the most widely-used llms (such as chatgpt) are fine-tuned using methods such as reinforcement learning from human feedback @xcite . some findings suggest that rlhf-lms may sacrifice well-calibrated predictions for the sake of closer adherence to user instructions in dialogue @xcite , as the reinforcement learning objective encourages the model to allocate probability mass to the most preferred answer(s), rather than matching the relative frequency of possible answers. this paper evaluates several methods for extracting confidences about model predictions from figure 2 : rlhf generally worsens the calibration of llama-70b's log probabilities, as measured by ece (lower is better) or auc (higher is better). however, this paper (tables 1-5) will show that for several strong rlhf-lms, the model's verbalized confidence is often better-calibrated than its log probabilities, reversing some of this degradation. this reversal is strongest for truthfulqa, an adversarial dataset testing common misconceptions and other difficult queries. rlhf-lms. due to concerns that rlhf may cause systematic overconfidence in the model's probabilities (figure 2 ), as well as the general unavailability of per-token log-probabilities in widely used rlhf-lms, we pay particular attention to prompts that elicit verbalized probabilities, i.e., the model expresses its confidence in token-space, as either numerical probabilities or another linguistic expression of uncertainty. we find that, surprisingly, popular rlhf-lms are able to directly verbalize confidence scores that are better-calibrated than the model's conditional probabilities (estimated via sampling), without any fine-tuning to learn verbalization. to further improve calibration, we take inspiration from research in human psychology showing that overconfidence can be mitigated by considering alternative answers before responding @xcite . we show that prompting a model to produce several answer choices before giving its confidence scores significantly improves calibration of verbalized probabilities. combined with temperature scaling @xcite , this approach generally provides better calibration than model probabilities for chatgpt foot_0 , gpt-4 foot_1 , and claude 2 foot_2 across three datasets, often reducing expected calibration error (ece) by over 50%. related work. several studies have examined the calibration of large lms @xcite @xcite @xcite , finding that combining large pre-trained lms with temperature scaling @xcite produces very well-calibrated predictions @xcite @xcite . other work focuses on the tendency of language and dialogue models to use linguistic expressions of uncertainty in a well-calibrated manner @xcite . however, existing studies focus on lms trained purely with unsupervised learning (although @xcite briefly examine rlhf-lms), while widely used models in practice are fine-tuned with instruction-tuning or rlhf @xcite . rlhf has been shown to effectively leverage annotations of human preferences to control sentiment @xcite , improve summarization or instruction-following quality @xcite , and inject behavioral priors of harmlessness @xcite . however, recent work has raised the question of whether or not rlhf harms calibration (openai, 2023). our work is the first to show that verbalized probabilities are often bettercalibrated than the model's conditional probabilities for rlhf-lms such as chatgpt, gpt-4, and claude, and llama-2-70b-chat. ## evaluating calibration in rlhf-lms to study the calibration of rlhf-lms, we conduct experiments with gpt-3.5-turbo (chatgpt), gpt-4 (gpt-4), claude-1 (claude 1), claude-2 (claude 2), and llama-2-70b-chat (llama-2-70b-chat). metrics. we measure calibration with multiple metrics. to measure ece (expected calibration error; @xcite ), we bin model predictions by their confidence and measure the average accuracy of predictions in each confidence bin. the ece is defined as the average (squared) error between the average accuracy and confidence within each bin, where each error is weighted by the fraction of samples falling within the bin. we report raw ece as well as ece with temperature scaling (ece-t). temperature scaling fits a single temperature value β to the model's confidences to minimize negative log likelihood (nll) on the data, giving scaled probability pi of class i as pi ∝ p β i . see figure 1 for a depiction of ece binning. although ece is a standard and interpretable measure of calibration error, it completely fails to capture the confidences' discriminative power. 5 we therefore also report brier score (bs; @xcite ) on temperaturescaled confidences (bs-t), a proper scoring rule @xcite ) that is the mean squared error between the confidences and the correctness labels. finally, we assess calibration using a metric from the selective classification literature @xcite , specifically, the area under the curve of selective accuracy and coverage (auc). ## results tables 1-5 show the results of evaluating various methods for extracting confidence from rlhf-lms on gpt-3.5-turbo, gpt-4, claude-1, claude-2, and llama-2-70b-chat, respectively. we distill several key conclusions from these experiments. 1. large rlhf-lms can often directly verbalize better-calibrated confidences (either a numerical confidence probability or an expression such as 'highly likely') than the models' conditional probabilities. 2. among the methods for verbalizing probabilities directly, we observe that generating and evaluating multiple hypotheses improves calibration (see figure 1 ), similarly to humans @xcite , and corroborating a similar finding in lms @xcite . ## discussion in summary, we study the calibration of widely used rlhf-lms. we first replicate the finding for gpt-4 (openai, 2023) that rlhf can worsen the calibration of a model's conditional probabilities using the open-source llama-2-70b base and chat models (figure 2 ). to mitigate this regression and ease extraction of calibrated confidence scores for models for which log probabilities are not available, we propose and study new methods that can elicit calibrated confidences from rlhf-lms by prompting the model to verbalize its confidence in token space. we find verbalized probabilities are better-calibrated than conditional probabilities across several closed models, with mixed results for llama-2-70b-chat. our results raise several questions for future work. most notably, the difference between gpt-*, claude-*, and llama-2's ability to verbalize confidence is significant. what factors are important for learning this skill? additionally, the 1-stage and 2-stage verbalized numerical confidence prompts sometimes differ drastically in the calibration of their confidences. how can we reduce sensitivity of a model's calibration to the prompt? going beyond question-answering, can we leverage good calibration in short-answer settings to improve the reliability of long-form generations, perhaps by breaking down long-form generation into a sequence of short questions? finally, to what extent does a language model's calibration depend on the domain; do our conclusions in the context of factual recall hold in the context of reasoning or arithmetic? answering these questions provides one path toward building more trustworthy and useful language systems. limitations. while our work demonstrates a promising new approach to generating calibrated confidences through verbalization, there are limitations that could be addressed in future work. first, our experiments are focused on factual recalloriented problems, and the extent to which our observations would hold for reasoning-heavy settings is an interesting open question. additionally, the lack of technical details available for many state-ofthe-art closed rlhf-lms may limit our ability to understand what factors enable a model to verbalize well-calibrated confidences and differences in this ability across different models. finally, our study is limited to short-form question-answering; future work should extend this analysis to longer-form generation settings.