Calibrating the Confidence of Large Language Models by Eliciting Fidelity (2024)

Abstract: Large language models optimized with techniques like RLHF have achieved good alignment in being helpful and harmless. However, post-alignment, these language models often exhibit overconfidence, where the expressed confidence does not accurately calibrate with their correctness rate. In this paper, we decompose the language model confidence into the Uncertainty about the question and the Fidelity to the answer generated by language models. Then, we propose a plug-and-play method, UF Calibration , to estimate the confidence of language models. Our method has shown good calibration performance by conducting experiments with 6 RLHF-LMs on four MCQA datasets. Moreover, we propose two novel metrics, IPR and CE, to evaluate the calibration of the model, and we have conducted a detailed discussion on Truly Well-Calibrated Confidence for large language models. Our method could serve as a strong baseline, and we hope that this work will provide some insights into the model confidence calibration.

## introduction large language models (llms) acquire vast world knowledge and demonstrate powerful capabilities through pre-training @xcite @xcite . with technologies like rlhf @xcite and rlaif @xcite , large language models can become more helpful and harmless to align with human preferences @xcite . however, how to build a more honest system has not yet been fully discussed. an honest model should have a certain understanding of the boundary of its knowledge, that is, knowing what it does not know @xcite @xcite . a plausible method is utilizing the calibrated confidence to estimate the knowledge boundary of language models. for pre-trained language models, the per-token logit can already be considered a well-calibrated confidence score, which implies that pre-trained language models (mostly) know what they know @xcite . however, recent studies have indicated that language models optimized with techniques like rlhf will exhibit issues of overconfidence @xcite @xcite @xcite @xcite . this issue could be reflected in multiple-choice question answering (mcqa) tasks, where the probability of rlhf-lms generating a token and the likelihood of that token being the correct answer are not well-calibrated. for example, an answer provided by rlhf-lms with 95% confidence does not mean that there is a 95% probability that the answer is correct. this phenomenon may be due to the optimization objective of rlhf, which is to make the model generate responses aligned with human preferences rather than fitting answers that appear more frequently in the corpus during the pre-training stage. to alleviate the issue of miscalibration, previous work focuses on two perspectives: the logit-based method and the verbalization-based method. logitbased methods are usually post-hoc. we need to find a higher temperature (usually above 2.0), known as temperature-tuning @xcite , to make the distribution of the model's token logit smoother for mitigating overconfidence @xcite . the verbalizationbased method usually requires prompt engineering to elicit the model's confidence, and it also necessitates the model to have strong self-awareness @xcite @xcite . aggregating the model's logit-based and verbalizationbased confidence can also calibrate the model confidence to some extent @xcite . as shown in figure 2 and appendix tabel 6, by replacing the model's answer with "all other options are wrong.", we can assess whether the model had high fidelity to its previously given answer. inspired by this phenomenon, we decompose the language model confidence into two dimensions: the uncertainty about the question and the fidelity to the answer generated by language models. first, if the answers provided by language model are consistent under multiple samplings, it indicates that language model has lower uncertainty regarding that question. thus, we could utilize the information entropy of the frequency distribution of sampled answers to calculate the model's uncertainty about a question. second, we design a novel method to estimate the model's fidelity to each of its sampled answers. last, the uncertainty regarding question q and the fidelity to the answer a i together determine the model's confidence. as shown in figure 1 , our proposed uf calibration achieved good calibration across different mcqa datasets. meanwhile, uf calibration does not require knowledge of the model's pertoken log-probability, making it broadly applicable to various black-box rlhf-lms, which do not provide the per-token log-probability. to have a closer look at the calibration of model confidence, we propose two novel metrics for evaluating and observation: 1) inverse pair ratio (ipr), which is the proportion of inverse pairs in the reliability diagram. this metric could reflect whether the model is well-calibrated from the perspective of the monotonicity of the reliability diagram. if the reliability diagram is monotonic, it indicates that the average accuracy of low-confidence an-swers is always lower than that of high-confidence answers. 2) as shown in ## related work recent work has focused on llm calibration @xcite @xcite . in this section, we will briefly introduce two mainstream methods for eliciting the confidence from language models, namely the logit-based method and the verbalization-based method. ## methodology in this section, we will introduce the method we propose. our method does not require any knowledge of the per-token logit of language models or trivial prompt engineering to make the language model output its confidence in a specified format. ## https://openai.com/chatgpt to reduce the cost of api calls, we sampled 1 8 of the data for testing for each subject. common-senseqa (talmor et al., 2019) is a dataset for commonsense question answering, and we use the validation split with 1,221 questions for experiments. truthfulqa @xcite is a dataset that contains 817 questions designed to evaluate language models' preference to mimic some human falsehoods. all the experiments are conducted under a 0-shot setting. metrics. we utilize multiple metrics to evaluate. we bin the predictions from the model by their confidence and report the ece (expected calibration error). we also report the brier score of different methods in in this paper, we adopt 10 equal-size bins to calculate ece 10 , ipr 10 and ce 10 . we also report the accuracy on these benchmarks to measure whether calibration reduces the accuracy. baselines. we compared our approach with different baselines for eliciting the confidence of language model. first, we reproduced the verb and ling method proposed by @xcite . the verb method involves prompting the model to output a floating-point number between 0 and 1 to represent its confidence immediately after providing an answer @xcite . the ling method entails having the language model express its confidence level in natural language @xcite . since commercial models like chatgpt do not provide per-token logits, we employed a sampling technique to estimate the probability of token generation, referred to as the sampled method. unless otherwise specified, the sampled method involves sampling 10 times. for open-source models like llama2-chat, we directly use the probability of token generation as the measure of the language model's confidence, which we refer to as the token method. we also compare the conformal prediction baseline proposed by @xcite with our uf calibration in appendix d.1. all the prompt templates we use are shown in appendix e. ## analysis and discussion to take a closer look at the difference between different calibration methods tailored for language models, in this section, we verify the robustness of our method from two aspects: temperature-scaling and parameter-scaling. meanwhile, we also con- temperature-scaling in the main experiments, we evaluate various methods using a constant temperature of 1.0. in this section, we will explore the influence of sampling temperature on the performance of different methods. as illustrated in figures 4 and 7 , our proposed calibration method consistently achieves the lowest expected calibration error across all temperatures, showing remarkable robustness to temperature variations. this is because, in eliciting model fidelity, our method always employs greedy decoding rather than sampling. thus, the hierarchical chains we obtain are usually consistent across different sampling temperatures. in contrast, the expected calibration error of logit-based methods is usually affected by temperature. for the sampling method with limited sampling budgets, the lower the temperature, the more significantly the diversity of the sampled results will decrease, exacerbating the overconfidence of language models. for the token method, the impact of temperature on its calibration shows a trend of "first increasing and then remaining relatively stable" or "first increasing and then decreasing". this is because we could directly utilize (1) to estimate the confidence of each option, and if the temperature is too low (i.e., 0.1), it will lead to the confidence of a large number of options approaching zero. this phenomenon might contribute to reducing expected calibration error, but it does not necessarily indicate that the model's confidence is well-calibrated. the verbalization-based method is less affected by temperature, which indicates that the expressions which language models prefer to output are relatively consistent across different temperatures. parameter-scaling as shown in figure 5 , we evaluate the calibration of various methods at different parameter scales on the llama2-chat series models. our proposed method exhibits good calibration across different amounts of model parameters. with the size of model parameters increasing, the calibration of the verbalization-based method and the logit-based method is improving. this phenomenon indicates that as the scale of model parameters increases, the model's self-awareness is improving. however, the relatively high expected calibration error suggests that language models still have issues with overconfidence. truly well-calibrated confidence previous work mainly evaluates the calibration of language models through ece. this section will discuss the research question: "what kind of confidence is truly well-calibrated?". figure 6 demonstrates the calibration of various methods. from the calibration perspective, we hope that the confidence and accuracy relationship is close to the curve @xmath0. thus, we need to reduce the ece by calibrating confidence. meanwhile, we hope that the reliability diagram should be as monotonic as possible to ensure that the accuracy of the results generated with low confidence is lower than that of the results with high confidence. therefore, we propose the inverse pair ratio (ipr) to evaluate monotonicity. from the perspective of building a more honest system, we hope the model's confidence should be distributed across different confidence intervals. for example, if a language model has an overall accuracy of 75% on the truthfulqa dataset and the confidence of each question from the language model is always 75%, its ece and ipr would be 0. and we find that different models tend to express confidence within a fixed interval. in this case, we think that the confidence may not necessarily be a truly well-calibrated confidence because we could not exclude some low-confidence results based on the confidence from the language model. although the prior distribution of the model's confidence is unknown, our confidence estimation method finds that language models have different confidence for different questions. thus, we propose a metric called confidence evenness (ce) to measure whether the model confidence always is located in a fixed interval. we believe ece, ipr, and ce evaluate calibration from different perspectives and there is a trade-off between these three metrics. we suggest that truly well-calibrated confidence should achieve a balance among ece, ipr, and ce, rather than over-optimizing any of them. ## conclusion in this paper, we decompose the language model confidence into the uncertainty about the question and the fidelity to the answer generated by language models. through the decomposition, we propose a plug-and-play method, uf calibra-tion, to calibrate the confidence of language models. through experiments with 6 rlhf-lms on 4 multiple-choice question answering benchmarks, our method exhibits good calibration. besides, we propose two novel metrics, ipr and ce, to evaluate the calibration of language models. finally, we conduct a detailed discussion on truly well-calibrated confidence. we believe our method can serve as a strong baseline, and we hope that this work could provide some insights into the language model confidence calibration.