Fact-and-Reflection ( F a R ) Improves Confidence Calibration of Large Language Models (2024)

Abstract: For a LLM to be trustworthy, its confidence level should be well-calibrated with its actual performance. While it is now common sense that LLM performances are greatly impacted by prompts, the confidence calibration in prompting LLMs has yet to be thoroughly explored.In this paper, we explore how different prompting strategies influence LLM confidence calibration and how it could be improved. We conduct extensive experiments on six prompting methods in the question-answering context and we observe that, while these methods help improve the expected LLM calibration, they also trigger LLMs to be over-confident when responding to some instances.Inspired by human cognition, we propose Fact-and-Reflection (FaR) prompting, which improves the LLM calibration in two steps. First, FaR elicits the known “facts” that are relevant to the input prompt from the LLM. And then it asks the model to “reflect” over them to generate the final answer.Experiments show that FaR prompting achieves significantly better calibration; it lowers the Expected Calibration Error by 23.5% on our multi-purpose QA tasks. Notably, FaR prompting even elicits the capability of verbally expressing concerns in less confident scenarios, which helps trigger retrieval augmentation for solving these harder instances.

## introduction with the emergence of large language models (llms) @xcite @xcite @xcite , various prompting strategies have been proposed for improving the llm performance. it is now common sense that well-designed prompts can help elicit desirable capabilities from llms that are important for various tasks. for example, chain-of-thought and its variants have been widely used for unlocking llm reasoning capabilities @xcite @xcite @xcite . however, what is less studied is the fact that prompting also influences the model confidence in their responses. for example, as shown in figure 1 , the llm's verbalized confidence @xcite would shift when it is asked to simply first seek supporting facts. effective confidence calibration of llms -output confidence scores matching model performance -is crucial because calibrated confidence ensures the model reliability and helps guide practical use cases, e.g., identifying potential hallucinations @xcite , applying additional fact-checking @xcite , etc. beyond inference time usage, it can also help guide the training process, e.g., improving instruction tuning @xcite . therefore, one central question we want to answer is: how do different prompting methods influence the confidence calibration? to do so, we first start by assessing six different prompting strategies, on question-answering (qa) datasets that rely on reasoning @xcite and knowledge @xcite . in our assessment, we employ confidence calibration, specifically expected calibration error (ece) @xcite and macro calibration error (macroce) @xcite , to measure the discrepancy between a model's performance and its confidence levels. they are widely recognized metrics for evaluating the quality of confidence scores in models, as discussed in prior studies @xcite . we find that different prompting methods generally suffer from over-confidence, and exhibit poor calibration at the instance level. psychological and cognitive research @xcite indicates that human's over-confidence can be mitigated by disentangling the processes of fact acquisition and reasoning. instead of reasoning while stating the facts, explicitly recalling all relevant facts before deliberation can avoid early anchoring the reasoning process onto the first upcoming fact in the context. accordingly, we propose our fact-and-reflection (far) prompting to improve model confidence calibration -see figure 2 for an example. specifically, it consists of three steps: first, prompting the model for potential known facts and their sources, second, eliciting reflective reasoning to connect all recalled knowledge, and finally, generating the answer. our experiments on aforementioned datasets demonstrate that far prompting significantly reduces the confidence calibration error across various calibration measures @xcite . further analysis reveals that the improvement comes from that far prompting intrigues the model to generate cautious answers that express concerns, such as adding a comment like "there is no sufficient evidence" after the answer. we observe that expressing concerns co-occurs with an average of 13.2% reduction in confidence relative to the situations without expressing concerns. this phenomenon can suggest a potential mechanism that helps detect hard instances that are not answerable with llm's internal knowledge and may benefit from retrieval augmented generation. in summary, our main contributions are: question: what does jamaican people speak? what are the facts needed to answer this question? supporting facts: 1.jamaican people typically speak english … 2.jamaican creole or patois is one commonly … given above facts you provided, what are the sources? sources: 1. https://www.babbel.com/en/magazine/whatlanguage-do-jamaicans-speak 2. https://en.wikipedia.org/wiki/jamaican_creole given above facts provided, what is your reasoning? reasoning: there are two languages they speak … therefore, choose a final answer. answer: english and creole fact reflection answer figure 2 : an example of the proposed far prompting, which consists of three steps: fact, reflection, and answer. before answering the question and extracting the confidence scores, far explicitly decomposes the entire process into fact elicitation (with corresponding sources) and reflectively reasoning over the facts. the answers will then be utilized in the prompt to generate the final answer. highlighted text denotes the input prompts provided to the model. ## how do prompting strategies influence confidence calibration in llms? to examine the influence of different prompting strategies on the confidence calibration of llms, we follow @xcite and conduct our experiments on qa. specifically, we use qa datasets that mainly examine either reasoning @xcite or knowledge (web questions. @xcite where the confidence calibration is important due to the reliability requirement. an example question would be, "did aristotle use a laptop?". the answers are either in the format of yes/no or a set of short phrases, e.g., "english and creole". we report the macro-average scores for two kinds of qa data. following @xcite , we conduct all of our experiments using openai's gpt-3 (text-davinci-003, @xcite foot_0 . we set the default max output length to 120 and the temperature to 1.2. ## related work prompting large language models. recent research @xcite on large language models shows that in-context learning (icl) achieves great effectiveness in using models as few-shot or zero-shot reasoners. different styles of prompting such as knowledge prompting @xcite , chain of thought (cot) prompting @xcite , self-consistency prompting @xcite , self-ask @xcite , tree-of-thought prompting @xcite , and skill-in-context @xcite are then proposed to guide the model to elicit its knowledge for reasoning in different ways. most previous work mainly focuses on how such a prompting method influences the model performance on various tasks. in this paper, we compare how confidence calibration is influenced by different prompting methods. confidence calibration of llms. extracting honest and constructive confidence scores of large language models is considered an important step towards building faithful and trustworthy ai systems @xcite . many methods have been proposed recently to get reliable confidence scores with different suffix prompts added after outputting possible answers, such as a follow of true or false multiple choice question @xcite , asking models to describe the likelihood of the answer @xcite , and describing the likelihood that the answer is correct with demonstrations @xcite . however, it remains unclear how robust the methods are and how good they are comparatively. our paper proposes far prompting as an orthogonal method to improve calibration and compare different extraction methods with our test bed. recently, @xcite discuss the honesty problem of models as part of the alignment. qian et al. ( 2023 ) study the confidence change when there is a conflict between in-context and model internal knowledge. another line of work links model confidence with human confidence @xcite @xcite . in our paper, we refer to the model trustworthiness based on confidence calibration. ## conclusion and discussion we closely examined how different prompting methods influence confidence calibration and found that over-confidence may lead to bad instancelevel calibration. to address that, we propose far prompting, which decomposes the fact elicitation and reflective reasoning steps, and shows that it provides a good way to calibrate the model performance across different metrics. our further analysis reveals that the reasons behind the performance can be that far prompting elicits the model to generate more honest answers, via expressing concerns in lower confidence situations. we encourage the designers of future prompting methods to evaluate their influence on the confidence calibration in addition to the performance. meanwhile, we further suggest that future confidence extraction methods should take into account their robustness to different prompting methods.