S ay S elf: Teaching LLM s to Express Confidence with Self-Reflective Rationales (2024)

Abstract: Large language models (LLMs) often generate inaccurate or fabricated information and generally fail to indicate their confidence, which limits their broader applications. Previous work has elicited confidence from LLMs by direct or self-consistency prompting, or constructing specific datasets for supervised finetuning. The prompting-based approaches have inferior performance, and the training-based approaches are limited to binary or inaccurate group-level confidence estimates. In this work, we present SaySelf, a novel training framework that teaches LLMs to express more fine-grained confidence estimates. In addition, beyond the confidence scores, SaySelf initiates the process of directing LLMs to produce self-reflective rationales that clearly identify gaps in their parametric knowledge and explain their uncertainty. This is achieved by using an LLM to automatically summarize the uncertainties in specific knowledge via natural language. The summarization is based on the analysis of the inconsistency in multiple sampled reasoning chains, and the resulting data is utilized for supervised fine-tuning. Moreover, we utilize reinforcement learning with a meticulously crafted reward function to calibrate the confidence estimates, motivating LLMs to deliver accurate, high-confidence predictions and to penalize overconfidence in erroneous outputs. Experimental results demonstrate the effectiveness of SaySelf in reducing the confidence calibration error and maintaining the task performance. The generated self-reflective rationales are also reasonable and can further contribute to the calibration. The code is made public at https://github.com/xu1868/SaySelf.

## introduction while large language models (llms) exhibit remarkable proficiency in reasoning and generating * equal contribution. in this work, we present sayself, a training framework that teaches llms to generate more accurate and fine-grained confidence estimates. it successfully tackles the aforementioned problems in previous work. more importantly, sayself also goes beyond the confidence elicitation in previous work, and further enables llms to generate self-reflective rationales that indicate their knowledge gap and explain their confidence estimates (figure 1 ). we accomplish this by automatically generating a model-specific dataset for supervised fine-tuning using an off-the-shelf llm (e.g., @xcite ). specifically, for each question, we sample multiple reasoning chains from llms. we then perform clustering of the reasoning chains based on the semantic similarity and retain one instance per cluster. gpt-4 is then tasked with analyzing these instances from various clusters, summarizing uncertainties in natural language from a first-person perspective, which is subsequently used for fine-tuning. for accurate and fine-grained confidence estimates, we employ reinforcement learning to calibrate llms' confidence estimate in each response. we design a reward function that incentivizes llms to produce accurate, high-confidence predictions and imposes penalties for overconfidence in incorrect responses. in addition, the self-reflective rationales and confidence estimates are generated without multiple sampling, significantly reducing the inference time. we evaluate sayself on multiple knowledgeextensive question-answering tasks. we show that sayself significantly reduces the confidence calibration error and maintains the task performance. the generated self-reflective rationales effectively capture the internal uncertainty and can further improve the calibration performance. in addition, we find that sayself enables llms to produce low confidence in unanswerable questions. our research has the potential to exert influence on both related academic research and real-world applications, including but not limited to the fol-lowing cases: (1) a clear confidence expression with explanations can promote trustworthiness in ai, from the perspective of llms' alignment. (2) the self-reflective rationales can guide llms to perform subsequent steps, like invoking external tools or asking clarification questions, for better interaction and performance. (3) we also anticipate promising developments in training protocols once llms are trained with sayself, including proactive learning algorithms that enhance llms' interactions with humans for continued learning. ## related work llms' confidence elicitation eliciting accurate confidence estimates for llm-generated answers that contain multiple tokens is challenging @xcite . previous work can be categorized into prompting-based and training-based approaches. prompting-based approaches use a specific prompt to guide llms to generate confidence scores for their predictions @xcite , or prompt llms to generate the answers multiple times and use the consistency levels as indicators of their confidence @xcite @xcite . these approaches can cause inferior performance or lead to extensive inference-time latency. training-based approaches construct a specialized dataset for supervised fine-tuning, encouraging llms to express their uncertainty. @xcite first group examples based on their types of question as the label, then obtains the confidence score for each example using the empirical accuracy for the whole group. this approach can lead to suboptimal confidence estimates since not all examples in the same group are equal. r-tuning @xcite reconstructs the sft data to add "i am sure/unsure" at the end of the correct/incorrect responses, which can only generate binary uncertainty estimates. as mentioned in section 1, sayself addresses the limitations of the previous methods and guides llms to generate more accurate and fine-grained confidence estimates. ## sayself we present sayself, a training framework to teach llms to express fine-grained confidence with selfreflective rationales (see figure 2 ). sayself consists of 2 essential stages: (1) supervised fine-tuning: we establish a model-specific dataset containing self-reflective rationales and confidence estimates. this dataset is built from multiple sampled responses from llms. (2) reinforcement learning from task supervision: we use reinforcement learning with a carefully designed reward function to further calibrate the confidence estimates for each instance. for both 2 stages, we adopt the training samples in hotpotqa @xcite , which typically require multi-step reasoning on knowledge facts to derive the answer. after the two-stage training, the trained models can directly answer questions with confidence estimates and self-reflective rationales without additional computational overhead. ## conclusion this paper presents a training framework sayself for eliciting more accurate and fine-grained confidence estimates and self-reflective rationales from llms. sayself involves supervised finetuning with a model-specific dataset constructed by summarizing the difference between multiple reasoning chains and reinforcement learning with a properly designed reward function. our evaluations across diverse datasets confirm that sayself reduces calibration errors, maintains performance, and generates insightful rationales.