A Survey of Confidence Estimation and Calibration in Large Language Models (2024)

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks in various domains. Despite their impressive performance, they can be unreliable due to factual errors in their generations. Assessing their confidence and calibrating them across different tasks can help mitigate risks and enable LLMs to produce better generations. There has been a lot of recent research aiming to address this, but there has been no comprehensive overview to organize it and to outline the main lessons learned. The present survey aims to bridge this gap. In particular, we outline the challenges and we summarize recent technical advancements for LLM confidence estimation and calibration. We further discuss their applications and suggest promising directions for future work.

## introduction large language models (llms) have demonstrated a wide range of capabilities, such as world knowledge storage, sophisticated reasoning, and incontext learning @xcite @xcite . however, llms do not achieve good performance on all tasks @xcite . their generation still includes biases @xcite and hallucinations that do not align with reality @xcite . assessing the trustworthiness of the generations of these models remains challenging @xcite . confidence (or uncertainty) estimation is crucial for tasks such as out-of-distribution detection and selective prediction @xcite , and it has been extensively studied and applied in various contexts @xcite . a related concept is that of model calibration, which focuses on aligning predictive probabilities (estimated confidence) to actual accuracy @xcite . however, applying these methods directly to llms presents several challenges. the output space of these models is significantly larger than that of discriminative models. the number of possible outcomes grows exponentially with the generation length, making it impossible to access all potential responses. additionally, different expressions may convey the same meaning, suggesting that confidence estimation should consider semantics @xcite . finally, llms demonstrate unique properties, such as expressing confidence in words @xcite and ability to perform zero-shot or few-shot learning @xcite . nonetheless, their responses can be sensitive to the prompts, e.g., the examples provided and their order, which can cause a lot of instability in the results @xcite . given this, confidence estimation and calibration for llms is growing as an emerging area of interest @xcite @xcite . while existing surveys mainly focused on issues such as hallucination and factuality @xcite @xcite , there are no comprehensive surveys discussing recent advancements in confidence estimation and calibration for llms; here we aim to bridge this gap. we explore the unique challenges posed by llms and examine the latest studies addressing these issues. in section 2, we first discuss key concepts such as confidence, uncertainty, and calibration in the context of neural models. we further describe different metrics for classification and generation tasks. then, we pursue two different directions: one addressing confidence estimation and calibration techniques for generation in section 3, and another one focusing on classification in section 4. we conclude by exploring their practical applicability (section 5) and looking at potential future research directions (section 6). figure 1 shows the work we explore in this survey, organized in a taxonomy. 2 preliminaries and background ## applications hallucination detection and mitigation confidence or uncertainty can be applied as a signal for detecting and mitigating hallucinations of llms @xcite . selfcheckgpt @xcite and sac 3 @xcite both explored hallucinations in the generation with self-consistency, while the latter also checked cross-model response consistency by taking generations from other models as a reference. @xcite leveraged the model's logits to identify potential hallucinations, checked their correctness through a validation procedure, appended the repaired sentence to the prompt, and continued to generate. @xcite proposed using token-level uncertainty quantification to detect hallucinations in biographies generated by llms. a similar idea was used to detect machine-generated text, based on perturbations in a white-box setup @xcite . ## future directions comprehensive benchmarks the extensive utility of confidence estimation and calibration across numerous applications calls for a robust, multidimensional benchmark that covers a diverse array of tasks and domains. moreover, fine-grained annotations of llms' responses, with an emphasis on long-form generation, are essential in fostering the development of more efficient approaches that improve the performance on intricate generation tasks @xcite . the extensive utility of confidence estimation and calibration across numerous applications calls for a robust, multidimensional benchmark that covers a diverse array of tasks and domains. moreover, fine-grained annotations of llms' responses, with an emphasis on long-form generation, are essential in fostering the development of more efficient approaches that improve the performance on intricate generation tasks @xcite . multi-modal llms by using additional pretraining with image-text pairings or by fine-tuning on specialized visual-instruction datasets, llms can be transited into the multimodal domain @xcite @xcite . however, it remains unclear whether these confidence estimation methods are effective for multimodal large language models (mllms) and whether these models are well-calibrated. @xcite found that on qa datasets focused on fact-checking, the ece of gpt-4v's verbalized confidence is much lower than that of open-source models, which tend to be overly confident. we look forward to more efforts in detecting hallucinations in mllms through confidence estimation and in calibrating these models to discern events that are impossible in the real world. calibration to human variation plank (2022) clarified the prevalent existence of human variation, i.e., humans have different opinions when labeling the same data. human disagreement (jiang and de marneffe, 2022) can be attributed to task ambiguity @xcite ), annotator's subjectivity @xcite , and input ambiguity @xcite . recent work @xcite demonstrated misalignment between llm calibration measures and human disagreement in various learning paradigms. expressing the concern regarding different types of ambiguity @xcite , abstaining from answering ambiguous questions @xcite , and further resolving ambiguity @xcite are necessary for trustworthy and reliable llms aligned with human variation. ## conclusion this survey highlights the critical role of confidence estimation and calibration in addressing errors and biases in llms. the evolution of llms has paved the way for novel research opportunities and presented distinctive challenges. we first introduced the fundamental concepts of confidence and uncertainty, along with common metrics, estimation methods, and calibration techniques used in traditional discriminative models. we then identified the challenges these methods face in llms. next, we delved into the latest research, introducing the principles, the advantages, and the drawbacks of various methods for generation and classification tasks. we concluded by discussing the current applications and future research directions.