Improving Large Language Model Confidence Estimates using Extractive Rationales for Classification (2025)

Abstract: The adoption of large language models (LLMs) in high-stake scenarios continues to be a challenge due to lack of effective confidence calibration. Although LLMs are capable of providing convincing self-explanations and verbalizing confidence in NLP tasks, they tend to exhibit overconfidence when using generative or free-text rationales (e.g. Chain-of-Thought), where reasoning steps tend to lack verifiable grounding.In this paper, we investigate whether adding explanations in the form of extractive rationales –snippets of the input text that directly support the predictions, can improve the confidence calibration of LLMs in classification tasks.We examine two approaches for integrating these rationales: (1) a one-stage rationale-generation with prediction and (2) a two-stage rationale-guided confidence calibration.We evaluate these approaches on a disaster tweet classification task using four different off-the-shelf LLMs. Our results show that extracting rationales both before and after prediction can improve the confidence estimates of the LLMs. Furthermore, we find that replacing valid extractive rationales with irrelevant ones significantly lowers model confidence, highlighting the importance of rationale quality.This simple yet effective method improves LLM verbalized confidence and reduces overconfidence in possible hallucination.

## introduction large language models (llms) have been shown to achieve state-of-the-art performance on various natural language processing tasks such as classification, information retrieval, summarization, and many more @xcite @xcite . however, the adoption of these llms in high-stake scenario tasks continues to be a challenge with their lack of explainability and transparency. accurately expressing llms confidence in their prediction can aid endusers in their decision-making process, i.e., knowing when to trust/not trust. llms can verbalize uncertainty and confidence in their prediction but several studies pointed out unsolved issues with these verbalizations @xcite @xcite . for example, a recent study has shown that llms, when verbalizing their confidence, tend to be overconfident @xcite , while another study @xcite found that verbalized confidences emitted as output tokens are typically better calibrated than model's conditional probabilities in certain tasks. recent studies demonstrate that integrating explanations with confidence calibration shows promise in language models achieving better calibrated models @xcite @xcite . @xcite used token attribution explanations during model training while @xcite utilized feature attribution explanations to train a separate calibrator model. @xcite show that free text explanations generated by the llms can be unreliable but still useful to train a separate calibration model. @xcite showed that models trained on counterfactual augmented data improve model calibration and that concise explanations are preferred by calibrator models. however, post-hoc calibrators require additional training data, limiting scalability especially in lowresource settings. in this paper, we investigate whether llm prompt-only extractive rationales as explanations improve the confidence calibration of llms. extractive rationales constrain llms by anchoring predictions to explicit textual evidence, reducing overconfidence in possible hallucinations. unlike prior methods that rely on separate training data or post-hoc verifier models, our framework integrates extractive rationales directly into prompting, reducing complexity while maintaining interpretability. we perform both explain-then-predict (@xmath0, in which the llm first generates the rationale explanation and then arrives at a prediction based on it, and predict-and-explain (@xmath1, in which the llm first generates the prediction and provides the rationale, setups in generating these rationales. we ask the research question: do rationales improve llm confidence score prediction? we run our experiments for a high stakes scenario setting: disaster risk management. llms have the potential to help disaster managers filter through massive amounts of online social media data for relevant, critical, and actionable information during disaster events. with the goal of helping disaster managers, we are focused on commonly available llms that allow the disaster managers independence from a complex pipeline and the maintenance it implies. we investigate two approaches for integrating rationales when eliciting confidence estimates as shown in figure 1: (1) verbalized 1-stage: asking the llm for the rationale along with the predicted label and confidence score. this approach minimizes computational overhead aligning with our task, maintains coherence and intrinsic connection with the rationale and predicted label, reducing context fragmentation, and (2) verbalized 2-stage: asking the rationale and label first, then, afterwards, in the separate prompt adding the rationale to get the confidence score. this approach decouples the separate tasks of rationale generation and prediction with confidence estimation, allowing independent verification, akin to having a separate calibration model. we run our experiments using both closed and open-sourced off-the-shelf llms: gpt-4o-mini (openai, 2024a), gpt-4o (openai, 2024b), llama 3.1 8b-instruct (llama team, 2024), mistral 7b-instruct v0.3 @xcite across a humanitarian aid information type classification task @xcite . our key contributions are as follows: related work. model explanations have been used for calibration post-hoc by training separate verifier or calibrator models @xcite @xcite . unlike the separate post-hoc calibrators from @xcite ; ?; ye and durrett, 2022b), our prompt-based approach requires no additional training, making it suitable for off-the-shelf llms and various disaster event types. the previous way to measure confidence in model predictions rely on model's internal logits but this has become less suitable with off-the-shelf decoder-only llms. this led to methods of prompting llms themselves to express uncertainty in natural language which is referred to as verbalized confidence (lin et al., 2022; tian et al., 2023; xiong et al., 2024) where xiong et al. (2024) found that llms are prone to overconfidence when generating free text explanations while tian et al. (2023) observed better calibration when confidence is explicitly verbalized. closely related to our study, zhao et al. (2024) proposed a prompt-based approach to improving calibration asking for "facts" and "reflection" from the llm while zhang et al. (2024) proposed fidelity elicitation techniques, which are both relevant for multipurpose qa tasks but may not be as suited for classification task. our method addresses three gaps from prior work, overconfidence in generative rationales, the need for lightweight calibration, and trust. by integrating extractive rationales directly into prompting, we show how minimal architectural changes can yield calibration improvements. we examine whether the extractive rationales are being used to improve the llm calibration for the verb 2s top-1 prompt, we replace them with irrelevant rationales and measure the changes in confidence estimates. we explore two "bad" rationale variants, non-rationales -random phrases (of similar length to original rationales) that do not include any of the original rationale explanation words selected and diff-task rationales -rationales that were extracted from a different disaster tweet classification task, where some words may overlap with the original rationales. the different task we used was the type of help-seeking tweet classification: identifying whether a tweet expresses need for instrumental or emotional help in a disaster scenario @xcite . ## discussion & conclusion in this paper, we proposed integrating extractive rationale explanations with the predictions to improve llm confidence calibration in classification tasks. first, we test whether these extractive rationales hurt classification performance. we found that this approach has slightly higher to similar performance compared to the predict-only baseline, contrary to findings from huang et al. (2023)'s prompting setup with feature attribution as explanation and @xcite 's supervised training method. our results show that llms can express confidence in numerical probabilities better by asking for rationale-based explanations for both before (explain-then-predict) and after (predictand-explain) predictions than direct predict-only prompt. we showed that improvement is achieved in the two confidence verbalization strategies investigated, verb 1s and verb 2s. in the verb 2s setting, replacing the extractive rationales with "bad" rationales, non-rationales that have no overlap with the original and diff-task rationales that are from a dif-ferent classification task, hurt the llm confidence scores,thus, showing that the original rationales are relevant to the llm calibration. however, we note that this finding for the verb 2s setting is not applicable to the verb 1s setting. our results show that our method offers a lightweight alternative to complex pipelines while maintaining interpretability.