Quantifying Uncertainty in Natural Language Explanations of Large Language Models for Question Answering (2025)

Abstract: Large language models (LLMs) have shown strong capabilities, enabling concise, context-aware answers in question answering (QA) tasks. The lack of transparency in complex LLMs has inspired extensive research aimed at developing methods to explain large language behaviors. Among existing explanation methods, natural language explanations stand out due to their ability to explain LLMs in a self-explanatory manner and enable the understanding of model behaviors even when the models are closed-source. However, despite these promising advancements, there is no existing work studying how to provide valid uncertainty guarantees for these generated natural language explanations. Such uncertainty quantification is critical in understanding the confidence behind these explanations. Notably, generating valid uncertainty estimates for natural language explanations is particularly challenging due to the auto-regressive generation process of LLMs and the presence of noise in medical inquiries. To bridge this gap, in this work, we first propose a novel uncertainty estimation framework for these generated natural language explanations, which provides valid uncertainty guarantees in a post-hoc and model-agnostic manner. Additionally, we also design a novel robust uncertainty estimation method that maintains valid uncertainty guarantees even under noise. Extensive experiments on QA tasks demonstrate the desired performance of our methods.

## introduction large language models (llms) such as gpt-4 have recently achieved impressive gains in natural-language understanding and generation, demonstrating near-human fluency across a wide range of tasks @xcite . when adapted for open-domain question answering (qa), these models exploit their vast parametric knowledge to deliver concise, context-aware answers that surpass traditional retrieval pipelines in reasoning depth and coverage @xcite . however, in llm-based qa systems, the underlying llms are complex models where their inner working mechanisms are not yet fully understood. this lack of interpretability poses a significant barrier to their deployment in high-stakes decision-making applications, where inappropriate guidance can have severe consequences. given the importance of explaining llms' behaviors, many interpretation methods have been proposed @xcite . among them, natural language explanations @xcite @xcite are appealing for their self-contained insights, even when the models are closed-source. however, there is no work studying rigorous uncertainties behind these explanations, making it difficult to understand the confidence level associated with them. traditional uncertainty estimation methods (e.g., perturbation and bayesian-based methods) @xcite @xcite face significant limitations when applied to natural language explanations. specifically, they either fail to provide valid uncertainty guarantees, or require the access of model logits and extensive model retraining. in this work, we aim to provide rigorous, post-hoc, and model-agnostic uncertainty guarantees for natural-language explanations in qa. conformal prediction offers a distribution-free framework with provably valid coverage @xcite @xcite @xcite @xcite . however, traditional conformal prediction methods cannot be directly applied to natural language explanations. the reason is that in conformal prediction, the underlying models are typically trained in a supervised manner, where the uncertainty sets correspond to predefined class labels. in contrast, llms are trained in an auto-regressive fashion, generating text one token at a time, with each token conditioned on the previously generated tokens. additionally, real-world qa queries often contain noise, such as ambiguous phrasing and typographical errors. such noise can violate the underlying exchangeability assumption required by conformal prediction @xcite . therefore, these generated uncertainties could become invalid, posing substantial challenges for generating reliable uncertainty estimates of natural language explanations. although several robust conformal methods have been proposed @xcite @xcite , they typically assume well-structured datasets and fail to account for discrete and tokenlevel noise that is inherent in natural language explanations generated by llms. such noise complicates valid uncertainty guarantees for natural language explanations in qa. to address the above challenges, in this work, we propose ulxmqa, a novel uncertainty method for natural language explanations for medical question answering, which can generate valid uncertainty guarantees in a post-hoc and modelagonistic way. specifically, in our method, we first design prompts, which can assign each input token an importance score. then, we design an uncertainty set construction function, which selects explanation tokens based on their assigned importance scores. for the constructed uncertainty sets, we provide theoretical guarantees by proving that the expected fraction of ground-truth tokens included in these uncertainty sets is theoretically guaranteed. additionally, to address noisy data that may undermine the validity of the generated uncertainty sets, we also design a robust uncertain estimation method for these generated natural language explanations (rulx), which can provide robust valid uncertainty guarantees under discrete and token-level noise in questions. we further conduct extensive experiments to verify the desired performance of our proposed methods across different question answering tasks. ## methodology here, we first introduce our valid uncertainty method for natural language explanations in llmbased qa systems. then, we present the proposed robust uncertainty method, designed to mitigate the effects of noise. without loss of generality, in this paper, we consider a vision-language model based qa system, which can output accurate answers to medical questions about the input medical image. let q represent the question space and a the answer space. we denote a language model as m : @xmath0, which takes a sequence of k question tokens @xmath1, q 2 , . . . , q k ), and produces a sequence of m answer tokens @xmath2, a 2 , . . . , a m ). a designed prompt p augments the input and instructs m to emit a natural language explanation e, which we model as a subset of question tokens deemed essential for predicting a. modeling uncertainty in natural language explanations. here, we propose a post-hoc, model-agnostic method that assigns provably valid uncertainty to natural language explanations e. ## conclusion to the best of our knowledge, this work is the first to introduce a rigorous uncertainty estimation framework for natural language explanations in llm-based qa systems. the post-hoc, model-agnostic method guarantees coverage by ensuring the expected fraction of non-ground-truth explanation tokens below a threshold. building upon this, we further propose a robust extension that maintains reliable and valid uncertainty guarantees in the presence of noise. we also conduct extensive experiments across various qa tasks to comprehensively evaluate the effectiveness of our proposed methods.