Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty (2025)

Abstract: Quantifying uncertainty in black-box LLMs is vital for reliable responses and scalable oversight. Existing methods, which gauge a model’s uncertainty through evaluating self-consistency in responses to the target query, can be misleading: an LLM may confidently provide an incorrect answer to a target query, yet give a confident and accurate answer to that same target query when answering a knowledge-preserving perturbation of the query. We systematically analyze the model behaviors and demonstrate that this discrepancy stems from suboptimal retrieval of parametric knowledge, often due to contextual biases that prevent consistent access to stored knowledge. We then introduce DiverseAgentEntropy, a novel, theoretically-grounded method employing multi-agent interaction across diverse query variations for uncertainty estimation of black-box LLMs. This approach more accurately assesses an LLM’s true uncertainty and improves hallucination detection, outperforming existing self-consistency based techniques.

## introduction large language models (llms) have shown remarkable abilities to encode and apply real-world knowledge through their learned parameters @xcite . however, when crucial knowledge is absent, unreliable, inaccurately stored, or simply not retrieved, even if embedded within the model, llms can produce hallucinations @xcite . in high-stakes areas like medicine, finance, and law, such errors can have serious consequences, making uncertainty estimation as important as predictive accuracy. looking ahead, the responsible development and deployment of increasingly capable ai systems will require scalable oversight @xcite , alignment strategies that evolve with model complexity. while external assistance can support this goal, it is often costly and difficult to scale. ideally, models should be able to assess their own uncertainty to support more autonomous alignment. as models grow more powerful yet remain prone to hallucinations @xcite , it becomes increasingly important for users to identify which outputs are trustworthy. in practice, this challenge is compounded by the fact that most users interact with llms via apis @xcite , and by the growing prevalence of proprietary models, our focus turns to the black-box setting. this leads us to a central research question: how can we develop a robust methodology to quantify a model's uncertainty regarding its parametric knowledge, and further enable it to refrain from generating hallucinated responses, without internal model access or external assistance? current research on black-box model uncertainty often focuses on evaluating self-consistency for a single target query @xcite @xcite , calculating uncertainty by sampling multiple responses to the same target query and measuring entropy as uncertainty across semantically clustered responses. however, these approaches do not necessarily capture a model's true uncertainty. as in fig. 1 , a model may exhibit low uncertainty for an incorrect answer when responding to the target query, while confidently providing an accurate answer to the same target query when answering varied knowledge-perserving queries that require the same underlying knowledge (left). it may also assign inaccurate uncertainty to correct answers (right). we systematically demonstrate that the model failures shown in fig. 1 arise from the suboptimal retrieval of parametric knowledge in llms, that is, their inconsistent ability to reliably access knowledge encoded in their parameters @xcite , 2023; @xcite . our analysis further reveals that this limitation is driven by the model's susceptibility to contextual biases: subtle changes in question, e.g., different phrasing or additional contextual information, can lead to divergent responses @xcite @xcite . to address this issue and demonstrate a proof of concept, we propose diverseagententropy, a theoretically grounded framework inspired by recent advances in multi-agent collaboration for factual accuracy @xcite , self-revision in llms @xcite especially under contextual guidance @xcite , and test-time scaling @xcite . diverseagententropy leverages controlled multi-agent interaction, treating diverse, knowledge-preserving variants of a target query, queried through the same model, as distinct agents. these agents collaboratively refine their responses to the target query, enabling more reliable uncertainty estimation. as illustrated in fig. 2 , the interaction leads agents to converge on the correct answer. uncertainty is then estimated via the weighted entropy of final responses, with an abstention policy applied when uncertainty is high. in §5, we show that our uncertainty metric is more calibrated and effectively detects hallucinations, outperforming self-consistency-based blackbox methods. it achieves higher auroc, with notable gains on long-tailed and realistic benchmarks (+3.8% popqa, +7.5% freshqa, +5.2% truthfulqa). sampling across abstention rates also improves accuracy on known questions by 2.5%. the major contributions of our paper are: ## related work uncertainty estimation of lms. recent work has quantified llm uncertainty using entropy over multiple sampled outputs @xcite @xcite @xcite , but these methods rely solely on selfconsistency with the original query, which can be misleading (fig. 1 ). some studies explore verbalizing uncertainty @xcite @xcite , though xiong et al. (2024) find llms tend to be overconfident. others examine uncertainty through model activations (chen et al., 2024b; ch-wang et al., 2024), which is not feasible in our black-box setting. consistency evaluation of lms. while wang et al. (2023) and manakul et al. (2023) show that self-consistency and sampling aid reasoning and fact-checking, others highlight limitations. zhang et al. (2023) and zhao et al. (2024) argue that detecting hallucinations requires consistency across semantically equivalent questions. chen et al. (2024a) further show that llms often fail to maintain compositional consistency. agent interaction for lms. recent works @xcite @xcite improve factuality in lms through multi-agent cooperation or debate, primarily using cross-model agents with no self-correction. ## background on llm uncertainty estimation we aim to quantify the uncertainty of black-box llms for a target query q. following @xcite , we evaluate uncertainty by treating uncertainty estimation as the problem of predicting whether to trust an answer the model generates. for a given model θ, for any query q, p θ (y | q), @xmath0. we then provide background on uncertainty estimation, focusing on entropy-based evaluation, as uncertainty is commonly measured by the entropy of predictions in the existing literature @xcite . for an input query q and output space y, the total uncertainty for a given model θ can be approximated by the entropy of the output distribution @xcite : existing self-consistency-based uncertainty estimation methods in the black-box setting @xcite @xcite @xcite follow a similar procedure: 1) for a given input q, generate n response samples. 2) calculate similarity scores for these n responses. 3) compute an uncertainty estimate u (q) using the similarity scores. we explain existing methods in detail in appendix a.1. the model's uncertainty about the answer to a target query q thus depends entirely on the responses sampled specifically for that query. ## diverseagententropy in this section, we show that black-box methods relying on self-consistency misestimate model uncertainty due to a mismatch between uncertainty estimated from the original query and the model's actual knowledge (fig. 1 ). this stems from suboptimal retrieval of parametric knowledge @xcite @xcite due to susceptibility to contextual biases @xcite . to address this, we introduce a theoretically grounded same-model multi-agent framework to better recover true uncertainty. ## conclusion accurately determining the uncertainty of blackbox llms is vital for reliable responses. we show that existing methods fall short because models suffer from suboptimal retrieval of parametric knowledge due to susceptibility to contextual bias. in this paper, we propose a novel method with a theoretical framework, diverseagententropy, to quantify an llm's uncertainty after multi-agent interaction. our method overcomes the limitations of self-consistency-based uncertainty estimation and delivers superior performance in detecting hallucinations.