Ambiguity Detection and Uncertainty Calibration for Question Answering with Large Language Models (2025)

Abstract: Large Language Models (LLMs) have demonstrated excellent capabilities in Question Answering (QA) tasks, yet their ability to identify and address ambiguous questions remains underdeveloped. Ambiguities in user queries often lead to inaccurate or misleading answers, undermining user trust in these systems. Despite prior attempts using prompt-based methods, performance has largely been equivalent to random guessing, leaving a significant gap in effective ambiguity detection. To address this, we propose a novel framework for detecting ambiguous questions within LLM-based QA systems. We first prompt an LLM to generate multiple answers to a question, and then analyze them to infer the ambiguity. We propose to use a lightweight Random Forest model, trained on a bootstrapped and shuffled 6-shot examples dataset. Experimental results on ASQA, PACIFIC, and ABG-COQA datasets demonstrate the effectiveness of our approach, with accuracy up to 70.8%. Furthermore, our framework enhances the confidence calibration of LLM outputs, leading to more trustworthy QA systems able to handle complex questions.

## introduction recent advancements in large language models (llm) @xcite @xcite have significantly improved their capabilities in question answering (qa). however, users often ask under-specified questions that can have multiple interpretations @xcite . those ambiguities typically lead to inaccurate or misleading answers, which undermine the user trust in the systems @xcite . identifying questions requiring clarification is thus a crucial task to build trustworthy nlp systems. ## task formulation we focus on the scenario where we prompt an llm to get an accurate answer to an unambiguous question or detect an ambiguous question under the few-shot setting. more specifically, given a user's question q, the qa system has access to some external information c relevant to the question. the contextual information is either provided with the question (e.g., a document-grounded conversation) or is retrieved from a collection of documents d (i.e., retrieval-augmented qa). given c and q, the goal of the qa system is to ( 1 ) find an accurate answer a to an unambiguous question; or (2) request clarification when the question is ambiguous (i.e., has multiple plausible interpretations). 1 in this paper, we focus on two tasks: ambiguity detection and confidence calibration. the goal of the ambiguity detection task is to identify whether a given question is ambiguous. as for confidence calibration, the goal is to measure the quality of the confidence estimation, which is crucial for avoiding inaccurate, incomplete, or misleading answers. ## our approach in this section, we describe our framework (see figure 2 ), aiming to (i) identify ambiguous questions; and (ii) avoid providing incorrect, incomplete, or misleading answers. we first prompt the llm to generate several answers (answer-oriented prompting; see §4.1) and then deduce the ambiguity by analyzing cues from the llm outputs (see §4.2). ## conclusion in this work, we introduce a novel framework that enables llms to recognize ambiguous questions. our approach prompts the llm to generate multiple answers that are then analyzed through an interpreter model (i.e., random forest) to detect ambiguity. the random forest is trained with only 6 examples that are bootstrapped and shuffled to create multiple answer distributions. our experiments on three datasets demonstrate the effectiveness of our approach in low-resource settings in identifying ambiguous questions. furthermore, our approach has been shown to effectively refine the confidence calibration of llm outputs: this improves the llms' ability to accurately interpret and respond to complex queries, contributing to more reliable and trustworthy qa systems.