Don’t Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models (2025)

Abstract: Large language models (LLMs) have witnessed rapid advancements, demonstrating remarkable capabilities. However, a notable vulnerability persists: LLMs often uncritically accept flawed or contradictory premises, leading to inefficient reasoning and unreliable outputs. This emphasizes the significance of possessing the **Premise Critique Ability** for LLMs, defined as the capacity to proactively identify and articulate errors in input premises. Most existing studies assess LLMs’ reasoning ability in ideal settings, largely ignoring their vulnerabilities when faced with flawed premises. Thus, we introduce the **Premise Critique Bench (PCBench)**, designed by incorporating four error types across three difficulty levels, paired with multi-faceted evaluation metrics. We conducted systematic evaluations of 15 representative LLMs, Our findings reveal: (1) Most models rely heavily on explicit prompts to detect errors, with limited autonomous critique; (2) Premise critique ability depends on question difficulty and error type, with direct contradictions being easier to be detected than complex or procedural errors; (3) Reasoning ability does not consistently correlate with the premise critique ability; (4) Flawed premises trigger overthinking in reasoning models, markedly lengthening responses due to repeated attempts at resolving conflicts. These insights underscore the urgent need to enhance LLMs’ proactive evaluation of input validity, positioning premise critique as a foundational capability for developing reliable, human-centric systems.

## introduction large language models (llms) have shown rapid progress in mathematical reasoning @xcite figure 1 : illustration of how llms handle a query containing contradictory premises about book percentages. the example presents conflicting statements regarding the proportion of german books and contrasts two model behaviors: one that passively accepts the flawed premises, and another that actively identifies and reports the inconsistency. this highlights the importance of premise critique ability, which refers to the capacity to detect and articulate flaws in the input premises. a notable vulnerability of current llms is their tendency to accept flawed premises uncritically. as shown in figure 1 , models often fail to question erroneous or logically flawed inputs, leading to unnecessary, verbose reasoning on invalid questions. this wastes resources and risks misleading answers in real applications. to be effective and trustworthy, llms must evolve from passive responders into active evaluators-capable of identifying ill-posed problems and providing critical feedback on input quality. this goes beyond general robustness, emphasizing proactive detection and clear articulation of errors, inconsistencies, or flawed assumptions within user premises. we term this skill the premise critique ability, which refers to the model's capacity to detect misleading or flawed information in the input and explicitly highlight these problems in a clear and understandable way. unlike general robustness, premise critique transforms models into critical collaborators rather than mere information providers. despite its importance, most existing research @xcite @xcite focuses on evaluating llms' reasoning abilities under ideal conditions with correct premises. while some studies @xcite @xcite consider inputs containing false premises, they primarily focus on factual inaccuracies, overlooking deeper logical issues such as contradictions or inconsistencies. this reveals an urgent need for dedicated evaluation methods to assess the premise critique ability of llms. to address this gap, we propose premise critique benchmark (pcbench), a comprehensive benchmark designed to evaluate the premise critique ability of llms. we meticulously curated this benchmark by adapting mathematical reasoning problems, strategically injecting various types of logical inconsistencies and misleading information. this process yielded problems across four distinct error categories, designed to expose models to a comprehensive range of premise flaws, alongside three levels of difficulty to understand the impact of problem complexity. for each problem scenario, we created multiple versions, allowing us to evaluate models' critique performance under different conditions, including when they must spontaneously identify errors and when they are explicitly prompted to check for flaws. our evaluation employs a multi-faceted protocol, pro-viding a comprehensive view of llms' critique capabilities by capturing both their inherent ability to spot errors and their efficiency in generating responses. we conducted comprehensive experiments evaluating 15 representative llms, including the gpt-4 series, qwen3 series, and gemini. through a detailed analysis of their performance, we identify four key findings: (1) most models show limited ability to autonomously critique flawed premises, relying heavily on explicit prompts to detect errors. our main contributions are as follows: ## further discussion do models know premises are flawed? as shown in figure 5 , deepseek-r1 identifies the contradiction in the problem: maria's use of "1/4 of a skein" and "182 yards" as conflicting conditions. its internal reasoning (e.g., repeated "wait, let me check again") reflects hesitation and efforts to reconcile the discrepancy, but it ultimately ignores the fractional information and calculates solely based on "182 yards," prioritizing conditions without user input. despite recognizing the contradiction internally, the model's final answer does not explicitly highlight this inconsistency. it provides a result based on its chosen condition instead of informing users of the input flaw or explaining its reasoning. this "decision-making on the user's behalf" undermines its reliability as a "critical collaborator," as users remain unaware of the problem's issues and the model's calculation path, contradicting our definition of premise critique ability-the capability that requires not only resistance to misdirection but also clear identification and explanation of input errors. this gap between internal recognition and external expression is not unique to a single instance but is observed across multiple reasoning models: models like o3-mini-medium and o4-mini-medium (see figure 6 in the appendix) also show low ppcr despite high apcr, likely because they detect premise errors internally but fail to demonstrate premise critique ability in final responses. the case also supports the earlier finding that flawed inputs deepen models' overthinking (evidenced by deepseek-r1's repeated deliberation). despite expending computational resources on this internal processing, these models fail to incorporate their critical awareness into their final response. ## conclusion in conclusion, we present pcbench, a benchmark designed to evaluate large language models' (llms) ability to critique flawed premises by incorporating problems with diverse logical inconsistencies across multiple difficulty levels. our evaluation of 15 llms reveals key insights: most models exhibit limited autonomous critique, relying heavily on explicit prompts to detect errors. performance declines as error complexity increases, while obvious contradictions are often detected, deepen or procedural flaws are frequently missed. reasoning ability doesn't always align with critique performance, while some models spot flaws