Can LLM s Recognize Their Own Analogical Hallucinations? Evaluating Uncertainty Estimation for Analogical Reasoning (2025)

Abstract: Large language models (LLMs) often demonstrate strong performance by leveraging implicit knowledge acquired during pretraining. Analogical reasoning, which solves new problems by referencing similar known examples, offers a structured way to utilize this knowledge, but can also lead to subtle factual errors and hallucinations. In this work, we investigate whether LLMs can recognize the reliability of their own analogical outputs using black-box uncertainty estimation (UE). We evaluate six UE metrics across two reasoning-intensive tasks: mathematical problem solving (GSM8K) and code generation (Codeforces). Our results show that Kernel Language Entropy (KLE) and Lexical Similarity (LexSim) are the most robust indicators of correctness. Moreover, while analogical prompting increases model confidence over direct prompting, most uncertainty arises during the analogy transfer step. These findings highlight the limitations of analogical knowledge transfer in LLMs and demonstrate the potential of UE methods for detecting hallucinated reasoning in black-box settings.

## introduction recent advances in large language models (llms) have highlighted their surprising ability to utilize internalized knowledge for solving complex tasks. this ability, often acquired through largescale pretraining, enables models to answer factual questions, reason about concepts, and even perform domain-specific tasks without explicit retrieval @xcite . however, such knowledge utilization remains opaque and error-prone. in particular, llms frequently produce responses that are fluent and confident but factually incorrect, which is a phenomenon known as hallucination @xcite . to better understand how knowledge is used, represented, and sometimes misapplied by llms, * corresponding author we focus on a specific form of structured reasoning: analogical reasoning. this strategy encourages the model to solve a target problem by referencing a related, known problem. analogical reasoning has roots in human cognition @xcite and has been shown to enhance llm performance across domains @xcite @xcite . conceptually, it involves two stages: retrieving or constructing an analogy, and transferring it to the new context @xcite . despite its potential, analogical reasoning is also prone to hallucination-like failure. models may select an irrelevant analogy, or fail to adapt it correctly, leading to incorrect answers that nonetheless appear coherent and justified. these subtle errors are particularly dangerous in deployment settings, as they can undermine user trust in the model's reasoning ability. this raises a key research question: can llms recognize when their analogical reasoning is unreliable? we address this question by investigating the utility of black-box uncertainty estimation (ue) metrics. these methods aim to quantify model uncertainty based solely on output patterns, without requiring access to internal activations or probabilities @xcite . prior work has applied ue to tasks such as translation and summarization @xcite , but its effectiveness in analogical reasoning, where hallucinations arise from multi-step failures, remains underexplored. in this paper, we evaluate six representative ue metrics in the context of analogical prompting. our experiments span two reasoning-intensive benchmarks: gsm8k for mathematical problem solving, and codeforces for code generation. we further dissect analogical responses into their subcomponents to understand where uncertainty arises: in the analogy itself or in its transfer. this work makes three main contributions: • we present the first systematic evaluation of uncertainty estimation metrics for analogical reasoning in black-box llms. our findings provide insights into the mechanisms and limits of knowledge utilization in llms, and offer a practical pathway toward detecting hallucinated reasoning in analogical contexts. our code can be found in https://github.com/ bellafc/analogyue/ . ## method the overall experimental pipeline is illustrated in figure 1 . as part of this procedure, we apply analogical prompting to two reasoning-intensive datasets: gsm8k for mathematical problem solving and codeforces for code generation. the specific analogical prompts used for these two datasets are provided in figure 2 . ## discussion our findings highlight two key insights into ue in analogical reasoning tasks. first, kle @xcite and lexsim @xcite emerge as robust and complementary ue metrics, each excelling in different domains due to their underlying assumptions about semantic and lexical similarity. second, analogical reasoning lowers model uncertainty, but this uncertainty increases during the transfer phase, underscoring a critical bottleneck in applying analogies to novel problems. graph-based metrics (e.g., eigv, deg, ecc) and nli-based luq underperform, suggesting a misalignment with the nature of analogical reasoning. these methods assume that surface-level similarity or binary entailment captures uncertainty effectively. however, analogical tasks often require recognizing logically valid yet structurally diverse answers. their poor average scores and high variances confirm their inadequacy in capturing nuanced analogical consistency. the second set of results reveals that analogical reasoning lowers the model's self-assessed uncertainty (as reflected by higher anprr), yet this uncertainty loss does not fully translate into successful application (lower arprr). this divergence points to a key challenge: while models can identify useful analogies, the process of adapting them to new contexts introduces epistemic uncertainty. the lowest scores observed in the dpprr condition further reinforce the value of analogy-based prompting over direct prompting. however, the drop from anprr to arprr indicates that the analogy transfer step is a critical weakness in current llm capabilities. these findings suggest that future uncertainty metrics should better account for the two-step nature of analogical reasoning: analogy retrieval and transfer. while kle and lexsim provide partial solutions, hybrid models or adaptive metrics that dynamically weigh lexical and semantic coherence may further improve reliability.