Relevant or Random: Can LLM s Truly Perform Analogical Reasoning? (2025)

Abstract: Analogical reasoning is a unique ability of humans to address unfamiliar challenges by transferring strategies from relevant past experiences. One key finding in psychology is that compared with irrelevant past experiences, recalling relevant ones can help humans better handle new tasks. Coincidentally, the NLP community has also recently found that self-generating relevant examples in the context can help large language models (LLMs) better solve a given problem than hand-crafted prompts. However, it is yet not clear whether relevance is the key factor eliciting such capability, i.e., can LLMs benefit more from self-generated relevant examples than irrelevant ones? In this work, we systematically explore whether LLMs can truly perform analogical reasoning on a diverse set of reasoning tasks. With extensive experiments and analysis, we show that self-generated random examples can surprisingly achieve comparable or even better performance on certain tasks, e.g., 4% performance boost on GSM8K with random biological examples. We find that the accuracy of self-generated examples is the key factor and subsequently design two novel methods with improved performance and significantly reduced inference costs. Overall, we aim to advance a deeper understanding of LLM analogical reasoning and hope this work stimulates further research in the design of self-generated contexts.

## introduction a hallmark of human intelligence is that they can solve novel problems by drawing analogy from relevant past experiences, a concept known as analogical reasoning in cognitive science @xcite . as indicated by the name, recalling previously acquired relevant experiences can facilitate humans to better tackle new tasks, figure 1 : illustration of llm analogical reasoning in @xcite . llms are prompted to selfgenerate relevant examples as context before solving the new problem. whereas irrelevant ones are rarely beneficial and can even be distracting @xcite . for instance, when faced with a novel math problem about determinants (e.g., calculating the value of a given fourth-order determinant), humans can resolve this by reflecting upon the methodology employed to ascertain the value of a third-order determinant, whereas biological knowledge (e.g., how the human body regulates its temperature) can generally be considered irrelevant. with the recent advancements in scaling up model size and data, llms have demonstrated impressive zero-shot and few-shot performance across various reasoning tasks, especially, through advanced prompting methods like chain-of-thought (cot) @xcite . compared to common approaches such as zero or few-shot cot @xcite @xcite , @xcite introduce llm analogical reasoning, i.e., llms self-generate examples rel-evant to the query as context to better solve new problems; see fig. 1 for an example. however, it remains unclear whether relevance is the key to eliciting such capability in llms. while several studies explore the influence of the relevance of demonstrations in in-context learning (icl) and cot @xcite @xcite @xcite @xcite @xcite , none of them investigate whether self-generated relevant examples consistently outperform irrelevant ones in llm analogical reasoning. in this paper, to systematically assess the capability of llms to perform analogical reasoning, we conduct a series of ablation experiments on a variety of reasoning tasks including problems from gsm8k @xcite , math @xcite , and big-bench hard (bbh) @xcite . furthermore, we evaluate the generalizability of our findings to other reasoning tasks, e.g., gpqa @xcite , in section 4.3. with extensive experiments, we aim to address the following two research questions: to answer these questions, we empirically analyze the analogical reasoning abilities of gpt-3.5 (turbo), gpt-4o-mini, the llama series @xcite , and qwen 2.5 @xcite models. surprisingly, experimental results show that prompting llms to self-generate random examples can achieve comparable or even better performance on certain tasks which is not in line with the key claim of analogical reasoning in @xcite , indicating that llms cannot always perform analogical reasoning. as for q2, we point out through controlled experiments that the key factor is the accuracy of self-generated examples. informed by these findings, we design two approaches that can outperform existing methods with significantly reduced inference costs. specifically, we ask llms to randomly generate a few problems and manually verify their correctness, then use this fixed set of problems as in-context learning demonstrations for all test samples. consistent observations across different model types and scales consolidate the conclusions. we summarize the major contributions of our work below: ## related work this work mainly explores whether llms can truly perform analogical reasoning. in light of this, we review two lines of research that form the basis of this work: chain-of-thought prompting and llm analogical reasoning. ## methodology our analysis is based on the analogical prompting approach outlined in @xcite . specifically, for a given target problem x, analogical prompting introduces instructions like: # problem: [x] # relevant problems: recall five relevant and diverse problems. for each problem, describe it and explain the solution. # solve the initial problem: the goal is to induce llms to self-generate relevant examples, aiding them to solve the target problem via in-context learning. to ensure better performance and efficiency, several key technical decisions are made in @xcite : in this work, we leverage similar prompts foot_0 to guide llms to generate different types of irrelevant examples as context; see fig. 2 for example prompts: yasunaga et al. (2024) demonstrate that self-generating relevant examples can consistently outperform zero-shot cot and few-shot cot (handcrafted examples or retrieved top-k most similar training samples) on different tasks. therefore, we do not include these two methods in our work. interested readers can refer to the corresponding results and analysis in yasunaga et al. (2024). in addition, we show prompts for different methods on all datasets in appendix a.1. ## conclusion in this work, we have systematically assessed the capability of llms to perform analogical reasoning. we have identified key research questions and empirically analyzed a representative set of llms on a diverse collection of reasoning tasks. extensive experimental results and analysis show that llms cannot always perform analogical reasoning and the key influencing factor is the accuracy of self-generated examples rather than their relevance. given these findings, we have designed two icl-based approaches with better performance and significantly reduced inference costs. in the future, we would like to investigate additional analogical prompting methods to generate more accurate examples.