From Prototypical to Relational: How LLM s Navigate Complex Analogies (2025)

Abstract: We introduce a comprehensive benchmark to assess the analogical reasoning capabilities of large language models (LLMs) on complex analogy tasks that go beyond conventional formats with single correct answers. Unlike standard benchmarks that assume a singular ground truth, our framework presents a four-way multiple-choice analogy task in which all target options are semantically plausible. Leveraging concept pairs from Wikidata and AnalogyKB, we construct analogy instances enriched with multiple overlapping relational structures, where the relations are mined with RAG and ranked in salience through a GPT-4-assisted Max-Diff survey. To enable systematic evaluation, we propose three complementary semantic measures i.e. ranked relational overlap, context embedding similarity, and prototypicality; each grounded in established literature on analogical reasoning. Our experiments span a range of LLMs, evaluated under zero-shot, few-shot, and knowledge-enhanced prompting conditions. While models such as GPT-4 perform well on embedding-based and prototypicality-based measures, they consistently underperform when tasked with capturing fine-grained relational mappings. These results reveal that, despite their impressive surface-level semantic fluency, current LLMs exhibit notable limitations in structured relational reasoning.

## introduction analogical reasoning is the cognitive ability to recognize, map, and apply structural relationships between seemingly disparate concepts. it is widely regarded as a cornerstone of human intelligence, creativity, and abstraction @xcite . by enabling the transfer of knowledge from a familiar domain (the source) to a novel one (the target), analogy not only facilitates comprehension but also supports reasoning, explanation, and flexible problem-solving across a wide range of contexts @xcite . beyond simple compar-isons, analogical reasoning provides a powerful mechanism for generating hypotheses, fostering conceptual change, and driving scientific discovery. for instance, the classical analogy comparing an atom to a solar system-where electrons orbit the nucleus as planets orbit the sun, illustrates how analogical thinking scaffolds the understanding of abstract or counterintuitive scientific ideas by relating them to more familiar and intuitive phenomena. such mappings, even if not scientifically precise, often serve as conceptual entry points that guide learners toward deeper theoretical insights. more broadly, analogical reasoning is central to language, metaphor, and creativity, underpinning the ability to extend prior knowledge to novel situations and to reframe problems in innovative ways. humans can navigate complex systems through familiar relational templates @xcite . recent research in natural language processing (nlp) has posited that analogical reasoning similarly enables large language models (llms) to generalize beyond explicit training examples @xcite . accordingly, numerous studies have explored the analogical capacities of llms across tasks such as 4-way analogies @xcite , narrative analogy, story analogy, and relation-mining @xcite @xcite . these findings often report that llms can solve analogy problems in zero-shot settings, exhibiting behaviors loosely aligned with human analogical inference @xcite . however, these capabilities remain fragile and inconsistent and are highly sensitive to prompt design, context, and task formulation. crucially, prior work has largely focused on traditional analogy benchmarks like 4-term analogy or sat-style tasks @xcite , which feature a single correct answer and relatively shallow relational mapping (only one relation of stem matches with the correct target). while recent studies have proposed new analogical datasets and tasks @xcite , they often fall short in providing a sufficiently detailed semantic framework, which is essential for systematically guiding and evaluating task performance. this paper addresses a critical gap in the literature by investigating how generative large language models (llms) perform on analogy tasks with multiple plausible solutions, framed in an sat-style format. we examine whether generative llms engage in relational reasoning, consistent with established cognitive theories of analogy @xcite , or whether their responses reflect a reliance on prototypical, surface-level features. to this end, we make the following contributions: ## related work the performance of language models on analogical reasoning has been a growing area of research, showing significant advancements over time. model performance on analogy task: this work assesses model performance on analogy tasks using novel prompting, task descriptions, finetuning, and other techniques. earlier research employs the word analogy task to assess the analogical reasoning abilities of language models @xcite @xcite . recent studies @xcite have introduced analogical prompting, where language models are guided to generate relevant reasoning exemplars before solving a problem. @xcite developed prompts based on structured mapping theory and explored whether models can abduce structure while concept mapping. @xcite introduced link-of-analogy prompting, which enables llms to process new situations by drawing analogies to known situations. some studies posit that highly scaled models, such as gpt3, may achieve performance levels comparable to those of humans; however, this performance is often task-specific @xcite @xcite . ## sat multiple choice analogy task a multiple-choice four-way analogy task @xcite involves presenting an analogy problem consisting of a pair of related words or concepts as the stem, followed by several word pair options as target choices (see figure 1 ). the goal is to find the target pair that best aligns with the stem, as solving anal-ogy tasks requires matching relational structures across pairs. turney et al. @xcite collected 374 multiple-choice questions from sat exams where each question has one stem and five targets. in this section, we evaluate the complete dataset using decoder llms of varying scales, including the open-source llama 2 (13b and 70b model) @xcite and gpt3.5, gpt4 (openai, 2023) and compare it with previous benchmarks done by @xcite on encoder llms. @xcite in their benchmark used two variants of bert, one where cos(a 1a 2 , b 1b 2 ) is minimized (bert-aa) and one where cos(a 1b 1 , a 2b 2 ) is minimized (bert-ab) for 4-way a 1 : a 2 :: b 1 : b 2 analogy task foot_1 . we benchmarked the decoder models under both zero-shot and one-shot settings to assess the extent of performance improvement across these configurations. for the one-shot setting, we designed prompts using techniques like chain of thought (cot) @xcite and automatic chain of thought (auto-cot) @xcite . for details on the prompts, see appendix 9.1. the results are summarized in figure 3 . our findings indicate that small-scale decoder llms achieve performance comparable to both human accuracy and encoder models. in contrast, large-scale decoder models, such as gpt4, surpass human-level performance by a substantial margin, highlighting a significant scaling advantage. furthermore, incorporating a single in-context example in a chain-of-thought (cot) framework yields only a marginal performance improvement in gpt models. these findings suggest that analogical reasoning in llms likely emerges implicitly as a byproduct of increased model scale and extensive data exposure, rather than through explicit programming, taskspecific training, or the introduction of in-context examples @xcite . building on this insight, this study systematically examines model performance on more challenging analogy tasks, providing a deeper analysis of their reasoning abilities and limitations. ## task formalization christoph lofi (2013) provided a foundational framework for analogy tasks. expanding on this, we investigate a four-way multiple-choice analogy task. each problem consists of a stem concept pair and four target concept pairs, with the goal of selecting the target pair that most closely aligns with the stem pair. however, in our method multiple target pairs may appear plausible, and the correct choice is determined based on predefined criteria, referred to as semantic measures. ## evaluation in this study, we have benchmarked several prominent generative large language models (llms), including gpt-3.5, gpt-4 (openai, 2023), llama-2 (7b and 13b) @xcite , llama-3 @xcite , mistral-7b, falcon-40b @xcite , and mixtral-13b @xcite , etc. the evaluation of these models was conducted under three distinct prompting scenarios: zero-shot, automatic @xcite , and knowledge-enhanced @xcite prompting. in the context of zero-shot prompting, models were presented with analogy questions devoid of any prior context and asked to guess the correct answer. in the automatic chain-of-thought setting, analogy prompts were augmented by appending the directive "think step by step", thereby eliciting explicit, stepwise reasoning from the model. in the knowledge-enhanced prompting setting, we explicitly guided models with reasoning directives tailored to each semantic measure. for the prototypicality measure, the prompt first introduced the concept of prototypicality, emphasizing how certain targets are more representative exemplars of a relation than others, and instructed the model to select its response accordingly. for the ranked relational overlap measure, the prompt provided the set of salient relations associated with the stem pair. the model was asked to first infer the corresponding relations of each candidate target pair, then evaluate their relative rankings, identify the degree of relational overlap with the stem, and finally assign scores to the targets in order to select the most aligned candidate. for the context embedding similarity measure, the model was instructed to construct contextual text for both the stem and the candidate target pairs. it was then directed to compare these text and score the targets based on their semantic closeness to the stem. please note that steering the model by prompts to compare embedding directly is non-trivial. this structured prompting framework ensured that the evaluation of each semantic measure was accompanied by a clear reasoning directive, allowing us to systematically probe whether llms can adapt their reasoning strategies in accordance with explicit semantic guidance. for each model prediction we also asked models to generate rationale to explain their answer choices. ## conclusion this work introduced a novel benchmark designed to probe the analogical reasoning abilities of large language models in settings where multiple target solutions are semantically plausible. by moving beyond traditional single-answer analogy tasks, we demonstrated how llms respond when confronted with relational complexity and overlapping semantic structures. our experiments revealed that while scaling and prompting strategies improve performance, llms overwhelmingly rely on prototypicality and distributional similarity rather than finegrained relational reasoning. even the strongest models consistently underperformed on relational overlap measures, underscoring a fundamental limitation in their ability to capture structured relational mappings. these findings highlight an important gap between surface-level semantic fluency and genuine analogical reasoning. although prompting techniques, such as knowledge-enhanced instructions, provide measurable gains, they fail to address the deeper structural deficiencies observed across models. this suggests that analogical reasoning in current llms is largely an emergent byproduct of scale and training data rather than the result of explicit relational abstraction.