Can Language Models Serve as Analogy Annotators? (2025)

Abstract: Conceptual abstraction and analogy-making are crucial for human learning, reasoning, and adapting to unfamiliar domains. Recently, large language models (LLMs) have made the synthesis of analogical data possible, which, however, still heavily relies on extensive human efforts to be annotated. This paper empirically examines the LLMs’ capability to annotate story-level analogical data. Specifically, we propose a novel multi-stage progressive reasoning prompt framework \texttt{A3E} (Automated Analogy Annotation Expert), which is based on the structure mapping theory from cognitive psychology and efficiently annotates candidate story pairs across six fine-grained categories. We use \texttt{A3E} to evaluate how well the state-of-the-art LLMs can serve as analogy annotators. Experimental results demonstrate that our proposed \texttt{A3E} achieves an average performance gain of + 73% across a range of prompting baselines and base LLMs. The code and data is available at https://github.com/zhangxjohn/A3E.

## introduction the ability to abstract concepts and form analogies is fundamental to human learning, reasoning, and the flexible application of knowledge to unfamiliar domains. analogies are essential for human cognition, and these abilities are equally critical to the development of artificial general intelligence (agi) that can adapt flexibly and effectively to various domains @xcite . recent studies have demonstrated that large language models (llms) possess the emergent capability to function as analogical reasoners in a wide spectrum of analogy problems @xcite . in addition, some works utilized llms to generate natural language analogous corpus, thereby leading to a spate of novel research @xcite . although analogy generation holds great promise, a significant challenge lies in the rigorthese studies underscore the essential role of human annotation in analogy research, but they also reveal its inherent limitation: the process is laborintensive and costly, imposing a bottleneck for analogy research. as llms can generate analogous corpora, can we use them as analogy annotators to effectively inspect the quality of the synthesized analogical data? we delve into this question at the level of story analogies, which compare entire narratives or coherent sequences of events that involve different entities but similar relations. these analogies enable intelligent agents to understand complex real-world phenomena @xcite and gain cognitive insights @xcite . in particular, we evaluated the state-of-the-art llms, lllama-3.1 @xcite , qwen-2.5 (yang et al., 2024), and gpt-4 (openai et al., 2023) on an extended set of analogies from @xcite . we experimentally demonstrate that the most sophisticated llms may not possess an emergent capability in analogy annotation tasks by common reasoning prompt methods, such as zero-shot, few-shot cot @xcite , and instructions like "think step by step" @xcite . the complexity of the story's analogical mapping process is manifested in the implicit causal relations between events -relations between relations. the higher-order relational analogical mapping plays a crucial role in the construction of the human mind but is lacking in llms. without explicit step-by-step heuristic prompts, llms often generate superficial or incorrect analogies. to trigger an outbreak of new research with high-quality datasets (e.g., imagenet @xcite )), we decide to drive the progress of llms in analogy annotation. in this paper, we propose a3e (automated analogy annotation expert), a multi-stage progressive reasoning prompt framework inspired by structure mapping theory @xcite , which pioneers the automated annotation of story-level analogies, leveraging the advanced natural language understanding capabilities of llms. instead of focusing solely on aligning entities and verbs across story pairs, we explicitly align sentences to capture their underlying causal structures. finally, we categorize the types of analogies into six specific labels. in summary, our contributions are: ## preliminaries gentner's structure mapping theory (smt) @xcite ) has become an important theoretical foundation for the study of analogical reasoning in cognitive science @xcite . according to smt, the core process in analogy is the analogical mapping from entities in base domain b to entities in target domain t . the mapping process relies on a common relational structure between b and t , rather than their object attributes. as shown in figure 1 , a personal computer being invaded by malware is analogous to the trojan horse incident. in this analogy, there are the following object attributes mapping: @xmath0, @xmath1, @xmath2, personal @xmath3, and sensitive personal @xmath4. it can be seen that the object attributes in the two situations are quite different. however, upon examining the structural relationship mapping: @xmath5, @xmath6, @xmath7, and @xmath8, we can realize that the causal structure of the two domains is similar and reflects the same fact: vulnerabilities will cause serious damage. we focus on story-level analogies, aiming to classify each analogy pair (b, t ) into one of six specific categories as identified by the llm annotators. inspired by smt and the verified story analogy sources @xcite , we use the following six labels: figure 2 shows a set of analogy samples with one base and six different targets to aid in further understanding the concepts of the labels mentioned above. base: an employee accepted a harmless looking attachment with contained malware. the malware invaded his personal computer and stole his sensitive personal information. ## methodology we develop a method that investigates and validates the abilities of modern llms to serve as analogy annotators concerning story analogies. annotating a pair of story narratives is a reasoning problem. to address this reasoning problem using llms, we have formulated it as a multi-step and cascading reasoning generation task, thereby simplifying the annotation task into more manageable sub-tasks. in contrast to traditional cot-based reasoning methods, each step in our method is more akin to an independent agent, responsible for addressing its specific segment of the reasoning process. this approach brings an additional advantage by reducing the complexity of analogical reasoning to a level accessible to small language models, as it decouples the long dependencies of complex reasoning. in the following sections, we introduce four steps to solve this annotation task: entity analysis ( §3.1), sentence mapping ( §3.2), relational alignment ( §3.3), and analogical conclusion ( §3.4). ## conclusion in this paper, we have made significant efforts to address the current laborious manual annotation issue for story-level analogy generation. we propose a novel automatic annotation framework, a3e, with a systematic prompting procedure inspired by the cognitive psychology, achieving an average performance gain of +73% for current state-of-the-art llms (e.g., gpt4, llama3.1, and qwen2.5) on the analogy annotation task. as a comparison, existing approaches relying on crowdsourced annotations (e.g., amazon mechanical turk) suffer from inconsistent annotation standards and quality across analogy pairs. hence our study fills one fundamental gap of annotation-to-generation pipeline. we believe our work could also contribute to the research on improving the analogy and more general reasoning capabilities for llms.