ARN : Analogical Reasoning on Narratives (2024)

Abstract: As a core cognitive skill that enables the transferability of information across domains, analogical reasoning has been extensively studied for both humans and computational models. However, while cognitive theories of analogy often focus on narratives and study the distinction between surface, relational, and system similarities, existing work in natural language processing has a narrower focus as far as relational analogies between word pairs. This gap brings a natural question: can state-of-the-art large language models (LLMs) detect system analogies between narratives? To gain insight into this question and extend word-based relational analogies to relational system analogies, we devise a comprehensive computational framework that operationalizes dominant theories of analogy, using narrative elements to create surface and system mappings. Leveraging the interplay between these mappings, we create a binary task and benchmark for Analogical Reasoning on Narratives (ARN), covering four categories of far (cross-domain)/near (within-domain) analogies and disanalogies. We show that while all LLMs can largely recognize near analogies, even the largest ones struggle with far analogies in a zero-shot setting, with GPT4.0 scoring below random. Guiding the models through solved examples and Chain-of-Thought reasoning enhances their analogical reasoning ability. Yet, since even in the few-shot setting, the best model only performs halfway between random and humans, ARN opens exciting directions for computational analogical reasoners.

## introduction analogical reasoning is a core cognitive skill unique to humans @xcite , defined as the ability to perceive and utilize the similarities between situations or events based on (systems of) relations rather than surface similarities @xcite . the dichotomy between relational and surface similarity is illustrated in figure 1 : while the narratives q and n overlap in terms of characters, locations, and actions (surface similarity), they fail to create a system of relational correspondences and thus are disanalogous as shown by their high-level message. meanwhile, q and a are dissimilar on the surface while forming a coherent relational system of correspondences through the high-level message: no pain, no gain. mappings based on systems of relations (system mappings) have priority over independent relations @xcite . further, depending on whether the corresponding situations/events share the same domain or not, analogies and disanalogies can be qualified as near or far @xcite . analogy enables creative inferences, explanations, and generalization of knowledge and has been used for scientific inventions @xcite , solving problems @xcite , and policy-making @xcite . it has also been the subject of cognitive theories and studies about humans for common processes, such as retrieval of memories @xcite and problem-solving @xcite , mostly leveraging narratives as their experimental medium (e.g., @xcite @xcite , given their multi-tiered nature and potential for abstraction. however, these studies focus on humans or models expecting well-structured inputs @xcite , are small-scale @xcite , and involve manually curated narratives, lacking an indication of how they can be applied to create scalable benchmarks for evaluating computational models. nevertheless, borrowing from narratology theories @xcite ; here, a represents a far analogous narrative (forming a relational system mapping) to q, while n is a near disanalogy (having only surface similarities). @xcite , elements of narratives may enable analogical theories to be scaled and formalized as computational frameworks and corresponding benchmarks; however, this avenue has not been explored to date. meanwhile, analogical reasoning has also been relatively popular in natural language processing (nlp), typically framed as an intelligence test for models compared against humans. so-called word-based, proportional analogies of the form (a : b :: c : d) (e.g., @xcite @xcite are often used to measure the potential of word embeddings and language models in terms of analogical reasoning. recent studies @xcite show a strong ability of state-of-the-art (sota) large language models (llms) to discover proportional word analogies, though this skill degrades with higher complexity @xcite or associative phrasing of the input @xcite . shifting toward more complex settings, narrative-based analogy benchmarks that involve system mappings rather than simple word-based relational mappings, and are aligned with cognitive theories have been rarely considered @xcite , with limitations in scope and gen-eralizability. thus, we note a significant gap between the expressivity of cognitive theories of analogies and the present analogical reasoning benchmarks in nlp. specifically, existing computational frameworks seldom include analogy over narratives and do not provide a formalization of key concepts such as system mappings and near/far analogs. consequently, the benchmarks used to evaluate sota models do not provide the ability for systematic exploration of the sensitivity of model performance to different analogical categories beyond word-level relations. this gap brings a natural question: can stateof-the-art llms detect system analogies between narratives? to gain insight into this question, we make three contributions: our experiments show a clear gap between sota llms and humans, especially in detecting far analogies. to support further research, we make the arn benchmark publicly available at https://bit.ly/3xvtjbl . ## related work we review existing theoretical frameworks of analogies and narrative elements that provide the foundation upon which we construct our framework. then, we survey analogical reasoning benchmarks and compare them to ours, arn . ## arn framework inspired by analogical reasoning frameworks in cogpsy (e.g., @xcite , we formulate analogical reasoning over narratives as a task where input consists of a query narrative q and two candidate narratives, a and n (see figure 1 ). the goal is to select the narrative a that is analogous to q rather than the disanalogous one n , based on system mappings. we frame it as a binary classification task to enable an unambiguous evaluation of the analogical reasoning abilities of models and humans @xcite . we realize this task through a compositional framework with three components (figure 2 ): 1. extraction of elements of narratives, such as characters, actions, and high-level messages (see section 3.1); 2. formation of surface and system mappings based on those elements (see section 3.2); 3. task formulation, by defining (dis)analogies based on the mappings and partitioning the data points according to their semantic distance to the query narratives (see section 3.3). ## arn benchmark in this section, we describe the realization of the framework in section 3 as a benchmark for analogical reasoning over narratives (arn ). without loss of generality, we use narratives from the epic crowdsourced dataset @xcite , which consists of 2,500 short narratives (on average 4-8 sentences) annotated with 250 corresponding proverbs (10 narratives per proverb). we chose the epic dataset because of its systematic annotation of proverbs and the wide range of topics that it covers, allowing for the extraction of diverse elements with corresponding mappings. in section 4.1, we explain how we extract elements from the narratives in epic. the mappings between these elements are formed by quantifying their similarity across pairs of narratives (see section 4.2). the formed surface mappings serve as a basis to find distractors (see section 4.3), whereas the system mappings form corresponding analogies (see section 4.4). the arn benchmark consists of (q, a, n ) triples, pairing analogies a and their corresponding distractors n for the same query narrative q. the semantic similarity between q and a, and between q and n , are defined by using the number of formed surface mappings. the statistics and quality analysis of arn are presented in section 4.5. for more details about the involved preprocessing steps and examples, refer to appendix a. ## experiments we evaluate the analogical reasoning abilities of language models over narratives in arn using two settings. 3 first, we report on their autonomous @xcite analogical reasoning abilities to assess the emergent abilities of llms @xcite and the latent analogical reasoning abilities they might have obtained during their training (see section 5.1). following a zero-shot formulation similar to figure 1 , we provide the model with the query narrative and the two candidate narratives, asking for the best analogical match to the query narrative. asking for the most similar narrative yielded similar results (see section d.3). we evaluate the autonomous analogical reasoning abilities of six llms: gpt3.5 (openai, 2022), gpt4.0 @xcite , unifiedqa @xcite , llama-2 (touvron et al., 2023), flant5 @xcite , and macaw @xcite using deterministic hyperparameters (t @xmath0. we also evaluate sentence bert (sbert, all-mpnet-base-v2; reimers and gurevych, 2019) by picking the narrative with a higher cosine similarity to the query narrative as a baseline. additionally, in a more guided scenario, we experiment with a few-shot (in-context learning; @xcite setting to enhance models' abilities with solved demonstrations (see section 5.2). we evaluate llama-2, gpt3.5, and gpt4.0 because of their in-context learning abilities and long context windows. solved demonstrations are chosen randomly, and performance is reported as the average on three runs. for more details about the experimental setup and prompts, refer to appendix b and appendix c, respectively. ## conclusions and future work this paper addressed a gap between analogical reasoning theories in cognitive psychology and analogical reasoning benchmarks in nlp. our proposed framework studies analogical reasoning over narratives by a three-tier process of extracting narrative elements following narratology, devising corresponding mappings according to analogical reasoning theories, and providing a formal method for quantifying near/far analogical distances to create near/far analogies and disanalogies. the framework provides a solid basis for a binary qa task that enables a systematic study of models' analogical reasoning abilities across various scenarios. leveraging data on narratives with high-level proverbial messages, we instantiate the devised task with the analogical reasoning on narratives (arn ) benchmark, consisting of 1096 triples of query narratives, analogies, and distractors. evaluating multiple llms on arn in a zero-shot setting suggests that while models can recognize near analogies, their analogical reasoning performance degrades when detecting far analogies, characterized by the absence of surface mappings. this trend also holds for gpt4.0, performing best on average but dropping to a below-random performance on detecting far analogies in a zero-shot setting. few-shot prompting with chain-of-thought reasoning enhances models' performance in far analogies while being detrimental to solving near analogies. overall, we show that llms' analogical reasoning over narratives lags behind humans, especially on far analogies, which motivates further research on devising computational analogical reasoners on narratives.