A na S core: Understanding Semantic Parallelism in Proportional Analogies (2025)

Abstract: Formulaic criteria for proportional analogies, which capture relational mappings between two ratios of terms, are mainly confined to the formal level. As analogy datasets grow more complex, especially in evaluating the cognitive abilities of Large Language Models (LLMs), assessing parallelism in them becomes increasingly challenging and often requires human annotation. In this work, we propose AnaScore, an automatic metric for evaluating the strength of semantic parallelism in sentence analogies. AnaScore systematically provides formalized explanations for shared relational patterns at the level of conceptual knowledge. We apply AnaScore to annotate several existing datasets, considering different directions of the relations, and uncover artifacts in data construction. Our experiments with various LLMs demonstrate the efficacy of the AnaScore metric in capturing the inherent quality of analogical relationships, showing a positive correlation between analogy quality and model performance. Thanks to this metric, we clearly demonstrate that formally explainable examples are more beneficial for analogical reasoning, while ambiguous analogies with no clear criterion tend to hinder inference.

## introduction analogy, which relies on the parallelism of relational structures, is a ubiquitous cognitive operation that basically adapts learnt knowledge to new tasks @xcite @xcite . how analogy facilitates inference in problem-solving has been formalized into various learning paradigms for a multitude of different tasks @xcite . recent research has explored the emergent capacity of large language models (llms) to perform analogical reasoning @xcite @xcite . research in deepening our understanding of the underlying processes involved in analogy is justified by results showing that models enhanced with analogical abilities through further training demonstrate improved performance on other tasks @xcite . in llm benchmarking, analogy datasets have been developed with increasing levels of complexity. from well-defined word analogies @xcite , they have expanded to sentence-level analogies @xcite with richer relational patterns, and further to mining intricate structural mappings in long contexts @xcite @xcite . as these relational structures become more sophisticated, defining analogical parallelism becomes increasingly challenging, often requiring human annotation @xcite . performance on such complex analogies is considered a hallmark of how well llms align with human cognition @xcite @xcite . however, whether complex analogies always benefit in model learning is left to be discussed. research in cognitive psychology @xcite suggests that people are particularly sensitive to transparent mappings, profiting most from those with a high degree of structural consistency. such structural mappings allow for accurate inference from known situations to new ones. if analogies are abstract without any clear criterion for explaining the mappings they encode, will they be useful to imbue the models with reasoning abilities? to ensure effective model learning, it is indispensable to verify the solidity of analogies in existing datasets. in this paper, we focus on studying proportional analogies between sentences in the format a : b :: c : d. parallelism in proportional analogies refers to the structural mapping between two paired relationships, such that the transformation from a to b align with the transformation from c to d. in formal analogies, this parallelism is defined by equivalent transformations that can be captured through common edit operations applied to both ratios @xcite . semantic parallelism involves parallel transformations in underlying meaning, which can be described using external knowledge sources like conceptnet @xcite , as illustrated in figure 1 . to evaluate semantic parallelism, previous methods such as vector arithmetic @xcite rely on the theory of geometric parallelogram in multidimensional euclidean spaces. however, it has been corroborated that vector offset approaches are ineffective at identifying analogical relationships in embedding spaces. these measurements are often limited to simple relational patterns and cannot be generalized to richer analogical structures @xcite . there remains a need for a robust and explanatory metric to evaluate semantic analogies between sentences. in this paper, we propose anascore, an automatic metric for evaluating the quantity of conceptual mappings in sentence analogies. we use con-ceptnet to reify relational structures between sentences, depicting the underlying transformations in ratios. anascore exhaustively identifies parallel transformations between concepts, quantifying the degree of semantic parallelism of analogies, with formalized explanations of their shared relational patterns. we apply anascore to annotate existing analogy datasets and analyze the artifacts emerging from data construction based on annotation statistics. in addition, our llm experiments show a positive correlation between model performance and analogy quality. ambiguous analogies devoid of clear relational criteria are prone to impede analogy inference in few-shot learning. ## conclusion this work introduced the anascore metric designed to measure the strength of parallelism in analogies at the conceptual level. the underlying meaning of each sentence is represented using con-ceptnet. the differences between the conceptual representations of two terms reify the transformations expressed by a ratio. anascore consists in computing the similarity between the conceptual transformations expressed by two ratios. we used anascore to annotate four analogy datasets, categorizing analogies in terms of parallism strength, by quantifying the number of possible formalized explanations. we then sampled analogies from these datasets and conducted fewshot prompting experiments in solving sentence analogies using three expert llms. the experimental results revealed that analogies with evident explanations, typically reflecting more structurally coherent relationships, enable models to deliver more accurate and meaningful answers. in addition, the quality of examples has a significant impact on analogy inference. explainable analogies can strengthen the ability to solve sentence analogies, whereas ambiguous ones always disrupt in-context learning.