Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction (2023)

Abstract: The vital role of analogical reasoning in human cognition allows us to grasp novel concepts by linking them with familiar ones through shared relational structures. Despite the attention previous research has given to word analogies, this work suggests that Large Language Models (LLMs) often overlook the structures that underpin these analogies, raising questions about the efficacy of word analogies as a measure of analogical reasoning skills akin to human cognition. In response to this, our paper introduces a task of analogical structure abduction, grounded in cognitive psychology, designed to abduce structures that form an analogy between two systems. In support of this task, we establish a benchmark called SCAR, containing 400 scientific analogies from 13 distinct fields, tailored for evaluating analogical reasoning with structure abduction. The empirical evidence underlines the continued challenges faced by LLMs, including ChatGPT and GPT-4, in mastering this task, signifying the need for future exploration to enhance their abilities.

## introduction analogical reasoning is one of the foundations of human cognition, which helps humans understand complex and unfamiliar concepts by relating them to familiar ones @xcite @xcite . in cognitive psychology, theories like the structure mapping theory (smt) have been proposed to explain the underlying mechanisms behind analogical reasoning @xcite . according to smt, individuals gain new knowledge by establishing mapping relations between familiar systems to unfamiliar systems (system analogy) @xcite . as an example in figure 1 , an engineer can learn the eye cross-section by taking the analogy aperture should be mapped to pupil since both are channels for light to enter. ## evaluation 4.1 evaluation settings to minimize the impact of instruction design on llms, we create 10 different instruction templates for llms in this task and select the best one to evaluate model performance. furthermore, we also explore the ability of models to use background knowledge and the cot prompting @xcite with explanations in this task. the templates are shown in ## related work analogical reasoning has been an area of interest in the ai community, primarily focusing on word analogies @xcite . early researches in word analogy focus on evaluating the quality of word embeddings, which examines linear relations between words @xcite @xcite and can be effectively addressed through vector arithmetic for neural word embeddings such as word2vec @xcite and glove @xcite . in recent years, some studies explore analogy understanding of lms on various benchmarks @xcite and fine-tuned lms using constructed knowledge bases of analogies to improve performance @xcite @xcite . new word analogy benchmarks with more complex relational structure @xcite and with multimodal elements @xcite ) @xcite are built to evaluate the performance of multilingual and multimodal models. however, a gap exists between word analogy formats and the nature of analogical reasoning in human cognition @xcite @xcite , limiting the word analogy task to reflect the llms' analogical reasoning ability aligning with humans. there has been a paradigm shift toward exploring analogies between situations @xcite . these works are inspired by the smt @xcite , which aims to establish mappings between concepts in two domains based on a common relational structure. nonetheless, @xcite focuses on simple commonsense relations. @xcite argue that two processes with similar questioning formats can be analogies, which does not address complex structures and yield unsatisfactory performance on discovering analogies between different domains. furthermore, some studies also explore the analogy generation of llms @xcite @xcite . however, they mostly evaluate word analogies or simple analogies between two sentences, leaving complex structures in analogical reasoning unstudied. @xcite and @xcite examine the abstract language-based analogy task and evaluate the analogical reasoning ability of llms on this task. compared to their task, our task requires the intensive involvement of commonsense, encyclopedic, and cultural (e.g., idiom and historical) knowledge. recent researchers study ai alignment to guide ai toward achieving human preferences and ethical principles @xcite @xcite . we explore the analogical reasoning ability of llms with complex structure abduction, which is more aligned with human cognition. ## conclusion in this paper, we explore the analogical reasoning ability of llms. we highlight word analogies neglect structures and thus can not evaluate llms in alignment with human cognition. to better evaluate llms aligning with humans, we propose an analogical structure abduction task with a new benchmark, scar. experiments show that llms struggle with this task, but incorporating background knowledge and cot prompting can improve their performance. we hope the scar can be a valuable resource to advance the research on analogical reasoning.