ANALOGYKB : Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base (2024)

Abstract: Analogical reasoning is a fundamental cognitive ability of humans. However, current language models (LMs) still struggle to achieve human-like performance in analogical reasoning tasks due to a lack of resources for model training. In this work, we address this gap by proposing ANALOGYKB, a million-scale analogy knowledge base (KB) derived from existing knowledge graphs (KGs). ANALOGYKB identifies two types of analogies from the KGs: 1) analogies of the same relations, which can be directly extracted from the KGs, and 2) analogies of analogous relations, which are identified with a selection and filtering pipeline enabled by large language models (LLMs), followed by minor human efforts for data quality control. Evaluations on a series of datasets of two analogical reasoning tasks (analogy recognition and generation) demonstrate that ANALOGYKB successfully enables both smaller LMs and LLMs to gain better analogical reasoning capabilities. Resources of this paper can be found at https://github.com/siyuyuan/analogykb.

## introduction making analogies requires identifying and mapping a familiar domain (i.e., source domain) to a less familiar domain (i.e., target domain) @xcite . as shown in figure 1 , utilizing the analogy of the solar system can facilitate comprehension of the complex structure of atoms. analogical reasoning is an important aspect of the cognitive intelligence of humans, allowing us to quickly adapt our knowledge to new domains @xcite , make decisions @xcite , and solve problems @xcite . as a result, the topic of analogy has been drawing significant research attention in the community. based on the relational knowledge triples from kgs, i.e., facts about the solar system and an atom structure, we can discover new analogies using the corresponding relations between concepts. however, resources for analogical reasoning are rather limited in scale @xcite @xcite , which usually consist of only hundreds or thousands of data samples. as a result, these datasets do not support effective training of language models to gain analogical reasoning abilities. although large language models (llms) can make some reasonable analogies without requiring gradient update, their performance still lies behind humans @xcite . therefore, larger-scale data sources are needed to facilitate the research in this area. with richer analogies, we can train specialized analogy-making models and retrieve high-quality examples to assist llms. therefore, the research question is: how to acquire largescale analogies at a moderate cost? an analogy is determined by the relational structure @xcite , e.g., a:b::c:d (i.e., a is to b as c is to d), where the relation between a and b is analogous to the relation between c and d. the concepts a, b, c, and d can be entities and events. as shown in figure 1 , the "solar system" and an "atom" share a similar structure, allowing us to quickly grasp the relation between an "electron" and a "nucleus" in concepts of their source domain counterparts. such relational structure can be derived from the triplet knowledge, e.g. (electron, orbit, nucleus) and @xcite , in knowledge graphs (kgs) @xcite . therefore, such structure knowledge can be utilized and reorganized to create new analogy knowledge, supporting large-scale knowledge acquisition. in this work, we aim to build a knowledge base (kb) for storing analogies derived from existing kgs to improve analogical reasoning. however, due to the complicated relational structures, discovering analogies from kgs is not a trivial task. although two pairs of concepts with the same relation can form a valid analogy (e.g., lion, isa, animal and apple, isa, fruit), interesting and diverse analogies are implicit in the kgs, with more complex relations. concepts under two distinct but similar relations in kgs can also form a reasonable analogy @xcite . for example, chief executive officer and head of state can both be abstracted into a meta relation @xcite , i.e., head of organization. therefore, they are analogous relations under a meta relation. it is important to generalize the finding of implicit analogies beyond the same relations within kgs. we present analogykb, which is a largescale analogy kb. we use wikidata @xcite and conceptnet @xcite as our seed kgs and discover two types of analogies from these kgs: analogies of 1) same relations and 2) analogous relations. analogies of the same relations can be directly extracted from existing kgs. in contrast, analogies of analogous relations are more implicit, requiring the finding of relation pairs from the kgs that can form valid analogies. however, it is costly to manually select analogous relation pairs. therefore, we use instructgpt 003 @xcite , a llm of great capabilities in nlp tasks, for finding and deciding the analogical semantics of relations. to eliminate the noise from the outputs of instructgpt 003 ( § 3.5), we devise two filtering rules based on 1) the symmetry of analogy and 2) meta relation summarization, which generalizes two relations into a more abstract meta relation. then, we manually review the filtered results to further ensure data quality. our analogykb comprises over 1 million analogies with 943 relations, including 103 analogous relations. smaller lms trained on anal-ogykb gain significant improvements over the previous methods, even rivaling human performance on some analogy recognition tasks. furthermore, we prove that analogykb can endow both smaller lms and llms with satisfactory analogymaking capabilities. our contributions are summarized as follows: ## analogykb construction this section details the framework for building analogykb. we first define the schema of analogykb ( § 3.1). then, we collect relations with concept pairs from existing kgs ( § 3.2, step 1) and directly obtain analogies of the same relations from kgs ( § 3.3, step 2). we propose adopting llms (ouyang et al., 2022) followed by minor human efforts to acquire analogies of analogous relations ( § 3.4, step 3). ## conclusion in this paper, we introduce analogykb, a million-scale analogy kb to improve model performance in analogical reasoning tasks. we identify two types of analogies in existing kgs, i.e., analogies of the same and analogous relations, and utilize llms with minor human examinations to find them. analogykb demonstrates its great value in assisting both smaller lms and llms with the resolution of analogy recognition and generation tasks, especially with analogies of analogous relations in analogykb.