P arallel PARC : A Scalable Pipeline for Generating Natural-Language Analogies (2024)

Abstract: Analogy-making is central to human cognition, allowing us to adapt to novel situations – an ability that current AI systems still lack. Most analogy datasets today focus on simple analogies (e.g., word analogies); datasets including complex types of analogies are typically manually curated and very small. We believe that this holds back progress in computational analogy.In this work, we design a data generation pipeline, ParallelPARC (Parallel Paragraph Creator) leveraging state-of-the-art Large Language Models (LLMs) to create complex, paragraph-based analogies, as well as distractors, both simple and challenging. We demonstrate our pipeline and create ProPara-Logy, a dataset of analogies between scientific processes. We publish a gold-set, validated by humans, and a silver-set, generated automatically. We test LLMs’ and humans’ analogy recognition in binary and multiple-choice settings, and found that humans outperform the best models (∼13% gap) after a light supervision. We demonstrate that our silver-set is useful for training models. Lastly, we show challenging distractors confuse LLMs, but not humans. We hope our pipeline will encourage research in this emerging field.

## introduction analogy-making is a central to human cognition. it allows us to abstract information and understand novel situations in terms of familiar ones @xcite @xcite , abilities that are still lacking in current ai systems. research suggests that these abilities are essential for robust ai that can effectively generalize and adapt to diverse domains @xcite . according to gentner's structure mapping theory (smt) @xcite , analogy is a mapping from entities in base b to entities in target t , relying on relational similarity, not object attributes. for example, in the analogy between an electrical circuit and a water pump, there is a mapping between @xmath0, @xmath1. while object attributes are different (water is liquid, electrons are not), the relations are similar (electrons move through wires like water flows in pipes). despite the importance of analogy, relatively few analogy resources exist today. most resources mainly focus on word-analogies ("a:b is like c:d"). we argue that this setting is too simplistic, often boiling down to a single relation ("partof", conjugation); in the real world, analogies are often complex, involving multiple entities and intricate relations between them. real-world analogies are often described in natural language, adding to the complexity of the problem. a very recent work employed llms to generate analogies at scale between 2-sentence snippets (∼20 tokens) @xcite . however, resources of more complex analogies (e.g., full paragraphs) are few and sparse (18 samples max). we believe this lack of data hinders progress in computational analogy; in the past, high-quality datasets have led to a burst of novel research (e.g., imagenet @xcite ). in this work, we design a pipeline, parallelparc (parallel paragraph creator) to scale up the process of generating analogies between paragraphs (see figure 1 ), leveraging recent progress in llms. we release a gold-set, validated by humans, and a silver-set, which is automatically generated. coming up with non-trivial negative examples (non-analogous paragraphs) is a challenging task. our pipeline generates, in addition to positives (analogies), both simple negatives (random paragraphs) and challenging negatives (distractors). to demonstrate our pipeline, we create propara-logy, a dataset of paragraphs describing scientific processes across various domains, meant for studying analogical reasoning. a sample in our data includes two processes, each described via a title ("how does a solar panel work?"), a domain ("en- generate analogy candidates auto-labeling the candidates human validation generate distractors human annotation of a random sample random split silver set gold set input paragraphs (propara) analogy (positive), random (simple negative), distractor (challenging negative) analogy (positive), random (simple negative), distractor (challenging negative) figure 1: our data generation pipeline. we generate analogy candidates, then collect human annotations on a random sample to be used as few-shot for an auto-labeling model. we run the model to label candidates at scale. we randomly split the data into silver-set and gold-set, which is validated by humans. in addition to positives (analogies), we include random target paragraphs (simple negatives), and generate distractors (challenging negatives). we evaluate llms and humans on binary and multiple-choice analogical reasoning tasks on propara-logy. we found that humans outperform the best models (∼13% gap) after a light supervision. we show the automatically-generated silverset is useful for training models, and can significantly improve their performance. finally, we demonstrate the distractors significantly reduce the performance of llms, but not of humans. our main contributions are: ## existing analogy datasets we now survey available analogy resources. word analogies. many analogy resources focus on word analogies ("a:b is like c:d") @xcite @xcite @xcite . such analogies are widely used in entrance tests like the sat in the us or ncee in china. this area has gained popularity in the nlp community after @xcite show that word embeddings can model some relational similarities in terms of word vector offsets. this method can find analogies relying on certain simple types of relations, but struggles with complex relations @xcite @xcite . more recently, several studies explored the use of llms in generating word analogies @xcite . in addition to the word analogy itself (a, b, c and d), some resources include extra information, such as explanations @xcite . other resources include multiple correct options, either close analogies (c, d are similar to a, b) or far (c, d are from a different domain than a, b) @xcite . some resources include wrong answers, but often quite simple (e.g., random words). visual analogies. this is the visual equivalent of word analogies (where a, b, c, d are images). there have been multiple attempts to represent transformations between pairs of images @xcite @xcite , typically stylistic or geometric, and several resources published @xcite . as in word analogies, generating wrong answers is challenging. they are often created by either using random images or images that contain elements of the correct answer but exclude another element that is crucial for the analogy. base target similar relations title: how does a solar panel work? domain: engineering paragraph: solar energy powers an electric current within a solar panel. the photovoltaic cells within the panel convert the energy from the sun into electricity. the electrical wires then spread this power throughout the panel. the electric current is then used to power whatever the panel is connected to. title: how does photosynthesis occur? domain: natural science paragraph: photosynthesis occurs when sunlight powers chemical reactions within the chloroplasts of a plant. the chloroplasts are able to transform the energy from the sunlight into usable energy for the plant. this energy is then used to produce nutrients for the plant, which are then distributed throughout the plant. (solar energy, powers, electric current) (sunlight, powers, chemical reactions) (photovoltaic cells, convert, energy) (chloroplasts, tranform, energy) (electrical wires, spread, power) (plants, distribute, nutrients) figure 2 : an example of an analogous sample from our dataset (generated by our pipeline). two scientific processes, base and target, are described via a title, a domain, and a paragraph of natural-language text. a sample also includes similar relations, hinting at why the processes are analogous. paragraph-level analogies. very recently, @xcite created a dataset of 24k story pairs. however, the pairs are short snippets (2 sentences, ∼20 tokens), and well-aligned, making the setting overly simplistic. moreover, their work does not assess directly whether a pair is analogous. there are few resources of analogies between full paragraphs, most notably stories from cognitivepsychology literature @xcite @xcite . these datasets are manually curated and very small (18 samples max), rendering them inadequate for training models. furthermore, the stories have a near-identical structure ("mr. newton was the manager of a company that made razors"/"mr. boyce was director of manufacturing shaving knives..."), again making the setting non-realistic. notably, the dataset of @xcite also includes false analogy stories, which are similar to the base paragraph in terms of first-order relations, but dissimilar in higher-order relations (relations between the first-order relations). @xcite includes simple (random) negatives and hard negatives (snippets with similar entities). a recent work focused on finding analogies between paragraphs describing processes @xcite . their method ranks pairs of paragraphs from a dataset, such that analogous pairs rank high. however, this is a noisy resource, as many non-analogies rank high, and many of the identified analogies are from very close topics. ## dataset generation our goal is to develop a pipeline for generating high-quality data that could drive forward research efforts in computational analogy. figure 2 illustrates the format of data generated by our pipeline. a records contains two processes, base b and target t . each process is described via a title ("how does a solar panel work?"), a domain ("engineering"), and a full paragraph. in addition to expressive natural-language paragraphs, the data also includes similar relations between the two processes, which is a core element in identifying analogies @xcite . figure 2 shows a positive example (analogy). in addition to positives, our pipeline generates simple negatives and challenging distractors, designed to fool both humans and models. the pipeline (see figure 1 ) begins by using llm for generating analogy candidates -paragraphs (and relations) that potentially describe analogous processes across diverse domains in science ( §3.1). then, we use human annotators to label a random sample of the candidates ( §3.2), and use the annotated data as a few-shot for automatic labeling of candidates ( §3.3). then, we filter the data based on the automatic labels, and randomly split the filtered data into two disjoint sets: our gold-set, further validated by humans, and our silver-set, which is not ( §3.4). finally, we employ an llm to generate challenging distractors ( §3.5). ## dataset analysis our gold-set contains 310 positives (analogies), each with one corresponding simple (random) distractor and one challenging distractor. our silverset contains 403 positives, again with corresponding distractors. we note this is a proof-of-concept, and it is possible to construct larger sets if desired. gold-set analysis. we first computed the distribution of close and far analogies (based on majority vote). when all three annotators voted positive, 40% were far analogies. when at least one voted positive, the number increased to 47%. we conclude that our dataset is relatively balanced between close/far analogies. not surprisingly, disagreements are more common for far analogies. ## evaluating humans and llms we use the data to develop the propara-logy benchmark of analogy recognition. we propose binary classification and multiple-choice tasks. we evaluate the performance of both humans and stateof-the-art models, experimenting in zero-shot and guided settings (using labeled examples). our research questions are: rq1: how well can humans and models recognize analogies? rq2: is the silver-set useful for training models and improving their performance? rq3: can the distractors fool humans and models? ## conclusions analogy-making is crucial for ai to generalize and adapt to unfamiliar contexts. we designed a pipeline, parallelparc, leveraging llms to generate complex analogies and distractors. we demonstrated our pipeline by creating propara-logy, a dataset of analogies between scientific processes. propara-logy is orders of magnitude larger than previous datasets of full paragraphs, and could easily be expanded via the pipeline. our experiments show humans outperform models after light supervision, and that even the best models are more sensitive to distractors than humans. we also show that automatically generated data is useful for training and improving models. our pipeline is easy to adapt to new domains, requiring only small changes in the prompts. we hope researchers will use it in domains where analogies have shown promise. for example, in education @xcite , analogies can be used to leverage students' existing knowledge to make abstract or challenging material easier to grasp (e.g., in biology, the heart is frequently compared to a pump to help students understand how it circulates blood throughout the body, similar to how a pump moves water); in computer-assisted creativity @xcite , analogies can be used to inspire designers and engineers in solving new problems by using existing ideas from another field (e.g., nasa has embraced the principles of origami to develop foldable solar panels and satellite antennas). we hope this work will spur more nlp work on analogies, leading to novel tasks and benchmarks.