S tory A nalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding (2023)

Abstract: Analogy-making between narratives is crucial for human reasoning. In this paper, we evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus, StoryAnalogy, which contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. We design a set of tests on StoryAnalogy, presenting the first evaluation of story-level analogy identification and generation. Interestingly, we find that the analogy identification tasks are incredibly difficult not only for sentence embedding models but also for the recent large language models (LLMs) such as ChatGPT and LLaMa. ChatGPT, for example, only achieved around 30% accuracy in multiple-choice questions (compared to over 85% accuracy for humans). Furthermore, we observe that the data in StoryAnalogy can improve the quality of analogy generation in LLMs, where a fine-tuned FlanT5-xxl model achieves comparable performance to zero-shot ChatGPT.

## introduction analogy-making plays a central role in human reasoning abilities. by drawing similarities between seemingly unrelated concepts (e.g., in figure 1 , "virus" v.s. "burglar") and processes ("the virus invades cells" v.s. "the burglar breaks into the house"), we can infer that the virus infiltrates and damages cells in a similar way to how a burglar breaks into a house to steal or cause harm. these story-level analogies, which involve comparing entire narratives or coherent sequences of events, enable intelligent agents to gain insights @xcite @xcite and understand complex phenomena @xcite . ## storyanalogy conventional benchmarks in computational analogy primarily focus on word-level analogies (e.g. word to language is like note to music). however, less attention has been given to more sophisticated analogies. we introduce storyanalogy, a dataset of 24,388 pairs of stories (e.g., "the virus invades cells and dnas are damaged." versus "a burglar breaks into the house and smashes the valuables inside."), each annotated with two dimensions of similarity based to smt. ## story analogy identification we begin by assessing the ability of models to identify story analogies using two different setups. the first evaluation setup is similar to semantic textual similarity (sts) tasks @xcite , where we calculate the spearman's correlation between models' predicted similarity and the analogy scores (α) derived from annotations ( § 3.1). for the second evaluation, we reframe our dataset as multiple-choice questions and evaluate llms on this set ( § 3.2). ## story analogy generation we examine whether the dataset storyanalogy can enhance the ability of analogy generation. we evaluate flant5 @xcite , llama @xcite , chatgpt (openai, 2022), and gpt-3.5 in zero-shot and few-shot settings using 40 source stories from the test set. to explore the potential of smaller models in generating high-quality analogies, we fine-tuned flant5-xl (3b parameters) and flant5-xxl (11b parameters) using the same template. a crowd annotation is conducted to evaluate the quality of the generated stories from the models mentioned above. workers are provided with a source story and its corresponding generated target story. they are then asked to assess the following: (1) whether the target story is an analogy for the under the zero-shot setting, we observe that flant5 and llama struggle to generate meaningful analogies. they often tend to repeat patterns from the source stories (e.g., only replacing one word). in contrast, chatgpt and gpt-3.5 produce more flexible stories that are frequently considered as analogies and novel. stories in storyanalogy can help models generate better analogies. with a few demonstrations, we observe a significant improvement in the generation quality of llama (+28.4% and +27.5%). moderate improvement on chatgpt and gpt-3.5 is also observed. notably, finetuning smaller lms enhanced their generation quality. the finetuned flant5-xxl model performs better than the zero-shot chatgpt and is comparable to the few-shot chatgpt and gpt-3.5, despite having fewer parameters. furthermore, while models become more creative through finetuning and in-context learning, their generation plausibility decreases, indicating an increase in hallucination. ## conclusion we introduce storyanalogy, a multi-domain story-level analogy corpus with 24k story analo-gies pairs annotated on two similarities under the extended smt. to assess the analogy identification and generation capabilities of various models, we have devised a series of tests based on storyanalogy. the experimental findings indicate that current encoder models and llms still fall short of human performance in analogy identification. additionally, we demonstrate that generative models can greatly benefit from our dataset.