A na DE 1.0: A Novel Data Set for Benchmarking Analogy Detection and Extraction (2024)

Abstract: Textual analogies that make comparisons between two concepts are often used for explaining complex ideas, creative writing, and scientific discovery. In this paper, we propose and study a new task, called Analogy Detection and Extraction (AnaDE), which includes three synergistic sub-tasks: 1) detecting documents containing analogies, 2) extracting text segments that make up the analogy, and 3) identifying the (source and target) concepts being compared. To facilitate the study of this new task, we create a benchmark dataset by scraping Metamia.com and investigate the performances of state-of-the-art models on all sub-tasks to establish the first-generation benchmark results for this new task. We find that the Longformer model achieves the best performance on all the three sub-tasks demonstrating its effectiveness for handling long texts. Moreover, smaller models fine-tuned on our dataset perform better than non-finetuned ChatGPT, suggesting high task difficulty. Overall, the models achieve a high performance on documents detection suggesting that it could be used to develop applications like analogy search engines. Further, there is a large room for improvement on the segment and concept extraction tasks.

## introduction by mapping a complex or an unfamiliar concept, called the target, onto a more familiar concept, called the source, analogies aid in explaining educational concepts @xcite , inspiring creativity and scientific discovery @xcite . while analogy has been studied from the perspective of nlp for a long time, most work has studied analogy-finding either based on semantic similarity between texts (e.g., @xcite ), or by detecting shorter nominal 1 data and code available at https://github.com/ bhaavya/analogy_classification a good movie script is like a blueprint, with a strong story as the foundation and multi-dimensional characters as the load-bearing walls. you can't make a good movie without them metaphors (e.g., time is money) in documents. @xcite . detecting longer and more detailed analogies (we call them as long form analogy) at the level beyond a sentence that explains the similarities between the source and target concepts has not been well studied. such analogies, however, are abundant on the web and, if detected and extracted, would enable a wide spectrum of applications, such as search engines and dialog agents that can retrieve and generate analogies for education, research, and writing assistance. in this paper, we conduct the first systematic study of such long-form analogies in documents and propose a new task, called analogy detection and extraction (anade), which includes three synergistic sub-tasks: 1) analogy document detection (anadet): detecting documents containing analogies, 2) analogy segment extraction (anase): extracting text segments that make up the analogy, and 3) analogy concept extraction (anacone): identifying the source and target concepts being compared in an analogy. the three subtasks are illustrated in figure 1 . the rationale behind the anade task is that effective algorithms for it would enable running the algorithms on large amounts of online information (even the entire web) to "harvest" all the analogies, which can then be used to power many downstream applications, such as a search engine for analogies. although recent generative models like chat-gpt could already be quite powerful for some of these downstream tasks, e.g., generating analogies for education @xcite , our proposed tasks and datasets are still quite useful because of the following reasons: (1) a reference dataset of analogies is required to quantitatively evaluate the analogy-generation capabilities of chatgpt and more powerful llms in future. technically, the proposed task anade is closely related to many well-studied nlp tasks, such as text categorization and information extraction. however, there are also significant differences between the anade subtasks and those existing tasks and the anade tasks present interesting new technical challenges for nlp research. for example, the anadet problem is essentially a special case of the general text categorization problem, where special attention needs to be given to how an analogy is discussed in a document (e.g., use of local, analogy-indicating phrases). more importantly, to the best of our knowledge, there is only one work on anadet @xcite , which only studied the special form of explanatory analogies for explaining educational concepts using a small data set that is not publicly available. to the best of our knowledge, the anase and anacone problems have never been studied, and naturally, there are no data sets available for studying them either. although they are related to extraction tasks like ner (yadav and bethard, 2019), they are clearly different in nature given that the criteria to identify them are specific to analogies. to facilitate the study of the new task anade, we create the first benchmark dataset (called anade1.0) by scraping metamia.com and investigate the performances of state-of-the-art, transformer-based models on all sub-tasks to establish the first-generation benchmark results for this new task. we find that the longformer model achieves the best performance on all the tasks, confirming its effectiveness in processing longer texts. overall, our findings suggest that the trained models already have a good performance on the anadet task and could potentially be used to develop useful applications, such as analogy search engines. additionally, the anase and anacone tasks were found to be challenging for all the models, leaving much room for future research as indicated by our error analyses. finally, we envision that once anade1.0 is released to the public, its quality and size can both be further improved continuously by the research community, leading to additional versions of the data set to better support the study of the proposed anade task in the long run. specifically, our contributions include the following: 1) our work advances our understanding of how to create a data set for studying long-form analogy in text, revealing challenges that can help future work on data set construction. 2) we create a new data set that, for the first time, enables quantitative evaluation of algorithms for long-form analogy detection and extraction. 3) we compare multiple representative state-of-the-art algorithms and establish the very first benchmark for the new computational problems we defined for long-form analogy. although the reliability of this data can be further improved, this benchmark, including the data set and performance of sota methods, is the necessary initial step toward advancing research on long-form analogy detection and extraction, which have widespread applications (e.g., help students understand complex concepts using analogies). ## related work we now describe related work on identifying analogies, text categorization and information extraction. ## anade1.0 dataset in anade, the input is a set of documents (e.g., webpages) and the output is a set of analogy segments along with the source and target concepts used to form the analogy. to our knowledge, there is no publicly available dataset for studying anade. a major contribution of our work is to construct the first dataset for studying anade. specifically, to facilitate research on anade with quantitative evaluation, we construct a dataset based on metamia.com foot_0 , called anade1.0. metamia is a crowdsourced website, where general web users submit analogies found on the web with the following information: a brief description of the analogy, the two concepts that are being compared to each other -a target (usually a more unfamiliar concept) and a source concept (usually the more familiar concept ), and a reference link to the webpage where the analogy was found. ## anade benchmarking in this section, we present our benchmarking results. as our data set is the first dataset for studying anade, our goal in evaluation is to compare a set of representative state-of-the-art methods in terms of their performances on all the 3 subtasks so that we can establish the very first-generation benchmark results to enable further improvement of the models for solving the anade problem. we report all implementation details in the appendix a. ## conclusion and future work automated detection and extraction of analogies from large amounts of online text has many applications such as search engines and dialog agents based on analogies. to this end, we proposed and studied a new task, called analogy detection and extraction (anade), which includes three synergistic sub-tasks: 1) analogy document detection (anadet), 2) analogy segment extraction (anase), and 3) analogy concept extraction (ana-cone). to facilitate the study of this new task, we created the first dataset, called anade1.0. we systematically investigated the performances of stateof-the-art transformer-based models on all subtasks and established the first-generation benchmark results for this new task. we found that the longformer model achieves high performance at the anadet task, suggesting that we can already use it to crawl many analogy documents from the web to build an analogy search engine. at the same time, anase and anacone are not only novel but also challenging, suggesting much room for future research based on our data set as suggested by our error analyses. further, since the longformer model can process a longer input context (e.g., compared to bert), it has much better performance, especially on tasks where the input is the full web page that is typically very long. this indicates that, in future, models that are better at processing longer input contexts would likely perform even better. moreover, smaller models fine-tuned on our dataset perform better than the non-finetuned chatgpt model, further suggesting the value of our dataset for training and the difficulty of our proposed tasks. overall, our work also paves the way for interesting applications and research in this area. for example, for analogy segment extraction simile detection methods @xcite could be leveraged to first identify the core comparative phrase (e.g., a is like b) and then boundaries of the full analogy around this phrase could be identified. further, a sentence-level formulation of analogy segment extraction could also be explored to potentially alleviate some of the issues due to subjectivity in segment boundary annotation. our dataset would enable studying such a formulation too after additional pre-processing, such as sentence-tokenizing the documents and analogy spans. finally, one limitation of the current version of our data set is the existence of some noise that we couldn't remove due to limited resources. naturally, an important future task is to further improve the quality of anade1.0 (e.g., by human-in-theloop annotation of errors made by our trained models) and further verify our findings.