Long-Form Analogy Evaluation Challenge (2024)

Abstract: Given the practical applications of analogies, recent work has studied analogy generation to explain concepts. However, not all generated analogies are of high quality and it is unclear how to measure the quality of this new kind of generated text. To address this challenge, we propose a shared task on automatically evaluating the quality of generated analogies based on seven comprehensive criteria. For this, we will set up a leader board based on our dataset annotated with manual ratings along the seven criteria, and provide a baseline solution leveraging GPT-4. We hope that this task would advance the progress in development of new evaluation metrics and methods for analogy generation in natural language, particularly for education.

## introduction analogies are integral to several practical applications. in education, they help explain complex concepts by mapping them to more familiar ones @xcite (e.g., "earth rotates on its axis like an ice skater doing a pirouette"). they also inspire creativity by connecting seemingly disparate concepts @xcite . since manually creating good analogies can be challenging and require domain expertise @xcite , recently, large language models (llms) like gpt-3 @xcite have been used to aid with all such applications @xcite @xcite . they have shown great promise in generating long-form analogies (i.e., natural language analogies, typically a few paragraphs long, that describe the similarities between concepts) that are meaningful, novel @xcite and useful for science writers @xcite . however, not all automatically generated analogies are accurate or useful. poor analogies can have negative consequences, such as, leading to misunderstanding or misconceptions @xcite . this effect can be particularly concerning when such analogies are used in educational contexts, where clarity and accuracy are crucial. thus, evaluating the quality of generated analogies is important to identify good analogies. although a human evaluation of all generated analogies would be ideal, it is impossible to scale up. thus, there is a need for automatic evaluation metrics. moreover, there is a need to develop evaluation metrics for this new type of generated text to measure the progress of analogy generation methods. while several automatic evaluation metrics have been developed to evaluate generated text @xcite , they are not directly applicable to evaluate analogies. limited work has been done on automatically evaluating generated analogies using reference-based metrics (e.g., bleurt @xcite ) and reference-free metrics (e.g., novelty estimation based on similarity to a reference corpus of analogies) @xcite . such metrics have mostly been found to be inadequate. moreover, it is unclear as to what precisely makes a good generated analogy since its goodness depends on multiple factors (e.g., accuracy, strength of analogical connections). to address these challenges, we propose a new shared task for developing evaluation metrics that measure the quality of generated analogies. specifically, we identify seven major criteria for evaluating their quality based on existing literature and our pre-pilot experiments, namely, target concept comprehensiveness, accessibility, source and target concept accuracy, mapping soundness, coherence, and repetition. based on these evaluation criteria, we will create a dataset of manually rated analogies that are generated by models like gpt-4 in domains like science. this dataset will be used to assess the performance of automatic evaluation metrics submitted to our task. since llms have recently shown great promise in evaluating generated text @xcite , we will provide a baseline method that prompts gpt-4 for evaluation in a reference-free setting. we've found this method to be reasonably accurate based on pre-pilot experiments. but, we encourage participants to develop metrics using smaller language models and other types of models too (e.g., fact verification models for accuracy). similar to shared tasks on evaluation metrics for other nlg tasks (e.g., machine translation @xcite )), we expect our proposed task to accelerate research in both evaluation metric and text generation methods, particularly in the context of long-form analogies. more broadly, the insights from the task would also be useful for evaluating other kinds of generated long and creative text (e.g., stories). with the advent of llms, generation of various kinds of text has become feasible and useful for many practical applications. therefore, we believe that this is a timely novel shared task. ## task description given a generated analogy to explain a target concept, the overall task is to rate its quality based on defined criteria. a leader board competition would be set up to evaluate the submissions on our task and dataset. in this section, we describe the criteria we plan to use for evaluation of analogies, our datasets of human ratings and evaluation metrics to quantitatively evaluate the automatic ratings submitted to the task, and our proposed schedule. ## pre-pilot study to understand the task feasibility and guide the task design, we conducted a pre-pilot study. below, we describe the initial evaluation criteria and manual rating datasets used in this study, the results of prompting gpt-4 for automatic evaluation, and qualitative discussions to refine these criteria and finalize the ones reported in section 2.1. ## related work prior work has studied the modeling and generation of various forms of analogies @xcite , such as, analogies between structured representations of concepts @xcite , relational and proportional analogies (e.g., king:queen::man:woman) @xcite @xcite , analogies relating longer text, such as, two sentences or stories @xcite @xcite , and more recently, long-form analogies that explain the relation between concepts using natural language @xcite @xcite @xcite . we aim to evaluate long-form analogies that are typically a few paragraphs long. human evaluation of generated text, although ideal, is highly resource extensive. accordingly, several automatic metrics have been developed for evaluating generated text @xcite , and shared tasks have been established to drive such efforts @xcite . we build upon recent work on holistic evaluation of other types of figurative and creative text @xcite , because it enables a fine-grained evaluation. however, for automatic evaluation of generated long-form analogies, there has been very limited work @xcite @xcite . we compile and refine seven major evaluation criteria based on these and prior work on analogical modeling and reasoning @xcite , aim to extend their datasets both in the number of samples and ratings based on our criteria, and call for development of suitable automatic evaluation metrics. ## conclusion we propose a new shared task for development of automatic metrics to evaluate generated long-form analogies, which describe the analogical relation between concepts in natural language, on seven comprehensive criteria. the submissions would be evaluated based on their agreement with human ratings on our datasets. with this shared task, we hope to accelerate the progress in evaluation metrics and generation methods for long-form analogies.