HAUSER : Towards Holistic and Automatic Evaluation of Simile Generation (2023)

Abstract: Similes play an imperative role in creative writing such as story and dialogue generation. Proper evaluation metrics are like a beacon guiding the research of simile generation (SG). However, it remains under-explored as to what criteria should be considered, how to quantify each criterion into metrics, and whether the metrics are effective for comprehensive, efficient, and reliable SG evaluation. To address the issues, we establish HAUSER, a holistic and automatic evaluation system for the SG task, which consists of five criteria from three perspectives and automatic metrics for each criterion. Through extensive experiments, we verify that our metrics are significantly more correlated with human ratings from each perspective compared with prior automatic metrics. Resources of HAUSER are publicly available at https://github.com/Abbey4799/HAUSER .

## introduction similes play a vital role in human expression, making literal sentences imaginative and graspable. for example, robert burns famously wrote "my luve is like a red, red rose" to metaphorically depict the beloved as being beautiful. in this simile, "luve" (a.k.a. topic) is compared with "red rose" (a.k.a. vehicle) via the implicit property "beautiful" and the event "is". here, topic, vehicle, property, and event are four main simile components @xcite . as a figure of speech, similes have been widely used in literature and conversations @xcite . simile generation (sg) is a crucial task in natural language processing @xcite @xcite , with the aim of polishing literal sentences into similes. in fig. 1 , the literal sentence "he yelps and howls." is polished into a simile by inserting the phrase "like a wolf ", resulting in "he yelps and the commonly used automatic metric bleu deems the second candidate as the most high-quality one among all the generated similes, while our proposed metrics hauser deem the first candidate as the best one regarding its quality, creativity and informativeness, which better correlates with human ratings and also provides more criteria for sg evaluation. automatic evaluation is critical for the sg task since it enables efficient, systematic, and scalable comparisons between models in general @xcite . however, existing studies are inadequate for effective sg evaluation. task-agnostic automatic metrics @xcite @xcite are widely adopted for sg evaluation @xcite , which have several limitations: (1) the simile components should receive more attention than other words during sg evaluation (e.g. "he" and "wolf " in fig. 1 ), while there are no automatic metrics that consider the key components. (2) the sg task is open-ended, allowing for multiple plausible generations for the same input @xcite ) (e.g. the howling man can be compared to "wolf ", "buffalo", or "tiger" in fig. 1 ). hence, the metrics based on word overlap with a few references are inadequate to accurately mea- ## hauser for sg evaluation we establish hauser, a holistic and automatic evaluation system for sg evaluation, containing five criteria from three perspectives, and further design automatic metrics for each criterion (fig. 2 ). ## hauser application we perform a case study to prove that our designed automatic metrics are effective for various methods. here, we apply our metrics to a retrieval method @xcite (denoted as bm25), which utilizes the 20 context words around the insertion position given by groundtruth to retrieve the 5 most similar samples based on the bm25 ranking score from the training set, and adopts the vehicles from these samples to be those of simile candidates. this method ensures the diversity of generated similes. the method introduced in sec. 4.1 is denoted as ours. given the candidates generated by each method, we rerank them using a weighted combination of quality, creativity, and informativeness rankings obtained by hauser, with a ratio of 2:2:1. from tab. 11 in appendix, the candidates generated by various methods can be more correlated with human rankings after being ranked by our metrics, thus proving the generality of our metrics. it is noticed that the insertion position for bm25 is provided by the groundtruth, while the insertion position for ours is predicted by the model, thus proving the effectiveness of our generation method. ## conclusion in this work, we systematically investigate the evaluation of the simile generation (sg) task. we establish a holistic and automatic evaluation system for the sg task, containing five criteria from three perspectives, and propose holistic automatic metrics for each criterion. extensive experiments verify the effectiveness of our metrics. more correlated with humans than prior referencebased metrics (e.g. bleu, rouge, bertscore), our metrics are still reference-based and rely on the quality and scale of referenced data. we have discussed the effect of referenced dataset size in our paper and will design reference-free metrics to further complement our metrics in future work. additionally, since our metrics utilize a million-scale simile knowledge base or large-scale simile sentences as references, the efficiency of our method is slightly lower than the automatic metrics based on a few references. nevertheless, this limitation does not prevent our metrics from performing systematic and scalable comparisons between sg models.