Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation (2022)

Abstract: Research on Automatic Story Generation (ASG) relies heavily on human and automatic evaluation. However, there is no consensus on which human evaluation criteria to use, and no analysis of how well automatic criteria correlate with them. In this paper, we propose to re-evaluate ASG evaluation. We introduce a set of 6 orthogonal and comprehensive human criteria, carefully motivated by the social sciences literature. We also present HANNA, an annotated dataset of 1,056 stories produced by 10 different ASG systems. HANNA allows us to quantitatively evaluate the correlations of 72 automatic metrics with human criteria. Our analysis highlights the weaknesses of current metrics for ASG and allows us to formulate practical recommendations for ASG evaluation.

## introduction storytelling is at the heart of human societies: skillful storytelling allows a narrator to connect more authentically with their audience and listeners, and to understand the essence of complex concepts better @xcite . numerous applications could benefit from strong automatic story generation systems, including gaming @xcite , and education @xcite . several approaches have been explored to generate stories automatically or with minimum editing efforts @xcite . automatic story generation (asg) takes as input a short sentence (a prompt) and aims at generating a narrative from it @xcite . advances in neural language models @xcite @xcite have allowed substantial progress in asg. to further improve the quality of generated stories, it is indispensable to systematically evaluate asg * previously from laboratoire des signaux et systèmes (l2s), centralesupélec, cnrs, université paris-saclay. models. however, there is little work that specifically studies asg evaluation. most research works rely on human criteria such as coherence @xcite @xcite , relevance @xcite ), overall quality @xcite , narrative flow @xcite , and creativity @xcite . however, taken individually, these criteria fail to encompass all aspects of the task, and there is no consensus on a set of criteria that would cover those aspects in a complete and non-redundant fashion. due to the high cost of human annotation, system quality is also often evaluated using automatic metrics. however, it is not clear how these metrics correlate with human judgment in asg, and thus how suitable they are at all for the evaluation of asg. contributions. in this work, we revisit both human and automatic evaluation of asg. we believe that this meta-evaluation is a missing piece in the asg literature and a crucial step to strengthening the foundations of asg. formally, our contributions to the asg field are: 1. a comprehensive set of non-redundant human criteria for asg evaluation. motivated by the social sciences literature @xcite @xcite , we introduce six human criteria: relevance, coherence, empathy, surprise, engagement and complexity. ## hanna analysis in this section, we analyse the scores of hanna in detail. tab. 4 shows that human stories achieve significantly higher scores than generated stories. following @xcite , who advise to remove outliers, we compute correlations with human stories removed foot_5 . ## conclusions our analysis yields the following conclusions: