O pen MEVA : A Benchmark for Evaluating Open-ended Story Generation Metrics (2021)

Abstract: Automatic metrics are essential for developing natural language generation (NLG) models, particularly for open-ended language generation tasks such as story generation. However, existing automatic metrics are observed to correlate poorly with human evaluation. The lack of standardized benchmark datasets makes it difficult to fully evaluate the capabilities of a metric and fairly compare different metrics. Therefore, we propose OpenMEVA, a benchmark for evaluating open-ended story generation metrics. OpenMEVA provides a comprehensive test suite to assess the capabilities of metrics, including (a) the correlation with human judgments, (b) the generalization to different model outputs and datasets, (c) the ability to judge story coherence, and (d) the robustness to perturbations. To this end, OpenMEVA includes both manually annotated stories and auto-constructed test examples. We evaluate existing metrics on OpenMEVA and observe that they have poor correlation with human judgments, fail to recognize discourse-level incoherence, and lack inferential knowledge (e.g., causal order between events), the generalization ability and robustness. Our study presents insights for developing NLG models and metrics in further research.

## introduction significant advances have been witnessed in many nlg tasks with pretraining models @xcite . however, existing generation models are still far behind the human-level performance to generate reasonable texts, particularly for open-ended generation tasks such as story generation @xcite . one critical obstacle is the lack of powerful metrics for measuring the quality of generation. ## related work various automatic metrics have been proposed for evaluating language generation. they can be roughly divided into referenced, unreferenced, and hybrid metrics, according to whether relying on human-written references when calculating the metric score. referenced metrics usually measure the similarity between a sample and some references based on word-overlap (e.g., bleu @xcite , rouge @xcite ) or word embedding (e.g., bertscore @xcite , moverscore @xcite ). however, referenced metrics were reported to correlate poorly with human judgments in open-ended generation tasks @xcite due to the one-to-many issue @xcite . to address the issue, un-referenced metrics were proposed to measure the quality of a sample without any reference, such as perplexity, discriminator-based metric @xcite , union @xcite and grade @xcite . besides, hybrid metrics combine referenced and unreferenced metrics (e.g., ruber and its variant @xcite ) or learn from the human-annotated score (e.g., adem @xcite , bleurt @xcite ). recently, there have been many criticisms for existing metrics. @xcite showed the poor generalization of discriminator-based metrics. @xcite demonstrated adem is not robust to simple attacks such as simple word substitution or random word shuffle. however, these criticisms only focus on individual metrics or capabilities. notably, @xcite proposed a framework checklist to evaluate different capabilities of general language understanding models by validating the input-output behavior. the test cases are created from scratch or by perturbing an existing dataset. similar to checklist, open-meva also employs automatically constructing examples for behavioral tests. however, checklist only focuses on single sentences, thereby lacking the ability to test models in understanding long texts with many discourse-level features (e.g., temporal relationship). moreover, the testing methods of checklist are not directly applicable for nlg metrics. specifically, checklist measures the performance of a model by calculating the failure rate between discrete model prediction and automatic labels. such failure rates are ineffective for measuring metrics since most metric scores are continuous. to address the above issues, we propose perturbation techniques and testing methods more applicable for story generation metrics. ## data collection we collect mans and autos based on rocstories (roc for short) @xcite and writingprompts (wp for short) @xcite , which are commonly used for story generation @xcite and evaluation @xcite . roc contains 98,162 five-sentence commonsense stories with about 50 words, while wp consists of 303,358 pairs of prompts and stories, which are usually unconstrained on writing topics. we retain about 250 words (with correct sentence boundary) for stories rating story a overall quality: 1 2 3 4 5 reasons: • local errors • repetitive plots (repeating similar texts) -1 -2 • unrelated events (with unrelated events to the input or within its own context) -1 -2 • conflicting logic (against common sense or with wrong causal or temporal relationship) -1 -2 • global errors • chaotic scenes (difficult to understand as a whole) -2 instruction 1. read the following and seven stories from to . 2. comparing the stories with one another in terms of overall quality . 3. rate each story in 1-5 range (5 is the best). and choose your reasons. story a: he bought himself a ... story b: he wanted to try ... story g: it has been a pretty great ... input: my dad likes watermelon very much. . . . input story a story g one human-written story five generated samples one negative sample sample rating human-written 5 negative 1 ... ... ... ... ✅ annotator #1 annotator #5 ❌ . . . rating story g . . . sample rating human-written 2 negative 4 ... ... ... ... in wp. although we only consider the stories in the two corpora, openmeva is designed to measure the capability of nlg metrics to evaluate general linguistic features such as coherence, which may pertain to other stories. besides, our idea that building datasets by manual annotation or automatic construction can be easily extended to evaluate specific aspects for other types of stories. ## evaluation we evaluated existing metrics on openmeva, and analyzed the strengths and weaknesses with extensive experiments. ## conclusion we present openmeva, a benchmark to comprehensively assess capabilities of metrics for evaluating open-ended story generation. openmeva includes test examples which are created by either annotating machine-generated stories or perturbing human-written stories in terms of each single aspect. we evaluate a number of existing metrics on openmeva and analyze their performance on each capability extensively. experiments demonstrate that existing metrics still correlate weakly with human judgments, fail to recognize discourselevel incoherence, and lack inferential knowledge, generalization and robustness. our study reveals the weaknesses of existing metrics and may inspire new research on designing nlg metrics. the datasets, data augmentation tools, and implemented metrics in this paper can facilitate further research on language generation and evaluation.