D elta S core: Fine-Grained Story Evaluation with Perturbations (2023)

Abstract: Numerous evaluation metrics have been developed for natural language generation tasks, but their effectiveness in evaluating stories is limited as they are not specifically tailored to assess intricate aspects of storytelling, such as fluency and interestingness. In this paper, we introduce DeltaScore, a novel methodology that uses perturbation techniques for the evaluation of nuanced story aspects. We posit that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations (e.g., the introduction of typos). Given this, we measure the quality of an aspect by calculating the likelihood difference between pre- and post-perturbation states using pre-trained language models. We compare DeltaScore with existing metrics on storytelling datasets from two domains in five fine-grained story aspects: fluency, coherence, relatedness, logicality, and interestingness. DeltaScore demonstrates strong performance, revealing a surprising finding that one specific perturbation proves highly effective in capturing multiple aspects. Source code is available on our GitHub repository.

## introduction the emergence of large pre-trained language models (plms) @xcite has empowered story generation models to generate plausible narratives @xcite @xcite . the most advanced models have achieved the ability to produce stories which are not easily distinguishable from human-authored ones @xcite @xcite . however, the development of automated evaluation metrics in this domain has not progressed at the same pace @xcite . human evaluation, though the supermarket has various kinds of goods. the has various kinds of . it is a nice day. considered the gold standard, is hindered by its time-consuming, costly, and non-reproducible nature @xcite . consequently, there is a demand for better automatic methods that can evaluate the quality of stories. the prevailing evaluation metrics for story assessment have primarily been adapted from other natural language generation (nlg) tasks, such as bleu @xcite for machine translation, or rouge @xcite for summarization. fortunately, recent progress has given rise to the emergence of new metrics explicitly tailored for story evaluation, with a focus on quantifying story coherence @xcite or capturing human preferences @xcite . other works have directly utilized the likelihood of a story under a plm @xcite or its conditional likelihood based on human references or other contextual factors, such as story title @xcite . nonetheless, these approaches often yield a singular score that provides an estimate of the overall quality. however, @xcite argue that the quality of a story is comprised of various fine-grained aspects, such as fluency and adherence to commonsense, suggesting that an overall quality score has limited utility for comprehensive story evaluation. in this paper, we present deltascore, a method that evaluates story quality by measuring the likelihood difference using a plm between an original story and its perturbed version. the idea is that higher quality stories will exhibit more significant effects from the perturbation compared to lower quality ones. to provide fine-grained assessment of story quality, we experiment with perturbations that target specific aspects. figure 1 presents two examples to demonstrate the intuition of our approach: 1) when we introduce random typos to modify the two stories shown in figure 1a , we observe that the story with higher fluency is affected more by the perturbation; 2) when we modify the two stories in figure 1b by removing relevant words, we observe that the perturbation affects the story that has a closer association with the title to a greater extent. our empirical analysis demonstrates the superior performance of deltascore compared to existing metrics in evaluating intricate story aspects. furthermore, our investigation reveals an interesting discovery: one of our simplest perturbation methods, which simply shuffles all the words in the story, is very effective in capturing multiple aspects. this points to a possible interpretation that the pertubation may be functioning as a normalisation factor to modulate the effects of word frequency and text length when estimating sequence likelihood. ## deltascore we now describe the idea of our approach. given a story condition (e.g., a story title) @xmath0, ..., c n containing n tokens, a model-generated story @xmath1, ..., s m containing m tokens, and a perturbed story s ′ = s ′ 1 , ..., s ′ m ′ containing m ′ tokens, deltascore calculates the likelihood difference under a language model: ## results we evaluate kendall correlation at the story level, which involves comparing the predicted metric score versus the aggregated human rating for each story on a specific aspect. we use this as our primary metric due to the non-linear relationship between automatic and human metrics, as well as the ordinal scale employed in human judgments @xcite . we explore different settings of our approach in § 5.1 and present a comparison of our best approach with existing evaluation metrics in § 5.2. note that we use two different set of judgments, as explained in § 4.1, to avoid tuning and testing on the same test set. ## discussion and conclusion initially, our aim was to investigate various types of perturbations for assessing fine-grained aspects of storytelling. but seeing that performance of each metric does not vary much across different aspects in figure 3 , it suggests these aspects may be somewhat inter-correlated. also, our findings revealed that one of the simplest perturbation methods, namely jumble, is exceptionally effective in measuring most aspects. one hypothesis could be that jumble is functioning as a normalisation factor to modulate word frequency and sentence length effects when estimating sequence quality. this finding aligns with prior study that used sequence probabilities for measuring sentence acceptability @xcite . they found that it is important to normalise the probabilities and introduced various normalization techniques to mitigate the impact of word frequency and sentence length. jumble can be interpreted as an alternative normalisation technique. given this insight, it may also mean that deltascore has broader application beyond the evaluation of story quality. for instance, it could be used to score sentences in machine translation and summarization. in conclusion, we introduce deltascore, a novel approach to assess fine-grained story aspects by comparing the likelihood difference between the original story and a perturbed version using a pretrained language model. surprisingly, we discovered that a small set of perturbation methods excel in measuring the majority of story aspects. furthermore, our findings demonstrate that deltascore shows stronger correlations with human judgments compared to a range of existing metrics across two different story domains.