S tory ER : Automatic Story Evaluation via Ranking, Rating and Reasoning (2022)

Abstract: Existing automatic story evaluation methods place a premium on story lexical level coherence, deviating from human preference.We go beyond this limitation by considering a novel Story Evaluation method that mimics human preference when judging a story, namely StoryER, which consists of three sub-tasks: Ranking, Rating and Reasoning.Given either a machine-generated or a human-written story, StoryER requires the machine to output 1) a preference score that corresponds to human preference, 2) specific ratings and their corresponding confidences and 3) comments for various aspects (e.g., opening, character-shaping).To support these tasks, we introduce a well-annotated dataset comprising (i) 100k ranked story pairs; and (ii) a set of 46k ratings and comments on various aspects of the story.We finetune Longformer-Encoder-Decoder (LED) on the collected dataset, with the encoder responsible for preference score and aspect prediction and the decoder for comment generation.Our comprehensive experiments result a competitive benchmark for each task, showing the high correlation to human preference.In addition, we have witnessed the joint learning of the preference scores, the aspect ratings, and the comments brings gain each single task.Our dataset and benchmarks are publicly available to advance the research of story evaluation tasks.

## introduction even for humans, evaluating story quality is a challenging task. although many literature criteria have been proposed, the most straightforward way is to count how many readers like the story which is referred as to human preference. bearing it in mind, story writing community usually uses upvote count 1 dataset and pre-trained model demo are available at anonymous website http://storytelling-lab.com/ eval and https://github.com/sairin1202/ storyer figure 1 : the existing story evaluation method (union) outputs a score for estimating the coherence of the stories, while human-written stories rarely suffer from this problem. our model (ours) which is trained by comparing two stories (ranking), evaluates the story based on human preference (i.e., upvote counts), produces scores for various aspects (rating), and leaves comments (reasoning). our model is applicable to both machine-generated and human-written stories. as a story quality criterion. as shown in fig. 1 , more readers like the left story (upvote @xmath0.8k) rather than the right one (upvote @xmath1. existing methods which use referenced metrics (e.g., bleu @xcite , meteor @xcite , rouge @xcite ) and unreferenced metrics (e.g., union @xcite , manplts @xcite ), deviate from human preference (fig. 1 ). on the contrary, we aim to explicitly evaluate a story, introducing a human preference-liked system consisting of three subtasks: ranking, rating and reasoning. we build a model upon longformer-encoder-decoder (led) @xcite , where the encoder predicts the preference score (ranking), aspect ratings and confidences (rating) while the decoder generates the comments (reasoning). inspired by widely-used pairwise comparison in story evaluation, we train our model with the ranking objectives. in this way, the score margin between figure 2 : the writing prompt dataset with metadata (left) contains prompt, story, upvotes, and comments from readers. our dataset collection pipeline (right) shows the template for data collection. we ask the workers to select 3-5 aspects, score each aspect 1-5 from poor to good and leave the comments that shows the reason for the score they rated. in aid of the proposed task, we present a wellannotated crowd-sourcing dataset, consisting of two parts. (i) one is built from 63,929 stories and their corresponding upvote counts provided in writ-ingprompt dataset (wp) @xcite (figure 2 (left)) by pairing one highly-upvoted story (@xmath2. as a result, we obtain 100k pairs of stories, namely 100k story ranking data, used to train and evaluate the preference score prediction. (ii) the other part is made up of 45,948 aspect comments and their respective rating scores (1-5) by amazon mechanical turk (amt) and augmented data (section 3.2), namely 46k aspect rating and reasoning data, used for model explanation. our contributions are three-fold: ## related work overlap-based metrics such as bleu @xcite ) and rouge @xcite calculate lexical matches (i.e., n-gram matching) and reward the words that resemble the reference in their surface form, even if they do not accurately capture meaning, and penalize other paraphrases. recent research @xcite indicates that these metrics do not reflect human preferences, particularly for open-ended text generation tasks. neural-based metrics are motivated by the success of transformers as multitask learners @xcite , and adapt them for the task of neural language evaluation. when compared to overlapbased metrics, bertscore (zhang et al., 2019), moverscore (zhao et al., 2019), bleurt (sellam et al., 2020) report stronger correlations with human judgment. for specific use, in open dialogue generation, adem (lowe et al., 2017) captures semantic similarity beyond word overlap statistics, and exploits both the context and the reference response to calculate its score for the model response. ruber (tao et al guish by a bert-based model @xcite . the coherence score they produce can be expressed as the probability of the story being identified as human-written story. in this paper, we require our model to follow human preference, not only the coherence, which we believe is a more general way of story evaluation. ## dataset our dataset comprises of two parts: 100k story ranking, and 46k aspect rating and reasoning. foot_0 ## conclusion in this paper, we investigate a novel task of preference-aware story evaluation, storyer, which produce a score with explanation through various aspects and comments, bringing gains on both machine-generated and human-written stories evaluation. to support the task, we present a new dataset consisting of paired ranked stories and more explicit annotation (i.e., rating and reasons) for predefined aspects. our comprehensive ablation studies and intensive analysis show the effectiveness of using aspect rating and reasoning on preference score prediction. with the development of story generation, we believe that preference-aware story evaluation will be the mainstream research when machine-generated stories do not suffer from serious coherence problems. further studies on our dataset can also be conducted to reveal the point that influence the readers to upvote the stories.