C o K e: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization (2025)

Abstract: Evaluating creative text such as human-written stories using language models has always been a challenging task – owing to the subjectivity of multi-annotator ratings. To mimic the thinking process of humans, chain of thought (Wei et al., 2023) (CoT) generates free-text explanations that help guide a model’s predictions and Self-Consistency (Wang et al., 2022) (SC) marginalizes predictions over multiple generated explanations. In this study, we discover that the widely-used self-consistency reasoning methods cause suboptimal results due to an objective mismatch between generating ‘fluent-looking’ explanations vs. actually leading to a good rating prediction for an aspect of a story. To overcome this challenge, we propose Chain-of-Keywords (CoKe), which generates a sequence of keywords before generating a free-text rationale, that guide the rating prediction of our evaluation language model. Then, we generate a diverse set of such keywords, and aggregate the scores corresponding to these generations. On the StoryER dataset, CoKe based on our small fine-tuned evaluation models not only reach human-level performance and significantly outperform GPT-4 with a 2x boost in correlation with human annotators, but also requires drastically less # of parameters.

## introduction evaluating stories is an important and timeconsuming job for professionals in the entertainment industry. for example, novel competition judges, book editors, or movie producers might need to select the best story from thousands of submissions according to their tastes and the understanding of the market. as llms get better at judging story quality, automatically evaluating human-written stories becomes practical. however, there are still several challenges to overcome. first, judgements from * work is mostly done at amazon off-the-shelf llms might be biased towards the preference of particular annotators during the alignment stage, which could be very different from the tastes of the desired population. second, humans are extremely subjective in judging creative writing like stories, which is often demonstrated in their creativity: some readers or professional reviewers would think character shaping is the most critical component for evaluating a story, whereas others might like or dislike the characters along with some other components, like the scene description mentioned in the story. this lack of consensus in likes and dislikes, along with differences across aspects (e.g. character shaping, ending, etc) in the story makes evaluating human-written stories an extremely difficult task. the desired human evaluation here would entail that we collect diverse opinions from different readers/reviewers to estimate a average opinion of the story from a desired population, but this is extremely tedious and expensive. this high cost has motivated automatic measures for evaluating the stories written by humans. in this study, we aim at building an automatic story evaluation system that can 1) provide fine-grained evaluation for a human-written story in predefined and/or customized aspects, 2) provide a set of rationales that model diverse opinions of multiple humans and help us better predict the average score for different aspects of the story, and 3) be easily customized toward the opinions of the desired population (i.e., fine-tunable using the collected human judgements and explanation). the reason-then-predict approaches like chain of thought (cot) @xcite not only improve the interpretability of the said predictions by generating rationales but also improve downstream performance in predictions @xcite . using these approaches, large language models (llms) can score arbitrary aspects of a story without any additional training. how-figure 1 : coke provides a low-cost, audience-oriented (customizable), and keyword-guided approach to evaluating stories by generating and scoring diverse keyword sequences that explain a fine-grained aspect-story pair. fine-tuning a small language model (lm) to directly predict the population average of annotators is a cheap viable alternative, but does not provide rationales while also being inflexible w.r.t. how granular we want the story to be evaluated (e.g., character shaping of the vampire, ending w.r.t. a certain character, etc). another option is finetuning a small lm to generate free-text rationals for cots and use the self-consistency @xcite approaches to marginalize over multiple sampled cots. however, we discover that the free-text rationals tend to reduce the diversity of cots' rating predictions and deviate the average prediction rating from the population average. in order to mitigate this shortcoming we propose chain-of-keywords, coke, which consists of two simple yet effective modifications to regular cot approaches. first, instead of just generating a freetext rationale, we generate a chain of keywords before generating a rationale that can describe salient concepts in and outside the story. our intuition is that keywords help prevent the learning and generation of annotator artifacts (like sentiment-laden words and other personal descriptors like 'i think, i feel', etc), which assists with the objective misalignment we see in cot approaches. like sc, instead of generating one rationale, it samples multiple keyword rationales, which simulates annotator diversity and helps better estimate the population average. therefore, coke uses the generated key-words to score a story, and the corresponding generated rationale for interpreting the story, as shown in figure 1 . on storyer @xcite , a fine-grained story evaluation benchmark @xcite , we show that coke can better estimate population averages as compared to llm baselines using gpt-3.5 (text-davinci-003) and gpt-4 (gpt-4-0613) @xcite , as well as open-source llms like llama-2-7b-chat @xcite and mistral-7b-instruct @xcite . we also show that coke consistently outperforms self-consistency and approaches based on supervised fine-tuning, including those where the rationale generated is specifically aligned to that of annotator-written explanations using reinforcement learning (rl), as well as improved correlations on human evaluations as compared to baselines. furthermore, we also show that coke can work effectively even when built on smaller lms as its backbone (approx. 58x fewer # of parameters than gpt-3.5), while surpassing gpt-3.5 by 2.18x improvement in correlation metrics with the target annotator population. to the best of our knowledge, coke is a first rationalizethen-predict approach for fine-grained story evaluation surpassing llm performance for this task, and reaches human-level performance in the storyer dataset @xcite . ## 2 problem formulation we begin by describing our task setup and why the task is challenging. we assume that there exists a dataset that consists of the story-aspect ratings and the explanations for the ratings. one story-aspect pair could be annotated by multiple annotators from our target audience. any automated story evaluation system should provide a single score for an aspect-story pair that is close to the average ratings from annotators, without modeling the individual annotator @xcite . ## chain-of-keywords (coke) there are three kinds of words in a free-text explanation: sentiment words, keywords referring to the concepts in the story, and the functional words (e.g., stop words). we view the sentiment and functional words as an artifact for story evaluation because they only provide the information that the rating has already provided and could induce a bias in cot's rating prediction. this is because the probability of generating a positive sentiment word might be affected more by the nearby function words than by the quality of the input story and thus, the positive sentiment in the explanation would heavily bias the cot to predict a high score. for example, we observe that most positive rationales in the storyer dataset are much more likely to contain "i" while the most negative rationales have much more "it". in the positive rationales, i is the 8th likely words (1.8%) while it is the 14th likely words (0.6%). in the negative rationales, i is the 16th likely words (0.9%) while it is the 7th likely words (1.5%). if we observe some rationales starting with "i like" or "i love" in the training vampire stories, "i" could become the most likely first word in the generated rationale for a bad testing vampire story, which bias the cot to output like/love and a high rating at the end. we leverage these intuitions to build coke in the following manner (shown in figure 3 ). first, a language model is fine-tuned to generate keywords, along with a free-text explanation conditioned on those keywords, that inspects the story w.r.t the aspect. these keywords are in the form of phrases (from the story itself) that specifically do not contain artifacts. from this language model's decoder, we sample multiple keyword sequences, intended to simulate diverse annotator opinions. a trained scorer model is then used to produce score predictions from aspect-story-keyword triples, and scores for all individual candidate keyword sequences are averaged to produce the final score. more concretely, let d t r and d t e be the training and test datasets respectively. they are composed of the story-aspect-explanation-rating tuple (s i , a i , e i , y i ). for example, in storyer, s i is a human-written story from writingprompts @xcite , a i is one of the predefined aspects, y i is the rating from an annotator, and e i is the text justification for y i . if two annotators label the same story and aspect, the s i and a i would be the same for the two tuples. coke consists of two components: a rationalizer model, θ r , and a scorer model, θ s . the rationalizer is a seq2seq language model that is fine-tuned to generate rationales, given an aspectstory pair as an input: kj ∼ g θ r (s j , a j ), while the scorer is a regression language model that is fine-tuned to predict a floating point score, given aspect-story-rationale triplets as an input: y @xmath0, a j , kj ). we detail the training and inference process of coke below and further conduct ablations on different components of coke to justify our keyword extraction step and other design decisions in section 4.5. ## experiments in this section, we evaluate coke, llms with sophisticated inference strategies, supervised finetuning, along with coke ablations. ## applications of keywords in coke the keyword rationales generated by coke not only significantly improve the performance, but also being faithful because they are used as input for the scorer, similar to other faithful rationalization approaches like @xcite . moreover, the keywords provide more interpretable evaluation and more fine-grained evaluation based on user-provided keywords. ## related work due to the importance of automatic story evaluation, several types of approaches have been proposed. rouge @xcite , bertscore @xcite , bartscore @xcite , and ctc @xcite compare the similarity between the generated text and the reference story. although being effective in many other text generation tasks, higher similarity to the reference story is not necessarily a better story. another type of evaluation method injects some noise into the humanwritten stories to create the low-quality stories and train a classifier to separate them. examples include union @xcite , @xcite , unieval @xcite , and deltascore @xcite . although these methods are good at discovering the incoherency from smaller language models, they cannot be used to evaluate a human-written story given a fine-grained aspect. recently, researchers propose many general-purpose evaluation methods based on llms. for example, gptscore @xcite and g-eval @xcite directly prompt the llm and several open-source models distill llms to reduce the evaluation cost @xcite . @xcite summarize these llm-as-judge studies well. in these papers, gpt-4 usually demonstrates the best correlation with human judgments. methodologically, our method is related to the llm rationale generation and minimum bayes risk (mbr) decoding @xcite . recent work in generating fluent free-text rationales has made use of two types of approaches -finetuning a small language model with gold human written rationales @xcite @xcite or zero-shot prompting llms to generate free-text rationales @xcite @xcite @xcite . some approaches also leverage few-shot training approaches with a handful of gold rationales @xcite . our method could also be viewed as a special case of mbr, which generally refers to the methods that merge multiple generated candidate answers to improve the output quality. other special cases of mrb include self-consistency prompting @xcite , crowd sampling @xcite , complex cot @xcite , and output ensembling @xcite . ## conclusion in this study, we look at a simple, yet efficient way to evaluate story-aspect pairs. we propose coke that samples multiple generated keyword sequences before explanations, and using the generated keywords to score an aspect-story pair. we posit that sampling helps us get diverse annotator ratings, and using keywords helps alleviate the objective mismatch between generating coherent explanations vs. usable explanations for downstream scoring. we show that that keywords not only improve the rating prediction performances, but also make the evaluation more interpretable and controllable.