An Analysis of Scoring Methods for Reranking in Large Language Model Story Generation (2025)

Abstract: Outline-conditioned story generation using Large Language Models (LLMs) offers a promising approach for automating narrative creation. Some outline-conditioned story generation methods use automatic scoring during the generation process in order to improve the story quality. However, current research has shown that automatic scoring is not ideal for assessing story quality. This paper evaluates three proposed automatic story-scoring methods to improve the reranking of outputs during the generation process. These scoring methods leverage different prompting strategies and fine-tuning techniques to enhance the accuracy and relevance of the assessments. By experimenting with these approaches within a beam search framework, we aim to identify the most effective methods for optimizing story-generation outcomes. While we have found no significant overall difference between these methods in terms of their agreement with human ratings during story generation, the overall story ratings by human evaluators are average. These findings motivate the need for improved automatic scoring techniques and datasets while also indicating that simpler, more easily implementable scoring methods for reranking perform comparably to more complex approaches.

## introduction recent advancements in ai, particularly in large language models (llms), have highlighted how these tools can assist in writing. in creative writing in particular, there has been exploration into how these tools can be used to write fiction stories such as in @xcite and @xcite . this issue is particularly challenging because fiction writing demands creativity, raising questions about how to assess the quality of outputs from llms in such a creative domain and how to encourage the generation of higher-quality, more engaging stories with these tools. in the task of outline-conditioned story generation, as first proposed in @xcite , the input is an outline consisting of key plot-points, characters and events, which are then used to generate a flowing narrative. each point primes the generation of a passage of text from its corresponding plot-point. subsequent passages are generated from the next plot-point and the context of the previously generated passages. in our paper, we use outlines consisting of character information, events, and settings in a shallow (two-level) hierarchical structure. as stated by @xcite , the difficulty of this task lies in the fact that a model must fluently connect the points given in the text, while still following the outline. papers such as @xcite and @xcite employ automatic scoring techniques with reranking during the generation process in order to ensure that these stories remain coherent and relevant to the given outline. most recent research has suggested that automatic scoring, including scoring by llms, is ineffective at assessing story quality. instead, most story generation papers use human raters as their primary scoring method @xcite @xcite . however, there are cases where automatic scoring is a necessary component in the story generation system. for example, @xcite use automatic scoring in a beam search to select the ideal sequence of passages in a story. this is referred to as reranking, the process of scoring multiple candidate outputs to select the one that best meets specific criteria @xcite . in the context of story generation, reranking could be done automatically or with humans in the loop. using human-in-the-loop in this context could be tedious and time consuming, however, making the task harder for the humans involved. in this paper, we investigate how automatic scoring can be used in the story generation process to improve the story outputs. we look at which scor-ing techniques are the best for this specific task. specifically, we look at three different techniques of automatic scoring using llms: we use each of these three methods in a reranking framework to generate stories. we then get human raters to evaluate which of these methods, when used for reranking, generates the best stories. we found that there was no significant difference between these methods when used in reranking for story generation. this means that simple methods like prompt-based scoring perform just as well as other methods which may be more time consuming to implement. it unfortunately also means that fine-tuning a model does not seem to improve the story quality when used for scoring in reranking. additionally, it further motivates the need for better automatic scoring techniques and datasets. our contributions are as follows: ## related work several previous papers have explored outlineconditioned story generation, where outlines are used as input to generate a story @xcite @xcite . the doc framework @xcite uses verbose outlines and a reranking system in their generation process to choose the best sequence of passages. this paper was inspired by an earlier paper called re3 @xcite which also used reranking to choose the best story continuations. they found this reranking component to be critical for plot coherence and premise relevance. both @xcite and @xcite use a trained model for this reranking. @xcite introduces a system with the modularity of the original doc framework, but which is able to be integrated with more modern llms. it also uses log-likelihood-based scoring rather than a trained model for the reranking component. however, the scoring method used for reranking in their system is never fully evaluated. additionally much previous research has shown that current automatic scoring techniques are not up-to-par with human scoring @xcite @xcite @xcite . in @xcite the authors create the hanna dataset consisting of human scores of different stories on 5 different criteria: relevance, coherence, empathy, surprise, and engagement. for each of these criteria, they ask humans to give a 5-point likert score to stories generated by 10 different story generation systems using prompts from the writingprompts dataset @xcite . they then compared different automatic scoring techniques to the human scores using kendall correlations. they found that the correlations between the automatic scores and human scores were weak and called for stronger automatic scoring methods. they found that larger pre-trained models like gpt-2 performed the best, however. this prompted them to write a follow up paper @xcite where they further compared human scoring to automatic scoring using different llms for prompting. they found that while llms are consistent and have slightly higher ratings, they have correlations with human scores that are fairly similar to those of other automatic scoring methods. they therefore conclude that llms are currently the best proxy for human scoring of story generation. they also recommend future work on the use of fine-tuning models for this task. @xcite address the overall low quality of automatic scoring methods by creating a benchmark called openmeva for them. in contrast to @xcite , they use a single 5-point overall quality score rather than individual scores for several criteria. this overall score should be low for stories that have repetitive plots, unrelated events and conflicting logic, or globally chaotic scenes. they also found that state-of-the art methods corre- prompt a story passage: king aldric, determined... event: king aldric issues a decree. did this event happen in the story passage? yes or no. prompt b story passage: king aldric, determined... event: king aldric issues a decree. rate the story on a scale from 1 to 5 on relevance (how closely the story passage follows the event). 1-the story has no relationship with the event at all. 2-the story only has a weak relationship with the event. 3-the story roughly matches the event. 4-the story matches the event, except for one or two small aspects. 5-the story matches the event exactly. do not include any numbers other than your rating in your answer. rating (1-5): prompt c prompt: king aldric issues a decree. target story: king aldric, determined... rate the story on a scale from 1 to 5 on relevance, coherence, empathy, surprise, engagement, and complexity. ratings: late poorly with the human methods on this scale in their dataset. yang and jin (2024) discusses different types of automatic scoring. in general, there are four different types that use llms. the first of these is embedding-based methods which use embeddings and matching algorithms to assign a score. these have many limitations. the next is probabilitybased methods. these methods use the generation probability from llms in computing their score. there are also generative-based methods, which simply prompt llms for a score. finally, there are trained methods that fine-tune an llm to assign a score. @xcite found that generative-based methods are more effective than probability-based methods that use log-likelihood because their smooth distributions allow for better differentiation than the narrow range and peak structure of the probability-based methods. ## methodology in this section, we describe our approach for evaluating automatic story-scoring methods for reranking in outline-conditioned story generation. we first outline the process we use for generating stories and how reranking is used within that. we then describe the criteria we use for the scoring within the reranking. finally, we describe the three scoring methods to evaluate: log-likelihood-based scoring, prompt-based scoring and fine-tuning. ## conclusion in this paper, we compared three different automatic scoring methods when used in a reranking framework for story generation. our experiments were unable to prove a significant difference among these methods when their outputs are assessed by human raters. future work could explore alternative reranking techniques, such as reinforcementlearning-based methods. additionally, there is a need for more annotated datasets of stories. we hope that this research contributes to more effective and controllable story generation systems in nlp.