Towards A “Novel” Benchmark: Evaluating Literary Fiction with Large Language Models (2025)

Abstract: Current exploration on creative generation focuses mainly on short stories, poetry, and scripts. With the expansion of Large Language Models (LLMs) context windows, “novel” avenues emerge. This study aims to extend the boundaries of Natural Language Generation (NLG) evaluation by exploring LLMs’ capabilities in more challenging long-form fiction. We propose a new multi-level evaluation framework that incorporates ten metrics across the Macro, Meso, and Micro levels. An annotated fiction dataset, sourced from human authors, LLMs, and human-AI collaborations in both English and Chinese is then constructed. Human evaluation reveals notable disparities between LLM-generated and human-authored fictions, particularly the “high-starting, low-ending” pattern in LLM outputs. We further probe ten high-performing LLMs through different prompt templates, achieving moderate correlations by strategically utilizing diverse LLMs tailored to different levels, as an initial step towards better automatic fiction evaluation. Finally, we offer a fine-grained analysis of LLMs capabilities through six issues, providing promising insights for future advancements.

## introduction fiction is widely recognized as a cornerstone of literary expression @xcite , which plays a crucial role in fostering empathy @xcite , moral reflection @xcite , and cultural engagement in an increasingly globalized world @xcite . recent advances has demonstrated the potential of large language models (llms) in various natural language generation (nlg) tasks, ranging from traditional tasks like automatic summarization @xcite to more creative domains such as storytelling @xcite ), yet their potential within the long-form literary fiction remains largely unexplored. with the recent orders-of-magnitude growth in llms context window sizes, enabling them to process inputs over 100k tokens @xcite and generate outputs of up to 10k tokens @xcite , exploring their potential in literary fiction has become more attainable. current efforts have incorporated long-form fiction texts to facilitate tasks in quotation attribution @xcite , and evaluate llms capabilities in comprehension and reasoning @xcite ), yet none have specifically targeted the fiction evaluation task. this paper aims to fill this gap by exploring automatic fiction evaluation using longcontext llms with context windows of at least 128k, thereby providing a more comprehensive understanding of llms' evaluation abilities. the overall pipeline of our work is illustrated in figure 1 . specifically, we constructed a diverse fiction dataset in both english and chinese, encompassing award-winning and ordinary works by human authors, generated fictions from two llms under different prompts, and human-ai collaborative works @xcite , spanning seven mainstream genres: fantasy, sci-fi, mystery, martial arts, romance, coming-of-age, and realism. our focus is on entertainment-oriented short fiction (3k-20k foot_2 ), targeted at adult audiences. given that literary fiction poses increased challenges for nlg evaluation @xcite , with complex plot arcs, diverse character ensembles, sophisticated narrative techniques, profound thematic depth, and high demands for originality and linguistic skill, across extended contexts and varied genres, we designed a robust fiction evaluation framework featuring 10 metrics across 3 levels, with specific exceptions tailored for certain genres. we affirm its validity both theoretically through domain experts and empirically through consistent human evaluation, achieving a krippendorff's α of 0.74. building upon the raw dataset and framework for literary fiction, rigorous human assessments and automated evaluation involving 5 openly available and 5 closed-source llms were systematically conducted, along with different prompt strategies. we conducted an in-depth analysis of llms capabilities, centering on three primary issues. based on the analysis, we found that llms produce strong writing in the first 40%-60% of the text, but decline as the fiction progresses. additionally, we highlight their potential in evaluation tasks with a moderate correlation, their inherent biases, and a strong correlation between their writing and evaluation capabilities within the same dimensions. to summarize, our contributions are: ## fiction evaluation framework the intricate nature of literary fiction can cause ambiguities in setting evaluation criteria. to address this, we developed a comprehensive evaluation framework through expert consultation and iterative refinement. it is a dual-part framework that assesses fiction utilizing 10 universal metrics encompassing cognitive and emotional factors, while also accommodating genre-specific exceptions, such as intentional violations of physics in fantasy genre 4 . each criterion is rated on a 5-point likert scale, with higher scores indicating better quality. ## raw fiction dataset we constructed a bilingual fiction dataset from three sources: existing literary works, llm generation, and human-ai collaboration, referred to as authentic fictions, synthetic fictions, and collaborative fictions, respectively. the context length is restricted to 3k-12k words for english fictions and 6k-20k characters for chinese foot_4 . ## human annotation the assessment of literary fiction differs from goldstandard evaluations as it cannot rely on references. therefore, both human annotation and llm's automatic evaluation are reference-free. ## experiments due to the limited training data, our goal in this section is to investigate the llms' current capabilities in fiction evaluation through prompting rather than developing an advanced system through training. long-context llms. we evaluated ten highperforming long-context llms, including 5 proprietary models, namely gpt-4-turbo, gpt-4o, o1-preview, claude-3.5-sonnet-v2, and gemini-2.0-flash, along with 5 open-source ## results and analysis this section aims at analyzing llms' capabilities in fiction evaluation through three questions. ## conclusion this work aims to enhance the understanding of llms' capabilities by incorporating challenging literary fiction tasks. we developed a multi-level evaluation framework and constructed the first fiction dataset through llm-assisted writing with quality annotations in both chinese and english. through human evaluation, we identified key distinctions between human and llm-generated capabilities in fiction creation. we also evaluated ten llms and, based on our analysis, proposed a promising evaluation strategy by applying various models to different levels, establishing a benchmark for future study. we believe that our efforts lays the groundwork for a more systematic exploration of literary fiction and hope that the framework, dataset and proposed perspectives will attract broader academic interest to advance literary fiction evaluation.