The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing (2025)

Abstract: Recent studies comparing AI-generated and human-authored literary texts have produced conflicting results: some suggest AI already surpasses human quality, while others argue it still falls short. We start from the hypothesis that such divergences can be largely explained by genuine differences in how readers interpret and value literature, rather than by an intrinsic quality of the texts evaluated. Using five public datasets (1,471 stories, 101 annotators including critics, students, and lay readers), we (i) extract 17 reference-less textual features (e.g., coherence, emotional variance, average sentence length...); (ii) model individual reader preferences, deriving feature importance vectors that reflect their textual priorities; and (iii) analyze these vectors in a shared “preference space”. Reader vectors cluster into two profiles: _surface-focused readers_ (mainly non-experts), who prioritize readability and textual richness; and _holistic readers_ (mainly experts), who value thematic development, rhetorical variety, and sentiment dynamics. Our results quantitatively explain how measurements of literary quality are a function of how text features align with each reader’s preferences. These findings advocate for reader-sensitive evaluation frameworks in the field of creative text generation.

## introduction large language models (llms) are increasingly able to generate short stories that resemble human writing, leading to a growing number of evaluation studies based on reader judgments. as a common approach in these studies is to aggregate ratings to establish a consensual measure of literary quality for an evaluative group. by design, this process averages out individual differences to identify a shared standard. our work, however, adopts a perspectivist view @xcite , focusing specifically on this variation as a meaningful signal. while many factors can influence literary judgment, including references to other texts, political alignment, or familiarity with genre conventions, this study focuses on intrinsic textual features: measurable properties related to style, structure, and content. these features provide a reasonable starting point for modeling variation in reader preferences, as they can be extracted directly from the text and interpreted consistently across corpora. inspired by multi-criteria decision-making theory @xcite , we conceptually frame each reader's evaluation as an additive utility function over textual features. for each reader, we estimate a weight vector that best predicts their observed rankings, providing an interpretable representation of their literary priorities. this approach provides a quantitative basis for explaining preference variability, allowing us to characterize the paper ai writer human writers assessors ai surpass humans @xcite ) slm average writers average readers ✓ @xcite llm top writers average readers ✓ @xcite llm writing students writing students ≈ @xcite llm average writers @xmath0: comparison of ai-generated text performance against human writers. to guide our investigation into these aspects, this work seeks to answer four research questions: (rq1) do ai and human stories differ in measurable textual features across datasets? (rq2) can reader preferences be modeled from these features? (rq3) do experts and non-experts prioritize different features? and (rq4) do these preference patterns remain consistent across datasets? in addressing these research questions, this paper makes the following contributions: ## related work research at the intersection of artificial intelligence (ai) and literature has explored several key areas. a prominent line of work directly compares the perceived quality of ai-generated and humanauthored texts, often framing the comparison as a competition @xcite @xcite @xcite . these studies reveal a complex picture, with outcomes varying significantly depending on the type of text (e.g., poetry vs. short stories), the expertise of the evaluators (e.g., average readers vs. literary critics), and the specific ai models employed. for instance, some studies indicate that ai-generated texts are sometimes preferred by lay readers @xcite , while others, particularly those involving literary experts, consistently show a preference for human-authored texts @xcite . building on the observation of varied outcomes in quality perception, recent research delves deeper by quantifying specific aspects like creativity. the "creativity index," for instance, assesses linguistic originality by how much a text can be reconstructed from web snippets, revealing that professional human authors exhibit a 66.2 % higher index than llms @xcite . furthermore, the process of aligning llms with human preferences through methods like rlhf has been shown to inadvertently reduce their creativity index by about 30.1% @xcite . this observation implies a trade-off: enhancing safety or coherence might occur at the expense of originality @xcite , a dynamic particularly relevant when considering how different reader profiles, especially experts valuing originality, might assess these out-puts. the current work builds upon a foundation of research in computational literary studies that seeks to identify textual correlates of literary quality. studies have explored how features such as global coherence, local unpredictability, sentiment arcs, and emotional volatility differentiate between canonical literature, bestsellers, and other categories @xcite @xcite . these insights inform the selection of our implemented textual metrics. however, our study does not directly use these metrics to predict literary quality. instead, they are leveraged to model how individual readers differentially weight these properties, reflecting their unique quality model. the importance of perspectivism in literary evaluation has been emphasized in prior work. @xcite propose a "mild perspectivist" approach that aggregates literary judgments within socially or professionally coherent reader classes. they show that each class yields internally consistent quality judgments and argue that such reader classes are the natural unit for modelling literary value. our study extends this perspectivist view by modeling reader preferences at the individual level, not based on predefined categories, but inferred directly from annotated comparisons and textual features. this enables a finer-grained, datadriven reconstruction of evaluative profiles, which can then be clustered to reveal latent reader types without pre-established assumptions. ## problem definition we consider a collection of t texts @xmath1, x 2 , . . . , x t }, each described by a d-dimensional vector of intrinsic, reference-less features additionally, we have a set @xmath2, r 2 , . . . , r l } of l readers. each reader r @xmath3. these diverse raw evaluations (e.g., multi-dimensional likert scores, collections of binary judgments) are first aggregated into a single raw preference score s j (x i ) for each text x i evaluated by reader r j . to enable comparison across heterogeneous scales and readers, these raw scores s j (x i ) are then mapped to a continuous preference value ρ j (x i ) ∈ [0, 1] via per-reader minmax normalisation: reader preference centroid for each reader r j , their reader preference centroid or "paradigmatic text" is profiled by x * j , the ρ j (x i )-weighted average of features from their top-rated texts x top j (e.g., top 25% of their evaluations): these profiles capture preferred textual characteristics within a specific dataset. reader utility model inspired by multi-criteria decision-making theory @xcite , we posit that each reader implicitly combines textual features through an additive utility function: this formalisation sets the stage for our empirical study: (1) characterizing human-and aiauthored texts within their shared feature space to examine their properties and interrelations, and (2) inferring reader-specific preferences (via proxies for w j and profiles x * j ) to analyse patterns of literary preference. ## methodology our methodology proceeds by: (1) analyzing the datasets and extracting textual features; (2) processing reader evaluations into unified preference scores and pairwise data; (3) conducting an exploratory analysis of text features and initial reader preferences to predict preferences (rq1); and (4) training individualized random forest models (rq2), from which feature importances are extracted and clustered to identify reader profiles (rq3-4). these steps are detailed below. ## results and discussion in this section, we examine whether ai-and humanauthored texts differ measurably in their feature distributions, how these distributions relate to reader preferences, and whether reader-specific models can capture consistent patterns of evaluation. we structure the analysis around our four research questions. ## conclusions and future work this study demonstrates that literary quality judgments emerge from a structured interplay between a text's features and individual reader perception. we reveal two key drivers of literary preference: first, ai-generated stories gain favor when their textual profiles successfully emulate characteristics valued by specific reader groups, often aligning with features present in human-authored narratives that resonate with those readers; second, distinct reader communities consistently prioritize different textual dimensions. the preference for ai-generated texts, therefore, often hinges on their ability to exhibit the feature combinations favored by a particular reader profile. these stable, divergent preference profiles across datasets suggest that the notion of a universal standard for "good writing" may be an oversimplification. instead, our interpretable framework offers a path toward more nuanced evaluations of llmgenerated texts. topic count entity coherence local coherence global coherence log-likelihood readability (smog) mean sentiment emotional swings sentiment variance lexical diversity (mtld) syntactic depth sentence length max subordination rhetorical variety sentence rhythm topic entropy topic density mean feature importance by cluster cluster 0 cluster 1 figure 3: radar chart illustrating the mean random forest feature importances for the two reader clusters. higher values indicate greater importance assigned to a given textual feature when predicting preferences.