Modeling Readers’ Appreciation of Literary Narratives Through Sentiment Arcs and Semantic Profiles (2023)

Abstract: Predicting literary quality and reader appreciation of narrative texts are highly complex challenges in quantitative and computational literary studies due to the fluid definitions of quality and the vast feature space that can be considered when modeling a literary work. This paper investigates the potential of sentiment arcs combined with topical-semantic profiling of literary narratives as indicators for their literary quality. Our experiments focus on a large corpus of 19th and 20the century English language literary fiction, using GoodReads’ ratings as an imperfect approximation of the diverse range of reader evaluations and preferences. By leveraging a stacked ensemble of regression models, we achieve a promising performance in predicting average readers’ scores, indicating the potential of our approach in modeling literary quality.

## introduction defining what contributes to the perceived literary quality of narrative texts (or lack thereof) is an ancient and highly complex challenge of quantitative literary studies. the versatility of narrative and the myriad of possible definitions of a text's quality ultimately complicate the issue. in addition, the diversity and size of the possible feature space for modeling a literary work contribute to the complexity of the matter. it can even be argued that the quality of a literary text is not systematic and that "quality" is an expression of noisy preferences, as it mostly encodes idiosyncratic tastes that depend on individual reader inclinations and capacities. however, various studies have shown that this 'literary preference as noise' position is not tenable because text-intrinsic features (e.g., text coherence, literary style) and text-extrinsic factors (e.g., reader demographics) systematically impact perceived literary quality @xcite @xcite . at the same time, the questions of how such features interplay and what kind of metric we should use to validate them remain open. thus, current research on the perception of literary quality implicitly tries to answer two primary questions: 1) is it possible to define literary quality at all, and 2) is it possible to identify the intrinsic or extrinsic features that contribute to the perception of literary quality? while quality as a single measure may be impossible to agree on @xcite , it is hard to refute that reader preferences can be measured in different ways, both in terms of consistent attention given to literary works over time, and to valuations made by critics and readers. the intrinsic qualities of texts are more difficult to agree upon as the quality of a literary work consist of many elements, some that are virtually impossible to grasp by computational methods (e.g. the effect of metaphors or images). in addition, there are text-extrinsic features, such as the public image of the author or author-gender @xcite , which influence reviews to a degree that is hard to account for. still, as mentioned there is evidence that intrinsic models do have some predictive value when considering an array of different features , which pertain to both style and narrative. as such, the difficulty is not to only to model literary quality, as including intrinsic and extrinsic features such as genre and author-gender in a models of quality has resulted in good performances @xcite -but in elucidating what to include in a feature-set and why, and in seeking a level of interpretability. in this study, we aim to investigate the relationship between a narrative's emotional trajectory, its fine-grained semantic profile, and its perceived literary quality. using the average of hundreds of thousands of readers' ratings, we examine how sentiment-arcs and semantic profiles of literary narratives influence their perceived quality, exploring the prediction of these factors through a machine learning model trained on multiple features, encompassing both sentiment-related aspects and their dynamic progression, as well as semantic categorization. we also claim that access to a diverse corpus of works with a significant representation of highly successful works in all genres is an essential prerequisite for developing models with a credible performance. without the inclusion of the best regarded works, it is not possible to produce a model that relates to what is commonly understood as the highest level of literary achievement. the 9,000 novels corpus used in our study contains several of such works from 1880 to 2000, including major modernist and postmodernist writers as well as fiction from a range of popular genres. ## related works studies that predict the perception of literary quality from textual features have primarily relied on classical stylometric features, such as sentencelength or readability @xcite , the percentage of word classes, such as adverbs or nouns @xcite or the frequencies of n-grams in the texts (van cranenburgh and @xcite . more recent work has tested the potential of alternative text or narrative features such as sentiment analysis @xcite as a proxy for meaningful aspects of the reading experience @xcite @xcite @xcite . such work has focused on sentiment valence, usually drawing scores from induced lexica @xcite or human annotations @xcite , modeling, for instance, novels' sentiment arcs @xcite , although without considering fundamental arc-dynamics (e.g., temporal structure of plot variability) or narrative progression. by simply clustering sentiment arcs, @xcite was however able to identify six fundamental narrative arcs that underly narrative construction, while more recently, @xcite and @xcite have modeled the persistence, coherence, and predictability of sentiment arcs by fractal analysis, a method to study the dynamics of complex systems @xcite and to assess the predictability and self-similarity of arcs, in order to model the relation of sentiment arcs with reader evaluation @xcite . similarly, @xcite conducted fractal analysis on classical stylometric and topical features to model the difference between canonical and non-canonical literary texts. beyond sentiment analysis, the narrative content of texts has also been shown to impact perceived quality. relying on topic modeling, @xcite has shown that a higher topic diversity in texts corresponds to higher perceived literary quality, suggesting that works with a less diverse topical palette, like genre fiction, are perceived as having overall less literary quality, while @xcite has claimed that words that refer to intimate and familiar relations are distinctive of lower-rated novels, which can be linked to the hypothesis that specific genres, especially those in which women authors are dominant, are perceived as less literary @xcite . these studies suggest that the distribution of topics touched upon in texts impacts literary quality perception. several works have widely used resources like liwc to model such distributions @xcite . however, building on the findings of jarmasz (2012) -i.e., that roget's thesaurus is an excellent resource for natural language processing @xcite has shown that roget outperforms other dictionary resources (e.g., liwc and nrc sentiment lexicons) in modeling literary quality by category frequency -which is an intriguing argument to use the roget categories for modelling the perception of literary quality on a larger scale. ## quality measures while it is clear that various studies have recently used conceptually different features as a basis for understanding or predicting perceived literary quality, reader appreciation, or success, it should be noted that each study has a slightly different take on "quality" and that terms like "prestige", "popularity", or "canonicity" are not synonymous -although they could all be argued as aspects of qual-ity, and a more comprehensive study would benefit from taking a stronger perspectivist approach, considering multiple definitions of quality together @xcite . for any study trying to assess the factors contributing to the perception of literary quality, determining the quality judgments themselves is often one, if not the first, of the most challenging tasks. computational studies assessing literary quality often use a single standard of evaluation, which may not capture the diverse preferences of various groups of readers. various quality measures have been used, such as readers' ratings on platforms such as goodreads @xcite , or a text's presence in established literary canons @xcite . despite their diversity, different conceptions of quality can display significant convergences @xcite . in this work, we have employed average book-ratings on goodreads, a popular online social platform for readers that allows users, among other things, to comment, recommend, and review a book on a scale. 1 this metric possesses obvious limitations: it doesn't explicitly represent "literary quality" but arguably an aspect of it, it potentially conflates genre-specific value-judgements, and it forces goodreads' users to reduce their literary evaluations to a mono-dimensional scale. the latter issue might also obscure important differences in rating behaviour. for example, readers of sci-fi may be inclined to give a higher average rating on goodreads, something that we do not take into account when using average rating as a quality metric. nevertheless, this limitation can also be an advantage: the simplicity of the goodreads rating system offers a streamlined approach to a problem that frequently proves overly complex for quantitative analysis. the single goodreads' rating, representing readers' impressions on a single scale, offers a practical starting point for identifying patterns or trends across a wide range of books, genres, and authors. on the other hand, with its 90 million users, goodreads is argued to offer a particularly valuable insight into reading culture "in the wild" @xcite , as it collects books from widely different genres and curricula @xcite , and derives ratings from a notably heterogeneous pool of readers in regard to backgrounds, gender, age, native languages and reading preferences @xcite . ## data we have used the chicago corpus as a dataset, encompassing more than 9,000 english-language novels penned or translated into english between 1880 and 2000. the selection criterion for these works is based on each novel's number of libraries holdings. this results in a diverse compilation that spans various literary styles, from popular fiction genres to highly esteemed works of literature. it comprises novels written by nobel prize laureates @xcite and recipients of other highly regarded literary awards, as well as texts featured in canonical collections such as the norton anthology @xcite . however, it is important to acknowledge the cultural and geographical bias present in the corpus, which exhibits a significant over-representation of anglophone authors, limiting the scope of the analysis to a predominantly english-speaking context. titles authors number 9089 3150 avg. rating below 2.5 140 118 avg. ratings 3.74 3.69 ## features we employ three types of features, representing three distinct approaches to modeling a literary narrative. see ## models for our prediction task, we used a stacked ensemble model featuring a support vector machine-based regressor (svr) @xcite and a random forest regressor @xcite , with a ridge regressor as a meta-classifier @xcite . the svr is a popular choice for its ability to handle high-dimensional data and its robustness against overfitting, while the random forest is an ensemble method that constructs multiple decision trees to yield more accurate and stable predictions. they both outperformed other models in preliminary tests, demonstrating their promise as suitable candidates for this task. the ridge regressor, acting as a meta-classifier in our stacked ensemble, takes the predictions from the base models as input and generates a final prediction, leveraging regularization to minimize multicollinearity issues and prevent overfitting. as we didn't find benefits in using grid search for parameter tuning, possibly due to the high computational cost and time-consuming nature of the method, we report only the results of the experiments that did not include a pre-grid search for parameter optimization, opting for a more efficient approach to model selection and training. all models were trained on whole (9089) @xmath0.5 ( 8949 ) @xmath1.1 0.8 -0.0041 0.11 0.0003 0.07 sentiment features 0.42 0.14 0.03 0.09 0.07 0.06 roget features 0.49 0.13 0.17 0.09 0.23 0.05 sentiment + roget features 0.50 0.13 0.18 0.08 0.24 0.04 feature selection @xmath2.41 0.14 0.16 0.09 0.22 0.06 ## inspecting the most rated individual titles to better understand and analyse the strength and weakness of our model, we inspected the works that elicited its most accurate and the least predictions, considering only the "elite" of the most widely read (and often canonical) titles setting, a rating count threshold at 90,000. we provide an example of the very top and bottom of the list in on top of the list of the worst predicted are both famous and infamous novels: ayn rand's atlas shrugged, william gibson's neuromancer, james joyce's ulysses, and isaac asimov's i: robot. one possible explanation for this phenomenon is that these are all works that have a devoted following. as the model is solely fed with text-intrinsic features it would not be able to predict a more cult-like admiration of works that may otherwise be considered to be either very complex stylistically, like ulysses, less literary, like atlas shrugged, or particularly simple in style, like i: robot. having a reputation that makes these "more than just novels", but cultural beacons of various kinds, may affect users' grading behaviour. looking at instances with the lowest error in predicting average goodreads rating, the best predicted titles in our model, it is clear that these are popular and accessible works rather than highly canonized works. genre fiction, such as sci-fi (dick, card, butler), fantasy (gabaldon), and mystery (evanovich) dominate the list of best-predicted titles. a bit further down the list, below rank 13th, authors such as toni morri- ulysses james joyce 120014 all are known for solid craftsmanship and accessible stories. only conjectures can be made from inspecting these lists, but we do seem to see contours of a skewed grading that is based on more than textintrinsic features, like a form of readerly devotion that may be playing a role in both the rating count and the average score of some titles. another possible interpretation of this distribution, sustained by the large amount of genre fiction among the best-predicted titles, is that the features we selected for our model, and in particular the roget categories, behave in a characteristic way in works of genre-fiction, while more general works of literature might be distinguished better by considering stylistic features (wholly bypassed in our model). as such, roget categories may be acting as a proxy for genre, which would be reasonable considering the ideational focus of the roget thesaurus. the predictability of genre fiction especially may be explained if we assume that genre-fiction tends to place in a narrower grade-interval proper to their genre, while more general or "literary fiction" falls more consistently in a widert interval of ratings (from very low to very high). an alternative hypothesis, not entirely incompatible with the above and in line with previous work @xcite , is that genre-fiction and lowerrated works tend to be more mono-topical, i.e., be less diverse in content, treating a smaller range of topics. as such, roget categories may also to some extent be measuring topic-diversity, accurately predicting works lower that are more mono-topical. all in all, it is essential to bear in mind that our feature set does not include any stylometric features (such as word choice, sentence structure, and the use of punctuation), leaving it blind to a crucial aspect of literature -or even to "literariness" as such: stylistics contribute significantly to the expe-rience of the uniqueness and richness of a literary work @xcite , and is a central part of the impact of fiction in non-genre-fiction in particular @xcite . since our feature-set only observes texts from the sentimental and semantic perspective, it is possible that elements central to the reading experience in some of these titles remain unobserved. finally, the model's sensitivity to topical interplays might enable it to more accurately identify popular trends and themes and have a skewed performance towards books that follow popular topical patterns rather than those that exhibit exceptional style or depth. ## conclusions and future works the present study has shown that a combination of sentiment arc features, including dynamic measures, and semantic profiling based on roget categories enhances the predictive power of regression models for perceived literary quality -as measured through average goodreads' scores -across thousands of novels from the 19 th and 20 th century. our findings indicate that by accounting for a diverse set of psycho-semantic features in combination with measures that consider both the dynamics and valences of the novels sentiment arcs, we can obtain a performance that is better than that of any of the latter two approaches in isolation. a surprising finding was that the worst-rated titles seem to exhibit a particular predictability, possessing a more distinguishable profile in comparison to other titles, which might have contributed to an artificial inflation of our model's performance. it suggests that these particular titles may share specific sentiment or topical features that make them stand out from the rest, by which our model can identify them more easily. our results also highlight that the sheer magnitude of readers' ratings consistently enhances model performance. this observation supports the idea that certain aspects of literary quality tap into aesthetic preferences that are shared among large numbers of readers, at least widely enough to make predictions based on text profiling more reliable with a larger pool of evaluators. moreover, the predictive capacity of roget categories and sentiment arcs for literary quality perception indicate that there exists a underlying structure in how readers perceive and evaluate literary works. roget categories enable us to capture a coarse representation of the semantic content within texts, offering insights into themes, motifs, and granular references to topics that might resonate with readers. our related measures of sentiment arcs, in contrast, capture the emotional dynamics of the narratives, allowing us to examine the progression of feelings and the level of consistency and predictability of the story as it unfolds. this aspect is crucial because it highlights the role of sentiments in shaping the reader's engagement and overall impression of a text. by combining these two dimensions -semantic content and sentimental dynamics -we can delve deeper into the complex interplay between emotional patterns and thematic elements which impacts the perception of literary quality. this holistic approach enables us to gain a more nuanced understanding of the factors that contribute to the appreciation of literary works and the ways in which readers discern quality in literature. additionally, this combined analysis might potentially unveil commonalities and differences among various genres, styles, and time periods, further enriching our understanding of the multifaceted nature of literary quality. our approach still has a large number of limitations that need to be acknowledged. first, our approach relies on a reductive representation of the narrative texts, overlooking all traditional stylometric measures. the perception of literary quality is an intricate concept that relies on numerous factors, ranging from the stylistics, characters, plot development and pace, to cultural contexts. by reducing each narrative text to a subset of chosen features, our approach inevitably discards much of the richness and subtlety of works, while the narrow range facilitated by goodreads' scores forces the models to discern nuanced differences in perceived quality among texts that may be considered generally good by readers. this clearly limits our understanding of literary quality, especially when it comes to the more linguistically or stylistically virtuous titles. secondly, the reliance on goodreads scores as the sole metric of quality introduces bi-ases, as these scores are inevitably influenced by factors such as genre preferences and reader demographics. finally, the analysis is based on a limited sample of english-language texts from the 19 th and 20 th centuries, potentially limiting the generalizability of our findings to other periods, languages, or contexts. for the same reason, our study cannot consider the potential impact of translation and its effect on the reception of the texts. at the same time, given the inherent complexity of these constraints and the subjective nature of literary evaluation, the performances achieved by our models in terms of r2 scores and mean squared errors, which would be modest for easier tasks, can be considered rather promising. naturally, there is much that can be done from here. in the future, we intend to compile an even larger data set, in terms of both texts and features. integrating stylometric and syntactic features, for instance, could provide additional insights into the complex nature of literary quality. furthermore, we plan to investigate genre-specific patterns, as observing the performance of our models across different genres may reveal unique patterns and relationships that are specific to particular types of literature. finally, we intend to use more diverse and sophisticated metrics than goodreads: exploring alternative sources such as anthologies, awards, and canon lists. leveraging a richer set of indicators for literary quality/qualities, we hope to gain clearer insights into the complex interplay of factors that contribute to the perception of literary quality.