Sentimental Matters - Predicting Literary Quality by Sentiment Analysis and Stylometric Features (2023)

Abstract: Over the years, the task of predicting reader appreciation or literary quality has been the object of several studies, but it remains a challenging problem in quantitative literary studies and computational linguistics alike, as its definition can vary a lot depending on the genre, the adopted features and the annotation system. This paper attempts to evaluate the impact of sentiment arc modelling versus more classical stylometric features for user-ratings of novels. We run our experiments on a corpus of English language narrative literary fiction from the 19th and 20th century, showing that syntactic and surface-level features can be powerful for the study of literary quality, but can be outperformed by sentiment-characteristics of a text.

## introduction the question of what literary quality "is" is as complex as it is old. it may be argued that "literary quality" is an empty concept, since individual tastes of narrative and literature can differ widely among readers. yet it is possible that a set of textual and narrative characteristics tend to improve or damage the appreciation of a literary piece independently from genre expectations and preferences. this persistent intuition, while controversial, has been amply discussed through the history of literary criticism, and also stands at the foundation of most rhetorical or writing advice. the idea of an intersubjective agreement on literary quality may be also sustained by the convergence of large numbers of readers (and when considering canons, generations of readers) on certain titles rather than others @xcite . in the quest of defining principles of literary quality, quantitative analyses ask two questions: whether it is possible to define literary quality at all; and whether it is possible to individuate textual patterns that contribute to make a text more appreciated. in this paper we aim to explore the interplay of the sentiment and stylometric characteristics of narrative texts and their role in the perception of literary quality. ## related works traditionally, quantitative studies of literary quality have relied on texts' stylometric properties, ranging from the percentage of adverbs @xcite to the count of the most frequent n-grams in a text @xcite , to model the success or quality of literary works. more recent works, nonetheless, have emphasized the potential of sentiment analysis @xcite , at the word @xcite , sentence @xcite or paragraph @xcite level, to uncover meaningful mechanisms in the reading experience @xcite @xcite @xcite , usually by drawing scores from human annotations @xcite or induced lexica @xcite .while most studies have focused on the valence of sentiment arcs, @xcite and @xcite have tried to model the persistence, coherence, and predictability of novels' sentiment arcs, using fractal analysis @xcite @xcite @xcite , a method of studying patterns in complex systems, exploring the degree of predictability or self-similarity of narratives -a method that appears to capture meaningful patterns impacting reading experience. naturally, beyond which features to consider, another great challenge of studying literary quality is that of finding "oracles" of quality. measures of quality have been approximated by looking at readers' ratings on platforms such as goodreads @xcite , or by relying on established literary canons @xcite . despite their diversity, different concepts of quality display large overlaps (walsh and antoniak, 2021a), thus to a degree allowing for the comparison across canons and preferences @xcite . ## results most models tested show predictive power, i.e., perform better than random. their performance is reported in filtered corpus pronoun freq. ## conclusion and future works we have tried a new set of experiments in the highly challenging task of modelling literary quality, represented as the online average ratings of readers, from a small set of textual and sentiment features. while a similar attempts have been made before (on smaller corpora), to the best of our knowledge, we are the first to show that the addition of several sentiment-related features improves the predictive power of most models. the sentiment features considered here were of two kinds: a global kind, such as the mean sentiment of a novel; and a dynamic kind, such as the level of entropy and fractality of the sentiment arcs. we have also found that the bottom 2% of titles elicit distinctly lower ratings, and that their appreciation is partly predictable through the textual features we have included. finally, we analysed the features needed to predict perceived literary quality, noting that a balance between simplicity and diversity seems to characterize more appreciated titles. naturally this is a study on a complex subject. in the future we aim to repeat the experiment optimizing for quality proxies beyond goodreads ratings to study convergences between ways of defining quality, and use a larger set of features. we may also set it as a classification problem, and attempt more sophisticated models, as long as some interpretability remains.