Dimensions of Quality: Contrasting Stylistic vs. Semantic Features for Modelling Literary Quality in 9,000 Novels (2023)

Abstract: In computational literary studies, the challenging task of predicting quality or reader-appreciation of narrative texts is confounded by volatile definitions of quality and the vast feature space that may be considered in modeling. In this paper, we explore two different types of feature sets: stylistic features on one hand, and semantic features on the other. We conduct experiments on a corpus of 9,089 English language literary novels published in the 19th and 20th century, using GoodReads’ ratings as a proxy for reader-appreciation. Examining the potential of both approaches, we find that some types of books are more predictable in one model than in the other, which may indicate that texts have different prominent characteristics (stylistic complexity, a certain narrative progression at the sentiment-level).

## introduction defining literary quality or reader appreciation is a complex challenge for quantitative literary studies due to the the heterogeneous nature of narrative texts, and the complexity of mechanisms of judgements and standards in the literary field. while recent studies demonstrate that literary quality appears above chance at the scale of large numbers, and that both text-extrinsic and text-intrinsic features systematically impact sales figures and reader judgements @xcite @xcite @xcite , the question of how these features interact, and what metrics can be used to validate them, remains open. the challenge lies not merely in modeling literary quality, but in selecting which features to include in a model, while ensuring a degree of interpretability. in this study, we examine two different sets of textual features for modelling literary quality: stylistic and syntactic characteristics vs. narrative and semantic features based on sentiment analysis and word-category profiling. ## related works generally, we may distinguish two types of featuresets used to model literary quality: stylistic features (the "how" of writing) and those that capture deeper structures and content @xcite . previous studies of literary quality have predominantly relied on stylistic features, such as sentence-length, lexical richness or redundancy @xcite , syntactic complexity @xcite , or n-gram frequencies @xcite . more recent works have tested the effect of alternative features, such as sentiment analysis on reader experience @xcite @xcite @xcite . studies relying on sentiment analysis usually draw scores from lexica @xcite or human annotations @xcite , to outline the sentiment arcs of narrative texts @xcite , and have shown a correlation between reader appreciation and sentiment @xcite . @xcite and @xcite modelled persistence, coherence, and predictability of sentiment arcs using fractal analysis, a method to study the dynamics of complex systems @xcite , finding correlations with reader appreciation @xcite . in summary, simple or more complex approaches methodologically based on sentiment-annotation show a predictive power for reader appreciation. beyond sentiment analysis, other approaches to modelling literary quality have focused on the semantic content of texts. using topic modeling, @xcite found that novels with a higher topic diversity elicited higher ratings, and less topically diverse works like genre fiction were perceived as less prestigious, while @xcite found that the specific topics in texts also indicate higher or lower literary quality -topics linked to intimate and familiar relations, for example, seem to indicate lower ratings, which can be linked to the hypothesis that specific genres, especially those in which women authors are dominant, are perceived less literary @xcite . while topic modelling or resources like linguistic inquiry and word count (liwc) foot_0 are often used to model semantics @xcite @xcite have shown that the roget thesaurus outperforms them in modeling literary quality. ## conclusions and future works we find that novels' stylistic and syntactic features, as well as the characteristics of their overall emotional tone, the dynamics of their sentiment arcs, and the semantic categories they cover appear to be indicative of their appeal to readers and their perceived overall quality. moreover, while a model based on the selected sentiment and semantic features clearly outperforms a model based on selected stylistic and syntactic features, each model might be best at modelling different types of literary texts, where the stylistic model is better at predicting canonical from non-canonical titles. interestingly, the models converge on struggling to predict some titles that are perhaps popular because of extratextual factors. naturally the subject of predicting reader appreiciation of literar texts is complex. in the future we aim to repeat the experiment looking at various quality proxies beyond goodreads ratings to study convergences between different perceptions of quality, as well as using a larger set of features. we may also attempt more sophisticated models, as long as some interpretability remains, as the main objective is not to effectively predict a score, but to understand more about how literary texts affect readers at various narrative levels. a appendix 1) baseline 2) stylistic model 3) narrative model figure 3: distribution of real and predicted avg. rating values, for models trained on the full corpus (above) and on titles @xmath0.