Perplexing Canon: A study on GPT -based perplexity of canonical and non-canonical literary works (2024)

Abstract: This study extends previous research on literary quality by using information theory-based methods to assess the level of perplexity recorded by three large language models when processing 20th-century English novels deemed to have high literary quality, recognized by experts as canonical, compared to a broader control group. We find that canonical texts appear to elicit a higher perplexity in the models, we explore which textual features might concur to create such an effect. We find that the usage of a more heavily nominal style, together with a more diverse vocabulary, is one of the leading causes of the difference between the two groups. These traits could reflect “strategies” to achieve an informationally dense literary style.

## introduction the question of what "literary quality" is has been at the center of a millennia-long debate in aesthetics and literary studies.while literary judgment is almost by definition subjective, reflecting individual reader preferences, quantitative studies have shown that such preferences tend to converge at the large scale, and that both textual features, like coherence and style @xcite @xcite , and text-extrinsic features, such as reader or critic demographics @xcite @xcite , significantly influence appreciation or perceived quality. most schools of thought in literary research tend to see literary quality as either a perceived quality -an effect of reception and cultural dynamics @xcite @xcite -or as the effect of certain textual features, such as, among others, authorial strategies of defamiliarization and foregrounding @xcite @xcite . while consensus on a single gold standard of quality is hard to achieve @xcite , reader preferences and expert valuations can offer a range of measurable levels of appreciation. an often discussed dimension of literary quality is that of the so-called "literary canon", a complex concept generally representing a set of works that have survived or/and (by the same token) remain distinguished in the memory of a literary culture @xcite . a community, usually over large periods of time, defines as outstanding and worthy of attention; yet this process is not devolved to any individual authority, which makes the very definition of what is within the canon complex. as such, the canon is often scrutinized in cultural approaches to literary quality. some schools of thought have seen it as representing nothing but entrenched interests @xcite and thus as the cultural capital of ruling classes @xcite , while others have considered "canonic" works to excel in terms of intrinsic features @xcite , whether stylistic @xcite @xcite or narrative @xcite . in his work on the dynamics in the literary field, @xcite placed "popular success" and "consecration" at opposed positions. 1 recent quantitative studies of the literary field often follow a similar distinction between popularity and prestige @xcite , where more prestigious books and genres are what we could call the more "literary" ones @xcite . supporting the idea that such literariness may be distinguished by certain textintrinsic features, @xcite have shown that readers agree more on the "literariness" of books, and that it is easier to model literariness ratings from textual features than overall enjoyment ratings. in this work, we approximate what can be considered canonical in a large corpus of around 9,000 novels. based on this corpus, we ask two main questions: (1) are canonical novels more "perplexing", as measured through different large language models, than a non-canonical control group? (2) if they are, which linguistic and stylistic features might contribute to the difference? our reason for using perplexity (see chapter 4 for the formal definition of perplexity) is at least two-fold. on one hand, canonical works of fiction are often examples of either "virtuous" @xcite or defamiliarizing usage of language @xcite , thus an uncommon usage of language. such characteristics may make canonical works of fiction more surprising with respect to non-canonical. even when perplexity is operationalized as a measure of information theory, these works might elicit a higher perplexity on average. on the other hand, perplexity is a central measure of informativity in information theory @xcite . since perplexity is a function of surprisal, more perplexing texts tend to be more informationally dense. a highly specialized scientific paper -like a highly complex and articulate page of james joyce -is unusually informative in the sense that it constantly relays novel information (highly specialized or new words -neologisms -or words in a new order) to the reader. in theory, a communicative system that manages to be as dense as possible without breaking down or being "too dense" for its own readers indicates elements of a heightened communicative efficiency -a feature that communities might tend to optimize over time @xcite . ## related works studies seeking to predict literary success or perceived literary quality have often followed the intuitive idea that readers perceive a difference between more difficult and easier texts, and approximate some form of stylistic complexity. such studies use features related to the readability indices de-veloped in linguistics research, such as sentence length, vocabulary richness, or redundancy @xcite @xcite @xcite . additionally, readability formulas find integration in editing tools such as the hemingway or marlowe applications, foot_1 which prioritize more "readable" texts. yet the relation between stylistic aspects of text complexity and reader appreciation appears complex: while it is suggested that readers prefer more stylistically complex or informationally dense texts @xcite , it is a widespread conception that bestsellers are easier to read @xcite . in literary studies, reading ease has also been proposed as a marker of "better" style as far back as 1893 @xcite . while @xcite and @xcite found that readability formulas were weak for predicting reader appreciation, more recent work has shown that preference for the type of text difficulty measured by readability formulas may vary across different audiences: novels with higher readability are preferred by raters on large online platforms, while award-winning novels tend to have lower scores @xcite . measures that are more explicitly related to information density or entropy, such as word and bigram entropy @xcite , surprisal @xcite , and text compressibility @xcite have also been used to assess the complexity of literary texts. 3 @xcite , for example, shows a diachronic evolution of literary texts towards a greater density of information. surprisal has been shown to correlate with the cognitive effort of processing words @xcite @xcite and is as such a measure of the information density of text. the connection of information density or surprisal with a text's relative "quality" (in this case intended as communicative effectiveness) has been linked more explicitly in studies about non-literary domains. for example, @xcite found that scientific prose has gradually developed informationally denser prose, optimized for expert-to-expert communication. perplexity, as a closely related measure of the probability of words in context, may be applied as another measure of difficulty or as a measure of the information density of a text @xcite . while perplexity is primarily used as an internal evaluation metric for the performance of language models, it has also been used variously as a descriptive and predictive metric to distinguish between the domain and style of texts, for example between formal and colloquial tweets @xcite or between speech production by people with dementia and without @xcite . like surprisal, llms' perplexity also shows a relationship to human word processing or perception of text difficulty, for example with gaze duration in reading @xcite , though the similarity of model perplexity to, for example, human reading time may change with larger model size @xcite . still, the relation between the "difficulty" level of a text and perplexity is not clear-cut, and perplexity seems to capture something different than what can be estimated with traditional readability formulas from linguistics research. @xcite show no relation between model perplexity and one readability measure, while @xcite suggest that models might actually attribute less perplexity to texts aimed at adults compared to texts aimed at younger audiences. similarly, there seems to be no clear connection of perplexity to stylistic features of texts connected to readability, suggesting that different textual features affect readability and model perplexity @xcite ). 4 some work has been done to estimate surprise or narrative coherence in fiction @xcite @xcite , still the question of quality or reader appreciation of more or less "surprising" texts remains underexplored. in this context, perplexity may constitute an additional measure easily related to different types of reader appreciation. notably, in text generation, model perplexity is explored to retrieve more or less diverse output, given that a higher likelihood text (with less perplexity) does not necessarily mean that it is of better quality for human raters @xcite . ## perplexity perplexity is an information-theoretic measurement of how well a probability model predicts a sample @xcite . the perplexity of a well-trained language model on a test text can be interpreted as the exponential of its average level of surprisal @xcite , namely in this study, we calculate the perplexity of three language model on the same corpus. we based all perplexity calculations on the byte pair encoding tokenization @xcite used in the series of gpt2 models. due to constraints imposed by the maximum input length, we employed a stridebased methodology to gauge perplexity at the text level. this method incorporates a strided sliding window, wherein the context is shifted by fixedlength strides, affording the model an expansive context for making predictions at each iteration. in this specific framework, the window's size is 1024 tokens long, with a stride length of 512 tokens. by doing so, the second half of the previous context window served as the first half of a new context window to calculate perplexity estimates for the remaining 512 tokens @xcite . therefore, the surprisal for each book is comprised of perplexity values for the initial 1024 tokens, intermediate segments of 512 tokens, and the residual tokens of varying lengths. the aggregate-mean value is designated as the textual perplexity within the confines of this study. ## models we use three alternative, large language models to assess the average perplexity of the novels (see the two standard versions of gpt2 models (radford et al., 2019), namely the gpt2 and the gpt2-xl, are used in this paper since they are based on neural networks with billions of parameters and trained on terabytes of text, achieving very good results both in generating natural text and in estimating the perplexity of unseen texts. however, there is a substantial risk that some of the books of the corpus may have been included within the dataset that these models were trained on, especially when openai has not yet published its dataset until now. therefore, the main methodology is to train a model of the same architecture as the series of gpt2 models from scratch using a dataset outside the corpus (hereinafter referred to as the self-trained model). if the corpus perplexity estimation observed from the self-trained model is in close correlation with results from the series of gpt2 models, then the potential bias risk can be excluded. in this context, a new text generator based on the gpt2 model is trained from the beginning on the "article" content of the cnn dailymail dataset foot_9 . the primary reason for not employing other literary works as the training set is due to the potential bias associated with the selection of these books. moreover, the cnn dailymail dataset is chosen for its compilation of approximately one million news stories designed for reading and comprehension tasks @xcite , offering a narrative consistency more closely aligned with that of novels than other datasets, such as wikitext. then, we use the adam optimizer with a learning rate of 5e-5 and a cross-entropy loss criterion to train the model for 10 epochs. the spearmann correlation test results presented in figure 3 show a robust correlation in perplexity values between the self-trained model and the other gpt2 models, indicating that a potential data bias can be excluded at least within this corpus. therefore, the models forming the final hierarchy can be viewed as a sequential examination on the hypotheses and the consistency of our results across the models of varying sizes, ranging from the smallest version of the self-trained model to the largest version of the gpt2-xl model (see ## perplexity & the canon these three variants of gpt2 models based on the transformer architecture are employed to calculate perplexity values using the stride-based method across the entirety of the novels in the chicago corpus. a first mean of evaluation is to observe whether the mean perplexity changes with the models' size, as we would expect larger models to display lower perplexity. consistently with our expectations, the mean perplexity values decrease when the model size increases, as delineated in despite some potential fluctuations, the outcomes exhibit general consistency across the three language models. notably, the highest and lowest perplexity values are elicited from the same two books, namely the graduate by charles richard webb (the least perplexing novel overall) and finnegans wake by james joyce (the most perplexing). then, the mann-whitney test is used to examine the perplexity difference between canonical and non-canonical works. as shown in when inspecting works of different types of canonicity (contrasting literary prizes with other types of collections) we find that works judged canonical by experts and that are more closely affiliated to institutions (the norton anthology, opensyllabus, and penguin classics) have a higher perplexity ( it is important here to note here once again that perplexity is not an absolute measure, but is the result of a model's training. models trained on large enough datasets will capture fundamental regularities and find idiosyncratic uses of language more perplexing, but every model will consider elements closer to its training set as more normal. in this paper we assume the large training sets of the models as representative enough of contemporary english. perplexity is a powerful measure of linguistic predictability, as it results from large-scale modelling of word sequences. it can also be, and usually is, the effect of a composition of several factors, so that it is not always easy to understand what elements are driving its values. the richness of a large corpus of narrative fiction only adds to this difficulty. according to all our models, the most perplexing "novel" in the corpus is james joyce's finnegans wake, while the least perplexing is webb's the graduate. a look at the first few lines of these books suffices to align our intuition to the models' results. 12 but the reasons for scoring a high perplexity can be different even among those texts that are "clearly" unusual: for example, another high-perplexing novel, harris' nights with uncle remus, often reads as a fairy tale, but is heavily interspersed with heavy use of almost unintelligible eye dialect. 13 while the models' scores clearly pick from the same elementsrecording internal spearman correlations between 0.89 and 0.93 (fig. 3 ) -it is not easy to determine which linguistic features have the highest role in determining a given level of perplexity, and, more importantly, in determining which are the perplexing elements in a text that help tell canonical from non-canonical works. in the next sections, we will check the correlation between perplexity and some textual features often considered in the discourse over literary quality and canonicity. we refer to figure 4 for a summary of the findings. ## conclusions we have explored some features of canonical vs non-canonical works based on a corpus of 9,000 novels from the late 19 the and the 20 th century. we first found that canonical novels seem to elicit higher perplexity scores based on three llms, with the difference remaining significant across different model sizes. perplexity seems to reflect a higher complexity in style of canonical novels, compared to that of non-canonical works that enjoy a vast readership. we have then explored some specific features that might contribute to this effect. based on our collection, the higher perplexity of the canonical group is linked to different distributions of grammatical constructions: heavier use of nominal phrases, paired with average longer sentences, words, and a higher lexical diversity. specifically, the presence of a more marked nominal style might be an important cause for the difference in perplexity between the two groups, although it is clear that the overall effect is a result of an ensemble of features at the syntactic, stylistic, and semantic level. the idea that "canonical" novels, on average, are more challenging for readers than non-canonical ones, while the opposite holds for widely spread but non-canonical texts (such as texts rated very often on goodreads), mirrors existent findings @xcite . 16 the characteristics of this difference are of particular interest as they seem to be at least partly linked to the communication efficiency observed in expert-domain prose for other fields. a heavily nominal style has been linked with the development of refined and diverse vocabulary, a higher cognitive load for the reader, and more effective communication of information, as nouns can be highly specific and diverse, bringing a higher amount of information at the cost of higher decoding effort. it is a hypothesis worth considering that canonical works might achieve a lower communicative immediacy in favor of a higher communicative efficiency. what this means for a literary work and what it implies for the reader's experiencegiven the unique communicative functions of literary texts -remains an open question to explore in future studies. finally, it is important to consider that in this work we have intentionally ignored the dimension of time: we are interested in which features distinguish canonical, awarded, best-selling texts etc., independently from their distribution within the corpus. this means that we are treating our models as an abstract "contemporary reader" who, in front of the corpus, reacts to the canonical group differently from the bestseller group and so on. in future, we intend to observe how the features we have studied in this work correlate with time.