Results of a Single Blind Literary Taste Test with Short Anonymized Novel Fragments (2020)

Abstract: It is an open question to what extent perceptions of literary quality are derived from text-intrinsic versus social factors. While supervised models can predict literary quality ratings from textual factors quite successfully, as shown in the Riddle of Literary Quality project (Koolen et al., 2020), this does not prove that social factors are not important, nor can we assume that readers make judgments on literary quality in the same way and based on the same information as machine learning models. We report the results of a pilot study to gauge the effect of textual features on literary ratings of Dutch-language novels by participants in a controlled experiment with 48 participants. In an exploratory analysis, we compare the ratings to those from the large reader survey of the Riddle in which social factors were not excluded, and to machine learning predictions of those literary ratings. We find moderate to strong correlations of questionnaire ratings with the survey ratings, but the predictions are closer to the survey ratings. Code and data: https://github.com/andreasvc/litquest

## introduction it remains an open question why some novels are considered literary, while other novels such as thrillers are considered less prestigious. one position in this debate is that formal features of the text play an important role, i.e., formalism. the notion that foregrounding language is involved with the aesthetic appreciation of literary texts is supported by experimental studies @xcite . recent work investigates the role of the text in literary quality empirically on a larger scale by collecting literary ratings of novels in a large reader survey (henceforth the riddle survey, @xcite . this survey provides a way to investigate the association of literary judgments and textual features directly. can human judgments of literary quality be predicted by machine learning models from purely textual features? the answer turns out to be yes, to a substantial extent. the model by @xcite showed that textual features can explain 61% of the variation in the literary ratings. however, machine learning may pick up on various subtle frequency differences that humans do not notice; compare the case of authorship attribution, which is arguably more difficult to do by humans by hand, than it is for a computer with a simple table of function word frequencies and cosine distances. in order to place the success of the machine learning model that can predict literariness in context, we would have to ask humans to assign ratings purely based on a text fragment, without being influenced by the prestige associated with an author's name or the title of the novel. the setup of such a 'challenge' was proposed in van cranenburgh and @xcite ; however, no results were presented. wine experts perform at chance level in blind taste tests @xcite ; will readers be able to recognize literature? in this paper we report on the results of a pilot version of such an experiment and analyze the results. this allows us to see the extent to which our participants agree with ratings in the riddle survey who rate based on author and title, as well as how they compare to the machine learning models that perform a similar task. this also allows us to estimate the importance of textual features for human readers when making judgments. by looking into the motivations that the participants have given for their ratings, we also hope to identify interesting linguistic features of literary quality, which can be investigated further in future computational and literary studies work. ## questionnaire setup we selected 8 originally dutch novels for the pilot; 4 of which were rated as highly literary (5.5-7.0) in the riddle survey, and 4 rated as neither non-literary or very literary (3-5-5.0) in that survey. we deliberately did not select non-literary novels with low ratings of less than 3.5, because previous research showed that the genre of such novels is very easy to recognize (van cranenburgh and bod, 2017), and we are interested in the more subtle differences between literary novels and 'not quite literary' novels. in addition, author gender is balanced, both for the literary and the not quite literary novels. from these novels we selected fragments of 250 words, from the beginning of a chapter, though not the first chapter. we rule out the influence of author prestige by not showing any metadata of the fragments that are presented to the participants. in addition to not presenting metadata, we also anonymize the fragments; i.e., names of the main characters in the text are abbreviated to initials. fragments are presented in an arbitrary order (although we did not shuffle the fragments across participants). we use the same 7-point likert scale as in the riddle survey, which ranges from not at all literary to very literary. as in that survey, we did not give a definition of what 'literary' is, because we did not want to influence the participants. for a given fragment, participants were asked the following questions: we recruited participants from our social network. they range from casual to enthusiastic readers, of various genres, with ages ranging from 30-60. we collected responses from 48 participants. compared to the 14k participants of the riddle survey, this is a small pilot. however, for the riddle survey, we look only at the ratings of books the participants had rated, which means that the number of ratings varies per book. it was shown that 50-100 ratings are sufficient to compute a reliable mean @xcite . in contrast, all fragments were rated by all 48 participants in our questionnaire. ## literary ratings in this first analysis, we consider the mean literary rating for each novel fragment in the questionnaire. we compare these means to the ratings in the riddle survey and to predicted ratings from supervised models. ## analysis of motivations overall, the common thread in the motivations is that elaborate and sophisticated language is seen as more literary. most motivations refer to specific stylistic aspects. participants also remarked that the task was hard due to the short text fragments. the following sentence was cited by multiple participants who agreed on its high literariness (our translation): "she brings out the world's blissful mongoloid smile, the grinning, wet shining zen of dumb objects, from which she blows the names off like chaff." however, the choice of fragment likely plays a large role. the fragments were selected arbitrarily, and are not necessarily representative of the whole novel. especially for dorrestein, this likely played a role. in the fragment, a woman observes a group of children, followed by the following free indirect discourse (our translation): "she notices that she moves her jaws. i am mrs pacman. bite, swallow, gone. bite swallow gone." this phrase struck many participants, but it is striking that they disagreed on whether it made the fragment more or less literary. one participant cited this phrase as motivation for a 6 out of 7 ('no fixed pattern, surprising, would like to read more'), while another cited it to give a rating of 1 out of 7 due to the sudden shift in style ('a hodgepodge of old fashioned words and modern imagery'). incidentally, such examples may be an issue for the notion of foregrounding as an explanation of literariness: while we find phrases that stand out for many of the participants, they do not agree on their effect on the literary status of the fragment. this requires further study; for example, would the participants' judgment change if more context is given? while the fragment by den tex scored high in the questionnaire, several of the participants cited the mention of a soccer player, or the fact that soccer was discussed in the fragment, as a reason for judging the fragment as non-literary. this suggests that there is a perception that references to low brow, popular culture are associated with lower prestige, while literature should be about more timeless and serious matters. we can hypothesize that this is part of a heuristic: by default, references to low-brow popular culture trigger a low rating; on the other hand, such references can also appear in high literature, but in this case more context (i.e., a longer fragment) may be needed to see how such references fit in the story. ## discussion and conclusion we found a reasonable consensus for the strongest style differences in the fragments. the correlations of the riddle survey ratings with our questionnaire ratings range from moderate to strong, which supports the construct validity of literariness as a variable. the difference between the ranking of the most and least literary fragment was preserved, and some of the changes in ranking could be explained as genre and gender effects that disappear when rating an anonymized fragment. participants found the task of rating short fragments hard, and the predictions by the machine learning models are closer to the survey ratings. however, the machine learning models are trained, while the participants are not. the ability of predictive models to reproduce the quality ratings should be interpreted carefully since the models may pick up on more than just literariness from the textual features, such as the aforementioned genre and gender effects. the participants agree on criteria (e.g., word usage), agree on salience of phrases, but sometimes disagree on how literary they are. this suggests that foregrounded language is not necessarily seen as literary language, and raises the question of how to identify potential linguistic markers of literary language for further study. our work is related to work on experimental aesthetics. in one branch of experimental aesthetics based on the ideas of fechner (1876), the subjects' ideas on aesthetics are researched. respondents are asked to recall as many adjectives related to aesthetic judgment as possible, for instance of literary novels. in @xcite , this approach leads to the conclusion that 'beautiful' and 'suspenseful' are central to descriptions of aesthetic judgment of fictional literature. this type of research is related to the influence of the text, but @xcite did not ask readers to reflect on specific works. other work on experimental aesthetics considers foregrounding effects and does present participants with novel fragments (e.g., hakemulder, 2004) or lines of poetry (e.g., @xcite . however, more generally, there is more to literary quality than aesthetics. even though there has been a focus on the aesthetic experience of literary language (cf. @xcite , ethical, moral, affective and other motivations can play a role in literary judgments as well. in future work, we want to determine the influence of author prestige by conducting a controlled experiment in which one group sees fragments with author names, while the other only sees the fragments. after this, a larger experiment should be conducted, with more novels, more fragments per novel, and more participants. it would be interesting to contrast general readers with readers with particular backgrounds (e.g., literature professors). once promising linguistic markers are identified, we want to manipulate these markers in the fragments, to confirm their effect on literariness, similar to the work of hakemulder (2004) on novel fragments and @xcite on poetry.