Computational Detection of Narrativity: A Comparison Using Textual Features and Reader Response (2022)

Abstract: The task of computational textual narrative detection focuses on detecting the presence of narrative parts, or the degree of narrativity in texts. In this work, we focus on detecting the local degree of narrativity in texts, using short text passages. We performed a human annotation experiment on 325 English texts ranging across 20 genres to capture readers’ perception by means of three cognitive aspects: suspense, curiosity, and surprise. We then employed a linear regression model to predict narrativity scores for 17,372 texts. When comparing our average annotation scores to similar annotation experiments with different cognitive aspects, we found that Pearson’s r ranges from .63 to .75. When looking at the calculated narrative probabilities, Pearson’s r is .91. We found that it is possible to use suspense, curiosity and surprise to detect narrativity. However, there are still differences between methods. This does not imply that there are inherently correct methods, but rather suggests that the underlying definition of narrativity is a determining factor for the results of the computational models employed.

## introduction storytelling is and has been a big part of the daily lives of many. people learn from stories, get inspired by stories, relate to stories and feel stories. we as humans can be seen as story-interpreting machines. however, how we perceive and interpret storytelling elements has been a point of discussion in the field of narratology for a number of years. previous research has attempted to define narrativity by means of cognitive processes @xcite @xcite . a notable way to define narrativity is discussed extensively by @xcite , who uses four perceptive elements to define what makes a text narrative: situatedness, event sequencing, world-making and, as he describes it, "feltness". * the first two authors contributed equally. another important change in narratology concerns the shift from an idea of narrativity as a binary class, i.e. a text is a narrative or not, to narrativity as "a local, multidimensional scalar property" @xcite . accordingly, @xcite have attempted to use the definition of narrativity as a scalar property to computationally detect narrativity, emulating human judgement using statements regarding the elements defined by @xcite . however, there are still a lot of aspects of narrativity detection left unexplored. alternative definitions heavily focus on readers' perception of texts @xcite @xcite . for instance, @xcite illustrates that narrativity can be defined by inherent human interpretation, which he refers to as the three "narrative universals": suspense, curiosity, and surprise. this research will attempt to explore the possibility of using this form of readers' perception as a way to detect narrativity, and will thus answer the questions: to what extent can suspense, surprise and curiosity as a form of readers' perception be employed to detect narrativity? is there a relation between textual features associated to narrativity and reader response identified by narrative universals? to answer these questions, we improve a textbased approach to detect the local narrativity of documents @xcite , provide new readerresponse-based annotations for an existing corpus @xcite , and discuss the results and implications of detecting narrativity using two different theoretical frameworks. ## related works the goal to define the concept of narrativity in order to detect said concept within literary texts has been a main drive within the study of narratology. one of the fundamental definitions of narrative has been proposed by @xcite , stating that the minimum requirements of a narrative should be that they represent a sequence of events by means of one or more characters. over time, researchers have elaborated on the aforementioned definition utilising two paradigms related to narrativity, as described by @xcite : "etic" and "emic". etic approaches to narrativity regard the concept as definable and detectable by means of textual and structural elements. whereas emic approaches utilise cognitive processes to classify texts by means of their degree of narrativity. a similar dichotomy has been described by @xcite by comparing objectivist and constructivist paradigms in the context of narrative theory. @xcite describe the objectivist paradigm as viewing textual aspects, such as semantics and syntactics, as defining features of a narrative. constructivist theory, however, regards the relation between the audience and the text as a way to detect narrativity, which @xcite described as being "the result of a process of construction and combination of certain processes and properties whose specificity is also dependent on extra-objectual factors". one approach grounded within constructivist narrative theory, that can thus be considered an emic approach, is the concept of "readers' perception". readers' perception, or often called "reader response" and "reception" in literary studies, depicts the way a reader interprets textual elements on an emotional or cognitive level @xcite . several researchers have attempted to define narrativity by means of readers' perception. these definitions have been employed by other research to detect narrativity in a handful of manners, including computational methods. @xcite suggests that the degree of narrativity in stories can be defined by the means of four perceptive elements: this idea has been successfully implemented by @xcite as well. while utilising etic, objectivist approaches to detect narrativity across long time scales, such as extracting narrativity by means of lexical and syntactic features, @xcite also utilized emic, constructivist approach similar to @xcite and @xcite using the perceptive elements by @xcite . @xcite assembled a team of three trained annotators to annotate 5sentence-long passages spanning across four "discursive domains": (1) non-fiction, (2) fiction, (3) poetry and (4) science. the annotators were assigned to rate the passages on a 5-point likert scale based on three elements: (1) "feltness", reworded as "agency", (2) event sequencing and (3) worldmaking. these elements have been explained in a codebook including annotation guidelines. while their approach to quantify readers' perception by means of herman's elements and human annotation as a way to detect narrativity within texts is valid, their data lacks linguistic and stylistic differences. @xcite uses an expanded dataset spanning across 19 genres, ranging from legal documents to aesop's fables, and from 19 th century literature to reddit stories. this extensive dataset is extremely helpful for narrativity detection, due to its divergent nature, and can also be used to detect narrativity by means of different forms of readers' perception. one other theory, also mentioned by @xcite , which acknowledges the human interpretative aspects of narratives, has evolved over several years @xcite . @xcite views narrativity as having the possibility to be defined by the effect that it can have on the reader, a view grounded within readers' perception. sternberg states that the degree of narrativity of literary texts can be defined through three "master effects", or "universals": (1) suspense, (2) curiosity, and (3) surprise: former research has used sternberg's constructive approach to narrativity to construct computa-tional models using some of the three universals. @xcite developed a computational model to detect suspense and found that, when compared with human judgements, their model predicted suspense rather well. however, their approach only used human judgements as a way to compare results, rather than attempting to use human judgements to train the model. a similar approach has also been used by @xcite . there is a lack of research where all three universals are utilised to detect narrativity. while researchers have attempted to model one of the three universals by means of textual elements, they used human judgements as a way of evaluation, rather than as an approach to model suspense. detection of narrativity by means of human judgements of suspense, curiosity and surprise can provide an insight into the relation between a story and its reader. therefore, the question which will be tested is whether it is possible to detect narrativity utilising all three of sternberg's universals. the approach taken will utilise human annotation to train a machine learning model and calculate narrativity scores based on readers' perception. ## data the data used within this research was a corpus constructed and provided by @xcite , consisting of 17,706 documents, ranging across 20 unique genres ( ## results and discussion the best predictive model for both types of narrativity (text-based and reader-based) is theil-sen regressor (tsr) with two features: tf-idf and concreteness ( these findings are also supported by the total annotation averages: there is a strong positive correlation between the total annotation averages (@xmath2.77), but there is some unexplained variance. moreover, kendall's τ shows a very weak correlation between the actual ranking of documents (τ = .03). this indicates that, even though there is a strong correlation between the predicted values, the way in which the documents are ranked by means of the predicted narrativity scores differ strongly between models. this does not imply that one of these theories is inherently correct, but that they are different ways of viewing narrativity. notably, the distribution of annotations shows that text-based annotation led to a majority of passages with the highest narrativity score, whereas reader-based annotation led to the opposite result: the majority of passages has been assigned the lowest narrativity score (figure 1 ). if we look at values predicted for the whole corpus, we see that they have an even stronger correlation (@xmath3.91) and span the whole narrativity spectrum in a more even way (although still skewed) than the annotated passages, confirming that high or low narrativity texts are not more frequent than those with a moderate degree of narrativity (figure 2 ). conversely, piper and bagga (2022)'s use of a binary classifier (logistic regression) pushed the predictions towards extreme values, biasing the interpretation. we also looked at the average predicted probabilities for each genre ( ## conclusion as narratology moved towards the idea that narrativity is a local, scalar, and multidimensional characteristic of texts, this research aimed to answer the questions "to what extent can suspense, surprise and curiosity as a form of readers' perception be employed to detect narrativity?" and "is there a to accomplish this, we performed multiple rounds of annotation to quantify readers' perception by means of three cognitive effects of narrative (suspense, curiosity, surprise). we found that the annotators generally agree with each other and there are moderate to strong correlations between text-based and reader-based narrative dimensions. as for the computational aspect, we trained a quite accurate regression model on a single highly predictive feature (tf-idf ). comparing our results to the results by @xcite , we found that there is a strong positive correlation between their and our approach to defining narrativity as a scalar property of texts. however, we also found that text-based and reader-based conceptions of narrativity do not completely overlap. while this research has been able to capture narrativity by means of readers' perception as defined by @xcite , it is not able to capture narrativity as a whole. as mentioned before, narrativity can be defined in various ways and thus there are many plausible ways to detect it. future research could combine both text-based and reader-based approaches to better grasp the complex, multidimensional nature of narrative. for instance, principal component analysis could help identify dimensions that could be conflated and dimensions that are irreducible to objective and textual properties. figure 5: passage id: 0f7c0e3f-3974-4271-af05-0d39cf3fba2e; genre: roc figure 6: passage id: ab7e5971-590b-4e59-8e47-ffd4f4c00e91; genre: roc figure 7: passage id: b49fdceb-29b8-4e83-9cf2-b3abd7b34aa5; genre: roc figure 8: passage id: code-breaker-the-walter-isaacson; genre: bio