N arra D etect: An annotated dataset for the task of narrative detection (2025)

Abstract: Narrative detection is an important task across diverse research domains where storytelling serves as a key mechanism for explaining human beliefs and behavior. However, the task faces three significant challenges: (1) inter-narrative heterogeneity, or the variation in narrative communication across social contexts; (2) intra-narrative heterogeneity, or the dynamic variation of narrative features within a single text over time; and (3) the lack of theoretical consensus regarding the concept of narrative. This paper introduces the NarraDetect dataset, a comprehensive resource comprising over 13,000 passages from 18 distinct narrative and non-narrative genres. Through a manually annotated subset of ~400 passages, we also introduce a novel theoretical framework for annotating for a scalar concept of “narrativity.” Our findings indicate that while supervised models outperform large language models (LLMs) on this dataset, LLMs exhibit stronger generalization and alignment with the scalar concept of narrativity.

## introduction narrative detection is an essential task in nlp and the subfield of computational narrative understanding @xcite @xcite @xcite . a growing body of research is developing across a variety of domains that focus on storytelling as a key mechanism for explaining human beliefs and behavior @xcite . being able to detect where, when and to what degree the act of narration is taking place among textual outputs will support research into the function of narration across a range of fields. we see three core challenges facing the task of narrative detection. first is the high degree of variety surrounding the social contexts of storytelling. this is called "situatedness" by @xcite and is one of the four essential elements of nar-rative in his scheme. stories can appear in the news media, on social media, in both fiction and non-fiction books, online fan writing sites, scattered throughout large cultural heritage archives, and multi-modally as well (graphic novels, comic books, and children's books) to name a few. while certain components of narrative behavior will likely change across contexts, we also expect some core behavior should remain consistent. we call this the problem of "inter-narrative" heterogeneity. the second main challenge is what we call "intranarrative" heterogeneity, i.e. the degree to which narrative communication can differ over narrative time. narrative practices do not consist of a single, fixed set of behaviors that occur always and everywhere in a story, but rather a dynamic combination of features that may wax and wane. one of the principal theoretical shifts to occur in the field of narratology over the past few decades has been this shift from understanding narrative as a matter of kind to one of degree @xcite @xcite . "narrativity" according to this theoretical framework is a quality that can best be understood not as a global binary class (a document either is or is not a narrative), but as a local, multi-dimensional scalar property @xcite . a narrative document, such as a novel, may exhibit greater or lesser degrees of narrativity at different moments in the text, just as ostensibly non-narrative documents, such as scientific reports, may also exhibit degrees of narrativity and in different ways. this stylistic heterogeneity introduces the third challenge facing the task of narrative detection, which is the theoretical heterogeneity underlying the task. the concept of "narrative" consists of a complex set of dimensions and different sources have proposed different frameworks for its study. not surprisingly, narrative continues to be understood and operationalized in different ways. a key goal for the field moving forward will be the devel-1 opment of more standardized narrative models. in this paper, we introduce the narradetect dataset, which aims to make the following contributions: ## prior work the creation of narrative datasets within the field can be divided into two principal areas: the first is the development of domain specific collections of stories or story dimensions for the purposes of narrative understanding. these include news stories @xcite , cultural heritage material @xcite @xcite , novels @xcite , birth stories @xcite , and artificial stories @xcite , to name but a few. datasets for the task of narrative detection are far more scarce and rely on both positive and negative examples. @xcite have created one of the few publicly available narrative detection datasets. the storyseeker corpus consists of an annotated dataset of narratives at the sentence level on a set of 502 reddit posts and comments drawn from over 100 different subreddits from the webis-tldr-17 dataset @xcite . they use a binary model of annotation applied to sentence spans and following @xcite define a narrative as "a sequence of events involving one or more people." doyle et al. ( 2024 ) have created a collection of 750 manually annotated reddit posts for the presence of narrative from the r/suicidebereavement subreddit. following @xcite , they annotate posts based on the following categories: the presence of a plot, characters, the author as a character, and a clear beginning, middle, and end. @xcite annotated a collection of 849 facebook posts related to the topic of breast cancer for the presence of narratives. in a follow-up study, @xcite annotated a collection of 3,000 tweets drawn from the antivax @xcite and cmu-miscov19 (memon and carley, 2020) datasets respectively. they annotate tweets for the presence of "narrative style," which they define as: "the presentation of a sequence of events experienced by a character or characters" following @xcite . narrative detection datasets to date can thus be characterized by the following qualities: narrative has only been operationalized as a binary category; annotation has largely been undertaken with respect to a specific domain (social media); and different theoretical constructs have been used to inform annotation, with events and event sequences being the most predominant category. ## evaluating the narradetect corpus for narrative detection we evaluate the utility of the narradetect dataset using both supervised and unsupervised methods. for supervised models, we experiment with two feature representations: (1) a semantically neutral feature space derived from part-of-speech (pos) tags excluding punctuation and (2) contextual embeddings obtained from the bert large cased model. an svm with a gaussian kernel serves as the classifier in both cases. in order to disentangle narrativity-related fea-tures from genre-specific signals, we employ an adversarial learning approach. a shared feature extractor, implemented as a feedforward neural network, generates input representations optimized for narrativity classification. the primary narrativity classifier predicts whether a passage is narrative or non-narrative, while an auxiliary genre predictor identifies the passage's genre. a gradient reversal layer between the extractor and genre predictor suppresses genre-specific signals, with a combined loss function balancing narrativity and genre prediction using a trade-off parameter λ. this approach enables the model to learn features capturing narrativity independently of genre. the adversarial learning process achieves an f1 score of 0.87 / 0.97 for narrativity classification using pos / bert features, while keeping genre prediction accuracy low at 0.18 / 0.19 on our manually annotated test set. these results demonstrate the model's ability to extract narrativity-relevant features with minimal genre interference. next we test our data alongside two other datasets discussed in prior work: storyseeker @xcite and the r/bereavement data @xcite . once again using our svm classifier and two feature representations, we rotate through all train / test splits and measure f1 scores for each scenario. as shown in figure 2 , there is high within-group accuracy, coupled with considerable decline on out-of-domain data. the exception is the storyseeker data which generalizes well to our data though the reverse is not the case. the r/bereavement data shows the lowest generalizability of all sets. for our unsupervised training, we employ gpt-4 (gpt-4-0613) as our frontier model and llama3.1:8b as our open-weight model. we use a zero shot prompting framework: "is this passage from a story? answer only with a number: 1 if yes, 0 if no." for the scalar task, we use similar prompts to what our human annotators received (e.g. "how strongly do you agree with this statement: this passage is organized around sequences of events that occur over time"). ## conclusion to advance the goal of narrative detection, we introduce the narradetect dataset, which formalizes "narrativity" theoretically and includes two sub-corpora. the large corpus captures diverse narrative practices across contexts, while the smaller, manually annotated dataset provides a novel scalar framework to address intra-narrative heterogeneity, grounded in foundational narrative theory @xcite . our models achieve high predictive accuracy, though supervised models show performance drops on out-of-domain data, warranting further investigation. unsupervised llms, however, demonstrate robustness across narrative datasets and align well with human annotations, reinforcing the validity of our framework. we hope narradetect enriches existing resources and aids in benchmarking llms for narrative understanding.