The Empirical Variability of Narrative Perceptions of Social Media Texts (2024)

Abstract: Most NLP work on narrative detection has focused on prescriptive definitions of stories crafted by researchers, leaving open the questions: how do crowd workers perceive texts to be a story, and why? We investigate this by building StoryPerceptions, a dataset of 2,496 perceptions of storytelling in 502 social media texts from 255 crowd workers, including categorical labels along with free-text storytelling rationales, authorial intent, and more. We construct a fine-grained bottom-up taxonomy of crowd workers’ varied and nuanced perceptions of storytelling by open-coding their free-text rationales. Through comparative analyses at the label and code level, we illuminate patterns of disagreement among crowd workers and across other annotation contexts, including prescriptive labeling from researchers and LLM-based predictions. Notably, plot complexity, references to generalized or abstract actions, and holistic aesthetic judgments (such as a sense of cohesion) are especially important in disagreements. Our empirical findings broaden understanding of the types, relative importance, and contentiousness of features relevant to narrative detection, highlighting opportunities for future work on reader-contextualized models of narrative reception.

## introduction identifying stories in social media texts provides a lens through which we can study how individuals and communities process and communicate experiences @xcite @xcite . however, despite narrative's omnipresence in our private @xcite and public lives @xcite , its generality and multi-faceted complexity makes modeling and detecting it a major challenge in nlp @xcite @xcite . thus far, most approaches to narrative detection in nlp have involved a small number of researchers and/or trained students who use a prescriptive annotation guideline concerned with textual features to obtain a gold label. in this way, prior efforts exhibit relative uniformity in terms of annotator types (researchers, sometimes alongside trained students), number of annotators (single to few), annotation paradigm (prescriptive), narrative feature type (textual), and label aggregation (single gold). 1 these prescriptive approaches can lead to annotations (and, by extension, trained models) that reflect singular definitions of storytelling, failing to reflect the true variability of narrative perceptions. there are alternative approaches worth considering across these dimensions, such as crowd workers' perceptions, descriptive (codebook-free) annotations @xcite , extra-textual features concerned with reader response, and different ways of handling human label variation @xcite . exploring these alternatives would help determine the extent to which narrative perceptions are invariant across substantially different annotation contexts, which affects the generalizability of llms and other models trained on prescriptive labels or prompted with prescriptive codebooks using in-context learning. moreover, insights into which features shape or divide crowd perceptions of narrative can guide model development concerned with the psychological, pragmatic, and social aspects of narrative reception. to these ends, we investigate descriptive perceptions of english-language stories from crowd workers, broadening beyond prescriptive labeling from researchers. we present storyperceptions, a dataset of 2,496 free-text perceptions of storytelling from 255 crowd workers, based on the 502 social media texts in the storyseeker dataset @xcite . through open coding of the crowd workers' perceptions, we develop a detailed taxonomy of 30 codes representative of the discourse categories (e.g., explanation), textual features (e.g., events), and extra-textual features (e.g., suspense) that crowd workers associate with narratives. using the taxonomy, we augment the dataset with meta-annotations of both the crowd workers' label rationales and perceptions of authorial goal. with storyperceptions, we explore the following questions. first, to bring a broad and fresh set of perspectives to the narrative detection task, we ask [rq1] what are crowd workers' descriptive perceptions of storytelling in social media texts? second, to illuminate the degree and nature of variation in descriptive perceptions, we ask [rq2] how do narrative perceptions differ among crowd workers? third, and finally, to understand how invariant perceptions are across annotation contexts, we ask [rq3] how do narrative perceptions differ across prescriptive labels from researchers, descriptive annotations from crowd workers, and predictions from llmbased classifiers? pairwise comparison of descriptive crowd annotations with @xcite prescriptive labels provides insight into the generalizability of prescriptive approaches. our motivation for comparing human annotations with llms is that deep learning models have been argued to capture "average" perceptions of concepts @xcite or otherwise fail to accurately represent the distributions of human opinions for many tasks @xcite . however, the extent to which this is true for subjective concepts like stories is not fully understood. we eval-uate, for the first time, how well descriptive story annotations from llms align with human labels from different contexts (crowd workers' descriptive labels and researchers' prescriptive labels). while we do not consider any set of human annotations as the gold standard, understanding the alignment between llm and crowd worker annotations is crucial for future computational social science research on narrative perceptions in social media. our comparisons illuminate the validity and risks of using llm-assisted annotations in this domain. we find that while crowd workers' label rationales frequently refer to a few core textual features (events, characters, plot, setting), they also appeal to cognitive and aesthetic responses (sense of conflict, cohesion, feeling like a story, etc), often alongside references to textual features. regarding variation among crowd workers, we find that disagreements are more likely to revolve around holistic assessments (e.g., about plot structure) rather than more straightforward textual features. furthermore, comparing crowd labels to the prescriptive labels from researchers or llm-based predictions shows that crowd workers have the highest requirements concerning sequential events and plot structure across annotator types. finally, we find that models in the gpt-4 family @xcite are typically less likely to identify stories in texts that describe abstract activities (e.g., habits, behaviors, processes) relative to human annotators. altogether, this study offers new insights from crowd workers on the narrative detection task, underscores the intricate nature of narrative discourse through its diverse features and varying levels of inter-annotator agreement, and identifies both opportunities and important questions for future work at the intersection of narrative modeling, reader response, and llm-assisted annotation. ## the storyperceptions dataset our texts are drawn from the storyseeker dataset @xcite , which contains posts and comments sampled from reddit communities @xcite . 2 the dataset includes researcher-annotated binary labels indicating whether a text contains a story, based on a prescriptive codebook that emphasizes characters and event sequences ("a story describes a sequence of events involving one or more people"). we refer the reader to @xcite for a complete discussion of their codebook. building storyperceptions on the story-seeker dataset has several advantages: the texts originate from diverse online communities, providing a broad range of features relevant to narrative perception; they are nearly evenly split between stories and non-stories; and the prescriptive labels, grounded in the researchers' codebook, serve as a useful comparison point with the descriptive labels and rationales collected from crowd workers. ## examining crowd perceptions [rq1] to understand the crowd workers' perceptions of storytelling, we first analyze the feature and discourse codes associated with stories and examine their co-occurrences. ## disagreement among the crowd [rq2] to further examine crowd perceptions of storytelling, we examine divergent perceptions among crowd workers. in storyperceptions, we observe substantial disagreement among the descriptive crowd labels (@xmath0.426) @xcite , compared to antoniak et al.'s (2024) prescriptive annotations (@xmath1.655), underscoring the subjectivity of the descriptive narrative detection task in the absence of prescriptive guidelines. ## disagreement across annotation contexts [rq3] finally, we use paired sets of annotations to explore disagreements across annotation contexts, including descriptive crowd annotations, prescriptive labels from researchers, and predictions from llms, including several models from the gpt-4 family 9 and the instruction-tuned llama 3 8b @xcite . we describe our prompts in appendix e. as shown in llama 3 8b achieves the lowest agreement scores with human annotators, primarily due to an overprediction of stories, which indicates the model's susceptibility to confirmation bias. examining story labeling rates, we find that prescriptive labeling from researchers and descriptive gpt-4 predictions identify more stories than crowd workers. the feature code most relatively prevalent in scenarios where crowd workers do not identify a story (but researchers or gpt-4 do) 10 however, with evidence that larger models retain longtail knowledge better than smaller models @xcite , we could conjecture that if gpt-4 is the largest among these models, its relatively larger size could partially explain its apparent retention of more nuance for the narrative detection task from training. is not_plot_sequence. 11 this suggests that crowd workers require greater structural complexity (e.g., with respect to the sequential chain of events comprising a plot) in stories. 12 although fully explaining divergences across annotator types and the influence of the term 'story' on crowd worker annotations is beyond our scope, we offer initial conjecture based on common intuitions and empirical findings. although storytelling is woven into much of human discourse, we do not always name it as such. typically, when we encounter the word 'story' (e.g., 'let me tell you a story'), there is a connoted expectation of significance or interestingness to the story. thus, crowd workers may be more hesitant to attach the term 'story' to a text unless that text surpasses a baseline threshold for interestingness and purpose, differing from antoniak et al.'s (2024) prescriptive labels, which focus on character and event sequences rather than story purpose or interestingness. concerning feature-level preferences in cases of disagreement between a human annotator type (researcher or crowd) and a gpt-4 family of models (including gpt-4o, gpt-4t, and gpt-4), the feature code most relatively prevalent in disagreements is behavior_strategy. this bolsters and extends our finding in section 5.2 regarding the contentiousness of behavior_strategy among crowd workers. evidently, behavior_strategy is also contentious across annotation contexts. excepting the gpt-4 / crowd comparison, where the disparity is less pronounced, the models in the gpt-4 family frequently avoid labeling texts as stories when they describe behaviors, habits, or plans, relative to human annotators. while this tendency ostensibly aligns with antoniak et al.'s (2024) codebook guidance not to conflate general descriptions of behavior with events bounded in space and time, this guidance does not preclude other aspects of the text from justifying a story label. we conclude that models in the gpt-4 family lack nuance in their ability to identify stories that both (1) contain general descriptions of behavior and (2) particularized events and/or other features that contribute to storytelling, relative to human annotators who, in aggregate, approach these texts in a more balanced manner. rq2: how do narrative perceptions differ among crowd workers? we find that while crowd workers generally agree on basic textual features, their holistic assessments of complex textual features (such as plot) and extra-textual aesthetic judgments (such as sense of cohesion) can diverge from one another. additionally, distinguishing between events and more general descriptions of behavior is a particularly challenging and contentious aspect of narrative detection for crowd workers. rq3: how do narrative perceptions differ across prescriptive labels from researchers, descriptive annotations from crowd workers, and predictions from llm-based classifiers? through pairwise label comparisons across annotation contexts, we conclude that prescriptive labels from researchers and descriptive gpt-4 predictions can approximate certain aggregated crowd perceptions of narrative status reasonably well for short text-based social media posts. important qualifications include differing thresholds for structural complexity (with crowd workers having a stricter definition of plot) and a tendency of gpt-4 family models to diverge from human perceptions for texts that describe behaviors, habits, or abstract plans. ## conclusion & future work in this paper, we introduced the storypercep-tions dataset to bring crowd workers' descriptive perceptions to bear on the narrative detection task. complementing prior work that uses prescriptive annotations from a small number of researchers, our empirical findings highlight the types, relative importance, and contentiousness of a broad range of features for narrative perception. our study points to several opportunities for further research. first, while we looked at simple co-occurrence of features, deeper statistical analysis and experimentation could more clearly illuminate interactions and causal relationships between features. while we offer a pilot experiment in training a story classifier with aggregated labels from crowd workers (see appendix f.5), future studies could survey more sophisticated methods to incorporate multiple perceptions during training. further research could also integrate a broader range of features from our taxonomy-both textual and extra-textual-into narrative detection models. moreover, replicating our crowd studies using alternative terminology, such as 'narrative,' would help assess whether our findings generalize or reveal important distinctions across different terms. finally, as attention in computational narrative understanding broadens beyond textual features to include reader reception, there are major outstanding ethical and epistemic questions concerning the use of llm-assisted annotation; an exploration of these questions tailored for the subfield of computational narrative understanding would be invaluable.