Where Do People Tell Stories Online? Story Detection Across Online Communities (2024)

Abstract: Story detection in online communities is a challenging task as stories are scattered across communities and interwoven with non-storytelling spans within a single text. We address this challenge by building and releasing the StorySeeker toolkit, including a richly annotated dataset of 502 Reddit posts and comments, a detailed codebook adapted to the social media context, and models to predict storytelling at the document and span levels. Our dataset is sampled from hundreds of popular English-language Reddit communities ranging across 33 topic categories, and it contains fine-grained expert annotations, including binary story labels, story spans, and event spans. We evaluate a range of detection methods using our data, and we identify the distinctive textual features of online storytelling, focusing on storytelling spans, which we introduce as a new task. We illuminate distributional characteristics of storytelling on a large community-centric social media platform, and we also conduct a case study on r/ChangeMyView, where storytelling is used as one of many persuasive strategies, illustrating that our data and models can be used for both inter- and intra-community research. Finally, we discuss implications of our tools and analyses for narratology and the study of online communities.

## introduction automatic detection of stories online is an important but complex task, as online stories can be topically varied and embedded within a longer document. for example, in several challenges have hindered the large scale analysis of storytelling online. to bridge this gap, we formalize the task of story detection by publicly releasing 1 storyseeker: the first dataset, codebook, and models for detecting stories and story boundaries across diverse online communities. in doing so, (1) we release a public, detailed codebook that can be used across a variety of data types, (2) we explicitly tie our story definition to the related task of event detection @xcite by an- ## related work narratology storytelling is a broad concept that has been explored by fields as diverse as economics (shiller, 2020), literary theory (bal and van boheemen, 2009), sociology @xcite , and nlp @xcite @xcite . arriving at agreement on a single story definition is a challenging task. in the field of narrative theory ("narratology"), storytelling has been defined by its emphasis on sequences of events, aspects of change and/or conflict, and embodiment or "feltness" @xcite @xcite . as @xcite writes, "narrative roots itself in the lived, felt experience of human or human-like agents interacting in an ongoing way with their surrounding environment. @xcite propose a minimum schema for capturing storytelling that emphasizes the presence of ten basic elements, including an agent, action, and location in time and space. storytelling datasets operationalizing storytelling schemas for nlp is an active area of research @xcite @xcite . nlp research on narratives and storytelling has deployed a wide range of definitional dimensions, the most common being sequences of events arranged temporally, causally related events leading to resolutions, and the presence of entities or characters, while a smaller number include a rhetorical purpose for the text and world building @xcite @xcite @xcite . see a.6 for an enumeration of story definition features used in prior work. prior work in story annotation has mostly focused on specific discourse domains such as healthcare @xcite , argumentation @xcite , book publishing @xcite , bereavement @xcite , or blogs (dos @xcite . to our knowledge, all prior work on story detection has focused on passageor document-level annotations, except for a set of preliminary annotation guidelines for embedded narratives without an associated dataset by @xcite and a domain-specific study by @xcite . much past annotation work is not publicly available @xcite , due to data agreements, sensitive content, and other constraints. we expand on these works by widening our view to many communities and topics, by using a spanbased approach, and by publicly releasing our annotations. our annotation guidelines presented in §3 most closely resemble the guidelines provided by @xcite and @xcite , which rely on events and characters, and draw most inspiration from the narrative annotation guidelines provided by @xcite and the event annotation guidelines provided by @xcite . story detection most prior work on automatic story detection has focused on using feature-based classification approaches, relying on features like n-grams, pos tags, and coreference chain length @xcite @xcite @xcite . see a.6 for an enumeration of story features used for prediction in prior work. these studies either had an explicit goal of using interpretable methods or were conducted prior to the arrival of large language models (llms). two newer works have attempted narrative detection using llms @xcite . both studies hand-annotate a small set of texts from online healthcare communities with binary story labels, and both find that fine-tuned bert-based models perform better than classical models at the document-level story prediction task. ## designing a story codebook our guidelines must work across diverse online communities while recognizing features that make storytelling in these contexts potentially unique from storytelling in literary or visual contexts. our interdisciplinary team worked iteratively to code data, compare labels, and design a codebook containing a story description that fit our context (diverse online communities) and research goal (measuring storytelling). drawing from prior work, we focused our story identification guidelines on agent-centered events, leading to a two-step annotation process: first, the annotation of event spans, and second, the annotation of story spans. we provide our full codebook instructions in a.7. our strong emphasis on events while annotating story spans is novel and adds consistency to the annotation of a diverse set of texts. it also allows us to build on prior work on event span annotation @xcite by reconsidering this task within the context of storytelling, leading us to a more flexible description of events than prior work. importantly, our codebook should be considered not as presenting complete or final definitions of either stories or events but rather providing actionable advice that captures features useful for the consistent annotation of these phenomena. ## creating a multi-community corpus we developed the storyseeker dataset, a collection of reddit posts that introduces the story span annotation task and covers a large number of online communities. to detect storytelling across diverse settings, we required training and evaluation data that contained both storytelling and non-storytelling communication mixed together by topic, context, and even within a single document. data source we sample texts from webis-tldr-17, a corpus of 3.8 million reddit posts and comments @xcite . this dataset was designed for summarization research, and so unlike a random sampling of reddit data, each text in this dataset contains contentful texts (texts with coherent sentences leading to a summary statement, rather than images, links, or other kinds of texts). sampling across the 500 most frequent subreddits in the dataset, we follow an open-coding approach to categorize the subreddits into 33 categories (e.g. professional advice). we use these categories to remove sensitive (e.g., r/confessions), toxic (e.g., r/pettyrevenge), explicit (e.g., r/sex), and non-english (e.g., r/mexico) subreddits from the human annotation tasks, and we use these categories to structure our analyses ( §6). we show the full set of categories and their member subreddits in a.2. we sample a balanced set of five texts from each of the filtered subreddits, downsampling gaming foot_2 and requiring that each text contains at least 100 tokens and no more than 500 tokens. annotation process our annotation process included highlighting both story and event spans, as we found that first identifying events was crucial in making the story span decision. after several rounds of codebook construction, two of the authors independently annotated the target data using prodigy; each annotator annotated the full set of texts. 3 annotation was challenging, as texts can contain many events, and we included frequent rounds of group deliberation for difficult examples as well as a final round of discussion to arrive at a single consensus label for each document. final corpus our final storyseeker dataset includes 502 texts with event-and story-spans (see ## developing a story detection model we release two fine-tuned classifiers as part of storyseeker to detect storytelling documents and spans across diverse online communities. prediction methods to predict the presence of storytelling, we evaluate a variety of prediction models and methods to cover reasonable baselines that vary in cost, gpu usage, and need for labeled data. these methods include a svms baseline 3 https://prodi.gy/ using tf-idf features, a fine-tuned roberta foot_3 model @xcite , and zero-shot and fewshot gpt-4 prompting @xcite . for the document classification task, we predict the binary presence of storytelling in the document, while for the span detection task, we predict whether each token is part of a story. we divide our expert-annotated data into a training/prompting set of 301 texts, a validation set of 100 texts, and a test set of 101 texts. we rely on the consensus labels, i.e., if the annotators agree after discussion that a text contains a story span, then we use this as either (i) a positive instance of storytelling or (ii) an indicator of whether to include the union of the annotator's spans for token prediction. we experimented with different prompts using our validation set, including variations of the below task that included examples, chain-of-thought questions, and guidelines. ## analysis based on prior literature and our iterations of annotation and codebook refinement, we identify a set of features that we expect may be associated with storytelling in social media. these include entity and pronoun rates @xcite , events @xcite @xcite , verb tense, concreteness @xcite , whether a text is a post or comment, and average sentence length. see a.9 for figure 2 : some subreddits contain stories on specific topics, while others draw a wide range of storytelling topics. we show the 500 most frequent subreddits, colored by category. the y-axis shows the rate of storytelling in the subreddit (predicted by our classifier), and the x-axis shows the distinctiveness of the subreddit's vocabulary (calculated only for texts containing stories). see figure 7 in the appendix to see overall category rankings and variation. using our consensus labels to compare story spans with non-storytelling texts, we test whether each feature is more prominent in one group than the other. specifically, we run t-tests on the features, applying the holm method @xcite to account for multiple comparisons (see the tests indicate that, in decreasing order of effect size, the frequency of our expert-annotated events, past-tense verbs, realis events @xcite , first-person singular pronouns, third-person singular pronouns, entity mentions, and concrete terms are significantly more frequent in stories. conversely, present-and future-tense verb tenses, the text type being a comment (instead of a post), and second-person pronouns are significantly less frequent in stories. ## case study: r/changemyview we demonstrate storyseeker's usefulness not only when measuring storytelling across communities but also when measuring storytelling across topics within a single community, by sharing a focused case study of how one subreddit uses storytelling as a rhetorical strategy. figure 3 : the debate topics plotted by the overall rate of storytelling arguments they receive (y-axis) vs. the rate of storytelling in winning comments (x-axis). some post topics, like music, elicit many storytelling comments but very few of those comments are persuasive, while other topics like race elicit fewer stories but responses containing stories more often tend to be persuasive. see figure 4 in a.5 for detailed views of both axes with standard deviation bounds. the poster, and this community has frequently been the subject of research about persuasion @xcite . using the winning arguments corpus @xcite , we explore how storytelling is used to persuade. figure 3 shows the post topics that elicit comments with more or less storytelling and whose storytelling comments are more or less likely to receive a winning "delta" point, indicating that the poster was convinced by the comment's argument. topics are trained on the posts via a latent dirichlet allocation (lda) @xcite model (@xmath1, and after examination of the highest probability words and documents, we assign each topic a descriptive name. the y-axis shows the difference between the mean post topic probabilities for storytelling and non-storytelling comments, while the x-axis shows the difference between the proportion of winning comments that include storytelling and the proportion of non-winning comments that include storytelling. more details about our methods and the topic model are given in a.3. we find that social and personal topics like lifestyle choices and music receive more storytelling comments, while abstract topics like economic systems receive fewer storytelling comments. however, topics that receive more storytelling comments are not necessarily the same topics where storytelling comments are more likely to win delta points. examining the outliers in figure 3 , we find that for topics like drug legalization and words & symbols, storytelling comments are more likely to be persuasive even though commenters are less likely to use storytelling in their arguments. conversely, topics like social norms receive many storytelling comments but those comments are less likely to win. our results add fine-grained distinctions to prior work that has examined "personal and anecdotal" arguments on r/changemyview (poulsen and dedeo, 2023). these patterns likely are driven not only by the topic's subject matter but also the particular framing used in this subreddit; topics like ethical animal treatment could be imagined to elicit to many stories, but do not in this community. ## discussion implications for narratology the features that are most strongly associated with storytelling support prior work's emphasis on storytelling's basis in agent-centered, event-driven forms of communication that are grounded in concrete settings. these features are believed to support social coordination by fostering joint attention around virtual events (known as the "deictic theory" of storytelling; @xcite . nevertheless, considering narratology's historical focus on long-form literary forms like the novel, elinor ochs's famous assertion that the "mundane conversational narratives of personal experience constitute the prototype of narrative activity rather than the flawed byproduct of more artful and planned narrative discourse" highlights the opportunities for narrative theorists to attend to "small stories" @xcite , such as social media stories profiled here. we hope that the storyseeker codebook, dataset, and models can serve as a cross-disciplinary bridge for narrative theorists, computational social scientists, and nlp researchers interested in large-scale analysis of narrativity across social media. implications for storytelling in online communities compared to prior work that detects storytelling in online healthcare communities @xcite , our model predicts an overall lower rate of storytelling, likely owing to the diversity of topics in our dataset. our classification performance is lower than the performance reported in those works, despite using similar finetuning techniques, signalling the challenge of iden-tifying stories across diverse topics and communities. while storytelling may be a sign of trust as a form of self-disclosure @xcite , we do not find a significant relationship between the overall storytelling rate in a specific community and the same community's toxicity (measured via the perspective api foot_4 ), size (number of members), or user activity (posts per member). finally, we provide some additional experiments about overall post and comment patterns in a.4. ## conclusion we formalize the task of story detection and story span detection by releasing storyseeker, a framework that contains a dataset, codebook, and finetuned models for online story detection. our analysis showcases how these tools can be used both across many diverse reddit communities and to probe a specific community. using our annotated story spans, we provide the first results indicating how story spans differ from non-storytelling spans cooccurring in the same texts, and we map where people tell stories. we hope our tools and analysis spark further research into online storytelling.