Story Embeddings — Narrative-Focused Representations of Fictional Stories (2024)

Abstract: We present a novel approach to modeling fictional narratives. The proposed model creates embeddings that represent a story such that similar narratives, that is, reformulations of the same story, will result in similar embeddings. We showcase the prowess of our narrative-focused embeddings on various datasets, exhibiting state-of-the-art performance on multiple retrieval tasks. The embeddings also show promising results on a narrative understanding task. Additionally, we perform an annotation-based evaluation to validate that our introduced computational notion of narrative similarity aligns with human perception. The approach can help to explore vast datasets of stories, with potential applications in recommender systems and in the computational analysis of literature.

## introduction narrative understanding is a field that has received much attention in the last few years. various approaches have tested models either on narrativebased question answering tasks or performed intrinsic evaluations, such as narrative cloze evaluations, where models need to predict missing events in a sequence. in this work, we seek to address the topic of story embeddings with a focus on narrative, meaning representations that prioritize the aspect of "what" is happening rather than the surface-level information of "how" it is being told. for example, a love story with a specific twist can be set in different settings (outer space or countryside), with a different cast (e.g., different names and some different traits for all characters), or in a shortened version, without fundamentally changing the narrative. after altering the story's final twist, the new narrative could still be considered similar without being identical. researchers in the acl community have, in the context of fictional works, often used the terms narrative and story without a clear distinction (e.g., @xcite . the field of narratology has a multitude of competing terms to offer, specifically to distinguish between the order of events as presented to the reader (commonly used terms are syuzhet, plot and discours) and that of the actual happenings in the narrated world (commonly used terms are fabula, story and histoire) @xcite . in this work, we refer to the story as the entirety of the narration abstracted from the individual formulation, whereas we use narrative specifically to refer to the story's structure. thus, a narrative could broadly be seen as the order and relationship of events in the story, but it does not include other information, such as the setting, tone, and style of the story. this work presents a contrastive-learning-based approach for training story embeddings using a preexisting dataset. we assume that any fictional text can be represented by its summary for our purposes of modeling the narrative. while there are various characteristics of a story that can not be gleaned from a summary, such as the style and mood of a text, the narrative is core to what is represented in a summary. thus, summaries are the perfect testing ground for narrative embeddings, although an expansion to full texts in the future is desirable. it has been observed that retellings of -specifically fairytales -have recently increasingly been published, with many retellings changing the setting to a modern-day one or introducing the representation of minorities @xcite . as such, they represent a structurally similar story, with a new setting and limited alterations to the narrative. other retellings, however, change the story significantly, sometimes merely retaining themes from the original work. on a limited scale, previous work has addressed the automatic identification of stories following the same plot @xcite . in this work, we consider this task as a possible application of story embeddings. ## related work a substantial line of work (e.g. @xcite @xcite has dealt with graph-based representations of narratives, specifically with predicting missing narrative triples and inferring schemas of commonly re-occurring narratives. @xcite take what can be considered a hybrid approach, building explicit networks but using contextual vector representations rather than lexical items to represent triples. similarly, using less contextual information, in prior work, we trained narrative triple embeddings based on narrative chains @xcite . following ever-increasing advancements in the field of language models and motivated by the information loss inherent to extracting narrative triples, this work seeks to apply a more distantly supervised approach to representing stories. our work builds on two previously released datasets @xcite . both datasets contain story summaries extracted from wikipedia. specifically, both seek to find different formulations of summaries for very similar stories. the movie remake dataset by @xcite contains a relatively small collection of summaries from multiple remakes of the same movie. in contrast, our previously released dataset, tell-me-again (hatzel and biemann, 2024), collects summaries from multiple wikipedia language versions of the same fictional work. the movie remake dataset only contains 266 summaries and is thus not suited for training, whereas tell-me-again contains roughly 30,000 stories. each story comes with up to five different summaries, originally extracted from multiple wikipedia language versions and automatically translated into english. the dataset additionally comes with a pseudonymized variant, explicitly created for training models that do not focus on entity names. in this variant, entity names are replaced in each summary by alternatives in an internally consistent manner. these pseudonymized versions are created using rule-based replacement strategies on top of a model-based coreference resolution system. rocstories is a dataset for testing commonsense reasoning, first released in 2016 @xcite with the introduction of the story cloze task. in the task, systems pick one of two sentences as the end of a five-sentence story. one choice is a logical conclusion to the story, but the other choice only matches in terms of vocabulary and is not a fitting conclusion to the story. as a result, humans can solve the story cloze task perfectly, but at the time of publication, the best-performing system in an accompanying shared task reached only around 75% accuracy. the original task formulation did not allow for supervised learning, providing only complete five-sentence stories without two choices as training data. the creation of semantic sentence representations with large language models (llms) has recently gained much interest. while @xcite train embeddings from last-token hidden states, it has been suggested that the causal attention mechanism in generative decoderonly models limits their effectiveness for embeddings @xcite . alternatives have been proposed in the form of adding bidirectional attention back into existing models @xcite or by duplicating the input sequence, thereby functionally allowing each token to attend to every other token @xcite . ultimately, the new approaches were shown to be more training-sample-efficient but did not show real inference quality gains over the extensively finetuned e5 model by @xcite . embedding approaches are typically focused on very short sequences of text, particularly individual sentences @xcite . doc2vec @xcite is a static-embedding-based approach to document embeddings. while it was primarily evaluated on short segments, it does not have a limitation regarding the input size, a common constraint in transformerbased approaches. the definition of what exactly constitutes narrative similarity has been addressed by @xcite in their corresponding codebook @xcite . in a pairwise similarity annotation task, they explicitly ask annotators to consider the narrative schemas and to ignore the specific names of entities, only considering their roles. they do not define an exact measure of how distances between schemas are determined, nor do they instruct annotators to write down explicit schemas. despite these limitations, they achieve comparatively good inter-annotator agreement (0.69 krippendorf's α) on narrative similarity of news articles. ## our approach our model, called storyemb, is a causal language model whose last token representation is fine-tuned on similarity tasks using augmented data. our model is trained to produce representations that are similar for multiple summaries of the same story. as a foundation model, we use mistral-7b @xcite . specifically, we use e5 @xcite , an adapter-finetuned variant, trained using synthetic data, for similarity modeling. as story similarity is a complex task, we assume that a more capable model would perform better; due to hardware constraints, we chose a 7b parameter model. we train our model using gradient cache @xcite to enable large batch sizes on limited hardware while reaching identical results to traditional similarity training. in training, we optimize for reducing the cosine similarity between pairs of summaries labeled as the same while maximizing the cosine similarity between those pairs that, by nature of belonging to different works, are implicitly labeled as different. our approach follows @xcite in using contrastive mse-loss for similarity training. we use a batch size of 1000 positive pairs and in-batch negatives. for the optimizer, we use adam with a learning rate of 5 × 10 -5 and perform early stopping on a subset of pseudonymized summaries from the tell-me-again dataset. the training is limited to the adapter parameters and, as we are training based on their weights, we follow @xcite and use lora with rank @xmath0. while our training setup differs in various details (we use a different loss and do not employ hard negatives), the training can be considered a continued fine-tuning of e5 with a similar objective, just focusing on narrative similarity. our training data is sampled from the tell-me-again dataset but limited to only summaries with a minimum of 10 and a maximum of 50 sentences in length. this is motivated by the desire to exclude (a) very short synopses and loglines on the low end and (b) documents that are too memory-demanding on the high end. the length limit could be subject to further experimentation in the future. we evaluate whether the data augmentation approach -replacing names with alternative ones in a consistent manner -proposed by hatzel and biemann (2024) can improve the performance of a similarity model. to this end, we compare an augmented version of our model, trained on pseudonymized versions of the original summaries, and a non-augmented version, trained on the original summaries. following the e5 paper, we add a query prefix to each document. through manual exploration on the development set, we selected the query, "retrieve stories with a similar narrative to the given story: ". while many of the original applications of e5 follow an asymmetric setup where the query and the document are encoded using separate prompts, our prompt aligns well with one of their evaluation prompts: "retrieve tweets that are semantically similar to the given tweet". behnamghader et al. ( 2024 ) have recently introduced a more sample-efficient way, called llm2vec, to train llms for sentence reprentations. in preliminary experiments, we found, perhaps in part due to length limitations in training as a result of the full-attention setup, an llm2vecbased model to perform inferiorly to our model. ## experiments after training, we perform several downstream task experiments to explore the capabilities and characteristics of our narrative embeddings. three experiments test narrative retrieval capabilities (section 4.1). we also perform an experiment focused on narrative understanding (section 4.2). all our experiments in this paper are limited to english data. recall that our training data consists of pairs of story summaries automatically translated from various languages to english. ## results we also test a pre-trained doc2vec model (lau and baldwin, 2016) as a more traditional baseline with no inherent length limitation. outside of our own model's performance, it is interesting to see doc2vec outperform e5 by far on the pseudonymized version of the dataset; the static-embedding model exhibits no noticeable drop in performance from the standard to the pseudonymized setting (in fact, the results on the pseudonymized version are marginally better for all metrics). upon consideration, this is not surprising as the static word embeddings in doc2vec may have a hard time with generic entity names, especially personal names. while the performance increase on the pseudonymized texts is expected, it is surprising that, even for the non-pseudonymized texts, the model trained on augmented data performs better. as noted earlier, our model's training was stopped early based on the performance on the pseudonymized texts (for both model variants). the training finished after just three training steps (after seeing no improvements for two more steps), with each step taking roughly 1 h 40 min on two a100 gpus. in fact, the unaugmented model continues to improve on the non-pseudonymized data afterward, presumably due to an ever-increasing focus on entity names as a shortcut to solving the task. ## approximate attribution to inspect which aspects our storyemb model focuses on, we apply an attribution approach for sentence encoders @xcite . the approach builds on the idea of integrated gradients (sundararajan et al., 2017), extending the concept to siamese networks, specifically sentence encoders. in essence, the approach samples gradients along an interpolation from a semantically neutral sequence to the analyzed sequence, identifying features that the output is sensitive to. the result is a token-token matrix across two input sequences, signifying the contribution of any two terms to the overall similarity of the two sequences. @xcite operate only on models that use average pooling across tokens. we adapt their implementation to work with decoder-only models and use 50 interpolation steps. e5 employs last-token pooling, using the last token's hidden state as a sentence representation. as a result, the attribution scores are muddled: information needs to flow to the last token, leading to the majority of the similarity being explained by changes in the last token's representation. in an effort to get interpretable attribution scores despite the pooling approach, we display the delta in attribution scores from the e5 model to our storyemb model instead. figure 1 illustrates that our augmented-data approach does, in fact, place less emphasis on named entities than the vanilla e5 model. we compare similarity across the two sentences "alice wakes up." and "alice falls down.". generally, negative values in an attribution indicate that two specific tokens interact to reduce the similarity of the two sentences. the attribution scores figure 1 tively less emphasis on said value. in this case, our fine-tuned model places much less importance on the name "alice" than the original e5 model does. to generalize from the single example, we collect attribution scores for 50 random sentences from the sts benchmark dataset's test set @xcite . we collect average attribution scores for part-of-speech and named-entity tags, showing the results in we have restricted the analysis to single sentences as it is computationally expensive and could not feasibly be performed on entire stories. ## qualitative exploration in general, the manual evaluation of similarity has the issue that many datasets contain some form of duplicates, leading to the issue that trivially similar narratives are retrieved first. we explored similar segments, using the approach from section 4.1.4, excluding trivially similar texts. we found many instances of meaningfully narratively similar texts being retrieved by storyemb. in figure 2 , we show a pair of segment summaries from the detective novel dataset, with the segment from "the triumphs of eugene valmont" being considered the 17th closest, by storyemb, to the "in the fog" segment. e5, meanwhile, only considers it to be the 221st closest segment. the two segments share a very similar narrative of a necklace theft with a subsequent arrest but lack any shared named entities. ## conclusion in this work, we presented an approach to creating embeddings that represent stories, specifically their narratives. our storyemb model, trained on the tell-me-again dataset, shows state-of-the-art performance on both the corresponding test split and on the movie remake dataset @xcite , far outperforming both recent llm-based models and the static-embedding baseline. we demonstrate the model's retrieval capability in practice on summaries of passages of literary texts. further, we explore the retrieval of retellings on a novel small-scale dataset, opening up another potential application avenue. on the movie remake dataset, we clearly demonstrate the effectiveness of the tell-me-again data augmentation approach, producing better results on both pseudonymized and non-pseudonymized versions of the texts. similarity score attribution indicates that the data augmentation techniques employed have the desired effect: making the model place less emphasis on names.