Modeling Emotional Trajectories in Written Stories Utilizing Transformers and Weakly-Supervised Learning (2024)

Abstract: Telling stories is an integral part of human communication which can evoke emotions and influence the affective states of the audience. Automatically modeling emotional trajectories in stories has thus attracted considerable scholarly interest. However, as most existing works have been limited to unsupervised dictionary-based approaches, there is no benchmark for this task. We address this gap by introducing continuous valence and arousal labels for an existing dataset of children’s stories originally annotated with discrete emotion categories. We collect additional annotations for this data and map the categorical labels to the continuous valence and arousal space. For predicting the thus obtained emotionality signals, we fine-tune a DeBERTa model and improve upon this baseline via a weakly supervised learning approach. The best configuration achieves a Concordance Correlation Coefficient (CCC) of .8221 for valence and .7125 for arousal on the test set, demonstrating the efficacy of our proposed approach. A detailed analysis shows the extent to which the results vary depending on factors such as the author, the individual story, or the section within the story. In addition, we uncover the weaknesses of our approach by investigating examples that prove to be difficult to predict.

## introduction stories are central to literature, movies, and music, but also human dreams and memories @xcite . storytelling has received widespread attention from various disciplines for many decades @xcite , e. g., in the fields of psychology @xcite , cognitive sciences @xcite , and history @xcite . a crucial aspect of stories is their emotionality, as stories typically evoke a range of different emotions in the listeners or readers, which also serves the purpose of keeping the audience interested @xcite . several efforts have been made to model emotionality in written stories computationally. however, these studies have often been constrained to dictionary-based methods @xcite . in addition, existing work often models emotions in stories on the sentence level only @xcite without taking into account surrounding sentences, missing out on important contextual information. in this study, we address the aforementioned issues by employing a pretrained large language model (llm) to predict emotionality in stories automatically. in combination with an emotional text-to-speech (tts) system @xcite , our system could serve naturalistic human-machine interaction, educational, and entertainment purposes @xcite . for example, stories could be automatically read to children @xcite by voice assistants. furthermore, the prediction of emotions in literary texts is of interest in the field of digital humanities @xcite , especially in computational narratology @xcite . we conduct our experiments on the children's story dataset created by @xcite . specifically, our contributions are the following. first, we extend the annotations provided by @xcite and, subsequently, map the originally discrete emotion labels to the continuous valence and arousal @xcite space (cf. section 3). we then employ debertav3 in combination with a weakly-supervised learning step to predict valence and arousal in the stories provided in the dataset (cf. section 4). to the best of our knowledge, our work is the first to model emotional trajectories in stories over the course of complete stories, also referred to as emotional arcs, using supervised machine learning, and, in particular, llms. while predicting such valence and arousal signals is common in the field of multimodal affect analysis @xcite @xcite , it has not been applied to textual stories, yet. ## related work various unsupervised, lexicon-based approaches to model emotional trajectories in narrative and literary texts have been proposed. with a lexicon-based method, @xcite identified six elementary sentiment-based emotional arcs such as ragsto-riches in a corpus of about 1,300 books. @xcite generate lexicon-based emotional arcs and demonstrate their usability in predicting the perceived literary quality of novels. further examples include the works of @xcite , @xcite , @xcite and @xcite . while these previous works use dictionaries to directly predict emotionality, we only utilize them to map existing annotations into the valence/arousal space. moreover, a range of datasets of narratives annotated for emotionality exists. in a corpus of 100 crowdsourced short stories, @xcite provided annotations both for character emotions as well as for emotions evoked in readers. the dataset for emotions of narrative sequences (dens) @xcite ) contains about 10,000 passages from modern as well as classic stories, labeled with 10 discrete emotions. in the authors' experiments, fine-tuning bert @xcite proved to be superior to more classic approaches such as recurrent neural networks (rnns). the relational emotion annotation (reman) dataset @xcite comprises 1,720 text segments from about 200 books. these passages are labeled on a phrase level regarding, among others, emotion, the emotion experiencer, the emotion's cause, and its target. @xcite conducted experiments with bilstms and conditional random fields (crfs) on reman. the stanford emotional narratives dataset (send) @xcite is a multimodal dataset containing 193 video clips of subjects narrating personal emotional events, annotated with valence values in a time-continuous manner. the corpus of children's stories @xcite we are using for our experiments is originally labeled for eight discrete emotions (cf. section 3). @xcite modeled emotional trajectories in a subset of this corpus, while in @xcite , the authors conducted machine learning ex-periments with several handcrafted features such as sentence length and pos-tags as well as bag of words. while the corpus has frequently served as a benchmark for textual emotion recognition, scholars have so far limited their experiments to subsets of this dataset, selected based on high agreement among the annotators or certain emotion labels. examples of such studies include an algorithm combining vector representations and syntactic dependencies by @xcite , the rule-based approach proposed by udochukwu and @xcite , and a combination of convolutional neural network (cnn) and long short-term memory (lstm) introduced by @xcite . no existing work, however, aims at modeling the complete stories provided in the dataset. ## data we choose the children's story dataset by @xcite , henceforth referred to as alm, for our experiments. from the mentioned datasets, the alm dataset is the only suitable one as it is reasonably large, comprising about 15,000 sentences, and contains complete, yet brief stories, with the longest story consisting of 530 sentences. moreover, the data is labeled per sentence, allowing us to model emotional trajectories for stories. we extend the dataset by a third annotation, as described in section 3.1, and modify the originally discrete annotation scheme by mapping it into the continuous valence/arousal space (cf. section 3.2). originally, the dataset comprises 176 stories from 3 authors. more precisely, 80 stories from the german brothers grimm, 77 stories by danish author hans-christian andersen, and 19 stories written by beatrix potter are contained. every sentence is annotated with the emotion experienced by the primary character (feeler) in the respective sentence, and the overall mood of the sentence. for both label types, two annotators had to select one out of eight discrete emotion labels, namely anger, disgust, fear, happiness, negative surprise, neutral, positive surprise, and sadness. for a detailed description of the original data, the reader is referred to @xcite . we limit our experiments to predicting the mood per sentence, as it refers to the sentence as a whole instead of a particular subject. ## experimental setup we fine-tune (cf. section 4.1) the 304m parameter large version of debertav3 @xcite , additionally utilizing a weakly supervised learning approach (section 4.2). further details regarding the computational resources can be found in appendix e. ## results ft alm [ccc↑] ft gbalm [ccc↑] ft gbalm + alm [ccc↑] valence arousal valence arousal valence arousal c dev test dev test dev test dev test dev test dev test 0 .6798 .6972 .5576 .5904 .6967 .7134 .5744 .6109 .7007 .7184 .5793 .6168 1 .7522 .7538 .6206 .6593 .7658 .7782 .6365 .6782 .7712 .7842 .6436 .6873 2 .7746 .7812 .6404 .6752 .7857 .7986 .6541 .6942 .7905 .8024 .6626 .7018 4 .7924 .7881 .6550 .6762 .7997 .8056 .6665 .6989 .8059 .8138 .6742 .7098 8 .7983 .7988 .6613 .6798 .8098 .8141 .6736 .7027 .8168 .8221 .6809 .7125 overall, the best results on the development set per prediction target and partition are boldfaced, and the best results for each context size are underlined. in figure 4 ) and c) additionally on alm (ft gbalm + alm, step 4 in figure 4 ). there is a clear trend for both arousal and valence to increase with larger cs. the models trained with a context size of 8 account for the best valence and arousal results in every set of experiments, e. g., ccc values of .8168 and .6809 for arousal and valence, respectively, on the development set when trained on both corpora. these are also the best results encountered overall. in contrast, the models with @xmath0, yielding e. g., only ccc values of .6798 (valence) and .5576 (arousal) on the development set in the alm-only configuration. this supports the assumption that the context of a sentence is oftentimes key to correctly assessing its mood. the gap between valence and arousal ccc values is in line with previous studies showing that text-based classifiers are typically better suited for valence prediction than for arousal prediction @xcite . further, our results demonstrate the benefits of the weakly supervised approach. training on gbalm always improves upon training on alm only, especially for smaller context sizes c. to give an example, both the valence and arousal ccc values on the development set increase by more than 1.5 for @xmath1. further tuning on alm afterward leads to additional performance gains for both prediction targets and all context sizes. however, the increase never exceeds 1 percentage point in comparison to training on gbalm. ## discussion we demonstrate the efficacy of our approach to model emotional trajectories via llms, achieving ccc values of .8221 and .6809 for valence and arousal on the test set, respectively. we find that considering a sentence's context is crucial for predicting its emotionality. furthermore, our analysis reveals the author-dependence of these results, which, in addition, vary from story to story. even within a story, certain parts (namely, beginning and ending) are often easier to predict than others. further analysis of our models' predictions uncovers additional challenges, such as assigning the correct role to protagonists and understanding the overall tone of a story. all these aspects combined shed light on the complexity of the task at hand. keeping this in mind, our methodology can be understood as a first benchmark for predicting emotional trajectories in a supervised manner. ## conclusion we proposed a valence/arousal-based gold standard for the alm dataset @xcite . moreover, we provide first results for the prediction of these signals via finetuning deberta combined with a weakly supervised learning step. we obtain promising results, but, at the same time, demonstrate the limits of this methodology in our analysis. future work may include attempts at a more holistic story understanding, involving e. g., the roles of protagonists. besides, our analysis of the results by author suggests that personalization methods (e. g., @xcite ) may improve the results. further, the potential of even larger llms such as llama @xcite or (chat-)gpt @xcite remains to be explored for this task. such models may even assist in refining the rather simplistic mapping method we utilized for the creation of the gold standard, as they have been shown to come with inherent emotional understanding capabilities @xcite . code, data, and model weights are released to the public foot_1 .