Emotion Arcs of Student Narratives (2020)

Abstract: This paper studies emotion arcs in student narratives. We construct emotion arcs based on event affect and implied sentiments, which correspond to plot elements in the story. We show that student narratives can show elements of plot structure in their emotion arcs and that properties of these arcs can be useful indicators of narrative quality. We build a system and perform analysis to show that our arc-based features are complementary to previously studied sentiment features in this area.

## introduction this work deals with the study of emotion arcs in student narratives. plots @xcite and emotion arcs @xcite @xcite form the foundations of storytelling. story-telling is an important literacy skill. children are taught to understand and write narratives in school, and literacy standards foot_0 require students to write increasingly competent narratives. while researchers have already introduced analysis of plots in well-written stories and emotion arcs in novels, not much attention has been paid to these phenomena in narratives written by novices. in this work we study student narratives along the dimension of emotion arcs. we show that even novice writing can show plot elements. we then investigate if the quality of narratives can be determined by measuring properties of the emotion arcs. there has been work investigating scoring of student narratives @xcite . however, previous focus has been on other aspects of the narrative, such as event progression, organization, vividness, detailing and subjectivity. we investigate if emotion arc characteristics can help to improve automated narrative scoring systems. plot analyses and research on constructing shapes of stories have considered the general sentiment or affect present in the text. in our work we focus on a specific type of sentiment/affect that we believe is closer to plot structure: events that produce good/bad effect, affective events and sentiment connotations. we show that while there is overlap with subjectivity and sentiment, our approach captures a different dimension of narrative quality. we believe that our work advances the understanding of plots elements in narratives written by novices. our work connects evidence of plots elements to the quality of narratives as judged by human raters using standard scoring rubrics. specifically, the contributions of our work can be summarized as follows: 1. we show that emotion arcs can be seen even in simple narratives written by novice writers. 2. our experiments show that emotion arc properties are indicative of the quality of narratives. they are related to other sentiment factors in narratives, but are distinct in what they capture about the narrative quality. 3. we show that encoding emotion arc characteristics help to improve narrative essay scoring systems. ## narrative data as our focus is student narratives, we use the annotated narrative dataset from @xcite . the data comprises of 942 narrative essays written by school students from the criterion r program 2 . criterion is an online writing evaluation service from educational testing service 3 . it is a web-based, instructor-led writing tool that helps students plan, write and revise their essays. nar-rative essays in this dataset belong to writers from three grade levels: grades 7, 10 and 12. each essay is in response to one of 18 story-telling prompts; prompts belong to topics related to personal experiences, hypothetical situations, and fictional stories. given below are example prompts: the average essay length in the data is 320 words, with a range of 3 to 1310 words and a standard deviation of 195. the rubric used for scoring the essays was created by education experts and teachers. it defines a separate score (0-4) each for essay organization and essay development. the dataset also provides a narrative score for each essay, which is the sum of the organization and development scores. the score is an integer value from 0 to 8, with 8 corresponding to perfect organization and development of the narrative. the human inter-annotator agreement for the narrative quality score is 0.76 qwk foot_3 . we use this score for our work. we refer the reader to the original paper for details on the data, rubrics and annotation. ## emotion arc an emotion arc involves the plotting or tracking of sentiment valence of some form along the time axis @xcite @xcite . however, we observed that sentiment words and phrases occurring in narratives serve different purposes, such as describing character and settings, embellishing the story, advancing the plot, etc. for example, senti-ment words may be used to describe a scene (e.g. "beautiful house"), a character (e.g. "smart girl"), a character's private state (e.g. "peter thought that was foolish") or emotions (e.g. "sally was furious"). in our work, we are primarily interested in sentiments and emotions as they relate to the plot. thus our focus is mainly on events and implicit sentiments. events are the core building blocks of narratives, and positive and negative events are closely tied to plot progression. this intuition is in line with lehnert's work on plot units @xcite , which also focuses on positive/negative events (called events that please, and events that displease). additionally, much of the plot movement is brought about by elements that have implicit sentiment value. for example, if "a kills b" in a story, it indicates an objective event on the surface, but denotes a conflict (or resolution, depending on whether b is an antagonist) in the story. given this focus, our emotion arcs are constructed based on the following phenomena that have been previously developed for other purposes in computational linguistics: good-for and bad-for events: good-for and badfor events, also known as benefactive and malefactive events, positively/negatively affect the entities on which they act @xcite . these events indicate someone (or something) doing something that affects someone (or something) in a positive or negative manner. in the context of stories, we hypothesize that such events can indicate elements of a plot, such as conflict, resolution and goal achievement. affective events: these are events that affect an experiencer in positive or negative ways @xcite ) even though they do not, in their surface form, hold a valence. the events are implicitly affective based on the human knowledge of the event itself, such as going on a vacation or breaking a record. sentiment connotation : these are words that imply a positive or negative sentiment even though they appear objective on the surface @xcite . for example, a gun-shot invariably indicates a conflict in the plot, even though it is objective on the surface. ## relationship to narrative quality we saw in the previous section that well-written student narratives generally tend to have emotion arcs corresponding to plot elements. the next question is whether these elements can be indicators of narrative quality score as defined by standardized essay scoring rubrics. in order to answer this, we study the relationship of narrative quality scores to properties of the emotion arcs. note that the narrative quality score is a function of a number of factors, such as organization of the story, effective use of transitioning, clear opening and closing, vivid description, character development, use of dialog, event sequencing, effective use of figurative language and other narrative techniques. hence we expect presence of plot element in a story to be just a component that would contribute to the determination of the score. while individual creativity makes it difficult to directly equate emotion arcs to scores, we can extract features that represent arc characteristics. we extract the following features from the arcs. for the ease of explanation, we denote the arc value at position i as d(i). we define local maximum as positions where d(i -1) < d(i) > d(i + 1), and local minimum as points where d(i -1) > d(i) < d(i + 1). the slope at each point is calculated as d(i) -d(i -1). ## related work narratives can be analyzed along many different dimensions, such as sentiment, emotion, plot, characters, engagement, creativity, and success of stories. computational linguistic analyses started with shorter texts, concentrating mostly on fables, folk stories and fairy tales. in the last decade they fully embraced analysis of full length novels and movie scripts. several studies focused on character traits and personas in stories. @xcite proposed a rich representation of story-characters for summarizing and representing novels. researchers have also studied social networks and have modeled relationships in stories @xcite @xcite . @xcite ; @xcite modeled character inter-relations and their development in novels. evolving relations were represented as relationship sequences/trajectories and learned using structured prediction techniques. @xcite analyzed personal narratives from a blogging platform for automatic detection of turning points in stories. @xcite presented the task of detecting turning points in movie scripts, as a particular way for analyzing narrative structure. they used neural network models for automatically detecting sequences of major explicit events in stories. sentiment analysis has been employed for narrative analysis in many studies. @xcite ) analyzed aesop's fables, producing automatic plotunit representations @xcite , using taskspecific knowledge base of affect. several studies focused on annotation of folk stories and fairy tales for emotions @xcite @xcite . alm et al. @xcite described a machine learning approach for multiclass classification of sentences for their emotional content. piper and so (2015) used a sentiment lexicon and compared the proportion of sentiment words between several groups of novels. they found that, on average, 19th century novels have a larger proportion of sentiment words (7%) than modern novels (about 5.5%). bostan and klinger (2018) presented a survey of recent corpora annotated for emotion classification in text, with a variety of clas-sification schemata. @xcite present a new dataset of classic and modern novels, with passages (sections of 40 to 200 words) manually annotated for emotion classes based on plutchik's eight basic emotions. the data was used for training and evaluating deep learning architectures for emotion classification. @xcite presented a novel challenge and datasets for affect and emotion detection in text, calling it emotional relationships classification: which character feels which emotion to which character. the idea that, in stories, emotion and sentiment is not static, but changes dynamically, is an old one. it is often presented as practical advice in writing guides for aspiring screenwriters and novelists @xcite . the idea of actually charting the emotional progression of scenes and stories (emotion arcs) is attributed to kurt vonnegut (see e.g. @xcite , described by jockers (2014) and del vecchio et al. ( 2018 )). the annotation study of @xcite was one of the earliest computational studies that considered a notion of emotional trajectory -plotting the emotional values for sequences of text segments in 22 grimms' fairy tales. @xcite used sentiment analysis to generate emotional profiles for full-length english novels. they implemented the notion of emotion arcs, tracking the level of emotion-laden content through consecutive segments (10k-long word segments) of literary works (from project gutenberg), using a lexicon of words with ratings on a single positivenegative scale -a sentiment polarity lexicon. they found that large-scale arcs cluster into six common arc shapes. a similar approach was described by jockers (2014) and @xcite . del vecchio et al. ( 2018 ) used a similar technique to track emotion arcs in about 6000 movie scripts. that study also found that large-scale arcs cluster into six common shapes. the study went further, relating the arc shapes to movie success, using movie gross revenue as success indicator. chu and roy (2017) performed a multi-modal analysis of emotion arcs. they used neural network models to construct emotional arc representations from movie clips, audio clips, images and two-word image captions (the latter analyzed with sentiwordnet lexicon, @xcite ). they applied clustering to find major groupings of emotional arcs. they evaluated their models by predicting viewer engagement with short online videos (measured as number of comments the videos received). @xcite used lexical expressions of emotion for genre-classification of whole novels. they extended the emotion arc approach, by using eight fundamental emotions defined by @xcite , instead of a single sentiment valence dimension. thus, their tracking reflects the development of each of the eight emotions throughout the time course of the narrative. overall, this method contributes to significant improvement of genre classification over a strong lexical baseline. ## discussion to the best of our knowledge, our work is a first attempt in exploring emotion arcs and plot elements in student writing via event affect and implied sentiment. the exploration is by no means complete. in this section we discuss the rationale of our choices, some limitations of the current study and challenges in this task. in order to construct emotion arcs in section 3 we relied on a number of resources and made a number of choices, which has influenced the precision of the resulting arcs. first, we used (semi) automatically created lexicons, and these have issues with both noise and coverage. we did not combine the lexicons or remove duplicates between lexicons. this could have led to a single instance of an easily recognizable emotional element in a sentence being counted more strongly than it otherwise should have. contextual polarity resolution was not performed, which could have influenced the determination of the sentence-level valence. finally, when employing the effectwordnet, we simply used the most frequent sense instead of per-forming complete word sense disambiguation. our curve-fitting function might have dampened some of the spurious errors, but there is obviously scope for improvement. in section 4 we sampled arc properties and employed them as features. our choices were driven by the requirements that the properties should be efficiently extracted, and be generalizable across narrative sub-genres and writing proficiency. more constrained environments/applications could afford closer modeling of arcs (e.g. finding components of the curve equation). in our work, we have used the simplifying assumption that there are emotional correlates to plot elements, and by tracking the emotion arc, we are able to capture some elements of the plot. while this assumption may hold for simple short stories, it is likely to collapse for stories from creative authors. accomplished authors can create tension and resolution without emotional accompaniment. finally, it is important to note that plot and components of a plot, such as rising action, conflict, falling action, resolution, etc., while easily discernible to human readers, pose significant challenges for a machine. world knowledge, human experiences, interpretation of motivations, inferences of human actions and context of the story, play an important part in the recognition. consequently, automated methods too will need to look beyond words, sentences and paragraphs. ## conclusion in this work, we studied emotion arcs in student narratives, and explored ways to harness them for automatically determining narrative quality. we showed that emotion arcs are manifest in student writing and intuitively correspond to plot elements. our analyses showed that simple arc properties correlate with narrative quality score, and that the features derived from the arcs are similar to, yet distinguished from previously explored subjectivity features. we built scoring systems and showed that adding our arc-based features have the potential to improve the scoring performance. our future work will include addressing the limitations and challenges discussed in section 6.