Exploring aspects of similarity between spoken personal narratives by disentangling them into narrative clause types (2020)

Abstract: Sharing personal narratives is a fundamental aspect of human social behavior as it helps share our life experiences. We can tell stories and rely on our background to understand their context, similarities, and differences. A substantial effort has been made towards developing storytelling machines or inferring characters’ features. However, we don’t usually find models that compare narratives. This task is remarkably challenging for machines since they, as sometimes we do, lack an understanding of what similarity means. To address this challenge, we first introduce a corpus of real-world spoken personal narratives comprising 10,296 narrative clauses from 594 video transcripts. Second, we ask non-narrative experts to annotate those clauses under Labov’s sociolinguistic model of personal narratives (i.e., action, orientation, and evaluation clause types) and train a classifier that reaches 84.7% F-score for the highest-agreed clauses. Finally, we match stories and explore whether people implicitly rely on Labov’s framework to compare narratives. We show that actions followed by the narrator’s evaluation of these are the aspects non-experts consider the most. Our approach is intended to help inform machine learning methods aimed at studying or representing personal narratives.

## introduction we can develop the ability to retrieve a story that we have experienced or heard when someone else is telling a story. we find ourselves thinking about our story, and so we think that we know what is coming next in our friend's story. however, in order for computers to match stories automatically, we need to understand what "matching" implies and what aspect of a story should be attended to. there have been some attempts to match stories @xcite and to understand human judgment about matched stories @xcite . nevertheless, these efforts have been mostly developed in supervised scenarios that already have a set of matched stories in hand, and they are mostly focused on non-personal narratives (e.g., fictional). from these insightful works, however, we want to explore the understanding that when we consider stories to be similar, we attend to some aspects more than others, stressing the need for comparison of different aspects rather than at a global level. as a first effort towards our purpose, we collect the largest annotated corpus of spoken personal narratives to our knowledge, comprising 10,296 narrative clauses from 594 stories. we use transcripts of roadtrip nation (rtn) videos foot_0 , where professionals share stories about their lives and career pathways. as for the annotation task, we ask mechanical turkers to annotate each clause under labov's sociolinguistic model of personal narratives @xcite , where a narrative is defined by a structural component, which includes a temporal organization (action clauses) and contextual orientation (orientation clauses), and an evaluation component (evaluation clauses), which represents storytellers'/characters' needs and desires (explained in more depth in section 3). next, aiming to automatically tag stories, we develop a model to classify these clauses that reaches 84.7% f-score for the highest-agreed clauses. once we can automatically differentiate among clause types, we would like to use them to compare stories, but, do ordinary people rely on these clause types to compare narratives? to approach that question, we pair stories and run experiments to understand to what extent ordinary people (as opposed to literary experts) rely on labov's model to think about similarities among these stories. our approach is intended to help inform machine learning methods aimed at studying personal narratives and at modeling abstract information extraction. to the best of our knowledge, this work is the first to propose and develop an approach to understand whether ordinary people rely on labov's framework to compare personal narratives and what they perceive as similarities among those narratives. we show that actions followed by the narrator's evaluation of these are the aspects non-experts consider the most when they compare stories. our main contributions can be summarized as follows: the rest of the paper is organized as follows. in section 2, we present some main related work. in section 3, we specify the story aspects to be used in our experiments. in section 4, we describe the uniqueness of our introduced narrative corpus. in section 5 and 6 we describe results, and we end with conclusion and future directions in section 7. ## related work our work is preceded by substantial efforts toward document @xcite @xcite and story @xcite @xcite @xcite representation. we find that most approaches to text similarity focus on non-narrative corpora (vor der brück and pouly, 2019; @xcite . we also observe that most works in stories have been developed for non-personal narratives. an specific approach to story matching was proposed by @xcite , who used movie remakes from wikipedia as paired stories and showed that even in that scenario it was challenging to match the remakes. additionally, their method does not generalize well to other story types (or even movie plots) since they include specific movie parameters, like characters' name and gender, as the basis of their solution, which does not apply to our case since we do not attempt to match stories based on these surface-level indicators. the closest work to ours was done by @xcite , who proposed a set of crowdsourcing tasks to analyze perception of similarity in folk narratives. they tried various approaches to retrieve these narratives. nevertheless, they had in hand a set of metadata labels that allowed them to match narratives prior to any experiment. how we narrate our stories was initially studied by @xcite . more recently, @xcite proposed the first mechanism to automatically classify labov's clauses (action, orientation, and evaluation-type clauses) in personal narratives based on clauses' syntactical structure, namely part-of-speech (pos). by using 50 short stories from online mini-blogs, of diverse topics and structures, they developed a well-defined set of instructions to properly annotate labov's clause types (referred to as baseline method and dataset onward). however, personal narratives from spoken stories set a more challenging context for both annotation and collection (see section 4). we get inspiration from these works to approach clause types classification using newer techniques like word embeddings @xcite and neural networks @xcite . furthermore, as we learn to disentangle narrative dimensions or aspects (namely, action, orientation, and evaluation-type clauses), we can use them for other story representation tasks. for instance, identifying the clauses within a story that tell people's intents/desires, reactions, and evaluation of the events (e.g., emotions) can help train and evaluate models aimed at detecting, or planning plots conditioned on, those underlying intentions and reactions @xcite . ## story aspects of comparison stories can be thought to be similar in a variety of dimensions; unlike most non-narrative texts, stories have "meta" dimensions that go beyond what is said (context of a story, actions that happen, emotional content, speaker's backgrounds, among others). in this work, we explore to what extent labov's model for personal narratives underlies how non-expert people perceive story similarities. we focus on the following three aspects: temporal organization (action clauses): these clauses express a series of events. the narrator might play with the story's chronology, causing differences between narrative structures of one narrator and another. contextual world (orientation clauses): these clauses describe information about the context in which actions occur; they serve to orient the audience about people, places, time, and behavioral situations. human needs and desires (evaluation clauses): these clauses give significance and tell about the purpose of telling that story; they express the narrator's needs and desires. see figure 1 for an example of a narrative annotated under labov's model for personal narratives. ## narrative corpus we introduce the largest dataset of annotated spoken personal narratives to our knowledge, from now on referenced as roadtrip nation or rtn corpus. these narratives were obtained from transcripts of stories video-recorded by roadtrip nation (rtn). in those videos, people from many backgrounds share stories about their lives and career pathways. the corpus comprises 10,296 narrative clauses from 594 stories (each one told by a different person), which account for more than 10 hours of people telling stories, each one averaging 17.1 clauses or 62 seconds long, where each clause has on average 11 tokens. to split narratives into clauses, we proceed as follows. for every sentence in the story, we take every independent clause along with its dependent clause, which account for one narrative clause. to determine clauses, we rely on top-level s* (s, sinv, sbar, sbarq, sq) tags from penn treebank ii @xcite . for each top-level s* tag, we take its subtree along with hanging prepositions, conjunctions, and adverbs. while we propose to automatically split our data, @xcite 's data (our baseline dataset) was split by trained humans. we compared our strategy implemented using nltk with their strategy by spliting their stories as well; we found that our method differs at most in one clause from their manually split stories. ## narrative clauses classification learning to classify narrative clauses can help us disentangle personal narratives' dimensions. our specific intention is to understand how this decomposition helps compare stories in different aspects (clause types are assumed to be aspects or dimensions within stories for this work). additionally, each of these clause types can be used independently for various objectives. for instance, actiontype clauses could guide events extraction where, even though the narrator might play with the story's chronology, having these clauses apart can help find causal or temporal orders. also, identifying orientation-type clause can help create a grounded understanding of the story, where actions and emotions depend on the story's environment described in these clauses. finally, evaluation-type clauses could bring to surface narrators' mental states, clause type rtn baseline action 26.7% (2.15) 24.2% evaluation 40.0% (2.29) 50.0% orientation 29.7% (2.24) 24.2% not story 3.6% (2.13) 1.6% total clauses 10,296 1,602 we propose to use a convolutional neural network (cnn) with max-over-time pooling to classify clauses @xcite . more specifically, our model consists of a non-static cnn as in @xcite , where we initialize embeddings using @xmath0. we perform 1-max pooling with relu activations over each map generated by filters of sizes 2, 3, and 4; we use 30 filters per size. then, we use two linear ((90, 45), (45, 3)) layers with dropout of 0.3 before the final softmax layer. we also explored fine-tuning bert @xcite and found that, in most tried scenarios, this simple word-based cnn-based model outperformed bert in accuracy, maybe due to the small fine-tuning dataset. we randomly split the rtn dataset into 86% for training, 7% for validation, and 7% for testing, removing the "not story"-clause type. this gives 7,698 training, 619 test, and 634 validation clauses with @xmath1. our vocabulary has around 6,000 tokens, including an unknown word token we use for uncommon words (≤ 2 appearances). for training, we used 60 epochs and early stopping based on the validation error. we trained with different number of filter, linear layer sizes, batch sizes and learning rates set through experimentation based on performance. we find our best results using adam with a learning rate of 5e-5 and use batch sizes of 64. ## people's perception of similarity aiming to understand the aspects (i.e., clause types) that ordinary people attend to the most when they think about similarities among stories, we proceeded as follows. we represent each story as a set of narrative clauses, where each clause is initially encoded into a high-dimensional vector by using the universal sentence encoder (use) introduced by @xcite . next, given stories s and s', for each clause in s we find the closest clause in cosine similarity in s' (@xmath2, and vice versa (s' → s), and obtain an average similarity score. using this mechanism, we match stories only at clausetype subsets (action, evaluation, or orientation-type only). finally, we sample 60 story pairs with average cosine @xmath3.5 for one of the clause types matches. see appendix a for some sample matched stories. for our experiments, we use these 60 stories, which are presented to turkers in audio form only (as opposed to transcript text). while reading and listening might require different attention spans, since labov's sociolinguistic model focuses on stories that are produced orally (just like these) and these are short stories -62 seconds long on average -we rely on turkers' auditory cognitive processing. ## conclusion we introduce the largest corpus of annotated spoken personal narratives, to our knowledge, and develop a straightforward method to classify these narratives' clauses using labov's sociolinguistic model. our model's high performance in classification reflects the disentangling power that labov's model offers for analyzing oral personal narratives. only by being trained in our introduced corpus, our model performs well in an earlier proposed dataset of written stories. furthermore, we propose the first attempt to understand whether ordinary people (i.e., non-narrative experts), such as mechanical turkers, rely on labov's model to compare personal stories, and show that these people do rely on two out of three labov's aspects of narrative. namely, action-type and evaluation-type clauses are perceived as central aspects of comparison, but the same does not apply to, and remains unresolved for, orientation-type clauses. one natural next step would entail shedding light on how different questions' wording and emphasis, aimed at matching stories, affect what people think of as similarity aspects. we hope that these precursory findings about the aspects that proved to underlie story-matching could also be used in a broader set of tasks, such as finding causal or temporal relationships between events, inferring mental states, or grounding actions and emotions in a story's set-up. finally, we acknowledge that we have only scratched the surface of this wonderfully rich space of personal narrative representations and of what people focus on when they compare stories. our overarching goal, of modeling human judgment of narrative similarity and building a machine capable of replicating that behavior, leaves untouched several questions that future research should explore. for example, what other aspects should be examined to represent personal narratives, how to decide the relative relevance of these aspects, and how to model similarity judgments within aspects.