Uncovering Surprising Event Boundaries in Narratives (2022)

Abstract: When reading stories, people can naturally identify sentences in which a new event starts, i.e., event boundaries, using their knowledge of how events typically unfold, but a computational model to detect event boundaries is not yet available. We characterize and detect sentences with expected or surprising event boundaries in an annotated corpus of short diary-like stories, using a model that combines commonsense knowledge and narrative flow features with a RoBERTa classifier. Our results show that, while commonsense and narrative features can help improve performance overall, detecting event boundaries that are more subjective remains challenging for our model. We also find that sentences marking surprising event boundaries are less likely to be causally related to the preceding sentence, but are more likely to express emotional reactions of story characters, compared to sentences with no event boundary.

## introduction when people read stories, they can easily detect the start of new events through changes in circumstances or in narrative development, i.e., event boundaries @xcite @xcite . these event boundaries can be expected or surprising. for example, in the story in figure 1 based on crowdsourced annotation, "getting along with a dog who does not generally like new people" marks a surprising new event, while "their playing fetch together for a long time" is an expected new event. we aim to study whether machines can detect these surprising or expected event boundaries, using commonsense knowledge and narrative flow features. characterizing features that are informative in detecting event boundaries can help determine how humans apply expectations on event relationships @xcite @xcite figure 1 : example story with sentences that contain either a surprising event boundary, no event boundary or an expected event boundary respectively. the annotations of reader perception are from the hippocorpus dataset @xcite . @xcite . furthermore, detection of sentences with event boundaries can also be useful when generating engaging stories with a good amount of surprises. @xcite @xcite . to differentiate sentences with surprising event boundaries, expected event boundaries, and no event boundaries, we train a classifier using 3925 story sentences with human annotation of event boundaries from diary-like stories about people's everyday lives @xcite . we extract various commonsense and narrative features on relationships between sentences of a story, which can predict the type of event boundaries. commonsense features include the likelihood that adjacent sentences are linked by commonsense relations from the knowledge graphs atomic @xcite and glucose @xcite . narrative features include realis @xcite that identifies the number of event-related words in a sentence, sequentiality @xcite based on the probability of generating a sentence with varying context and simgen (rosset, 2020), which measures the similarity between a sentence and the sentence that is most likely to 1 be generated given the previous sentence. we then combine the prediction based on these features with the prediction from a roberta classifier @xcite , to form overall predictions. we evaluate the performance of the classification model by measuring f1 of the predictions and compare various configurations of the model to a baseline roberta model. we find that integrating narrative and commonsense features with roberta leads to a significant improvement (+2.2% f1) over a simple roberta classifier. there are also individual differences on the subjective judgment of which sentences contain a surprising or an expected event boundary, that is reflected in the detection model's performance. the performance of our model increases with increasing agreement across the human annotators. additionally, by interpreting the trained parameters of our model, we find that the absence of causal links between sentences is a strong predictor of surprising event boundaries. to further analyze how surprising event boundaries relate to deviation from commonsense understanding, we compare the performance of the classification model on the related task of roc story cloze test @xcite . this task concerns whether the ending sentence of a story follows/violates commonsense based on earlier sentences, which can be linked to whether sentences are expected or surprising. our model performs significantly higher on the roc story cloze test (87.9% f1 vs 78.0% f1 on our task), showing that surprising event boundaries go beyond merely violating commonsense and therefore can be seen as more challenging to detect. together, our results suggests that while detecting surprising event boundaries remains a challenging task for machines, a promising direction lies in utilizing commonsense knowledge and narrative features to augment language models. ## event boundary detection task events have been widely studied in natural language processing. they have often been represented in highly structured formats with wordspecific triggers and arguments @xcite @xcite @xcite or as subject-verb-object-style (svo) tuples extracted from syntactic parses @xcite @xcite . in narratives, events are represented as a continuous flow with multiple boundaries marking new events @xcite @xcite ; however, we lack a model to detect the boundary events that mark the meaningful segmentation of a continuous story into discrete events. in this work, we study stories from a cognitive angle to detect event boundaries. such event boundaries relate to our narrative schema understanding @xcite @xcite , commonsense knowledge @xcite and world knowledge @xcite . existing work has studied on salient (i.e. important/most report-able ) event boundaries within a story @xcite @xcite . however, missing from literature is how salient event boundary can either be surprising or expected based on the knowledge of how a flow of events should unfold. for example, events can be surprising when they deviate from commonsense in terms of what people would predict (e.g., if someone won something, they should not be sad; @xcite . surprising events can also be low likelihood events (foster and keane, 2015) such as seeing someone wear shorts outside in winter, or due to a rapid shift in emotional valence between events @xcite such as seeing a protagonist being defeated. importantly, there are individual differences in how humans segment narratives into events @xcite . we tackle event boundary detection as a threeway classification task that involves distinguishing surprising but plausible event boundaries in story sentences from expected event boundaries and no event boundaries. to mirror how humans read stories, we predict the event boundary label for a sentence using all of its preceding sentences in the story, as well as the general story topic as context. surprising event boundaries are novel events that are unexpected given their context, such as a dog getting along with someone despite not typically liking new people. expected event boundaries are novel events that are not surprising, such as a person playing a new game with a dog for a long time given that they like each other. in contrast, sentences with no event boundary typically continue or elaborate on the preceding event, such as a person liking a dog given that they get along with the dog (figure 1 ). majority label #samples (%) % majority agreement (std) no event 2255 (57.5) 68.1 (13.9) expected 650 (16.6) 58.8 (10.6) surprising 509 (13.0) 61.7 (11.9) tied 511 (13.0) 41.1 (5.7) total 3925 (100) 62.2 (15.2) ## event-annotated data we use the english-based event-annotated sentences from stories in the hippocorpus dataset to study event boundaries. this dataset contains 240 diary-like crowdsourced stories about everyday life experiences, each containing an average of 16.4 sentences and are annotated at the sentence level @xcite . stories were inspected for the absence of offensive or person-identifying content. for the annotation, eight crowdworkers were shown a story sentence by sentence and were asked to mark whether each sentence contained a new surprising or expected event boundary, or no event boundary at all, based on their subjective judgment @xcite . summarized in due to the inherent subjectivity of the task, aggregating labels into a majority label yields low agreement (e.g., 61.7% for surprising event boundaries; ## event boundary detection model we first describe informative commonsense and narrative features that we extract for the event boundary detection model. then, we describe how we integrate these features with a roberta classifier in our model before detailing our experimental setup. figure 2 depicts an overview of our model. ## results and discussion we first quantify the performance of our model in detecting event boundaries, using a coarse-grained performance measure on f1 with respect to majority vote. then, we investigate how the performance varies based on annotation subjectivity. finally, we inspect the model parameters to identify commonsense and narrative features that are most informative in detecting event boundaries. ## comparison with story cloze test to better understand how surprising event boundaries relate to deviation from commonsense reasoning, we compare our event boundary detection task to the roc story cloze test @xcite . this test involves identifying whether a candidate ending sentence follows commonsense (commonsense ending) or deviates from commonsense (nonsense ending) given the first four sentences of a english short story. the roc story cloze test dataset contains 3142 samples with 1571 commonsense endings and 1571 nonsense endings. 3 we train a separate event boundary de-tector model on the roc story cloze test, using the same experimental setup as for event boundary detection, except the loss function; we use the cross-entropy loss since only one label is available for each sample. ## conclusion we tackle the task of identifying event boundaries in stories. we propose a model that combines predictions made using commonsense and narrative features with a roberta classifier. we found that integrating commonsense and narrative features can significantly improve the prediction of surprising event boundaries through detecting violations to commonsense relations (especially relating to the absence of causality), low likelihood events, and changes in emotional valence. our model is capable in detecting event boundaries with high annotator agreement but limited in detecting those with lower agreement. compared to identifying commonsense and nonsense story endings in story cloze test, our task is found to be only tagentially related. our results suggest that considering commonsense knowledge and narrative features can be a promising direction towards characterizing and detecting event boundaries in stories.