Narrative Embedding: R e- C ontextualization Through Attention (2021)

Abstract: Narrative analysis is becoming increasingly important for a number of linguistic tasks including summarization, knowledge extraction, and question answering. We present a novel approach for narrative event representation using attention to re-contextualize events across the whole story. Comparing to previous analysis we find an unexpected attachment of event semantics to predicate tokens within a popular transformer model. We test the utility of our approach on narrative completion prediction, achieving state of the art performance on Multiple Choice Narrative Cloze and scoring competitively on the Story Cloze Task.

## introduction common sequences of events describing real-world interactions are a fount of useful information for how we interact with our world. for example, the prototypical "restaurant schema" of @xcite might contain the sequence of events in story a of figure 1 . knowledge of this plausible sequence of events allows us to both predict likely subsequent events in a novel 'restaurant experience' as well as infer unstated events when observing a recounting of the same. since written stories often provide the minimum required for human comprehension, the stated narrative is liable to be rife with omission, leaving the reader to infer the missing events. for this reason, understanding these chains of events is central to understanding the narrative as a whole. each story is made up of constituent events, and by developing representations of events we can in turn produce a representation of the story at large. event representation has been a topic of substantial interest to the field @xcite @xcite @xcite and as the wealth of unstructured narrative data continues to grow, to industry as well. indeed, industry has seen narrative representations applied both directly in the case of film and book analysis @xcite as well as indirectly as a means of communication in automated data explanation applications (narrative science, 2021). our goal is to provide an unsupervised approach to narrative representation by utilizing and combining advances in representational techniques at both the semantic and structural level. our approach is based on 're-contextualization', a process of embedding representations through transformer blocks at two levels of granularity. first, similarly to standard transformer models, we contextualize events within a single sentence to provide local context. second, we extract events from each sentence to contextualize them across an entire story or story arc. the motivation for this approach is to allow our representation to differentiate between events which, though potentially drawn from identical or nearly-identical sentences, represent substantially different narrative components (e.g. "john orders fish" from story a versus story b in figure 1 ). in the process, we produce two models that demonstrate competitive performance on the multiple choice narrative cloze (mcnc) task and the story cloze task (sct) respectively. specifically, our contributions herein can be summarized by the following: ## datasets for our experiments, we make use of two datasets: english gigaword corpus for multiple choice narrative cloze task and the story cloze task corpus for the story cloze task @xcite . the english gigaword corpus consists of new york times news articles containing a training set of 830,643 documents. this dataset was then parsed with spacy @xcite and embedded with the 'bert-base-uncased' pretrained model @xcite . train/test splitting was done the same as in @xcite . the story cloze task (sct), a premiere benchmark narrative task featured as the shared task of lsdsem 2017 @xcite consists of 49255 five sentence stories and 3744 sct examples which have two candidate completions. sct is an especially good dataset for evaluation of our approach since one of the goals in its construction was to make the story completions as close to statistically indistinguishable from one another as possible. this then requires a more complete narrative understanding to differentiate between alternative endings. as a result, sct places a stronger emphasis on event disambiguation and cohesive interpretation than on joint probabilities of events. more specifically, we use the sct_1.5 dataset provided by @xcite which addresses inherent story-completion biases found in the original sct dataset. the improvement afforded by sct_1.5 is especially important for transformer based models since it is impractical to know what elements of the original sentence bled into the semantics of any given token. for instance, the sentiment of the sentence as a whole may adjust the semantic representation of an otherwise semantically neutral verb in the sentence, allowing our classifier to pick up on sentiment shift through sentences (a feature for which sct_1.0 was strongly biased). the training and test sets of sct_1.0 and sct_1.5 have numerous instances of overlap with sct_1.5 sharing 1556 of its 1571 stories with sct_1.0. the test set overlap is less ubiquitous, with partial and complete overlaps including instances of word adjustment (e.g. "forever 21" → "forever twenty-one") of 913 out of 1872 test stories. when removing all of the overlapping instances, we find an additional 959 instances from sct_1.0 testing which we can add to sct_1.5 training, alongiside the validation set of sct_1.5. for the purposes of comparison, comparative results reported here will be on a withheld portion of the sct_1.5 validation set since the accompanying test set is blind, and total submissions to determine accuracy are limited. final best models' results are also submitted to the public leader-boards for a ground-truth blind assessment. for both mcnc and story cloze, in keeping with common practice on the tasks, we report a raw accuracy metric. foot_1 ## methods we explore narrative representation through its application to two tasks using three models: two models for multiple choice narrative cloze (mcnc) -transformer event composition (event-comp) & transformer event sequence with attention (re-context) -and one for the story cloze task. these three models aim to answer the following 3 questions: what impact does re-contextualization have on bert embeddings for event prediction? 3. story cloze: do our results generalize to more controlled and higher-level narrative completion tasks such as sct? we present and analyze two models on mcnc. model 1 provides a comparison against the original event-comp model from @xcite and allows determination of the direct impact of transformer (here bert) embeddings on mcnc. the second model aims to capture narrative context via a novel architecture using an attention layer over a larger-scope window of narrative events. the idea behind this is to take locally (intrasentential) contextualized event embeddings and re-contextualize them within the scope of the greater narrative (intersentential). this mechanism is what allows us to tackle the type of event confusion displayed in figure 1 . the story cloze task model, model 3, follows the same representational architecture as our second mcnc model. ## discussion & conclusion we find that bert embeddings represent a substantial improvement to the mcnc state of the art. our novel attentional re-contextualization of events provides a competitive story cloze task performance yielding the highest score amongst models using explicit event representation. of broader interest to the field are our findings that for cloze tasks bert attaches enough event semantics to predicates that they can function as full event proxies without loss of performance. this finding suggests possible avenues for further research to discover what other syntactic entities carry broader semantic aggregates in bert. our overall findings and contributions can be summarized as follows: