Narrative Cloze as a Training Objective: Towards Modeling Stories Using Narrative Chain Embeddings (2023)

Abstract: We present a novel approach to modeling narratives using narrative chain embeddings.A new dataset of narrative chains extracted from German news texts is presented. With neural methods, we produce models for both German and English that achieve state-of-the-art performance on the Multiple Choice Narrative Cloze task. Subsequently, we perform an extrinsic evaluation of the embeddings our models produce and show that they perform rather poorly in identifying narratively similar texts. We explore some of the reasons for this underperformance and discuss the upsides of our approach. We provide an outlook on alternative ways to model narratives, as well as techniques for evaluating such models.

## introduction the narrative cloze task was originally introduced by @xcite and is the task of, given a sequence of narrative triples, predicting a masked triple. such triples are made up of subject, verb, and object, and the triples in one chain share a common participant, referred to as the protagonist. their subsequent work @xcite improved upon the results from the original paper and formulated the task slightly differently, expanding it to schemas with multiple participants. @xcite extract additional information and introduce evaluation metrics. an excerpt from one of their automatically extracted chains goes as follows: @xcite ), (_, heartbroken, a), (a, die, _), where a and b are entities, and each triple represents a verb with its arguments. one of the early motivations for the narrative cloze task was modeling narrative contexts and inferring narrative schemas @xcite . we aim to adapt the semantic modeling performed as part of the task to identify documents that share similar narrative schemas rather than ex-plicitly inferring such schemas. that is to say: analogously to masked language modeling, we use narrative cloze as a training objective to train narrative understanding, rather than language understanding. the motivation being that abstract story similarities may be found, eventually enabling computational comparisons of stories rather than texts. such an approach could, for example, be useful in digital humanities with researchers already experimenting with word embeddings to identify and compare adaptations of the same story @xcite . our approach to modeling narratives constitutes a continuous and embedding-based approach to schemas like propp's model of russian folklore @xcite . the chain-based approach has the upside of allowing for abstracting over information that is not relevant to the actual narrative, but that will be captured by more recent semantic embedding methods like sentencebert @xcite . the method's potential downside, however, as discussed by @xcite is that too much contextual information is lost, making the task of predicting triples ambiguous or impossible. through the use of contextual embeddings and an optional additional re-contextualization process, they improve on existing narrative cloze results by using additional information. our ultimate goal of this work is to enable embedding-based computational narrative similarity comparisons of texts, a task we see as closely related but not identical to the popular narrative generation field (see e.g. @xcite . the three key contributions of this work are (1) a dataset of german narrative chains and (2) the application of narrative embeddings to a down-stream task in the form of replicating human narrative similarity judgments, as well as (3) state-of-the-art models on english and german for narrative chains without external information from contextual embeddings. ## background to evaluate the capability of our embeddings in recognizing similar narratives, we rely on comparisons to human annotations. conceptual work on text similarity @xcite pointed out that text similarity is not inherently well defined by showing that, without further instructions, some annotators focus strictly on content, whereas others additionally take the text's structure into account. accordingly, our task calls for a dataset that explicitly annotates narrative schema similarity. @xcite introduced such a dataset in the form of a multilingual news similarity dataset containing the similarity of news article pairs along seven dimensions. according to their annotation code book @xcite , dimensions are to be rated independently of each other, with the narrative dimensions focusing on similarity in narrative schemas as defined by @xcite ; the dataset thus contains human ratings of schema similarity. since its inception, the narrative cloze task has seen work in different directions. @xcite has criticized newer approaches to the task as deviating from its original formulation, focusing on extracted events in text order rather than manually annotated ones; they emphasize that the automated approach is much more aligned with the capabilities of language models. @xcite approach the narrative cloze task but reformulate it to use contextual embeddings instead of verb lemmas. while this approach yields much higher accuracies and can help disambiguate events, we feel that in the light of modeling narrative disjointly from the surface form, such contextual embeddings would potentially hamper the model's performance in any downstream application. in the narrative cloze task, the model is asked to predict a masked triple describing an event. in practice, this is a four-tuple of the subject, verb, indirect object, and object in more recent implementations like the one by @xcite . evaluation has, as suggested by @xcite , in the recent past been performed in a mcnc (multiple choice narrative cloze) setup where the model is asked to pick the most fitting triple for a corresponding masked triple in a chain given exactly 5 options that are randomly sampled from the entire corpus. this evaluation setup was introduced to enable more interpretable results and pays tribute to the fact that, in most cases, the exact triple is ambiguous not just by virtue of synonymous verb lemmas but also due to contextual ambiguity. the work by granroth-wilding and clark (2016) discusses multiple models, with the best score being achieved by a model that calculates the compatibility of a given candidate triple by averaging across its compatibility (as scored by a neural model) with all other elements of the chain. ## datasets we use the gigaword dataset with the preprocessing pipeline presented by @xcite . in addition, we build a german dataset based on scraped german news data. the data is extracted using a german coreference resolution system by @xcite and dependency parsing from spacy @xcite . we produce a dataset of around 1.8 × 10 6 german narrative chains; we filter out any chains shorter than three at dataset creation time. compared to the approximately 5.7 × 10 6 chains with a length of at least three in the gigaword-derived dataset, this is a relatively small collection but still allows us to explore the adaption to a different language. while we rely on the intrinsic mcnc evaluation for comparison to existing work, for assessing the use for narrative modeling, we need a downstream evaluation, and only limited data is available for this purpose. the multilingual news-similarity se-meval dataset @xcite ) is, at first sight, a great fit; the pairs of articles making up the dataset are each annotated with regard to their similarity along seven specific dimensions, with each being dimension scored on a scale of 1-4. the dataset's narrative dimension is, however, highly (ρ=.88) correlated with its overall dimensions, meaning that when articles are narratively similar, they are likely to also be similar in a general sense. it may seem that, due to this alignment in similarity, no differentiation needs to be made for modeling the two dimensions, but we believe this difference is crucial in identifying texts that deal with the same narrative in different circumstances. this difference may also be interesting for other domains, especially narrative literary texts, where the correlation may, in practice, not be as high. in these texts, two scenes telling a similar narrative may not share any concrete entities; for example, the circumstances of two arguments between multiple characters may be entirely different with different surroundings and differently named characters, yet share some conceptual similarity. as the overall dimension can be modeled well using existing text similarity models, however, it seems unlikely that our approach based on narrative chains will be able to outperform existing models for the news domain. still, we employ the dataset as a testbed for extrinsic evaluation for the narrative cloze task. we make all our extracted chains, the ones from the nyt dataset, our german dataset, and the se-meval dataset, available for download to enable further research. foot_0 ## experimental setup to enable some comparison with prior work, we replicate the testing setup by @xcite wherever possible. in this section, we discuss the specifics of the task and provide an embedding baseline for our downstream evaluation. ## model setup and architecture we present a neural model that, using static word embeddings as input features, performs state-ofthe-art narrative cloze prediction. 3 to provide an additional point of comparison, we build a baseline based on modern techniques, specifically the t5 @xcite architecture and training setup. transformer emb out emb out emb out emb out emb out emb out emb out emb out emb out emb out emb out emb out emb out emb out emb out emb out emb out target classification loss cosine-similarity loss × vocab vocab vocab vocab vocab vocab vocab vocab vocab vocab vocab vocab vocab vocab vocab vocab @xmath0 ## results in as supported by the much worse performance of the t5 model without access to entity strings (a drop by over 60 percentage points), we strongly suspect that the t5 model is only looking for compatible mentions and will often only find one such option in the five choices presented. we manually confirmed that this strategy works in the majority of cases. the performance compared to that of the @xcite model can be explained by the fact that this model only compares pairs of triples, averaging across their coherence scores, and can thus not look for mention compatibility globally in the entire chain. overall, removing entity surface forms leads the t5 model to underperform drastically, whereas our static-embedding-based model only suffers a minor performance penalty. as previously discussed, we suspect this setup may lead to more meaningful narrative modeling. as a downstream evaluation of our embeddings, in after qualitative analysis, we suspected that our model might only be a topic model of sorts that considers the domain of verbs rather than any sequential nature of them. this is supported by the fact that it is overall still comparable in mcnc performance with the coherence based @xcite model. further news articles often do not tell happenings in their chronological order while our extraction pipelines rely on text order, meaning that the order does not necessarily follow logical sequences of actions. we test this hypothesis of no sequential understanding in ## conclusion in this work, we presented models with state-ofthe-art mcnc performance in two different setups and on german and english datasets. we produced vector embeddings as narrative representations and performed extrinsic evaluations of our narrative cloze models using the comparison to human narrative similarity ratings. in a qualitative review of our model outputs, we illustrated that the model captures sequential information. the performance of our embeddings indicates that narrative-cloze may not be a perfect fit for narrative similarity modeling; on the other hand, we were able to, in some scenarios, produce embeddings that model narrative similarity better than overall similarity, placing emphasis on the desired aspects of a text. in almost all cases, our models were also able to place less emphasis on entities than plain word embedding and especially sentence encoder models did. overall, it can be concluded that limiting the model's access to information can help create embeddings that represent a specific aspect of the text. it is also clear that in the current state, in almost all setups, our chain embeddings are outperformed even by static verb embeddings. we see two major roadblocks to applying this approach to the computational modeling of narratives. the first is the limited evaluation data: while the semeval dataset by @xcite is a step in the right direction, it fails to clearly demonstrate the need for narrative modeling as, in the news domain, dimensions are strongly correlated. a dataset on another domain is needed; this is something we seek to address in upcoming work. the second is the actual quality of predictions. in preliminary annotation experiments, we were unable to perform on par with the predictions system. while further analysis is required, we suspect that this is attributable to the fact that the chains provide too little information.