Memory and Knowledge Augmented Language Models for Inferring Salience in Long-Form Stories (2021)

Abstract: Measuring event salience is essential in the understanding of stories. This paper takes a recent unsupervised method for salience detection derived from Barthes Cardinal Functions and theories of surprise and applies it to longer narrative forms. We improve the standard transformer language model by incorporating an external knowledgebase (derived from Retrieval Augmented Generation) and adding a memory mechanism to enhance performance on longer works. We use a novel approach to derive salience annotation using chapter-aligned summaries from the Shmoop corpus for classic literary works. Our evaluation against this data demonstrates that our salience detection model improves performance over and above a non-knowledgebase and memory augmented language model, both of which are crucial to this improvement.

## introduction forster @xcite compared a story to a wriggling worm of time that can be seen as a series of events arranged in order (see also @xcite -dinner comes after breakfast, night after day, nemesis follows hubris. not all events are of equal importance, and some far more salient than others. for example, the beginning of dickens' great expectations -keep still, you little devil, or i'll cut your throat! -is more salient to the story than events such as my sister had a trenchant way of cutting our bread and butter for us. salient events in storytelling are those that drive the plot forward, change the state in the story world, as opposed to descriptive details or non-consequential activities. as such, detecting salience is an essential part of understanding narrative. detecting salient events has important downstream applications such as summarisation; salient events are the core of plots and can aid storyline writing and story generation; they represent essential information and are relevant to question answering. this paper builds on the work of @xcite , who use barthes cardinal functions (bcf) for unsupervised salience detection. we augment this approach with a knowledgebase (kb) and memory. barthes cardinal functions @xcite are hinge events that cannot be deleted without altering the story; they are the decision points between alternative consequential paths. barthes and duisit also define catalysers, which are inconsequential events such as the bread and butter example, indices, which are descriptive, referring to a character or situation, and informants, which identify time and space. these latter types can be seen as satellites around the nuclei, or filling in gaps between cardinal functions. hence to identify bcf is to identify the main skeleton of the plot. we treat the bcf events as the salient events in a story. this scheme relates in narratology with @xcite kernels and satellites model, as well as with discourse theory in rst @xcite , which similarly has nuclei and satellites and more loosely with sdrt @xcite with coordinating and subordinating relations. the key to the otake et al. method is that it can be implemented using any lm (language model) on any text and does not require a large corpus of annotated training data. in this paper, we extend the bcf concept by exploring new measures of salience derived from structural manipulations: we infer swap salience, which is swapping rather than deleting an event within the bcf framework. @xcite discusses how an event can be salient if a reader expects it, but it is unexpected to the character in the story. the reader puts themselves into the character's shoes. @xcite emphasises how suspense is driven by anticipation and apprehension on behalf of characters the reader cares about. @xcite propose to use this knowledge disparity between the reader and the character to create more suspenseful plots and hence more important events. we model knowledge salience as the difference between an expert-informed reader versus a naive one by taking the difference between the average log-likelihood of a base lm and an lm enriched with memory and a kb. we also take inspiration from the model of @xcite , who compute suspense and surprise in short stories using vector states from a hierarchical model; this follows from theoretical work by @xcite , and cognitive work from @xcite . we show how a vector salience measure can be computed based on this approach. in addition to exploring new salience measures, our work aims to overcome limitations of existing work on salience modeling. otake et al. ( 2020 ) only evaluate their model on a single type of narrative (russian fairytales) and on a very small annotated dataset. we address this by using aligned summaries from the shmoop corpus @xcite to provide salience labels. this results in a large dataset of long works (novels and plays) annotated with silver-standard salience labels. a second limitation of otake et al. is that they use gpt-2 @xcite as their lm, which has a relatively short context of a few hundred wordpieces. while this works for short stories, the context is too short to track lengthy novels or plays. often a character will disappear for a long period; for example, abel magwitch in great expectations. plots are often non-linear with recalls and flash-forwards, and the same characters and places reoccur at intermittent points in the story. at any moment in the story, the most relevant passages are not the most recent but the previous actions of the characters, places, and situations involved. we address this limitation by incorporating an episodic knowledge retrieval mechanism (derived from rag; @xcite ) and fuse this with a short-term memory mechanism that can extend the capabilities of a transformer lm. the intent is that the memory will learn to recall the most relevant parts of the story, act as an implicit index into these dimensions, and the kb will supplement this with typical plot knowledge. this memory mechanism is much more suitable than recent work on extended transformers for longer sequences, see @xcite and @xcite for thorough reviews. characters, places, subplots ebb and flow in long stories, so the most relevant information may be hundreds of pages previous with mainly irrelevant information in-between, which suits indexed episodic memory rather than a transformer that must filter out the mainly irrelevant details in-between. for example, abel magwitch in great expectations is in the first two chapters and then reappears explicitly in chapter 40. our results show that integrating kb and memory components improves the overall performance of salience detection. using a vector alternative to infer salience is a slight improvement over the lm. other measures such as swap salience and knowledge salience perform worse than the main salience measures but still show improvements over our baseline model. ## related work the main architectural innovation is to use an external knowledgebase, based on rag @xcite , and combine this seamlessly with a memory mechanism to improve the model's predictive performance. the main structure of this model is to use a question and document encoder, both transformers, to learn and look up passages of text from a knowledgebase (based on dpr; @xcite ) and then fuse this knowledge into a transformer encoder/decoder model such as bart @xcite or t5 @xcite . similar models including realm @xcite , hard em @xcite , spanseqgen @xcite , and fusion-in-decoder (izacard and grave, 2021) have achieved state-of-theart results in factual domains such as answering natural language questions, trivia or games such as jeopardy. in these domains, the key insight is that offloading knowledge externally allows models to perform better than much larger transformers that need to encode all knowledge in their weights. these methods that rely on retrieving raw text are also competitive with those that have tried to incorporate structured information such as graphretriever @xcite or pathretriever @xcite . we experiment both with a wikipedia kb and wikiplots, a kb of story plot summaries. the motive for the latter is that these plot fragments or vignettes act as a planning system (or schema; @xcite guiding expectations. @xcite used a similar concept in a rule-based system. @xcite also use a bag-like episodic memory mechanism for inference in stories without the more sophisticated transformer encoders of the rag model. after the experimental work in this paper, a follow-up paper by @xcite on several rag variants found that the kb was able to reduce the amount of hallucination in generating dialogue. the kb grounds the text generation in relevant facts retrieved from the kb. while the story domain is different intuitively, the same effect is desirable; inferring salience should be grounded either in plot knowledge from wikiplots or general knowledge from wikipedia, and also the memory of the previous character actions and plot developments. ## conclusion the main overall finding is that the bcf method can infer salience over and above baselines with an improvement on much longer works. we find that augmenting an lm with memory and an external kb can improve the detection of salience and increase the predictive power of the lm on narrative text. we also find that a vector-based version of the concept can perform slightly better than using the log-likelihood from an lm. therefore, this paper demonstrates that it is feasible to run an unsupervised method on novels from dickens or plays by shakespeare and achieve correlation with an automated silver label benchmark. nevertheless, the map results are around 0.3, and rouge-l is 0.4, which leaves room for improvement. one factor in the moderate increase could be that the salience modelling is explicitly local over the label of the n next tokens. this is more a local view of salience as intended from the reader perspective. the model may flag up false leads that are locally important but not globally for the plot. in contrast, the shmoop is written with the knowledge of the whole story, and so will exclude them. a more reader orientated evaluation is for future work. although the shmoop alignment is generally strong, there are occasions where arguably multiple sentences could be deemed the correct one, and the silver label is one and the salient peak the other. with this unsupervised approach, performance is likely to be underestimated as the labels are entirely independent. in contrast to much recent supervised work, such as pegasus @xcite summarization has additional human evaluation on some datasets. system, use silver labels created with proxies such as rouge. the labels both train the system and are evaluated on. even with a separate test set performance, the system is more likely to replicate any noisy misalignments in the labelling process and overestimate performance. on future work, if rag can improve lm prediction performance and help infer salience, then the same models would seem to hold promise in improving text generation, including story generation. the knowledge salience approach is a simple attempt to model the informed reader versus the naive one. in narratology, the characters perspective is crucial in for example eventfulness @xcite @xcite notion of characters crossing a forbidden semantic border; or suspense as per @xcite , or @xcite concept of the reader as problem solver. there is, therefore, rich potential work in modelling character states, knowledge, intents and contrasting them with the readers' expectations, and the norms of the narrative world in inferring concepts such as salience, suspense, and surprise. characters could be implicitly modelled using a per entity memory model extending the current rag approach. or take a more structured approach inspired by recent work such as sims and bamman (2020) modelling literary character communication, or story generation systems such as cast @xcite that model multiple characters goals or c2po @xcite that more explicitly models causal chain relations.