Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions (2024)

Abstract: This work introduces an original and practical paradigm for narrative comprehension, stemming from the characteristics that individual passages within narratives tend to be more cohesively related than isolated.Complementary to the common end-to-end paradigm, we propose a fine-grained modeling of narrative context, by formulating a graph dubbed NarCo, which explicitly depicts task-agnostic coherence dependencies that are ready to be consumed by various downstream tasks. In particular, edges in NarCo encompass free-form retrospective questions between context snippets, inspired by human cognitive perception that constantly reinstates relevant events from prior context. Importantly, our graph formalism is practically instantiated by LLMs without human annotations, through our designed two-stage prompting scheme.To examine the graph properties and its utility, we conduct three studies in narratives, each from a unique angle: edge relation efficacy, local context enrichment, and broader application in QA. All tasks could benefit from the explicit coherence captured by NarCo.

## introduction since the advent of large language models (llms), document comprehension has been improved significantly by simply employing the endto-end generative paradigm. especially, with long context window enabled via techniques such as position interpolation @xcite , cached or efficient attention @xcite @xcite , context compression or pruning @xcite @xcite , the end-to-end paradigm is deemed undoubtedly simple and effective for comprehension tasks (e.g. question answering) on various documents. however, while the typical benchmarks are continually enhanced by more advanced llms (tou-* corresponding author. @xcite @xcite , the end-to-end paradigm may not suffice for all comprehension scenarios. in this work, we focus around the narrative context, i.e. stories or novels, and propose a conceptually original framework of fine-grained context modeling: a graph is formulated that depicts the relations between context snippets, abstracting over the context to reflect a high-level understanding of the narrative. the graph itself is practically realized by llms to harness their rapidly evolving strengths, and the resulting graph could serve to facilitate various downstream narrative comprehension tasks. our motivation arises from the distinctive nature of narratives: multiple development of characters or events in a story could be entangled over long context ranges, where each local passage usually serves specific purposes for others. thus, individual passages tend to be cohesively interconnected than being isolated. as the end-to-end paradigm implicitly grasps these context connections through sequence modeling, our approach explicitly models these dependency relations to capture coherence, offering a directly-applicable alternative path orthogonal to the end-to-end paradigm. concretely, drawing inspiration from the cognitive process on narrative perception, whereas humans can constantly reinstate relevant or causal events from past context during reading @xcite , our formalism, termed narrative cognition graph (narco), splits the entire context into chunks that act as graph nodes, with edges representing the relations between node pairs. in particular, edge relations are constituted by free-form questions. as humans could relate to past context in retrospect, accordingly, each question in narco edges arises from the succeeding node (latter context), asking necessary background or causes that can be clarified by the preceding node (prior context). hence, graph edges consist of inquisitive questions that naturally reflect retrospection. overall, the resulting graph explicitly depicts task-agnostic understanding of fine-grained coherence flow that could be flexibly utilized by downstream tasks. though our graph formulation partially shares motivations with discourse parsing that characterizes how each proposition relates to others within a close context @xcite , our method targets on practical utility for narrative comprehension, where edges in narco are designed to be easily obtained and effectively consumed by downstream tasks. consequently, narco is formulated in a different scope from discourse parsing by two main perspectives. first, as most discourse frameworks, such as rhetorical structure theory @xcite , penn discourse treebank @xcite , or the recent questions under discussion (qud) @xcite are rooted upon linguistic principles, their relation types are oriented for formal discourse analysis, requiring trained experts to annotate edges according to a defined linguistic taxonomy. whilst for narco, the relation space is larger without taxonomy constraints, offering diverse high-level semantic signals for narrative tasks. second, narco practically leverages llms to derive edge relations, without reliance on human annotations. thus, the edge quality is not restricted by annotation resources, and shall be continuously enhanced along with the ongoing llm advancement. the key difficulty of narco lies in the edge realization between two nodes, which itself demands strong context understanding to determine which aspects to inquire upon the context, and to assess their saliency for comprehension. such process is especially strenuous due to the large hypothesis space compared to conventional discourse formalisms. to this end, we pose soft semantic constraints on relations, and employ llm's capabilities to construct edges automatically through our proposed prompting scheme, of which consists a question generation stage and a self verification stage (section 3). the obtained edges could then be utilized by downstream tasks in two primary ways. first, edges themselves directly provide information flow to guide various comprehension tasks. second, they offer global coherence view for each node, thereby augmenting the local context to deepen the digest of independent passages. to empirically demonstrate the practical utility of narco, we present three studies on narrative comprehension tasks, each from a unique angle: overall, our key contributions in this work are: • we propose a new paradigm of fine-grained context modeling to facilitate narrative comprehension, orthogonal to the end-to-end paradigm. ## related work questions under discussion qud is a linguistic framework with rich history that approaches discourse and pragmatics analysis by repeatedly resolving queries triggered by prior context @xcite @xcite . qud has been adapted by recent works for discourse analysis @xcite @xcite @xcite or other applications @xcite . our proposed narco also adopts question-form relations; though, the scope and motivation is different from discourse analysis. consequently, narco differs from qud works considerably on the following design choices. ## narco: narrative cognition graph in this section, we start by delineating our graph formulation, which is itself not tied to any particular implementation. subsequently, we elaborate our specific graph realization using llms, without dependence on human annotations. ## study i: edge efficacy our first study examines the graph edges on whether they express useful relations, such that the generated retrospective questions should bridge the coherence between two context. for appropriate assessment, we conduct recap identification on recident dataset @xcite , a task on narratives that identifies whether certain preceding snippets can function as a recap or prelude to the audience in regards to a current context. concretely, the input takes a short snippet from a novel or show script, along with a provided list of its preceding snippets. the task resolves which preceding snippets are directly related with the current one in terms of plot progression, requiring contextual understanding of narrative development. as narco is designed to capture the coherence relations between context, edges of retrospective questions could be leveraged to link the current snippet to related preceding ones. therefore, recident serves as a natural testbed for comprehensive evaluation of edge efficacy. ## study ii: node augmentation our second study underscores the narco utility of local context augmentation, examining whether the graph typology could enrich node representation with global contextual information. specifically, for a node v j , a preceding node v i and succeeding node v k such that @xmath0, e ij depicts outgoing questions arising from v j to v i , and e jk specifies incoming questions from v k that can be clarified by e j . these questions either highlight important aspects of events or situations in the current context, or provide implication of subsequent development. such auxiliary information from neighboring nodes is especially useful for retrieval on narratives, as each passage tends to be more interconnected with others than isolated. we hence investigate if an embedding function on top of narco could lead to enriched local representation. towards this objective, we consider the plot retrieval task defined in @xcite , which aims to find the most relevant story snippets given a query of short plot description. it is challenging as queries are often abstract based on readers' overall understanding of the stories, requiring essential background information clarified on candidates, analogous to the concept of decontexualization @xcite . retrieval on narratives thereby fits our evaluation purpose well. ## study iii: broader application our last study sheds light on the potentials of graph utility in broader applications. as a first step towards this new direction, in this work, we evaluate with retrieval-augmented generation (rag) @xcite in the task of long document question answering. experiments are conducted on @xcite , a multi-choice qa dataset on narrative documents, mostly being fiction stories from project gutenberg. with an averaged length of 5k+ tokens per document, we adopt the retrieval-based approaches, where relevant snippets conditioned on the question are retrieved first, then fed to a llm to generate answers, following a standard rag paradigm. especially, quality was constructed with global evidences in mind: questions may require multiple parts in the document to reason upon. therefore, narco may assist to recognize more relevant snippets through the extracted relations across the narrative context, leading to improved qa performance benefited from enhanced retrieval. methods retrieval-based approaches are commonly adopted for tackling long context, which have been evaluated on quality by previous works @xcite @xcite . following these setup, we split the full document by short snippets, and retrieve relevant snippets with regard to the question, which are then concatenated as the shortened context for subsequent zero-shot qa inference by llms. to leverage narco, we apply the same retrieval process described in section 5.1.1 to identify relevant snippets, where the query-edge similarity is interpolated as in eq (2) using bge-large encoder. experiments we employ llama2 @xcite and chatgpt for the zero-shot qa inference. as evaluation on the test set requires submission to the zeroscrolls leaderboard @xcite , we first perform fine-grained performance analysis on the dev set with short retrieved context (<1k tokens), then submit the final test set results using chatgpt with 1.5k context limit, aligned with @xcite for direct comparison. the baseline retrieval method and our enhanced retrieval are denoted by r and er respectively. having demonstrated that narco can improve rag in narratives through enhanced retrieval, its utility beyond the retrieval process may be further exploited, e.g. potential facilitation on llm pretraining or inference directly. we leave future research to explore additional integration of finegrained context modeling. ## conclusion we address the distinctive characteristics of narratives, and propose a novel paradigm of fine-grained context modeling, which explicitly captures the inter-connective coherence within narrative context. a graph is thereby formulated, dubbed narco, with edges encompassing free-form retrospective questions to depict the relational dependencies. narco is practically realized by llms via our designed two-stage prompting scheme, leveraging the promising development of llms without reliance on human annotations. to examine the graph properties and its utility, three unique studies are conducted, where narco is shown to bring empirical improvement on various narrative applications.