Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding (2023)

Abstract: Narrative understanding involves capturing the author’s cognitive processes, providing insights into their knowledge, intentions, beliefs, and desires. Although large language models (LLMs) excel in generating grammatically coherent text, their ability to comprehend the author’s thoughts remains uncertain. This limitation hinders the practical applications of narrative understanding. In this paper, we conduct a comprehensive survey of narrative understanding tasks, thoroughly examining their key features, definitions, taxonomy, associated datasets, training objectives, evaluation metrics, and limitations. Furthermore, we explore the potential of expanding the capabilities of modularized LLMs to address novel narrative understanding tasks. By framing narrative understanding as the retrieval of the author’s imaginative cues that outline the narrative structure, our study introduces a fresh perspective on enhancing narrative comprehension.

## introduction when reading a narrative, it is common for readers to analyze the author's cognitive processes, including their knowledge, intentions, beliefs, and desires @xcite . in general, narrative is a medium for personal experiences @xcite . although large language models (llms) have the capability to generate grammatically coherent texts, their ability to accurately capture the author's thoughts, such as the underlying skeletons or outline prompts devised by the authors themselves @xcite , remains questionable. this is supported by cognitive research that bilingual individuals tend to convey more precise thoughts compared to monolingual english speakers @xcite . the potential deficiency in tracing thoughts within narratives would hinder the practical application of narrative understanding and, thereby preventing readers from fully understanding the true intention of the authors. narrative understanding has been explored through various approaches that aim to recognize thoughts within narratives @xcite @xcite . still, these approaches are often fragmented, focusing on diverse tasks scattered across multiple datasets, obfuscating the fundamental elements (e.g., the characters, the events and their relationships) of the narrative structure @xcite @xcite . to address this gap, in this paper, we lay the foundation and provide a comprehensive synthesis of the aforementioned narrative understanding tasks. we start with the key features of this genre, followed by a formal definition of narrative understanding. we then present a taxonomy of narrative understanding tasks and their associated datasets, exploring how these datasets are constructed, the training objectives, and the evaluation metrics employed. we proceed to investigate the limitations of existing approaches and provide insights into new frontiers that can be explored by leveraging current modularized llms (e.g., gpt with rlhf @xcite ), with a particular focus on potential new tasks. to sum up, our work firstly aligns disparate tasks with the llm paradigm, and categorizes them based on the choices of context and inputoutput format. then it catalogues datasets based on the established taxonomy. subsequently, it introduces bayesian prompt selection as an alternative approach to define the task of narrative understanding. finally, it outlines open research directions. ## definition of narrative understanding narrative texts possess distinct characteristics, which are different from other forms of discourse. elements such as point of view, salient characters, and events, which are associated or arranged in a particular order @xcite @xcite , giving rise to a cohesive story synopsis known as the plots @xcite . its scope spans across various genres, including novels, fiction, films, theatre, and more, within the domain of literary theory @xcite . although it is unnecessary to endorse a particular narrative theory, some elements are commonly encountered in comprehension. for example, readers have to understand the causation or relationship that goes beyond a timeline and delve into the relationships between the characters @xcite . from a model-theoretic perspective, narrative understanding can be described as a process through which the audience perceives the narrator's constructed plot or thoughts @xcite . to this end, we define narrative understanding as the process of reconstructing the writer's creative prompts that sketch the narrative structure @xcite . in line with @xcite , we adopt the practice of using the descriptions of nlp tasks as context to accommodate different paradigms. additionally, we employ the llama @xcite taxonomy to dichotomize this data-oriented task as either multiple-choice or free-form text completion. let {x n , y n } n @xmath0:n are narratives, and y 1:n are the annotated sketches, narrative understanding aims to predict y given x by optimizing p θ (y 1:n |x 1:n , context). existing literature can be roughly categorized based on the format of y n and how context is described. to align with the classical nlp taxonomy, we specify context as a single prompt from either reading comprehension (section 2.1), summarisation (section 2.2), or question answering (section 2.3). in the rest of this section, the context will be further elaborated and specialized in more narrowly-defined tasks to refine the taxonomy, resulting in the reformatting of y n accordingly. ## dataset we have conducted a review of datasets ( ## evaluation methods for classification tasks, such as multiple-choice qa and next sentence prediction, accuracy, precision, recall, and the f1 score serve as the most suitable evaluation metrics. here we mainly discuss evaluation metrics for free-form or generative tasks, crucial yet challenging for narrative understanding. traditional metrics based on n-gram overlap like bleu @xcite , rouge @xcite , meteor @xcite , cider @xcite , wmd @xcite , and their variants, such as nist @xcite , bleus @xcite , sacrebleu @xcite remain popular due to their reproducibility. however, their correlation with human judgment is weak for certain tasks @xcite . alternative content overlap measures have been proposed, such as peak @xcite , which compares weighted summarization content units, and spice @xcite , which evaluates overlap by parsing candidate and reference texts into scene graphs representing objects and relations. the advent of bert introduced a new approach to evaluation, based on contextualized embeddings, as seen in metrics such as bertscore @xcite , moverscore @xcite , and bartscore @xcite . taskspecific evaluations using bert-based classifiers, such as factcc @xcite for summary consistency and qags @xcite to determine if answers are sourced from the document, have been explored. recently, llms have also been employed for evaluation tasks. for instance, canarex (anantharama et al., 2022) assesses time-series cluster recovery from gpt-3 generated synthetic data. unieval @xcite uses a pretrained t5 model for evaluation score computation. gptscore @xcite leverages zero-shot capabilities of chatgpt for text scoring, while g-eval @xcite applies llms within a chain-of-thoughts framework and form-filling paradigm for output quality assessment. to facilitate tracking the research progress, we list the performance of current sota models on representative benchmarks in ## applications narrative assessment the "good at language -> good at thought" fallacy @xcite has spurred the application of narrative understanding to the automatic assessment of student essays. studies on automatic story evaluation @xcite reveal that the referenced metrics, e.g., bleu and rouge scores, deviate from human preferences such as aesthetic and intrigue. this calls for the identification of narrative elements and their relations. for example, @xcite builds a graph of discourse proximity of essay concepts to predict the essay quality w.r.t. the development of ideas and exemplification. @xcite annotates multiple dimensions of narrative quality, such as narrative development and narrative organization, to combat the scarcity of scored essays. such narrativity is also evaluated in @xcite with a focus on detecting the cognitive aspects, i.e., suspense, curiosity, and surprise. other sub-tasks, such as comment generation, are studied in @xcite . ## challenges and future directions prompt tuning and author prompt recapitulation in lieu of the prompt-driven nature of llms that elicits tasks with task descriptions or few-shot examples rather than task-specific finetuning @xcite @xcite @xcite , we propose a unified approach for narrative understanding. this approach involves a single supervised training process, denoted as p θ (y 1:n |x 1:n , context), where context represents a prompt of task description or few-shot examples, and y 1:n denotes predictable annotations produced by crowd-sourced platforms such as amazon mechanical turk @xcite @xcite . while the framework is universally applicable, the practice of relying on annotators to infer authors' thoughts or construct skeleton prompts is inefficient, inaccurate and unreliable @xcite , direct consultation with the authors is often impractical, particularly for posthumous masterworks. as suggested in the model-theoretic view @xcite , a gap needs to be closed between the narrator and the audience to overcome the reader's uncertainty and other environmental limitations. in this regard, we envisage a bayesian perspective @xcite for the recovery of the author's thoughts. let p ϕ (narrative|sketch, task) denote the oracle model that simulates the author's composition process, where the author fills in the conceived skeleton out of their own initiative (where sketch is the skeleton prompts and task is the task description). the recapitulation of the author's prompt can then be expressed as argmax sketch p ϕ (narrative|sketch). previous ef-forts have formulated the objective as predicting the narrative elements or the narrative structure (as discussed in section 2), given the intractability of the likelihood of generating the narrative, p ϕ (narrative|sketch, task). it is also viable, however, to probe and optimize sketch directly, considering the ability of llms to compute the error in close proximation. hence, p ϕ (narrative|sketch) could be derived as the marginal likelihood σ task p ϕ (narrative, task|sketch), and its expectation form e p(task|sketch) p ϕ (narrative|sketch, task) can be approximated by sampling an appropriate task. the dependence between task and sketch aligns well with the composition practice of choosing a suitable genre to match the author's creative intent. in more complex cases where the narrator is known to have employed particular narratology techniques (e.g., flashback @xcite ), task and sketch can be parameterized by generative models (e.g., lm-driven prompt engineers @xcite ), which can be tuned by a prompt base through gradient descent optimization. interactive narrative llms, with their ability to carry out numerous language processing tasks effectively in a zero-shot manner @xcite , are paving the way towards a future of immersive and interactive narrative environments. these environments could resemble the dynamic storylines experienced by individuals, as depicted in the tv series "westworld". agent recent studies @xcite have shown promising results in using a database to store an agent's experiences and thought processes, effectively serving as a personality repository. by retrieving relevant memories from this database and incorporating them into prompts, llms can be guided to predict behaviors, thereby producing human-like responses. however, extracting comprehensive character-centric memory from narratives, encompassing aspects such as "who", "when", "where", "action", "feeling", "causal relation", "outcome", and "prediction" @xcite , remain largely unexplored. current studies primarily focus on simple corpora and there is ample room for further investigation in this area. environment creating immersive and interactive environments for users and agents presents several challenges, primarily due to three key factors: (1) environment extraction. character locations and environments, often vaguely defined unless crucial to the plot, have to be clarified. most works rely on pre-built sandbox environments @xcite @xcite to address this issue. however, challenges remain in extracting and representing the environment accurately. (2) environment generation. interactive narratives aim to provide users with greater freedom, but this poses the challenge of automatically generating reasonable and coherent details within the narrative's world. it is crucial to maintain consistency and engagement in storytelling, despite varying user inputs and directions. (3) environment update. agents' text commands may change the state of the world, requiring accurate and costeffective updates. current systems update environment states using predefined rules @xcite . however, using llms to derive and generate narrative environments challenges the use of predefined rules, making efficient and large-scale environment updates a future research direction. open world knowledge incorporating external knowledge and commonsense has been a longstanding challenge in both dataset construction and model design @xcite @xcite . efforts to address this challenge have been made over the years, with the emergence of llms providing a source potential of commonsense knowledge @xcite . notably, the text world theory (twt) @xcite has been leveraged to provide world knowledge relevant to everyday life, which is simulated through natural language descriptions @xcite . similarly, in @xcite , a text world framework is established, in which the worldbuilding elements (e.g., characters, time and space) are annotated to enhance readers' perception. the text world entails nuanced knowledge derived from the interplay of various elements. however, such supervision provided by the static world is somewhat limited, as the reward is implicit and the model needs to extrapolate from the annotations to exploit the world knowledge. in contrast, the open world @xcite presents an ideal source of supervisor, where the model can be rewarded with incentives derived from world mechanisms @xcite that synergistically complements the audience model with the everyday commonsense. to this end, the rlhf (reinforce-ment learning with human feedback) @xcite strategy could be applied to the openworld system, which enables the iterative training of the narrative understanding process. ## conclusion in this paper, we have systematically examined the emerging field of narrative understanding, cataloguing the approaches and highlighting the unique challenges it poses along with the potential solutions that have emerged. we have emphasized the crucial role of llms in advancing narrative understanding. our intention is for this survey to serve as a thorough guide for researchers navigating this intricate domain, drawing attention to both the commonalities and unique aspects of narrative understanding in relation to other nlp research paradigms. we aspire to bridge the gap between existing works and potential avenues for further development, thus inspiring meaningful and innovative progress in this fascinating field.