Event Causality Is Key to Computational Story Understanding (2024)

Abstract: Cognitive science and symbolic AI research suggest that event causality provides vital information for story understanding. However, machine learning systems for story understanding rarely employ event causality, partially due to the lack of methods that reliably identify open-world causal event relations. Leveraging recent progress in large language models, we present the first method for event causality identification that leads to material improvements in computational story understanding. Our technique sets a new state of the art on the COPES dataset (Wang et al., 2023c) for causal event relation identification. Further, in the downstream story quality evaluation task, the identified causal relations lead to 3.6-16.6% relative improvement on correlation with human ratings. In the multimodal story video-text alignment task, we attain 4.1-10.9% increase on Clip Accuracy and 4.2-13.5% increase on Sentence IoU. The findings indicate substantial untapped potential for event causality in computational story understanding. The codebase is at https://github.com/insundaycathy/Event-Causality-Extraction.

## introduction stories manifest in various forms in modern society, such as myths, fables, gossip, comic books, bedtime rituals for children, and million-dollar theatrical productions. stories are theorized to play crucial roles in civilization, from building collective identities @xcite to familiarizing readers with social skills @xcite . research on computational story generation @xcite @xcite @xcite and understanding @xcite @xcite @xcite has gained traction in recent years. ## methodology our objective is to acquire a causal graph, a directed graph that contains events as nodes and causal relations as directed edges. at the core of here is a list of nodes (events) from a story event graph. we want you to fill in the edges of the event graph with causal connections between nodes. an event graph contains nodes and edges. each node represents an event, and each edge represents the causal connection between two events. example input: node 0: when dan goes to school in the morning, he has to take the bus. node 1: one day dan was running late, and missed the bus to school. our approach is an llm prompt that includes an instruction and a list of story events, as shown in figure 1 . the prompt can contain a number of examples, though we show only one due to length considerations. the prompt requests the llm to detect and output causal relations among the events. for simplicity, we consider each sentence in the story as an event. the output format for the causal relations is edge: (node a -> node b). in preliminary experiments, we find that the arrow "->" notation, similar to the influential dot language @xcite of graph representation, tends to yield better results than other notations we tried. ## event causality extraction to assess the quality of llm-extracted event causal relations, we compare against two human-annotated benchmarks: copes @xcite and glucose @xcite . it is worth noting that the purpose of these experiments is not to seek state-of-the-art performance, but to verify the identified causal relations are of decent quality. however, we still manage to beat state-of-the-art baselines on copes. task definition and datasets for the copes task, the input is a pair of events and the output is whether or not a causal relationship exists between the events. copes contains 340 stories and 1360 event pairs from rocstories, split 50/50 into the validation set and the test set. the glucose dataset contains a number of causal dimensions. we select only dimensions 1 and 6, which concern causality between events. given a story and one of its events, the task is to identify all of the direct causes or effects of the event from the story. glucose paraphrases the identified causes and effects as well as the current event in a subject-verb-object format and applies reference-based evaluation such as bleu. as our technique only outputs causal relations and does not perform paraphrasing, we directly use the original sentences from the story. ## open-domain story evaluation having established that the event causality that we identified are quite accurate, we still need to verify if the event causality provide valuable information to actual story understanding tasks. to this end, we conduct two tests. this section describes the first test, automatic story quality evaluation. we generate the quality ratings from chatgpt with and without the automatically extracted event causal relations, and compare how they correlate with human ratings. ## story video-text alignment the second test task for the automatically extracted event causal relations is story video-text alignment. due to the wide use of storytelling techniques, aligning the video and the text modalities requires video-text similarity ground-truth or predicted alignment causal context v5 temporal context v6 t0 t1 t2 t3 t4 v1 v2 t1 t3 text video segments figure 3: the process of video context identification. the causal context is marked in green, the temporal context in purple, and the current item in red. for illustration purposes, the number of causal and temporal context items are both set to 2. we show that preceding events on the causal graph provide crucial context for this task. considering the domain gaps between this multimodal task and previous pure text tasks, this experiment supports the argument that our proposed technique can cope with a broad range of real-world stories. ## conclusion in this paper, we propose a simple and effective in-context-learning method for extracting event causality from stories with llms. we match and outperform supervised baselines in event causality extraction. furthermore, we validate the quality of the extracted event causality by applying them in downstream story understanding tasks. experiments show that event causality assists story evaluation and video-text alignment, indicating the critical role of event causality in story understanding.