F act T rack: Time-Aware World State Tracking in Story Outlines (2025)

Abstract: While accurately detecting and correcting factual contradictions in language model outputs has become increasingly important as their capabilities improve, doing so is highly challenging. We propose a novel method, FactTrack, for tracking atomic facts and addressing factual contradictions. Crucially, FactTrack also maintains time-aware validity intervals for each fact, allowing for change over time. At a high level, FactTrack consists of a four-step pipeline to update a world state data structure for each new event: (1) decompose the event into directional atomic facts; (2) determine the validity interval of each atomic fact using the world state; (3) detect contradictions with existing facts in the world state; and finally (4) add new facts to the world state and update existing atomic facts. When we apply FactTrack to contradiction detection on structured story outlines, we find that FactTrack using LLaMA2-7B-Chat substantially outperforms a fair baseline using LLaMA2-7B-Chat, and achieves performance comparable to a GPT4 baseline. Moreover, when using GPT4, FactTrack significantly outperforms the GPT4 baseline.

## introduction large language models (llms) have recently surpassed human performance across a wide range of tasks @xcite , yet generating long-form text remains fraught with challenges compared to tasks with shorter outputs. even when models are trained to support context windows of hundreds of thousands of tokens, they may still struggle to retrieve and reason over such long context @xcite . the most advanced existing language models still take long context generation as a direction for further improvement in the future. 1 our code and data are at https://github.com/ cogito233/fact-track . they do not contradict each other, rather, they can be seen as complementary parts of a larger narrative. after event 1, the monster is already killed by eva, it can not be defeated again in event n. so they are contradictory. event 1 indicates that john cannot use the laptop, but in event 2, john uses his laptop for work. so, they are contradictory. the relationship between john and his laptop is updated by event 1.5, so event 2 is not contradicted. figure 1 : facttrack tackles the problems of factual inconsistency and plot redundancy. note that those problems are based on our observations, to provide a clearer understanding of the problem, with both issues considered together in our pipeline and evaluation. for factual inconsistency detection, facttrack tracks a validity interval for each fact to distinguish legitimate contradictions from facts simply changing over time. for plot redundancy detection, our method can represent the timeline in more structured form, making detection easier. existing works using llms in hierarchical generation have explored structured approaches to maintain strong internal coherence within extensive texts of thousands of words @xcite . however, challenges remain in maintaining factual consistency and avoiding hallucinations during the generation. especially when these problems occur in a high-level planning stage, they can greatly damage performance on downstream tasks. therefore, a system capable of detecting factual contradictions and correcting them is essential. in figure 1 , we take story planning as an example, depicting two common issues encountered: factual inconsistency and plot redundancy. such issues are expected across various domains, as training corpus documents often suffer from length limitations and originate from fragmented sources, impairing the model's ability to establish long-distance, multi-state connections. moreover, generalizing current sentence-level fact verification models @xcite to accommodate longer texts remains a challenge. we highlight two main distinctions that arise when comparing sentence-level fact statements to lengthier texts: the complexity of multi-fact contexts, and the dynamic, stateful nature of content over time. to tackle these challenges, section 3 introduces the concept of directional atomic facts, establishing a universal framework for time-aware contradiction detection and state tracking. section 3.1 details the analysis of structures to identify atomic facts and detect contradictions around key events, employing llms for event decomposition into pre-facts and post-facts. the concept of pre-facts and postfacts is akin to preconditions and postconditions but formulated in natural text, and implying truthfulness for the entire validity time interval. this approach facilitates tracking the state changes over time prompted by events. maintaining a comprehensive non-contradictory set of facts (world state) also allows for detection of contradictions by verifying and updating the atomic facts within the world state. section 3.2 explores the broader implications of atomic facts through a more flexible and time-aware framework, introducing a timeline for events and atomic facts to identify contradictions or updates through their temporal overlaps. in section 4, we describe the main implementation of our method, facttrack. as shown in figure 5 , we first maintain a list of pre-facts and a list of post-facts as our data structure. to update our data structure for a new event, we use a four-step pipeline: decompose-determine-contradiction-update. with this data structure, we can detect whether two events contradict each other and identify the specific pair of atomic facts in conflict, guiding the subsequent correction procedure. furthermore, the validity interval of facts may be useful structural information in and of itself for downstream tasks. to measure the effectiveness of our approach, we introduce a task for detecting contradictions in the planning procedure, with story outlines serving as our test domain. for event pairs flagged by a method as contradictory, we use gpt-4 to annotate whether they are actually contradictory, on a 1 to 5 scale. experimental results show that contradictions flagged by facttrack (using llama2-7b-chat as a base llm) are judged to be real contradictions by a score margin of 0.384 more than a fair baseline using llama2-7b-chat. additionally, when facttrack is run on gpt4, the performance significantly surpasses all baselines, including the one using gpt-4 as a base model. in summary, our contributions are as follows: state tracking. prior work on state tracking ranges from dialogue tracking @xcite and memory and entities networks @xcite to using neural checklists @xcite and story planning @xcite . with the advent of llms, state tracking in long-form story planning and generation has shifted towards explicit tracking using natural language, ranging from unstructured text @xcite to structured dictionaries @xcite . additionally, there also exists some work on generating better temporal fact validity by predicting the validity interval @xcite , jointly modeling text with its timestamp @xcite , or utilizing wikipedia timestamps @xcite . our approach is related to the latter, but the main difference is our method operates on the generated output from a language model, and focuses on post-hoc detection rather than the calibration of a language model. hierarchical generation. hierarchical generation is widely applicable across various domains of long-form content creation, such as story generation. it can be implemented through the internal hidden states of the model @xcite @xcite , or explicitly via natural language text or structured schema @xcite @xcite @xcite . the paradigm of hierarchical generation brings both benefits and challenges to our work. on one hand, it facilitates the resolution of contradictions at higher levels compared to those detected at the bottom level or during sequential generation. on the other hand, it may require a more complex data structure to maintain the validity intervals of facts. ## time-aware atomic facts existing work focuses on leveraging the semantic decomposition capabilities of llms for finegrained fact-checking @xcite . in this session, we propose an event-centric, time-aware atomic fact decomposition method. we track facts on a timeline, building toward the design of our facttrack system. by analyzing the structure of an event and the representation of the world, we develop a method to better decompose events into atomic facts (section 3.1) with forward and backward directions. we then discuss how to use knowledge of contradictions between atomic facts to determine the validity intervals of these atomic facts on the timeline (section 3.2). through this decomposition of facts and by maintaining validity intervals, our method is particularly effective for time-varying facts as is common in domains such as story writing. examples illustrating the concepts in this section are provided in appendix a. ## facttrack we are now ready to introduce our method, fact-track (figure 5 ). facttrack can be understood as a data structure for representing a world state (section 4.1) coupled with a pipeline of operations for updating this data structure given a new event that we want to insert (section 4.2). notably, events do not need to be inserted in chronological order, which is useful for hierarchical inputs such as story outlines. in our pipeline, we start by breaking down the event into several pre-facts and post-facts. then for each fact, we conduct a series of interval operations to determine its validity interval, identify any contradictions, and finally update the validity interval of facts in world state. we also discuss how our method recognizes contradictions between two atomic facts (section 4.3). ## evaluation experiment setup. to examine our method's empirical effectiveness, we apply facttrack to the task of detecting the contradictions in a story outline. while our method can also work online, detecting problems during the outline generation process and providing feedback, we evaluate only offline in this work. we employ an outline generator similar to the detailed outliner from @xcite , modifying the prompt to include more factual information (appendix c.2). in particular, we follow their paradigm of breadth-first outline expansion. to balance length and knowledge density, we estimate there are 3-5 true contradictions per outline on average (appendix b.5). in the evaluation process, we randomly select 90 premises from the writingprompts dataset @xcite and use llama2-7b-chat @xcite to generate the outlines; note that we run fact-track on llama2-7b-chat as well. the task can be understood as a detection task: given the outline, we want to retrieve some candidate event pairs, indicating that the model considers them to be contradictory. outline statistics are shown in appendix d. baselines. with no directly applicable prior methods, we designed two baselines: ## conclusion and future work as the complexity of texts generated by llms increases, understanding their structures and tracking time-varying factual information becomes an increasingly important bottleneck in long-form generation. in this work, we have introduced fact-track, a fact-tracking framework that decomposes events into pre-facts and post-facts and maps them onto a timeline, facilitating the tracking of narrative progress and detecting contradictions between events. experimental results show that when we apply facttrack to contradiction detection on structured story outlines using llama2-7b-chat as the base model, our method achieves performance comparable to gpt4 and significantly outperforms baselines using the same base llm. furthermore, when we run our method on gpt4, we find that its performance significantly outperforms all baselines. by effectively maintaining factual consistency over extended contexts, we envision facttrack serving not only as a fact-tracking module for complex content planning but also as an efficient automatic evaluation metric for contradictions in extensive texts. we additionally envision several further possibilities for improving our system, such as enhancing decomposition accuracy, structuring atomic facts more effectively (for instance, based on entities), and/or maintaining timelines in narratives that do not follow a strict chronological order. moreover, although we only experiment on story outlines in this work, facttrack is in principle generally applicable to other domains, and we hope that our framework's capacity for managing timespecific knowledge could be of use in other areas as well, such as detecting fake news and dynamically updating knowledge bases.