P ro P ara- CRTS : Canonical Referent Tracking for Reliable Evaluation of Entity State Tracking in Process Narratives (2025)

Abstract: Despite the abundance of datasets for procedural texts such as cooking recipes, resources that capture full process narratives, paragraph-long descriptions that follow how multiple entities evolve across a sequence of steps, remain scarce.Although synthetic resources offer useful toy settings, they fail to capture the linguistic variability of naturally occurring prose. ProPara remains the only sizeable, naturally occurring corpus of process narratives, yet ambiguities and inconsistencies in its schema and annotations hinder reliable evaluation of its core task Entity State Tracking (EST).In this paper, we introduce a Canonical Referent Tracking Schema (CRTS) that assigns every surface mention to a unique, immutable discourse referent and records that referent’s existence and location at each step. Applying CRTS to ProPara, we release the re-annotated result as ProPara-CRTS. The new corpus resolves ambiguous participant mentions in ProPara and consistently boosts performance across a variety of models.This suggests that principled schema design and targeted re-annotation can unlock measurable improvements in EST, providing a sharper diagnostic of model capacity in process narratives understanding without any changes to model architecture.

## introduction comprehending changes in a dynamic world is difficult. it requires the model not only to reason about state transitions across multiple steps but also to infer from knowledge of the world. there has been considerable recent progress in understanding naturally occurring procedural texts, such as cooking recipes, wikihow, etc. @xcite , and establishing benchmarks over these datasets has helped drive research in this area of nlp. unlike procedural texts, process narratives describe a process in sequential steps in descrip-figure 1 : an annotated paragraph in propara. each row shows the existence and location of participants before and after each step ("?" denotes "unknown", "-" denotes "not exist"). this example demonstrates the problem of referential confusion where the participant "tadpole" can refer to more than one entity. red denotes the paragraph and annotation we find problematic, green denotes the annotation based on our new schema. although there exist a few datasets related to process narratives, many of them are not in natural language but are synthetic texts @xcite . recently there have been more datasets using natural language, but due to the rarity of the real-world data and expert knowledge needed to create these sets, most of these datasets are in a specific domain @xcite @xcite @xcite . propara @xcite is the only existing dataset of process narratives in natural language, covering diverse domains. it is a dataset of human-authored paragraphs of realworld processes, along with annotations about the changing states (existence and location) of entities in these processes. figure 1 shows an annotated paragraph in propara. annotators first construct the process steps given the prompt "describe the life cycle of a frog". then they select entities of interests of the process as participants. later, annotators track the existence (i.e., create, destroy, and none) and location changes of the participants. the resulting benchmark underpins the entity state tracking (est) task, which requires models to predict those step-wise state transitions. despite its value as a comparatively large resource in a data-scarce genre, propara's original annotation schema is not fully aligned with the formal requirements of est evaluation. in particular, the schema tolerates referential ambiguity and allows multiple surface mentions to be conflated under a single participant label, leading to inconsistent state chains. figure 1 illustrates the problem: the label tadpole is used for two distinct entities introduced in step 1 and step 5, but both are erroneously merged. consequently, a system that correctly predicts the destruction of the first tadpole is penalized because the gold annotation wrongly asserts that tadpole is (re)created later in the narrative. such annotation artifacts obscure true model performance and impede principled analyses of reasoning errors. to address these problems, we propose a canonical referent tracking schema (crts) and introduce a re-annotated dataset propara-crts. crts is a tightly specified annotation framework that assigns every surface mention in a process narrative to a unique, immutable discourse referent and obliges annotators to record that referent's existence and location at every step in the text. the re-annotation proceeds in three interconnected stages. first, at the paragraph level, we apply dense paraphrasing @xcite to rewrite or split sentences so that every entity mention is self-contained, eliminating referential ambiguity in the running text. next, at the participant level, we merge coreferential mentions and assign each cluster a single canonical name-its crts referent-thereby establishing a strict one-to-one correspondence between discourse entity and participant label. finally, at the state-label level, we traverse the revised step sequence and re-annotate existence and location, adding previously implicit transitions, sharpening coarse location spans, and guaranteeing that each referent experiences at most one state change per step. to investigate the effectiveness of our reannotation on the evaluation of the est task, we train the state-of-the-art models on the re-annotated data and are able to achieve higher performances and the predictions are more reasonable based on human inspection. we evaluate llms with a range of reasoning scaffolds and observe modest but consistent gains on propara-crts relative to the original corpus. this suggests that the cleaner reference mapping removes label noise that previously suppressed zeroshot scores. even with best performing prompting, however, the model still trails supervised systems by a large margin. when we fine-tune a parameter-efficient llama-3-8b on the crts training split, the model surpasses zero-shot llms and approaches the performance of fully supervised baselines, confirming that llms can internalize canonical referent tracking once given sufficient task-specific examples. foot_0 2 related work procedural texts comprehension wiqa (tandon et al., 2019) evaluates models' performance on "what if" questions regarding procedural texts. trip @xcite is a dataset created to evaluate the reasoning ability of language models related to procedural physics texts. mars @xcite evaluates the understanding of event and state changes in processes and how metaphysical changes to certain aspects of the process impact the process. @xcite propose a new model, cgli, that builds local and global representations to track entities in procedural texts showing improvement on propara and trip. other datasets focusing on the est task include openpi @xcite and its derivations and iterations, openpi-c @xcite , and openpi2.0 @xcite . at each step of a process, these datasets ask models to produce the entities involved in that step, the states about them that change, and the before and after states. @xcite propose a new task, tne (text-based np enrichment) which aims to collect all relevant information about an np from a paragraph. throughout a given text, this task challenges models to track the attributes of and participation of entities in events. est is both a complement to and a component of this task. ## canonical referent tracking schema canonical referent tracking schema (crts) is an annotation framework that maps every surface mention to a unique and immutable discourse referent-the single authoritative or canonical representation of that entity, corresponding to the participant in propara-and obliges annotators to record that referent's existence and location at every step in the discourse. three constraints follow: (i) referent uniqueness-a one-to-one mapping between participants and mentions, so that referent of distinct entities are never conflated; (ii) temporal atomicity-a referent can undergo at most one state transition per step; and (iii) complete state accounting-consistent state and location values are obligatory for every mention at every step, including transitions that are only implicit in the text. when any of these constraints is violated, gold annotations no longer reflect the ground-truth reasoning problem, and evaluation scores conflate model error with annotation noise. in this section, we show how propara systematically violates each principle and why those violations obscure a model's true competence in est. ## re-annotating propara to align propara with the crts constraints in §3, we perform a systematic re-annotation based on the dense paraphrasing (dp) technique @xcite . we treat dp as a truthpreserving textual enrichment strategy ϕ that maps a textual unit u (clause/sentence) to an enriched form u + = ϕ(u) in which otherwise implicit event roles, entity distinctions, and state transitions licensed by local lexical and discourse context are made explicit, with semantic fidelity u + ⇒ u. in this work, dp targets explicitness over economy by realizing missing arguments/roles, canonicalizing discourse referents, and expressing step-level state changes that are only implicit in u. ## experiments in order to investigate the effectiveness of the new annotation, we evaluate different models against propara-crts and compare the results with those of the original propara dataset. ## results evaluation scheme following the original propara setup, models are judged by their ability to answer three qa categories derived from an entity-step grid: (cat1) whether an entity is created/destroyed/moved, (cat2) at which step the change occurs, and (cat3) where it occurs @xcite . in our reporting, sentence-level scores evaluate these predictions at the granularity of entity-step pairs, aggregated as macro/micro f 1 over cat1-3. complementarily, document-level scores treat all answers for a paragraph as a set of predicted tuples (entity, change-type, step, locations) and compute precision/recall/f 1 against the gold set, emphasizing global consistency across the whole process @xcite . effect of canonical reference across all settings propara-crts yields higher f1 scores than the original annotation except for gpt-4o-mini at sentence-level. for gpt-4o the absolute gain ranges from +0.4 points with the direct prompt to +1.7 points with self-consistency. the trend is even clearer for trained models: llama-3-8b fine-tuned on crts improves by +4.1 points at the sentence level and by +1.0 points at the document level; cgli and meet register gains of +4.1 and +7.0 points respectively. these consistent improvements confirm that enforcing a canonical, one-to-one referent mapping removes annotation noise that previously capped model scores. ## analysis to further investigate the effectiveness of the reannotation, we look into the prediction differences in propara and propara-crts and see if the models trained on propara-crts understand process narratives better. ## conclusion we have presented propara-crts, a rigorously reannotated version of the propara corpus that re-places the original, ambiguity-prone schema with a canonical referent tracking schema. crts enforces one-to-one mention-referent mapping, stepwise atomicity, and exhaustive state accounting. dp supplies the minimal lexical edits needed to make colliding mentions distinguishable while preserving the original prose. during re-annotation we also corrected recurrent paragraph and statelabel errors, yielding 452 paragraphs with 13,417 noise-free state triples. experiments spanning from llm prompting, llm fine-tuning and supervised models show consistent gains on crts. the results confirm three claims: (i) referential canonicalization removes label noise that previously suppressed scores; (ii) prompts that elicit multi-step reasoning profit most from the cleaner supervision; and (iii) despite these gains, est remains challenging-supervised models still outperform purely prompted llms, underscoring the importance of dedicated training data. we release propara-crts, annotation guidelines, and validation scripts to facilitate future work on robust, semantics-aware evaluation of est in natural-language process narratives.