O pen PI 2.0: An Improved Dataset for Entity Tracking in Texts (2024)

Abstract: Much texts describe a changing world (e.g., procedures, stories, newswires), and understanding them requires tracking how entities change. An earlier dataset, OpenPI, provided crowdsourced annotations of entity state changes in text. However, a major limitation was that those annotations were free-form and did not identify salient changes, hampering model evaluation. To overcome these limitations, we present an improved dataset, OpenPI2.0, where entities and attributes are fully canonicalized and additional entity salience annotations are added. On our fairer evaluation setting, we find that current state-of-the-art language models are far from competent. We also show that using state changes of salient entities as a chain-of-thought prompt, downstream performance is improved on tasks such as question answering and classical planning, outperforming the setting involving all related entities indiscriminately. We offer OpenPI2.0 for the continued development of models that can understand the dynamics of entities in text.

## introduction tracking entity states in procedural texts @xcite @xcite is closely related to many nlp reasoning tasks. to name a few, question answering about events (e.g., should one use gloves when retrieving the tray from the oven) often require knowledge of entity states (e.g., the tray becomes very hot while in the oven; gloves insulate heat) @xcite @xcite ; planning @xcite largely involves actions upon entities resulting in state changes. while most recent work has relied on end-to-end language models (lms) @xcite , recent work has shown that explicit modeling entity states benefits lms in such tasks @xcite . procedural entity tracking is challenging in itself, requiring much understanding of an implicit environment as well as external knowledge of events and entities. we propose the openpi2.0 dataset which builds on openpi (open procedural inference) @xcite , a large-scale dataset for tracking entity states in procedural texts. openpi contains annotations of entities, attributes, and state changes for each step (e.g., after the step "set the pan in a heated oven", the pan's temperature was cool before and hot afterwards). openpi2.0 features two critical improvements (see figure 1 for a demonstration of key features of openpi and openpi2.0): 1. canonicalization. originally, different mentions of the same entity or attribute render evaluation difficult. here, we prompt lms to effec-figure 2 : an example from the original openpi dataset @xcite . regarding entity salience, we provide both manually annotated and automatically predicted labels. we evaluate them based on correlation with groundtruth data, and show that lms can reliably predict entity salience with a close-to-human performance. we argue that salient entities act as a means of compression of the most critical information in procedural texts, similar to saliency maps in computer vision @xcite . we proceed to qualitatively and quantitatively show that salient entities, as chain-of-though of lm prompting, benefit downstream tasks such as question answering and classical planning, while reducing cost by excluding less important entities in the prompt. openpi2.0 have following advantages: ## the original openpi dataset our work openpi2.0 builds upon the openpi dataset @xcite that tracks entity state changes in procedural texts with an open vocabulary. the procedures are extracted from wikihow, a web resource containing instructions of everyday tasks. as exemplified in figure 2 , the input is a procedure which includes a goal (e.g., "remove fog using a potato") and a sequence of ordered steps (e.g., "rub the cut side of potato on the window"). for each step, the output is an array of 4-tuples describing an entity state change. each 4-tuple contains an entity, an attribute, a state before the step, and a state after the step (e.g., the window's texture was smooth before and sticky after). the task is thus equivalent to predicting an entity state matrix given a procedure, where the axes are step, entity, and attribute, while the value is the before and after states. the data is annotated via crowdsourcing and manually validated. however, openpi lacks canonicalization of entities and the differentiation of salient entities. in our work of openpi2.0, we will address both issues using state-of-the-art models. ## canonicalization in the original openpi dataset, the entities and attributes that undergo change were written by crowd workers. consequently, the dataset contains different ways of expressing the same entity (e.g., coffee maker, coffee machine, espresso machine in a coffee-making procedure) or attribute (e.g., texture, smoothness, sheen of a paint). canonicalization by clustering the entities and procedures is thus important for two reasons: 1) it facilitates evaluation, especially in a generation setting, where a model might be wrongly penalized for predicting the paraphrase of some correct entity or attribute; 2) it facilitates further annotation of features such as salience ( §4) of the entities and attributes. here, we describe efforts to canonicalize the entities and attributes in the evaluation set of openpi. ## salience the original openpi is annotated with many parallel entities in each procedure. often, they vary greatly by importance in accomplishing the task. for example, in a procedure of "cooking a steak", entities fish, oven, gloves, and spice rack might all be involved, while some are more indispensable than the rest. intuitively, the knowledge of entity salience helps models focus on what matters in downstream tasks ( §4.3). in openpi2.0, we define two types of entity salience: the global salience refers to the importance of an entity in accomplishing the goal of the procedure, whereas the local salience refers to that in a step. ## resulting dataset: openpi2.0 by adding canonicalization of entities and attributes as well as salience of entities to the evaluation set of the openpi dataset, we now fully present openpi2.0. as the procedures and entity state annotations have not changed, openpi2.0 still has 55 procedures with 5.0 steps on average. these procedures are collected from wikihow and their topics are everyday activities. openpi2.0 also inherits the original entityattribute-state changes annotated by crowd workers. after canonicalization, there are 356 canon entities each with 7.6 unique mentions and 5.5 expanded mentions on average, 3240 canon attributes, each with 3.0 unique mentions and 3.3 expanded mentions on average, and 1193 before-after states in the development set. the quality of clustering and expansion and be evidenced in §3.1. regarding salience labels (on a scale of 1 to 5), the global salience of entities has a mean of 3.5 and standard deviation of 1.4; the local salience of entities has a mean of 3.4 and standard deviation of 1.5. ## related work entity state tracking prior work on entity state tracking spans various disciplines of ai. for instance, object tracking, a sub-task of entity state tracking, has led to much work in both robotics @xcite . in nlp, early efforts focus on synthetic, closed-domain data @xcite and more recent ones shift attention to real-world procedures @xcite @xcite @xcite with a closed set of entities and attributes. the only open-ended dataset to our knowledge is still openpi @xcite which we build on. entity salience a small body of work on entity salience has focused on annotating entity salience in news articles and web pages for better information retrieval, recommendation, and linking @xcite ; @xcite ; @xcite ; @xcite ; @xcite . in contrast, we focus on entities in procedural texts, situating our work in script learning, robotic execution, automatic planning and reasoning, etc. due to this mismatch of purpose, the definition, annotation process, and downstream applications of our entity salience and theirs are all fundamentally different. procedures and scripts script learning @xcite is an umbrella discipline that focuses on groups of human actions under certain scenarios. regarding domain, procedural texts are an attractive data source to reason about entities which undergo frequent changes. there has been steady efforts in computer vision @xcite , robotics @xcite , and language (mujtaba and mahapatra, 2019; zhang, 2022). in nlp specifically, work on procedures includes extracting them from instructional texts @xcite @xcite , reasoning about relations among events @xcite @xcite @xcite , knowledge-base construction @xcite @xcite , or applying them to downstream applications @xcite @xcite @xcite . as discussed in many of these cited works, knowledge acquired from learning scripts and procedures has been known to benefit robotics and planning. ## conclusion we propose openpi2.0, an improved dataset on open-domain entity tracking in procedural texts. openpi2.0 features canonicalization of entities and attributes, based on which we perform a comprehensive benchmarking evaluation of state-ofthe-art lms. openpi2.0 also provides human annotation, model prediction, and analyses of entity salience, using which we show qualitative examples on its effective on various downstream tasks.