A Dataset for Tracking Entities in Open Domain Procedural Text (2020)

Abstract: We present the first dataset for tracking state changes in procedural text from arbitrary domains by using an unrestricted (open) vocabulary. For example, in a text describing fog removal using potatoes, a car window may transition between being foggy, sticky, opaque, and clear. Previous formulations of this task provide the text and entities involved, and ask how those entities change for just a small, pre-defined set of attributes (e.g., location), limiting their fidelity. Our solution is a new task formulation where given just a procedural text as input, the task is to generate a set of state change tuples (entity, attribute, before-state, after-state) for each step, where the entity, attribute, and state values must be predicted from an open vocabulary. Using crowdsourcing, we create OPENPI, a high-quality (91.5% coverage as judged by humans and completely vetted), and large-scale dataset comprising 29,928 state changes over 4,050 sentences from 810 procedural real-world paragraphs from WikiHow.com. A current state-of-the-art generation model on this task achieves 16.1% F1 based on BLEU metric, leaving enough room for novel model architectures.

## introduction by one estimate, only about 12% of what we understand from text is expressed explicitly @xcite . this is especially apparent in text about actions where the effects of actions are left unstated. humans fill that gap easily with their commonsense but machines need to model these effects in the form of state changes. for example, when a potato is rubbed on a car window (to defog it), then the unstated effects of this action are the following state changes: windows becomes sticky, opaque, and the potato becomes dirty, etc. these changes can be tracked across the paragraph. an exemplary use case of text with actions is procedural text @xcite where modeling such state changes helps in various reasoning-based end tasks, e.g. automatic execution of biology experiments @xcite , cooking recipes @xcite and everyday activities @xcite . while there has been great progress in tracking entity states in scientific processes @xcite , tracking ingredients in cooking recipes @xcite , and tracking the emotional reactions and motivations of characters in simple stories @xcite , prior tasks are restricted to a fixed, small set of state change types thus covering only a small fraction of the entire world state. figure 1 illustrates this for a real-world procedure "how to keep car windows fog free using a potato". existing datasets such as propara @xcite only model the existence and location attributes, limiting the fidelity with which they model the world. specifically: addressing these limitations, our solution is a new task formulation to track an unrestricted (open) set of state changes: rather than provide the text and entities, and ask how those entities change for a pre-defined set of attributes at each step, we instead provide just the input text, and ask for the set of state changes at each step, each describing the before and after values of an attribute of an entity in the form (attribute of entity was value before before and value after afterwards.). importantly, the vocabularies for attributes, entities, and values is open (not pre-defined). our contributions are: from a procedural paragraph with sentences (i.e., steps) step 1 . . . step k , construct k data points, one per step. input: as input we are given a procedural text comprising current step step i as query and all past step as context step 1 • • • step i-1 . we denote the input as @xmath0, x c ), where x q is the step for which we need the state changes (i.e. the query) and x c is the context. here, we use the common assumption (dalvi et al., 2018) that the steps in procedural text are ordered such that the context required for step i is mentioned in step 1 • • • step i-1 . output: the output is a set of zero or more state changes @xmath1. a state change y i is of the form: attr of ent was val pre before and val post afterwards here, attr is the attribute or state change type, and ent is the changed entity. val pre is the precondition (i.e., the state value before), and val post is the postcondition (i.e., the state value afterwards). pre/ postcondition adj or relp(y pre i ) can be an adjectival phrase or a relational phrase. in this task, attr, ent, val pre and val post are open form text i.e. they are not tied to any fixed, constrained vocabulary. example: consider the running example: @xmath2: the window of your car is foggy, query: rub half potato on the window). then, {y} = { transparency of window was fogged before and partially clear afterwards, stickiness of window was smooth before and sticky afterwards }. in y 1 , @xmath3, @xmath4, val @xmath5.1 unique challenges openpi has two unique challenges that are not found in any existing state change dataset. ## conclusion we presented the first dataset to track entities in open domain procedural text. to this end, we crowdsourced a large, high-quality dataset with examples for this task. we also established a strong generation baseline highlighting the difficulty of this task. as future work, we will explore more sophisticated models that can address the highlighted shortcomings of the current model. an exciting direction is to leverage visuals of each step to deal with unmentioned entities and indirect effects.