Reasoning over Entity-Action-Location Graph for Procedural Text Understanding (2021)

Abstract: Procedural text understanding aims at tracking the states (e.g., create, move, destroy) and locations of the entities mentioned in a given paragraph. To effectively track the states and locations, it is essential to capture the rich semantic relations between entities, actions, and locations in the paragraph. Although recent works have achieved substantial progress, most of them focus on leveraging the inherent constraints or incorporating external knowledge for state prediction. The rich semantic relations in the given paragraph are largely overlooked. In this paper, we propose a novel approach (REAL) to procedural text understanding, where we build a general framework to systematically model the entity-entity, entity-action, and entity-location relations using a graph neural network. We further develop algorithms for graph construction, representation learning, and state and location tracking. We evaluate the proposed approach on two benchmark datasets, ProPara, and Recipes. The experimental results show that our method outperforms strong baselines by a large margin, i.e., 5.0% on ProPara and 3.2% on Recipes, illustrating the utility of semantic relations and the effectiveness of the graph-based reasoning model.

## introduction procedural text often consists of a sequence of sentences describing processes, such as a phenomenon in nature (e.g., how sedimentary rock forms) @xcite or instructions to complete a task (e.g., the recipe of mac and cheese) @xcite . given a paragraph and its participant entities, the task of procedural text understanding is to track the states (e.g., create, move, destroy) and locations (a span in the text) of the entities. compared with traditional machine reading task, which mainly focuses on the static relations among entities, procedural text understanding is more challenging since it involves discovering complex temporal-spatial relations among various entities from the process dynamics. to effectively track the states and locations of entities, it is crucial to systematically model rich relations among various concepts in the paragraph, including entities, actions, and locations. three types of relations are of particular interest. first, mentions of the same entity in different sentences are related. the inherent relation among these mentions may provide clues for a model to generate consistent predictions about the entity. for example, the entity electrical pulses are mentioned in two sentences "the retina's rods and cones convert it to electrical pulses. the optic nerve carries electrical pulses through the optic canal.". connecting its two mentions in two sentences helps to infer its location in the first sentence using the second sentence's information. second, detecting connections between an entity and the corresponding actions helps to make state predictions more accurate. take the sentence "as the encased bones decay, minerals seep in replacing the organic material." as an example. the entity bone is related to decay which indicates the state destroy, while it is not connected to seep indicating the state move. given the relation between bone and decay, it is easier for the model to predict the state of bone as destroy, instead of being misled by the action seep. last, when the state or location of one entity changes, it may impact all associated entities. for example, in sentence "trashbags are thrown into trashcans.", trashbags are associated with trashcans. then, in the following sentence "the trashcan is emptied by a large trash truck.", although trashbags are not explicitly mentioned, their loca-tions are changed by the association with trashcan. recent works on procedural text understanding have achieved remarkable progress @xcite @xcite @xcite . however, the existing methods do not systematically model the relations among entities, actions, and locations. instead, most methods either leverage inherent constraints on entity states or exploit external knowledge to make predictions. for example, @xcite propose a structural neural network to track each entity's hidden state and summarize the global state transitions with a crf model. @xcite inject commonsense knowledge into a neural model with soft and hard constraints. although @xcite model the relation between entities and locations, there is no general framework to model the relations, and some important relations, such as entityaction and entity-entity relations, are ignored. a general framework to systematically model the rich types of relations among entities, actions, and locations is essential to procedural text understanding. to the best of our knowledge, we are the first to explore comprehensive relation modeling, representation, and reasoning systematically. specifically, we first construct an entity-actionlocation graph from a given paragraph, where three types of concepts (i.e., entities, locations, and actions) are identified and extracted as nodes. we then detect critical connections among those concepts and represent them as edges. finally, we adopt a graph attention network to conduct reasoning over the entity-action-location graph (real), which provides expressive representations for downstream state and location predictions. we evaluate the proposed approach on two benchmark datasets for procedural text understanding, propara @xcite and recipes @xcite . our approach outperforms the state-of-the-art strong baselines by a large marge, i.e., 5.0% on propara and 3.2% on recipes. the ablation study and analysis show that the graph-based reasoning approach generates better representations for entities, locations, and actions. thus, it is highly valuable for both state and location tracking of entities. ## experiments this section describes the evaluation results of real on two datasets (propara @xcite and recipes @xcite ). we also provide ablation study and case analysis to illustrate the effectiveness of graph-based reasoning. propara contains procedural texts about scientific processes, e.g., photosynthesis, fossil formulation. it contains about 1.9k instances (one entityparagraph pair as an instance) written and annotated by human crowd workers. we follow the official split @xcite for train/dev/test set. the recipes dataset consists of paragraphs describing cooking procedures and their ingredients as entities. we only use the human-labeled data in our experiment, with 80%/10%/10% of the data for train/dev/test, respectively. detail statistics for the two datasets can be found in we follow previous work's setting @xcite and evaluate the proposed approach on two types of tasks on the propara dataset, documentlevel task and sentence-level task. document-level task focuses on figuring out input entities, output entities, entity conversions, and entity movements by answering corresponding questions. more details can be found in the official script foot_3 . following the official script, we evaluate models with averaged precision, recall, and f1 scores. in sentencelevel task, we need to answer three categories of questions: (cat-1) is entity e created (destroyed, moved) in the process? (cat-2) when is e created (destroyed, moved)? (cat-3) where is e created (destroyed, moved from/to)? for this task, we take macro-average and micro-average of the score for three sets of questions as evaluation metrics foot_4 . for the recipes dataset, we take the same setting as @xcite , where the goal is to predict the ingredients' location changes during the process. we take precision, recall, and f1 scores to evaluate models foot_5 . ## conclusion and future work in this work, we propose a novel approach real for procedural text understanding. unlike all previous works, we systematically exploit the rich semantic relations between entities, location, and actions. we design an entity-action-location graph to systematically model various types of concepts and their relations and develop the algorithms for graph construction, representation, and reasoning. we comprehensively conduct a quantitative and qualitative comparison of the proposed approach with strong baselines on two popular benchmark datasets for procedural text understanding and demonstrate the effectiveness of our approach. in the future, we will investigate approaches to further advance the procedural text understanding task, such as incorporating entity disambiguation and external knowledge in our approach.