The Role of Semantic Parsing in Understanding Procedural Text (2023)

Abstract: In this paper, we investigate whether symbolic semantic representations, extracted from deep semantic parsers, can help reasoning over the states of involved entities in a procedural text. We consider a deep semantic parser (TRIPS) and semantic role labeling as two sources of semantic parsing knowledge. First, we propose PROPOLIS, a symbolic parsing-based procedural reasoning framework. Second, we integrate semantic parsing information into state-of-the-art neural models to conduct procedural reasoning. Our experiments indicate that explicitly incorporating such semantic knowledge improves procedural understanding. This paper presents new metrics for evaluating procedural reasoning tasks that clarify the challenges and identify differences among neural, symbolic, and integrated models.

## introduction procedural reasoning is the ability to track entities and understand their evolution given a sequence of actions @xcite . this kind of reasoning is crucial in understanding recipes @xcite , manuals and tutorials @xcite , cybersecurity text @xcite , natural events @xcite , and even stories @xcite . an example of a procedural text in the natural event domain, its entities of interest, and their state changes are shown in figure 1 . inferring actions and their impact on entities involved in a procedural text can be challenging in various aspects. first, there are dependencies between steps to be considered in predicting a plausible action set. for instance, an entity destroyed at step t of the process cannot be moved again at step t + 1. second, some sentences contain ambiguous local signals by including multiple action verbs. for example, "the oxygen is consumed in the process of forming carbon dioxide.", where the oxygen is being destroyed, and the carbon dioxide is being created. third, the sentences are incomplete in some steps. for instance, a step of the process might only indicate "is buried in mud", which cannot be understood without context. fourth, finding the properties of some entities may require reasoning over both the global context and local relations. for instance, in the sentences "1. magma rises to the surface. 2. magma cools to form lava", the location of 'lava' after step 2 should be inferred from the prior location of magma, which is indicated in its previous step. fifth, common sense is required to understand some consequences. for example, in figure 1 , step 3, one should use common sense to realize that 'decomposing body' would expose the 'bones', which will be left behind in the 'sediment'. sixth, understanding some relations requires an advanced co-reference resolution. in figure 1 , step 4, a complex co-reference resolution is required to understand that the 'buried material' refers to both 'plants and animals bones' and that they are transforming into the 'oil'. except for the common-sense @xcite and the ability to make consistent global decisions actions @xcite , the other challenges might have only been indirectly tackled in the recent research @xcite , but have neither been addressed explicitly nor properly evalu-ated to measure their success on resolving these challenges. in this paper, we evaluate whether semantic parsers can alleviate some of these challenges. semantic parsers provide semantic frames that identify predicates and their arguments in a sentence. for instance, in the sentence 'move bag to the yard', "move" is the predicate, "bag" and "the yard" are the arguments with types "affected" foot_0 and "location" respectively. such semantic information can help disambiguate multi-verb local connections between predicates and arguments @xcite . they can also provide meaningful local relations, making it easier to connect global information to infer entities' states. for instance, in the same sentence, "magma cools to form lava", "magma" is noted as the 'affected' and 'lava' is the result of the predicate 'form'. this makes it easier to infer that the location of 'lava' should match the last location of 'magma'. for our study, we consider both the classic semantic role labeling (srl) foot_1 , based on @xcite , which is a relatively shallow semantic parsing model, as well as the deep semantic parser trips foot_2 @xcite . to investigate the effect of semantic parsing on procedural reasoning, we analyze its effect as a standalone symbolic model as well as its integration in a neuro-symbolic model that combines semantic parsing with state-of-the-art neural models to solve the procedural reasoning task. first, we design a set of heuristics to extract a symbolic abstraction from the trips parser, called propolis. we use this baseline to further showcase the effectiveness of semantic parsing information in solving the procedural task. next, we integrate the semantic parsers with two well-established procedural reasoning neural backbones, namely ncet @xcite and tslm (faghihi and kordjamshidi, 2021) (and its extension cgli @xcite ), through encoding the semantic relations as a graph attention neural network (gat) @xcite . for our experiments, we use propara dataset @xcite ) that introduces the procedural reasoning task over natural events that are described in english. we realized the existing evaluation metrics of this dataset do not reflect the actual performance of the models and fail to identify the challenges and shortcomings of the models. consequently, we propose new evaluation criteria to shed light on the differences between the models, even when they perform similarly based on the prior metrics. in summary, our contributions are (1) proposing a symbolic model (propolis) to solve the procedural reasoning task based on semantic parsing, (2) proposing a set of new evaluation metrics which can identify the strength and weaknesses of the models, and (3) showcase the benefits of integrating semantic parsing into the neural models. the code and models proposed in this work are all available in github foot_3 . ## related research procedural text understanding has been investigated in many benchmarks such as scone @xcite , babi @xcite , and pro-cessbank @xcite . recent research has focused on procedural reasoning as tracking entities throughout a procedural text. datasets such as propara @xcite , recipes @xcite , procedural cyber-security text @xcite , and openpi @xcite are in the same direction. procedural reasoning can also be influential in addressing causal reasoning (wiqa) @xcite , story understanding (trip) @xcite , and abstractive multi-modal question answering (recipeqa) @xcite . this paper primarily focuses on tracking entities' states and properties throughout a procedural text. recent research has addressed this problem by predicting actions and properties on local context (prolocal) @xcite , autoregressive global predictions based on distance vectors (proglobal) @xcite , integrating structural common-sense knowledge built over verbnet (prostruct) @xcite , building dynamic knowledge graphs over entities (kg-mrc) @xcite , explicitly encoding the model to explain dependencies between actions (xpad) @xcite , formulating local predictions and global sequential information flow and sequential constraints (ncet) @xcite , formulating the task in a qa setting (dynapro, tslm) @xcite , inte-grating common-sense knowledge from concetp-net (koala) @xcite , utilizing large generative language models (lemon) @xcite , or using both the question answering setting and sequential structural constraints at the same time (cgli) @xcite . all the models mentioned above investigate different neural architectures to tackle the task, while we are more interested in augmenting them with additional knowledge from semantic parsers. recent research has also investigated the integration of semantic role labeling into the procedural reasoning task (real) @xcite , which is very close to our goal in this paper. however, in this work, we propose and investigate a variety of combinations, a deeper semantic representation (trips) and named relations, in addition to a symbolic approach for solving the procedural reasoning task solely based on semantic parsing. ## technical approach problem definition the procedural reasoning task can be formally defined by a procedural text including m steps, @xmath0, s 2 , ..., s m }, a set of entities @xmath1, e 2 , ..., e n }, where n is the number of entities, and a set of properties. specifically, in the propara dataset, the property of interest is only the location of the entities p @xmath2, p l 1 1 , ..., p l m n }, where p l t i denotes the jth entity at step t. in propara, the location prediction starts at step 0, which indicates the entity's location before the process begins. the location of an entity can either be known (represented by a string) or unknown (represented by "?"). similar to prior research @xcite , the location property is used to infer a set of actions @xmath3, a 1 2 , .., a m n }, where a j t denotes the action type applied to entity j at step t. following the prior research @xcite , we extract all the noun phrases from the sentences and only consider those as location candidates. we investigate two different modeling approaches to solve this problem. first, we use a symbolic and parsing-based model, and second, we integrate semantic parsing with neural models. we use two different sources for semantic extraction: srl and trips. in general, srl is coarse-grained and shallow compared to trips. the connections in trips are not limited to the pairwise connections between predicates and arguments but are extended to the semantic connections between any two words. since trips relies on a general pur-pose ontology, it also augments the arguments and predicates with additional information about a set of possible features (mobility, container, negation) and mapping of the words to hierarchical ontology classes (i.e., mapping "water" to "beverage"). srl is centered around the semantic frames of the verbs ( predicates) and identifies each predicate's main and adjunct (mainly time and location) arguments in the sentence. figure 2 and 3 show examples of the srl and trips parses, respectively. the symbolic model only uses the trips parser as it provides more extended extractions and meaningful relations, while both srl and trips are used for integration with the neural baselines. ## evaluation we use three evaluation metrics to analyze the performance of the symbolic, sub-symbolic, and neural baselines. the first metric is sentence-level and proposed in @xcite . the second metric is a document-level evaluation proposed by @xcite . both of these metrics evaluate higher-level procedural concepts that can be inferred from the predictions of the model rather than the raw decisions. these metrics give more importance to the actions compared to the location decisions. although they can successfully evaluate some aspects of the models, they fail to measure the research progress in addressing the challenges of the procedural reasoning task. we extend these evaluations with a new decision-level evaluation metric that considers almost all model decisions with a similar weight and evaluates the models based on the difficulty of the reasoning process. ## experiments here, we summarize the performance of strong baselines compared with the symbolic (propo-lis) and integrated models. the implementation details of the models are available in appendix a. ## discussion here, we discuss some of the potential concerns that may arise with the usage of symbolic systems such as trips and the new evaluation criteria. coverage and rule crafting of propolis. our implementation of the symbolic method and the integrated models rely on the knowledge extracted from very fine-grained semantics covered in trips. consequently, a small mapping effort was needed to create such a system. the mapping between actions in propara and verbs is straightforward since verbs are automatically mapped onto ontological classes that provide the type of actions based on the parse. hence, defining the mapping rules for the most general relevant ontology types of verbs is sufficient because all the descendent types will fol-low the same mapping (see ## conclusion we investigated whether semantic parsers could help with reasoning over procedural text. we proposed propolis, a symbolic model operating on deep semantic parsers to solve the procedural reasoning task. for this task, the symbolic model outperformed many recent neural architectures. we then evaluated the effects of integrating semantic parsers with two well-known sota neural backbones. all integrated models outperformed baseline architectures, particularly when the parser provided more detailed information and rich semantic frames. furthermore, we proposed new evaluation metrics that show the pros and cons of the models and help identify the key challenges in reasoning over procedural text.