Event Semantic Knowledge in Procedural Text Understanding (2023)

Abstract: The task of entity state tracking aims to automatically analyze procedural texts – texts that describe a step-by-step process (e.g. a baking recipe). Specifically, the goal is to track various states of the entities participating in a given process. Some of the challenges for this NLP task include annotated data scarcity and annotators’ reliance on commonsense knowledge to annotate implicit state information. Zhang et al. (2021) successfully incorporated commonsense entity-centric knowledge from ConceptNet into their BERT-based neural-symbolic architecture. Since English mostly encodes state change information in verbs, we attempted to test whether injecting semantic knowledge of events (retrieved from the state-of-the-art VerbNet parser) into a neural model can also improve the performance on this task. To achieve this, we adapt the methodology introduced by Zhang et al. (2021) for incorporating symbolic entity information from ConceptNet to the incorporation of VerbNet event semantics. We evaluate the performance of our model on the ProPara dataset (Mishra et al., 2018). In addition, we introduce a purely symbolic model for entity state tracking that uses a simple set of case statements, and is informed mostly by linguistic knowledge retrieved from various computational lexical resources. Our approach is inherently domain-agnostic, and our model is explainable and achieves state-of-the-art results on the Recipes dataset (Bosselut et al., 2017).

## introduction language understanding in humans requires at least the knowledge of the semantics of events and entities. one needs to know the sequences of subevents that together make up a 'throwing' event, as well as the causal and temporal relationships between the subevents that distinguish a 'throwing' event from a 'pouring' event, or a 'running' event. furthermore, reasoning about entities that are participating in these events requires a deep understanding of the properties of an entity. it is the distinction between such entity properties that enables us, for example, to distinguish between 'throwing a ball' vs. 'throwing a molotov cocktail'. in contrast to humans, many high-performing nlp models do not depend on explicit knowledge of events and entities to process natural language; rather, they rely on the surface forms and patterns of word co-occurances in colossal amounts of language data to learn the mechanics of language as well as the interpretation of linguistic forms. since human knowledge and reasoning capabilities benefit from knowledge of events and entities, we suggest that a neural model may also benefit from such explicit symbolic knowledge. this requires successful incorporation of such symbolic knowledge into a subsymbolic system. explicit semantic knowledge, such as entity knowledge extracted from ontologies, has often been used in the field of natural language grounding, where the connection between natural language and the physical world is sought @xcite . there are yet other nlp tasks that are likely to benefit from explicit semantic knowledge as well, such as tasks focusing on machine comprehension of how things work (e.g. how plants make food), or how a certain physical result is achieved (e.g. how to make pizza using some ingredients). the nlp task that focuses on the machine reading comprehension of texts describing processes is called procedural text understanding @xcite @xcite . one of the subtasks in this field is entity state tracking @xcite @xcite @xcite , formally defined as: given a paragraph p that describes a process, and an entity e that is one of the participants in that process, did the state of e change during the process? if so, what was the type of change that occurred to e (usually to be chosen among a desired set of types of state change)? when did the change happen (i.e. at which time step during the process)? and finally, what was the locus of change (i.e. the location of e before and after the change) @xcite ? there are two main challenges in solving this problem. first, the size of annotated data for this task is usually small since achieving reasonable inter-annotator agreement for the task is hard, making it expensive and time-consuming. second, when facing implicit information, annotators frequently resort to commonsense knowledge -knowledge that state-of-the-art nlp models are not explicitly aware of. existing models for this challenging problem use some flavor of learning-based approaches to nlp (see section 2). one of the existing approaches that is closest in theory to ours is koala @xcite ) -a neurosymbolic model encoding entity-centric knowledge into a neural network that is used to track entity states and locations during a process. we re-implemented this model and adopted it as our baseline. one of our contributions in this work is offering a method for encoding symbolic event semantic knowledge into a neural model. in practice, we are proposing an approach to expose a neural model to sequences of latent universal concepts composing an event, allowing the network to learn from the spelled out event semantics as well as the surface forms of the events realized mostly as verbs. in addition to our neurosymbolic model (skip: semantic knowledge in procedural text understanding), we have also developed a purely symbolic model 1 called lexis. error analysis and ablation tests on this model demonstrate other sources of external knowledge that show promise for inclusion in a neural model in future work. in addition, we show that our theory and approach are dataset-and domain-independent, and can be used in any nlp task where knowledge of event semantics plays a major role for humans to achieve the goal of the task. we will also briefly illustrate our explanation module for lexis. we evaluated skip on the propara dataset @xcite , and lexis on both the propara and recipes @xcite ) datasets 2 . lexis achieved a new state-of-the-art 1 here, purely symbolic is used as opposed to sub-symbolic models that learn by example. @xcite 2 the reason we did not evaluate skip on the recipes dataset was that we only exposed skip to the knowledge performance on the recipes dataset (70.1% f1, improving over the existing state-of-the-art model by 11.7%), and skip performed better than our adopted neurosymbolic baseline model, (71.8% f1, improving over the re-implemented baseline model by 4.1%) foot_0 . our contributions are two-fold: (1) we adapt the methodology introduced by @xcite for incorporating symbolic entity-centric knowledge to the incorporation of verbnet event semantics. we extract and encode event semantic knowledge for injection into a neural network, and present skip, an end-to-end neurosymbolic model developed using this method, in conjunction with data augmentation and transfer learning techniques. (2) we present a general knowledge-based approach to text understanding using existing nlp resources, and present lexis, a purely symbolic model we developed for entity state tracking that achieves a new state-of-the-art on the recipes dataset, with an architecture that is adaptable to different genres of natural language text, and is explainable foot_1 . ## related work this work is inspired by the concept of event semantics and event structure offered by the generative lexicon theory, in efforts such as @xcite , @xcite , and @xcite , where event structure is enriched to encode and dynamically track object attributes that are modified during an event. the idea is that a complex event can be decomposed into simpler ordered subevents that explicitly label the transitions between entity states. with regard to entity state tracking, most recent existing models mainly rely on large language models @xcite @xcite , while earlier models (prior to 2020) rely on neural @xcite @xcite or learningbased approaches @xcite . the only existing neurosymbolic model (to our knowledge) is koala @xcite , which retrieves extracted directly from the verbnet parser, which does not shed light on the types of state change the model is expected to predict in the recipes dataset. we have access to such knowledge and will perform this evaluation in future work. the main difference between koala and skip is the type of external symbolic knowledge introduced to the model. whereas koala only leverages entity-centric knowledge, we introduce event semantic knowledge based on the generative lexicon theory and its implementation in the verbnet lexical resource @xcite @xcite . this allows the model to have access to direct and explicit knowledge about entity state transitions for all the participants in an event, the roles of each participant in the event, as well as causal relationships and temporal links between subevents @xcite . on the recipes dataset, @xcite evaluate only for location prediction, because location change is one of the state change types needed by propara as well. to enable prediction for the rest of the state change types required by the recipes dataset , a previously lacking knowledge resource has recently become available @xcite which explicitly provides the lexical semantic components indicating state changes such as changes in temperature or form, giving our symbolic model (lexis) an edge over other competing models foot_2 . ## methodology following @xcite , we develop skip by neural encoding of symbolic knowledge and allowing the model to selectively pay more attention to knowledge that is conducive to more accurate predictions. as mentioned in section 1, our main contribution is proposing a way to make a neural model utilize event semantic knowledge in its predictions, and use the obtained neurosymbolic model for downstream nlp tasks where knowledge of event semantics tends to be beneficial according to linguistic theory. in order to obtain logical representations of subevent semantics as well as temporal and causal relations between the subevents for encoding into our neural model, we rely on verbnet -a large english verb lexicon which expands event semantics into sequences of subevents. to automate this process, we use the state-of-the-art verbnet semantic parser @xcite and obtain the symbolic logical representations for individual sentences corresponding to the steps in each process. these logical representations, illustrated in in verbnet, verbs are classified into different classes based on similarities in their syntactic and semantic behavior. for example, all verbs belonging to the break-45.1 class ( the verbnet-extracted event semantic knowledge is then translated into natural language so that it is neurally encodable. we choose this method of encoding over direct encoding of the logical representations, because llms such as bert are already familiar with the structure of natural language, and we want to hone this existing power instead of introducing a whole new representation system which might be harder to learn, especially given the small size of the dataset. in order to acquaint a vanilla text encoder with the language translated from the event semantic logical representations, we fine-tune a bert encoder (kenton and toutanova, 2019) on the translated knowledge extracted for the training data. this will be explained in more detail in 3.3. ## experiments we evaluate skip on the propara dataset @xcite , which is an entity state tracking dataset developed by ai2, containing 488 humanauthored paragraphs describing scientific processes, with an 80/10/10 data split. while state change types (move, create, and destroy) were expertly annotated, entity location annotation was crowedsourced, resulting in lower quality and consistency. we perform document-level evaluation on propara, using the official evaluation code 8 . in re-implementing the baseline model (koala), we only changed the batch size, downsizing from 32 to 16 due to hardware limitations foot_6 . koala's reported results along with our re-implementation results are demonstrated in the first two rows of the whole model contains 235m parameters including 2 bert encoders. in lm fine-tuning, we used the uncased bert base model, and manually tuned hyper-parameters, setting the batch size to 16 and learning rate to 5 × 10 -5 . while we used the same text encoder developed by @xcite , our knowledge encoder was different. it was trained for 2 epochs on external knowledge. in the final training stage, we used a batch size of 10 and a learning rate of 3 × 10 -5 on the adam optimizer. the hidden size of the lstms was set to 256 and the dropout rate to 0.4. we performed early stopping with an impatience of 20 epochs, by evaluating changes in model accuracy over the dev set (∼1.5 gpu hours). we selected the best checkpoint in prediction accuracy on the dev set. ## discussion our experimental results were interesting in two ways. first, the fact that lm fine-tuning on both verbnet and conceptnet lowers the performance compared to fine-tuning on only one knowledge source could be an indication that, given the size of the data, two different sources of knowledge seems to confuse the knowledge encoder more than helping it. secondly, comparing fine-tuning on verbnet only vs. conceptnet only, the former proved to be more effective. this might indicate that knowledge of event semantics may better help the model track entity states and locations during a process, just as we had initially hypothesized based on lexical semantic theories. while entity-centric knowledge may give the model a better understanding of entities and their properties, such as their typical loca-tion, state changes are eventive concepts and often lexically encoded in verbs. since verbnet provides explicit labels for transitions between entity states, a successful verbnet parse ensures explicit symbolic knowledge which clarifies the types of state change lexically encoded in verbs. ## conclusions and future work we presented a method to extract event semantic knowledge and encode it in neural architectures for nlp applications where event semantics theoretically promises to enhance the predictive power of the model. we showed that this method was effective in skip -our neurosymbolic model designed for procedural text understanding. our error analysis demonstrated that skip can be relied on to perform annotation quality control. furthermore, lexis, our purely symbolic entity state tracking model designed based on our domain-independent approach, achieved a new state-of-the-art on the recipes dataset. we explained why this approach is domain-independent and can be adapted to other domains and nlp tasks. in future work, we would like to expand our neurosymbolic model to use other sources of linguistic knowledge that proved useful in lexis ablation tests. it would also be interesting to assess the success of this approach in other nlu tasks, such as causal inference and textual entailment, where event semantic knowledge is again theoretically important.