Are All Steps Equally Important? Benchmarking Essentiality Detection in Event Processes (2023)

Abstract: Natural language often describes events in different granularities, such that more coarse-grained (goal) events can often be decomposed into fine-grained sequences of (step) events. A critical but overlooked challenge in understanding an event process lies in the fact that the step events are not equally important to the central goal. In this paper, we seek to fill this gap by studying how well current models can understand the essentiality of different step events towards a goal event. As discussed by cognitive studies, such an ability enables the machine to mimic human’s commonsense reasoning about preconditions and necessary efforts of daily-life tasks. Our work contributes with a high-quality corpus of (goal, step) pairs from a community guideline website WikiHow, where the steps are manually annotated with their essentiality w.r.t. the goal. The high IAA indicates that humans have a consistent understanding of the events. Despite evaluating various statistical and massive pre-trained NLU models, we observe that existing SOTA models all perform drastically behind humans, indicating the need for future investigation of this crucial yet challenging task.

## introduction as a fundamental semantic primitive unit in human language (jackendoff, 1992), events play a pivotal role in facilitating efficient communication among humans and safe interactions with the world. recently, the natural language processing (nlp) community has made significant strides in helping machines comprehend events through various directions, such as event extraction @xcite , event relation extraction @xcite , event schema induction @xcite figure 1 : illustration of steps to obtain a ph.d. degree, with essential steps marked by red stars. successfully achieving the overall goal typically necessitates the completion of these crucial steps. @xcite , and event-centric knowledge graph construction @xcite . however, most of these studies primarily concentrate on modeling horizontal relationships between events, neglecting the internal components of an event (i.e., how an individual perceives an event mention). computational and cognitive studies @xcite indicate that humans can deconstruct a goal event into a discrete representation of finer-grained step events, ultimately facilitating the hierarchical organization of event-related knowledge. as illustrated in figure 1 , when discussing the goal event of "obtaining a ph.d. degree", we understand that several steps may occur along the way. for instance, one might receive the offer, pass the qualification exam, complete internships, publish papers, and defend the dissertation. among these steps, some are deemed essential to the goal, while others are not. for instance, passing the qualification exam is crucial for earning a ph.d. degree, whereas securing an internship is often not a requirement. this ability to discern the essentiality of steps pertaining to various goals equips humans with the commonsense needed to address problems and carry out daily tasks. similarly, understanding which steps are essential can profoundly benefit numerous nlp applications. for instance, event schema induction @xcite relies on eventcentric information extraction to derive graphical representations of events from text. in this context, understanding essentiality can enhance the quality of induced schemas by eliminating hallucinations and suggesting the addition of missing crucial events. moreover, grasping essentiality can potentially benefit intelligent systems for qa tasks @xcite and task-oriented dialogue processing @xcite . in this paper, we aim to assess the depth of understanding that current nlu models possess regarding events in comparison to human cognition. to accomplish this, we introduce a new cognitively inspired problem of detecting essential step events in goal event processes and establish a novel benchmark, essential step detection (esd), to promote research in this area. specifically, we gather goals and their corresponding steps from wikihow 2 and manually annotate the essentiality of various steps in relation to the goal. our experimental findings reveal that although humans consistently perceive event essentiality, current models still have a long way to go to match this level of understanding. ## task and data the essential step detection task is defined as follows: for each goal g and one of its sub-steps s, the objective is to predict whether the failure of s will result in the failure of g. in our formulation, g and s are presented as natural language sentences. the construction of esd includes two steps: (1) data preparation and (2) essentiality annotation. details of these steps are provided below. ## experiments recently, large-scale pre-trained language models have exhibited impressive language understanding capabilities. to assess the extent to which these models truly understand events, we evaluate them using esd. specifically, we benchmark their performance by examining a range of inference meth-ods as detailed below. ## related works the nlp community has increasingly focused on event understanding @xcite , with research divided into event-centric information extraction (ie) @xcite @xcite @xcite and structural event prediction @xcite . event-centric ie includes recognizing, typing events and inferring their relations @xcite @xcite , while structural event prediction involves context-independent inferences about event structures, causality @xcite @xcite , discourse @xcite , summaries @xcite , and memberships @xcite @xcite . this work pertains to the second research direction, focusing on the internal structure of events rather than relations between them. unlike previous event membership studies @xcite , this work predicts the essentiality of decomposed subevents, assessing the understanding of internal steps by sota llms. ## conclusion we introduce esd, an event understanding task assessing state-of-the-art nlp models' comprehension of events by identifying essential ones for goal achievement. experiments show that complex event knowledge is rarely expressed in text, and current large-scale language models struggle with complex event understanding. we will release all data and code to encourage research on complex event knowledge collection and improved reasoning for deeper event understanding.