Coalescing Global and Local Information for Procedural Text Understanding (2022)

Abstract: Procedural text understanding is a challenging language reasoning task that requires models to track entity states across the development of a narrative. We identify three core aspects required for modeling this task, namely the local and global view of the inputs, as well as the global view of outputs. Prior methods have considered a subset of these aspects, which leads to either low precision or low recall. In this paper, we propose a new model Coalescing Global and Local Information (CGLI), which builds entity- and timestep-aware input representations (local input) considering the whole context (global input), and we jointly model the entity states with a structured prediction objective (global output). Thus, CGLI simultaneously optimizes for both precision and recall. Moreover, we extend CGLI with additional output layers and integrate it into a story reasoning framework. Extensive experiments on a popular procedural text understanding dataset show that our model achieves state-of-the-art results, while experiments on a story reasoning benchmark show the positive impact of our model on downstream reasoning.

## introduction understanding the causal links of events in procedures is a key aspect of intelligence, facilitating human narration and dialogue. for instance, understanding why story b is plausible and why story a is not (figure 1 ) requires procedural understanding of the causes of john leaving his notebook at home, as opposed to him taking out his notebook from his bag: writing in a notebook is counterfactual in the former case, and intuitive in the latter. understanding stories requires procedural models that can reason consistently about event implications, and do so at different granularities. for a model to decide whether a story is plausible, it has to track the entity states over time, understand the effects of figure 1 : an example story of understanding task. given two stories, the task is to judge which story is more plausible, find the conflicting sentence pair in the implausible story, and predict entity states at every step. the described actions (green arrows), and consider the preconditions for a given action (pink arrows). meanwhile, the model must reconcile the causes and effects of all events described in the story, to provide a globally consistent interpretation. while procedural reasoning research reports steady progress in recent years @xcite @xcite , story understanding and procedural reasoning have rarely been considered together @xcite . works have attended only to complementary aspects of the procedural reasoning problem, e.g., @xcite build entity-centric context representations and ignoring timestep-wise representation modeling; @xcite later proposed a timestep-specific model providing unique context encoding at every step to enable modeling flexibility. however, these methods predict independent step-wise entity states, thus compromising the dependency of outputs across different steps-yielding high recall but low precision. global-output methods @xcite explicitly leverage the strong dependency across steps by jointly modeling the entity actions from all steps, but these methods only have one context encoding for all entities and steps, thus providing sub-optimal input representationsyielding high precision but low recall. in this paper, we propose coalescing global and local information (cgli): a new model for procedural text understanding that makes global decisions in consideration of entity-, timestep-centric, and global views of the input. to do so, our model builds a separate input view for every entity, at every step, while providing the whole context. meanwhile, cgli represents the entity actions across steps jointly with a structured prediction objective, thus achieving high consistency between different steps. the contributions of this paper are: 1. a novel procedural understanding method, cgli, which produces global outputs of narrative procedures based on a unified view of the input, combining both local (entity-centric, timestepspecific) and global (document-wide) views-thus optimizing precision and recall, simultaneously. 2. a story understanding framework, which builds upon our procedural understanding model, to enable story understanding with explicit and explainable understanding of event procedures, captured through entity precondition and effect states. procedural text understanding. the task input consists of an n-sentence paragraph @xmath0, s 2 , ...s n } , and k entities {e 1 , e 2 , ...e k }. the goal is to predict precondition state s p i,t and effect state s e i,t , for every entity at every step, as well as the action a i,t performed by the entity at every step; @xmath1, 2, ..k}, @xmath2, 2, ...n}. the effect state at t -1 is the same as precondition state at step t, i.e., s e i,t-1 = s p i,t , hence s i is a sequence of length n + 1. following prior work (mishra et al., 2018), a i,@xmath3, exist, move, destroy}, s e i,@xmath4, unknown-location, location}, and for location, a span of text in p needs be identified for the prediction. action a i,t describes the entity state changes from precondition to effect, thus it can be inferred from the state sequence s i , and vice versa-e.g., if s p i,1 = non-existence and s e i,1 = location, then a i,1 = create. procedural story understanding. the input to the procedural story understanding task consists of two parallel stories, p 1 , p 2 = {s 1 , s 2 , ...s n }, each consisting of n sentences and differing only in one of the sentences. following storks et al. (2021), the task is to identify which story is more plausible, identify the conflicting pair of sentences (s c1 and s c2 ) in the implausible story, and list the preconditions s e i and effects s p i of all entities at every step of a story. here, multiple attributes are considered for precondition and effect states. unlike in the procedural text understanding task, the story completion task does not require that the effect state at step t -1 should match the precondition state at step t, i.e., s e it-1 and s p it are not necessarily equal. ## cgli: coalescing global and local information in this section, we describe the input representation, the architecture, and the training details of our model, as illustrated in figure 2 . ## story understanding with cgli we integrate cgli into a story understanding framework with minimum modifications following the task definition, and the overall model is shown in figure 3 . as the story understanding tasks do not require location extraction, we remove the span extraction layer, which makes the input representation of step 0 obsolete. given that the continuity of effects to preconditions between consecutive steps does not hold in this task, we directly use c n ′ @xmath5. given b number of attributes for precondition and effect states, we apply an in-batch crf module for each attribute. specifically, we apply equations 7 and 8 for every attribute, yielding 2b such modules in total. to detect conflicting sentences, we concatenate every pair of sentence representations, and pass it through a linear layer to find the conflicting pair. for story classification, we take the mean of sentence representations for story representation, and pass it through a linear layer for binary classification. formally, we test story understanding on trip @xcite , which contains crowdsourced plausible and implausible story pairs. in each pair, the plausible story label and the conflicting sentence pair label in implausible story are annotated. trip annotates 20 attributes with predefined set of possible values. the annotations are given for all entities at every timestep of the two stories. @xcite y y n n 62.7 30.5 10.4 34.5 34.0 81.7 36.8 50.7 proglobal @xcite y y y n 63.0 36.4 35.9 45.1 45.4 61.7 48.8 51.9 prostruct @xcite y y n y -----74.3 43.0 54.5 kg-mrc @xcite n y n n 62.9 40.0 38.2 47.0 46.6 64.5 50.7 56.8 ncet @xcite n n y y 73.7 47.1 41.0 53.9 54.0 67.1 58.5 62.5 ien @xcite n n y y 71.8 47.6 40.5 53.3 53.0 69.8 56.3 62.3 dynapro @xcite y y n n 72.4 49.3 44.5 55.4 55.5 75.2 58.0 65.5 tslm (2021) y y y n 78.8 56.8 40.9 58.8 58.4 68.4 68.9 68.6 koala @xcite n we provide datasets splits details in for trip, we follow the original work and report the following metrics: accuracy of classifying the plausible story, consistency of finding the conflicting sentence pairs when the story classification is correct, and verifiability, which evaluates the prediction of the entities' effects at s c1 and the entities' preconditions at s c2 . we also report the average f1-score for preconditions and effects across the 20 attributes to better understand the model's procedural understanding ability. baselines. for propara, we directly report baseline results from the official leaderboard. for trip, we report the results from the best model released by @xcite . training details. for propara, we define two additional action types to represent the entity transitions, namely out-of-create, out-of-destroy similar to @xcite . hence, the total size of the action space is six. for evaluation, these two types would be mapped to none transition, and they are defined to help the model differentiate the none types during training, i.e., if the entity has not being created or if it has been destroyed. to facilitate model's learning on location predictions, we initialized our model with a roberta-large @xcite model pretrained on squad 2.0 @xcite . we run our model five times with different random seeds and report the maximum scores in we break down the results on propara by the document-level question types defined in §5 and compare our best model with the best results reported by tslm and koala. the precision and recall per question type are shown in figure 4 . consistent with the overall results, koala is particularly strong on precision for all types and tslm is much better on recall. cgli is able to maintain a balance between those two extremes and achieve overall better results. all three models perform similarly when predicting the inputs and the outputs of a procedure. yet, cgli achieves much higher performance on transitional questions regarding entity conversions and moves, which are notably harder to predict. these results suggest that the gains of cgli over previous works are mostly due to hard-to-answer categories. ## related work recent procedural text understanding benchmarks including scone @xcite , babi @xcite , processbank @xcite , propara @xcite , recipe @xcite , and openpi @xcite have inspired a series of methods. @xcite propose prolocal that encodes each step of a procedure separately and proglobal that encodes the full paragraph at every step. kg-mrc @xcite builds a dynamic knowledge graph of entity and location mentions to communicate across time steps. dynapro @xcite employs pre-trained lm to jointly predict entity attributes and their transitions. tslm (rajaby faghihi and kordjamshidi, 2021) formulates procedural understanding as a question answering task, and leverages models pretrained on squad @xcite enhanced with a timestamp encoding. although equipped with various ways to pass information across time steps, these methods still make local predictions thus they may compromise the global dependency of outputs. another line of work focuses on jointly modeling the entity action sequence, aiming to ensure global structure and consistency. prostruct @xcite aims to find the globally optimal entity action sequence using beam search. @xcite devise a structured neural architecture ncet, modeled with a crf, which recurrently updates the hidden representation of each entity at each step. ien @xcite builds upon ncet and augments the entity-to-entity attention. koala @xcite further enhances the ncet by pretraining on wikipedia and concept-net @xcite . the key shortcoming of these global methods is that they rely on entity mentions extracted from a single copy of encoded context shared by all entities and all steps, which limits their modeling capacity. our proposed method stands out from all previous works by coalescing complementary granularities of procedural text modeling, by building specific and informative input representations while modeling output dependency. concurrent to our work, @xcite proposed lemon for language-based environment manipulation. their focus on model pretraining is orthogonal to cgli. there are also numerous recent story understanding benchmarks @xcite @xcite , and modeling methods @xcite @xcite . the trip task @xcite integrates a procedural understanding component in story understanding to enable consistent and interpretable reasoning over narratives. to our knowledge, we are the first work to bridge the gap of modeling methods between procedural understanding and story comprehension. other tasks that require rea-soning over procedures exist, including defeasible reasoning @xcite , abductive commonsense inference @xcite , reasoning over preconditions @xcite , script reasoning @xcite and multimodal script reasoning @xcite , are typically solved by specialized methods, without separately modeling procedural and causal links. we intend to apply cgli on these tasks in the future to bridge this gap. ## conclusions & future work we proposed cgli: a novel procedural understanding method that combined global and local information. recognizing the key role of procedural understanding in downstream tasks, we also integrated cgli in a story understanding framework. our experiments showed the benefit of our coalesced method, with the global views providing optimal precision, while the local view boosting its recall, ultimately achieving new state-of-the-art results. we demonstrated that cgli can help with classifying stories, and identifying the conflicting sentence for inconsistent stories. future work should investigate how to enhance the commonsense ability of our procedural understanding model, e.g., by injecting commonsense knowledge during finetuning @xcite or by pretraining on commonsense knowledge bases @xcite @xcite , and how to apply procedural understanding to other downstream tasks, such as dialogue modelling @xcite and planning @xcite . also, it's worth exploring the lightweight-tuning methods @xcite to enhance the model's generalization and reduce computation cost.