Order-Based Pre-training Strategies for Procedural Text Understanding (2024)

Abstract: In this paper, we propose sequence-based pre-training methods to enhance procedural understanding in natural language processing. Procedural text, containing sequential instructions to accomplish a task, is difficult to understand due to the changing attributes of entities in the context. We focus on recipes as they are commonly represented as ordered instructions, and use this order as a supervision signal. Our work is one of the first to compare several ‘order-as-supervision’ transformer pre-training methods, including Permutation Classification, Embedding Regression, and Skip-Clip, and show that these methods give improved results compared to baselines and SoTA LLMs on two downstream Entity-Tracking datasets: NPN-Cooking dataset in recipe domain and ProPara dataset in open domain. Our proposed methods address the non-trivial Entity Tracking Task that requires prediction of entity states across procedure steps, which requires understanding the order of steps. These methods show an improvement over the best baseline by 1.6% and 7-9% on NPN-Cooking and ProPara Datasets respectively across metrics.

## introduction procedural text comprises a series of sequential instructions aimed at guiding individuals through a task by presenting information in a step-by-step manner. a procedure describes a step-wise interaction between multiple participating entities and their attribute changes. for instance, "photosynthesis" as a procedure consists of interaction between entities such as water, light, co2, sugar, etc. recently, there has been an increase in the number of studies in nlp that use procedural texts. procedural text is common in natural language in recipes @xcite @xcite @xcite , how-to guides @xcite , and scientific processes @xcite . in this study, we focus on recipes as they are commonly represented as ordered instructions. we utilize this order as a supervision signal to develop customized pretraining techniques to solve non-trivial tasks that require anticipating the implicit effects of actions on entities. understanding procedural text is difficult due to the changing attributes of entities in the context. previous works such as @xcite used sentence-level language modeling (slm) to learn contextualized sentence-level representation by training a hierarchical transformer to reconstruct the original order of a shuffled sequence, @xcite proposed interactive entity network (ien) to model different types of entity interactions using a recurrent network with memory for state tracking, and @xcite combined external knowledge with a bert model to improve entity tracking. however, such works do not compare pre-training techniques which consider sequential order of the steps of the procedure. in this paper, we try to solve the non-trivial entity tracking task that requires prediction of entity states across procedure steps. solving such a task requires understanding the sequential nature/order of the steps. explicitly learning the order within data has been shown to enhance performance of tasks such as video representation learning, solving jigsaw puzzles to learn image representations @xcite , etc. similarly, albert @xcite shows that sentence-order prediction between two sentences is a useful pre-training objective to improve performance on various downstream nlp tasks. inspired by such works, our work is one of the first to introduce and compare several novel 'order-assupervision' pre-training methods such as permutation classification, skip-clip, and embedding regression to enhance procedural understanding. these pre-training methods give a significant improvement of 1.6% and 7 -9% compared to baselines across metrics on two downstream entity-tracking datasets, namely, npn-cooking dataset @xcite in the recipe domain and propara dataset @xcite in the open domain. our methods also outperform sota llms in terms of average accuracy on propara. ## pre-training methods we propose three new pre-training methods that help in learning sequential context for procedural texts: permutation classification, embedding regression, and skip-clip. for all the methods, a set of recipes with the same number of steps are sampled. such a recipe of n steps can be represented by @xmath0, x 2 , ..., x n ). ## downstream tasks entity tracking consists of two sub-tasks -entity state and location tracking. both tasks output the result for every entity at each step of the process. entity state tracking task is a 4-way classification task that predicts if the entity is created, moved, unchanged, or destroyed at that step of the process. entity location tracking is formulated as a spanbased question-answering problem that outputs the location of the entity at a particular step taking the entire text of the process as the input. we perform this task on npn-cooking and propara datasets 2 . npn-cooking dataset @xcite consists of 65,816 training, 175 development, and 700 evaluation recipes. propara dataset (mishra et al., 2018) consists of 488 human-authored procedures (split 80/10/10 into train/dev/test) with 81k annotations regarding changing states (existence and location) of entities in those paragraphs. the model is evaluated in 3 ways corresponding to a given entity e. category 1: which of the three transitions -created, destroyed, or moved undergone by e over the lifetime of a procedure; category 2: steps at which e is created, destroyed, and/or moved; and category 3: the location (span of text) which indicates e's creation, destruction or movement. following faghihi and kordjamshidi (2021), entity tracking is formulated as a question-answering problem. fine-tuning hyperparameters are the same as in the default open-source implementation foot_2 . 4 experiments and results 4.1 pre-training setup 2019 ). for each recipe in the dataset, a sentence with the ingredients is also added as a step before the original recipe. the dataset is filtered to include recipes with more than 4 steps. the statistics of the dataset is shown in