CLMSM : A Multi-Task Learning Framework for Pre-training on Procedural Text (2023)

Abstract: In this paper, we propose ***CLMSM***, a domain-specific, continual pre-training framework, that learns from a large set of procedural recipes. ***CLMSM*** uses a Multi-Task Learning Framework to optimize two objectives - a) Contrastive Learning using hard triplets to learn fine-grained differences across entities in the procedures, and b) a novel Mask-Step Modelling objective to learn step-wise context of a procedure. We test the performance of ***CLMSM*** on the downstream tasks of tracking entities and aligning actions between two procedures on three datasets, one of which is an open-domain dataset not conforming with the pre-training dataset. We show that ***CLMSM*** not only outperforms baselines on recipes (in-domain) but is also able to generalize to open-domain procedural NLP tasks.

## introduction procedural text consists of a set of instructions that provide information in a step-by-step format to accomplish a task. procedural text can be found in different domains such as recipes @xcite @xcite @xcite , e-manuals @xcite , scientific processes @xcite , etc. procedural reasoning requires understanding the interaction between participating entities and actions associated with such entities in procedures. recent works have made attempts at tracking entities @xcite @xcite and aligning actions @xcite across the different steps of procedures to evaluate procedural reasoning. the performance of these tasks can possibly be enhanced by specialized pre-training on large procedural corpora, such that the pre-training objectives are aligned with the downstream tasks at hand. however, none of the prior art has studied the effects of pre-training to solve such procedural nlp tasks thoroughly. current self-supervised pre-training techniques such as masked language modeling (mlm) @xcite , next sentence prediction (nsp) @xcite ), text denoising @xcite , etc. have been shown to perform well on downstream nlp tasks such as question answering (qa), named entity recognition (ner), natural language inference (nli), etc. while such methods could capture local contexts proficiently @xcite , these methods fail in capturing procedural, step-wise contexts required for tracking entities and aligning actions in procedural texts @xcite as the learning objectives do not consider the context of the entire procedure, and the interaction between entities and actions. to address these challenges, in this paper, we propose clmsm foot_0 , which is a combination of contrastive learning (cl) using domain-specific metadata, and a self-supervised masked step modeling (msm) to perform domain-specific continual pretraining for procedural text. we use cl with hard triplets @xcite which enables learning fine-grained contexts associated with participating entities and actions, by pulling representations of highly similar procedures closer and pushing representations of slightly dissimilar procedures further away from each other. prior works on instructional video understanding such as @xcite have also shown to benefit from the use of msm in learning stepwise representations to capture global context of the entire video. inspired from such works, we incorporate msm for procedural text. msm considers a step as a single entity and learns step-wise context, as it predicts tokens of randomly masked step(s), given the other steps of the procedure. we pre-train our model in the recipe domain, as recipes provide step-wise instructions on how to make a dish, making them an ideal candidate for pre-training. we experiment with various downstream procedural reasoning tasks that require understanding of the changes in states of entities across steps of the procedure, triggered due to actions carried out in each step. the first task of tracking the movement of constituent entities through a procedure is carried out on the npn-cooking dataset @xcite of the recipe domain, and propara dataset @xcite of the open-domain (to evaluate the generalizability of domain-specific pre-training), while the second task is to align actions across different recipes of the ara dataset @xcite . we show that clmsm performs better than several baselines when evaluated on domain-specific as well as open-domain downstream tasks. specifically, clmsm provides performance improvements of 1.28%, 2.56%, and 4.28% on the npn-cooking, propara, and ara action alignment datasets, respectively. our extensive ablation study suggests that a) the proposed multi-task learning framework outperforms using any one of the two objectives or using these objectives sequentially, b) masking an entire step is much more effective than masking a random span, and c) the use of hard triplets during cl is much more effective than easy triplets. the remainder of the paper is organized as follows: we begin by introducing the proposed pretraining framework in section 2, followed by the experimental setup in section 3, and a description of the datasets, fine-tuning, and evaluation frameworks used for the two downstream tasks, along with experiments, results and analysis in sections 4 and 5. finally, we conclude with a summary of our findings in section 6. the code and models used in this paper are available on github foot_1 . ## experimental setup we evaluate clmsm on the tasks of entity tracking and recipe alignment 7 . we assess the performance of clmsm vis-a-vis baselines and various variations of clmsm, and list inferences that can be drawn. note that, from here onwards, the subscript in any model notation represents the encoder used (rb -roberta-base, rl -roberta-large, bert -bert-base-uncased). ## experimental details: tracking entities for the task of entity tracking, we evaluate on (1) npn-cooking dataset in the recipe domain to see how helpful domain-specific training is for in-domain tasks, and (2) propara dataset in the open-domain to see how generalizable a domainspecific pre-trained model is. ## experimental details: aligning actions as mentioned in @xcite , action alignment implies each action in one recipe being independently assigned to an action in another similar recipe or left unassigned. given two similar recipes r 1 (source recipe) and r 2 (target recipe), we need to decide for each step in r 1 the aligning step in the r 2 , by mapping the word token corresponding to an action in r 1 to one or more word tokens corresponding to the same action in r2. however, some actions in the source recipe may not be aligned with actions in the target recipe. an example of the task is explained as follows -let r 1 and r 2 be recipes to make waffles. we observe the alignment of the step "ladle the batter into the waffle iron and cook [...]" in r 1 with the step "spoon the batter into preheated waffle iron in batches and cook [..]" in r 2 (words in bold are the actions). the relationship between the specific actions of "ladle" in r 1 and "spoon" in r 2 as well as between both instances of "cook" may not be easy to capture when using a coarsegrained sentence alignment, which makes the task challenging. the details about fine-tuning on the task are described in section e of appendix.