Non-Sequential Graph Script Induction via Multimedia Grounding (2023)

Abstract: Online resources such as WikiHow compile a wide range of scripts for performing everyday tasks, which can assist models in learning to reason about procedures. However, the scripts are always presented in a linear manner, which does not reflect the flexibility displayed by people executing tasks in real life. For example, in the CrossTask Dataset, 64.5% of consecutive step pairs are also observed in the reverse order, suggesting their ordering is not fixed. In addition, each step has an average of 2.56 frequent next steps, demonstrating “branching”. In this paper, we propose the new challenging task of non-sequential graph script induction, aiming to capture optional and interchangeable steps in procedural planning. To automate the induction of such graph scripts for given tasks, we propose to take advantage of loosely aligned videos of people performing the tasks. In particular, we design a multimodal framework to ground procedural videos to WikiHow textual steps and thus transform each video into an observed step path on the latent ground truth graph script. This key transformation enables us to train a script knowledge model capable of both generating explicit graph scripts for learnt tasks and predicting future steps given a partial step sequence. Our best model outperforms the strongest pure text/vision baselines by 17.52% absolute gains on F1@3 for next step prediction and 13.8% absolute gains on Acc@1 for partial sequence completion. Human evaluation shows our model outperforming the WikiHow linear baseline by 48.76% absolute gains in capturing sequential and non-sequential step relationships.

## introduction a script consists of typical actions that are performed to complete a given task. online re1 our data and code are publicly available for research purposes at https://github.com/bryanzhou008/ multimodal-graph-script-learning/ 2 occurred in more than 10 videos. sources such as wikihow foot_0 provide a wide variety of community-edited scripts for everyday tasks (fig. 1 ). such a large library of linear scripts can serve as a starting point for learning goal-step knowledge @xcite . however, as the saying goes, "all roads lead to rome". there is usually more than one way to achieve any given goal. practically speaking, users should be presented with multiple alternative step sequences so that they can pick the most suitable route according to their unique situations and preferences. robots and virtual assistants also stand to gain the crucial abilities of global planning optimization and on-the-spot improvisation from alternative step paths. ## methodology to learn a graph script induction model, we first ground the video dataset to textual steps on both task-level and step-level (fig. 2 ). after grounding, each video can be seen as a valid step sequence sampled from the ground truth graph script. then, we use such grounded step sequences to train our graph script model and enhance model learning 4 or a large repository of videos from which we can find matching videos using retrieval. ## experiments to evaluate our non-sequential graph script induction model, we propose 3 new downstream tasks: ## conclusions and future work we propose the new task of non-sequential graph script induction to capture optional and interchangeable steps in procedural tasks. instead of relying on the script annotation, we automatically induce graph scripts by grounding procedural videos to a wikihow textual step library. we transform the graph generation problem to a path generation problem that can better aligned with video observations, and train a seq2seq model using our grounded step sequences while imposing path-level constraints via a contrastive loss. experiments demonstrate our model's superiority on downstream tasks including next step prediction and partial sequence completion. human evaluation confirms our model's ability to generate graph scripts that correctly capture optional and interchangeable steps. future work will focus on incorporating more video supervision signals such as enriching steps from videos and adding the repeatable steps.