Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals (2022)

Abstract: The ability to sequence unordered events is evidence of comprehension and reasoning about real world tasks/procedures. It is essential for applications such as task planning and multi-source instruction summarization. It often requires thorough understanding of temporal common sense and multimodal information, since these procedures are often conveyed by a combination of texts and images. While humans are capable of reasoning about and sequencing unordered procedural instructions, the extent to which the current machine learning methods possess such capability is still an open question. In this work, we benchmark models’ capability of reasoning over and sequencing unordered multimodal instructions by curating datasets from online instructional manuals and collecting comprehensive human annotations. We find current state-of-the-art models not only perform significantly worse than humans but also seem incapable of efficiently utilizing multimodal information. To improve machines’ performance on multimodal event sequencing, we propose sequence-aware pretraining techniques exploiting the sequential alignment properties of both texts and images, resulting in > 5% improvements on perfect match ratio.

## introduction instructions are essential sources for agents to learn how to complete complex tasks composed of multiple steps (e.g., "making a wood sign from scratch"). however, instructions do not always come in a proper sequential order, for example, when instructions must be combined across sources (e.g., to accomplish a complex task there might be multiple useful resources for certain task-steps come out from a single google search). therefore, sequencing unordered task-steps is crucial for comprehending and inferring task procedures, which requires thorough understanding of event causal and temporal common sense. it is essential for sand the wood block. press a sandpaper on and rub it. ## problem definition given a task procedure s consisting of n steps, where each step s @xmath0: a textual description t i of tokens {t i,k } n t @xmath1,k } n i @xmath2. foot_1 a model is required to take as inputs a random permutation of s, i.e. s @xmath3, ..., s p n }, where p is a permutation (s p j can take one of the following three modalities: t p j , i p j , and {t p j , i p j }), and predict the correct order of s p , i.e. argsort(s p ). ## datasets and human annotation we are interested in understanding the current stateof-the-art models' performance on this multimodal instruction sequencing task. to this end, we curate instruction datasets to support our study. ## models to benchmark the proposed task, we construct models comprising: (1) an encoder which encodes multimodal or text/image-only inputs, and (2) an order decoder which utilizes the encoded representations to predict the orders. to help models capture sequentiality in task-steps better as well as adapt to our target task domains, we pretrain the encoders with several self-supervised objectives on the instructions before integrating them with the decoder. ## experiments and analysis our experiments seek to answer these questions: ## conclusions in this work we present studies of language and multimodal models on procedure sequencing, leveraging popular online instructional manuals. our experiments show that both multimodality and our proposed sequence-aware pretraining are helpful for multimodal sequencing, however, the results also highlight significant gaps below human performance (∼ 15% on pmr). we provide insights as well as resources, such as the multi-reference annotations of the sequencing task, to spur future relevant research. we also anticipate that the alternative orders defined and annotated in our work can benefit more comprehensive task-procedure understanding. future work such as predicting task steps which can be parallel or interchangeable, and understanding step dependencies can be explored.