A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos (2020)

Abstract: Watching instructional videos are often used to learn about procedures. Video captioning is one way of automatically collecting such knowledge. However, it provides only an indirect, overall evaluation of multimodal models with no finer-grained quantitative measure of what they have learned. We propose instead, a benchmark of structured procedural knowledge extracted from cooking videos. This work is complementary to existing tasks, but requires models to produce interpretable structured knowledge in the form of verb-argument tuples. Our manually annotated open-vocabulary resource includes 356 instructional cooking videos and 15,523 video clip/sentence-level annotations. Our analysis shows that the proposed task is challenging and standard modeling approaches like unsupervised segmentation, semantic role labeling, and visual action detection perform poorly when forced to predict every action of a procedure in a structured form.

## introduction instructional videos are a convenient way to learn a new skill. although learning from video seems natural to humans, it requires identifying and understanding procedures and grounding them to the real world. in this paper, we propose a new task and dataset for extracting procedural knowledge into a fine-grained structured representation from multimodal information contained in a largescale archive of open-vocabulary narrative videos with noisy transcripts. while there is a significant amount of related work (summarized in §3 & 7), to our knowledge there is no dataset similar in scope, with previous attempts focusing only on a single 4 i 'm heating it up with a medium, medium high flame, and i 'm going to put some bacon in there and fry it up. 5 and this is not a diet recipe. 6 sorry they making clam chowder. 7 an eye you can see the photo montage before this what i did cool. 8 somehow i fried some bacon. ## problem definition the output will be: each procedural tuple s @xmath0, arg 1 , ..., arg k ) ∈ s r consists of a verb phrase and its arguments. only the "verb" field is required, and thus the tuple size ranges from 1 to k + 1. all fields can be either a word or a phrase. not every clip/sentence describes procedures, as most videos include an intro, an outro, nonprocedural narration, or off-topic chit-chat. key clips v r are clips associated with one or more procedures in p r , with some clips/sentences associated with multiple procedure tuples. conversely, each procedure tuple will be associated with only a single clip/sentence. ## dataset & analysis while others have created related datasets, they fall short on key dimensions which we remedy in our work. specifically, in ## extraction stage 1: key clip selection in this and the following section, we describe our two-step pipeline for procedural knowledge extraction (also in figure 4 ). this section describes the first stage of determining which clips are "key clips" that contribute to the description of the procedure. we describe several key clip selection models, which consume the transcript and/or the video within the clip and decide whether it is a key clip. ## extraction stage 2: structured knowledge extraction with the identified key clips and corresponding transcript sentences, we proceed to the second stage that performs clip/sentence-level procedural knowledge extraction from key clips. in this stage, the extraction is done from clips that are identified at first as "key clips". ## evaluation we propose evaluation metrics and provide evaluation results on our annotated dataset for both of the two stages: key clip selection and structured procedural extraction. detailed reproducibility information about the experiments are in appendix f. besides quantitative evaluation and qualitative evaluations, we also analyze the key challenges of this task. ## related work text-based procedural knowledge extraction. procedural text understanding and knowledge extraction @xcite @xcite @xcite @xcite @xcite unlike existing work on action graph/dependency extraction @xcite , our approach differs as we extract knowledge from the visual signals and transcripts directly, not from imperative recipe texts. instructional video understanding. beyond image semantics @xcite , unlike existing tasks for learning from instructional video @xcite @xcite @xcite @xcite @xcite , combining video & text information in procedures @xcite , visual-linguistic reference resolution @xcite , visual planning @xcite , joint learning of object and actions @xcite @xcite , pretraining joint embedding of high level sentence with video clips @xcite , our task proposal requires explicit structured knowledge tuple extraction. in addition to closely related work ( §3) there is a wide literature @xcite @xcite @xcite @xcite that aims to predict/align dense procedural captions given the video, which are the most similar works to ours. @xcite extracted temporal procedures and then generated captioning for each procedure. @xcite proposes a multimodal abstractive summarization for how-to videos with either human labeled or speech-to-text transcript. @xcite also introduces an unsupervised step learning method from instructional videos. inspired by cross-task sharing @xcite , which is a weakly supervised method to learn shared actions between tasks, fine grained action and entity are important for sharing similar knowledge between various tasks. we focus on structured knowledge of fine-grained actions and entities.visual-linguistic coreference resolution @xcite is among one of the open challenges for our proposed task. ## conclusions & open challenges we propose a multimodal open procedural knowledge extraction task, present a new evaluation dataset, produce benchmarks with various methods, and analyze the difficulties in the task. meanwhile we investigate the limit of existing methods and many open challenges for procedural knowledge acquisition, including: to better deal with cases of coreference and ellipsis in visual-grounded languages; exploit cross-modalities of information with more robust, semi/un-supervised models; potential improvement from structured knowledge in downstream tasks (e.g., video captioning).