A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks (2020)

Abstract: Many high-level procedural tasks can be decomposed into sequences of instructions that vary in their order and choice of tools. In the cooking domain, the web offers many, partially-overlapping, text and video recipes (i.e. procedures) that describe how to make the same dish (i.e. high-level task). Aligning instructions for the same dish across different sources can yield descriptive visual explanations that are far richer semantically than conventional textual instructions, providing commonsense insight into how real-world procedures are structured. Learning to align these different instruction sets is challenging because: a) different recipes vary in their order of instructions and use of ingredients; and b) video instructions can be noisy and tend to contain far more information than text instructions. To address these challenges, we use an unsupervised alignment algorithm that learns pairwise alignments between instructions of different recipes for the same dish. We then use a graph algorithm to derive a joint alignment between multiple text and multiple video recipes for the same dish. We release the Microsoft Research Multimodal Aligned Recipe Corpus containing ~150K pairwise alignments between recipes across 4262 dishes with rich commonsense information.

## introduction although machine learning has seen tremendous recent success in challenging game environments such as go @xcite , dota (ope-nai, 2019), and starcraft (deepmind, 2019), we have not seen similar progress toward algorithms that might one day help humans perform everyday tasks like assembling furniture, applying makeup, 4. stir fry until tender. recipes in the cooking domain provide procedural instruction sets that are captured -in large volume -both in video and text-only forms. instruction sets in these two modalities overlap sufficiently to allow for an alignment that reveals interestingly different information in the linguistic and visual realms. in figure 1 , for instance, the text recipe (left) and the transcribed video recipe (right) for shrimp fried rice vary in word usage, order of instructions and use of ingredients. know-5. add carrots, onion, peas and garlic and season with a pinch of salt and pepper. ## recipe data collection we describe our approach for collecting large-scale text and video recipes; and constructing recipe pairs for training our unsupervised alignment algorithm. ## recipe alignment algorithm we first describe our unsupervised pairwise alignment model trained to learn alignments between text-text, text-video, and video-video recipes pairs. we then describe our graph algorithm, which derives joint alignments between multiple text and video recipes given the pairwise alignments. ## experimental results we describe how we evaluate our pairwise alignment algorithm (from §3.1). we answer the following research questions using our experimentation: ## data release we describe the data released in our microsoft research multimodal aligned recipe corpus. in all our released data, for text recipes, we include the actual text of the instructions. whereas, for video recipes, we release the url to the youtube video with timestamps corresponding to the aligned video segments. ## applications of our corpus we believe that our data release will help advance research at the intersection of language, vision and robotics. the pairwise alignment between recipes within a dish could be useful in training models that learn to rewrite recipes given ingredient or cooking method based constraints. the joint alignment over multiple text recipes within a dish should prove useful for learning the types of ingredient substitutions and instruction reordering that come naturally to expert cooks. the textual and visual paraphrases will, we believe, have implications for tasks like textual similarity, image and video captioning, dense video captioning and action recognition. the single-step to multi-step breakdown derived from our pairwise alignments may also prove useful for understanding task simplification, an important problem for agents performing complex actions. such multimodal data at scale is a crucial ingredient for robots to learn-from-demonstrations of procedural tasks in a variety of environments. collecting such large scale data is prohibitively expensive in robotics since it requires extensive instrumentation of many different environments. other example applications are learning to ground natural language to physical objects in the environment, and catching when humans are about to commit critical errors in a complicated task and offering to help with corrective instructions. ## conclusion we introduce a novel two-stage unsupervised algorithm for aligning multiple text and multiple video recipes. we use an existing algorithm to first learn pairwise alignments and then use a graph-based algorithm to derive the joint alignments across multiple recipes describing the same dish. we release a large-scale dataset constructed using this algorithm consisting of joint alignments between multiple text and video recipes along with useful commonsense information such as textual and visual paraphrases; and single-step to multi-step breakdown. although our dataset focuses on the cooking domain, our framework should generalize to any domain with abundant volumes of unstructured-butalignable multi-modal data. diy (do-it-yourself) videos and websites, for instance, are an obvious next target. we also envision extending this work by including audio and video features to enhance the quality of our alignment algorithm. ultimately, we believe this work will further the goal of building agents that can work with human collaborators to carry out complex tasks in the real world.