Multimodal Procedural Planning via Dual Text-Image Prompting (2024)

Abstract: Embodied agents have achieved prominent performance in following human instructions to complete tasks. However, the potential of providing instructions informed by texts and images to assist humans in completing tasks remains underexplored. To uncover this capability, we present the multimodal procedural planning (MPP) task, in which models are given a high-level goal and generate plans of paired text-image steps, providing more complementary and informative guidance than unimodal plans. The key challenges of MPP are to ensure the informativeness, temporal coherence,and accuracy of plans across modalities. To tackle this, we propose Text-Image Prompting (TIP), a dual-modality prompting method that jointly leverages zero-shot reasoning ability in large language models (LLMs) and compelling text-to-image generation ability from diffusion-based models. TIP improves the interaction in the dual modalities using Text-to-Image Bridge and Image-to-Text Bridge, allowing LLMs to guide the textual-grounded image plan generation and leveraging the descriptions of image plans to ground the textual plan reversely. To address the lack of relevant datasets, we collect WIKIPLAN and RECIPEPLAN as a testbed for MPP. Our results show compelling human preferences and automatic scores against unimodal and multimodal baselines on WIKIPLAN and RECIPEPLAN in terms of informativeness, temporal coherence, and plan accuracy.

## introduction recent advances in embodied @xcite and conversational @xcite agents achieve prominent performance in task completion as humans by following instructions informed by texts and images. however, to what extent the models can provide useful guidelines for humans to complete the task remains underexplored. to uncover this, we propose the multimodal procedural planning task (as shown figure 1 ). the task aims to generate goal-conditioned (e.g. "how to make traditional szechuan chicken") text (e.g. "a teaspoon of light soy sauce" explain how to marinade chicken in step 2) and image (e.g. help identify the ingredients "chicken, garlic, ginger, and light soy sauce" in step 1) plans as useful guidelines to assist humans in task completion. previous work @xcite explored the generation of procedural plans in text-only form. in contrast, we generate both text and image plans, which provide guidance to perform tasks that acquire complementary information from multimodal contexts. generating plans in both text and image form poses new challenges since the generated plans should: a) be informative enough in both the text and image modalities, b) obey commonsense temporal coherence, such as the order of steps, and c) achieve high plan accuracy, indicating the complementary and alignment among multimodal plans. despite significant progress @xcite in the development of large language models (llms), they are unable to generate images. existing text-to-image (t2i) models can generate high-quality images conditioned on textual instructions @xcite @xcite . however, they are limited in their ability to generate images that require complex text comprehension, such as temporal reasoning (e.g. "learn basic surf safety before hitting the waves") and physical reasoning (e.g. "pick up the wine glass"). additionally, generating text and image plans separately using llms and t2i models results in inconsistency and incoherence between the two modalities. in this paper, we propose text-image prompting (tip), a novel dual-modality prompting framework that jointly leverages the capabilities of llms and t2i models for multimodal procedural planning. we first generate vanilla text plans by querying llms @xcite for step-by-step proce-dures. to generate textual-grounded image plans, we devise the text-to-image bridge (t2i-b), which elicits the complex language comprehension abilities of llms to assist t2i models in generating informative and text-aligned image plans. we generate visual-grounded text plans using the image-to-text bridge (t2i-b), which verbalizes the image plans and injects them back into llms to assist the text plans revision, thereby improving informativeness. the temporal coherence of the generated plans is improved considering the context of both text and image. benefiting from dual-modality prompting, generated plans are complementary and aligned across text and image modalities. to address the lack of suitable datasets for evaluating multimodal procedural planning, we collect the wikiplan and recipeplan datasets for benchmarking the task. we empirically evaluate the effectiveness of tip on wikiplan and recipeplan in a zero-shot setting and compare it with various baselines. our results demonstrate that tip generate plausible multimodal plans that are informative, temporally coherent, and accurate. our work highlights the potential of combining knowledge from llms and t2i models to uncover multimodal zero-shot planning capabilities. our main contributions are as follows: ## related work procedural planning this task @xcite has gain much attention in various aspects, including robotics @xcite @xcite , visionand-language navigation @xcite , conversational assistants @xcite @xcite , and animation @xcite . recent work is extended to the multimodal scenarios @xcite @xcite . in this work, task: how to make a candy bouquet? textual-grounded image plan step 1 step 2 … step 6 visual-grounded text plan step 1: gather materials: candy, foam blocks, cellophane wrap, ribbon or bows from the table. step 2: cut the foam block into desired size and shape of bouquet using scissors or knife like how you would arrange flowers in a vase. … step 6: wrap bouquet in cellophane wrap and tie off with ribbon or bow for extra decorations. vanilla text plan step 1: gather materials such as candy, foam blocks, cellophane wrap, ribbon or bows. step 2: cut the foam block to size and shape of desired bouquet. … step 6: wrap bouquet in cellophane wrap and tie off with ribbon or bow. we explore the multimodal procedural planning that generates goal-conditioned text and image sequences grounded in a multimodal context. ## multimodal procedural planning dataset collection our datasets are collected and repurposed from wikihow foot_1 and recipeqa @xcite due to their temporal relatedness among texts and images. we collect wikiplan by crawling the household "how to" articles from wikihow and then repurpose them into a multimodal procedural planning dataset by formulating the article title as the task name and content as the textual steps, with the pictures as the visual steps. we collect recipeplan from recipeqa dataset for multimodal procedural planning by sequencing all the given text-image pairs as the text and image step … gather materials such as candy, foam blocks, cellophane wrap, ribbon or bows from the table. what do i need to draw in the picture to describe the above text? step 2 gather materials such as candy, foam blocks, cellophane wrap, ribbon or bows from the table. what do i need to draw in the picture to describe the above text? step 1 gather materials such as candy, foam blocks, cellophane wrap, ribbon or bows from the table. what do i need to draw in the picture to describe the above text? prompt … items such as various kinds of candy, foam blocks, cellophane wrap, and ribbons or bows. items such as various kinds of candy, foam blocks, cellophane wrap, and ribbons or bows. prompt 1 items such as various kinds of candy, foam blocks, cellophane wrap, and ribbons or bows. llm t2i-bridge vanilla plan imagination prompt generation text-to-image generation textual-grounded image plan text-to-image model visualized instruction: caption 1: a table topped with lots of candy and candies … caption 6: a bouquet of colorful ribbons and bows on a table task: how to make a candy bouquet? step 1: gather materials such as candy, foam blocks, cellophane wrap, ribbon or bows from the table. … step 6: wrap bouquet in cellophane wrap and tie off with ribbon or bow for extra decorations . plan correspondingly, with the main title as the task name. we conduct zero-shot experiments on 1, 000 distinct, randomly sampled tasks from each dataset. more dataset details are in appendix c. ## conclusions we introduce the multimodal procedural planning task that aims to generate goal-conditioned text and image subsequences and benchmark models' performance with our curated testbed wikiplan and recipeplan. our proposed dual-modality prompting framework enables plausible multimodal procedural plan generation.