Enhance Multimodal Consistency and Coherence for Text-Image Plan Generation (2025)

Abstract: People get informed of a daily task plan through diverse media involving both texts and images. However, most prior research only focuses on LLM’s capability of textual plan generation. The potential of large-scale models in providing text-image plans remains understudied. Generating high-quality text-image plans faces two main challenges: ensuring consistent alignment between two modalities and keeping coherence among visual steps. To address these challenges, we propose a novel framework that generates and refines text-image plans step-by-step. At each iteration, our framework (1) drafts the next textual step based on the prediction history; (2) edits the last visual step to obtain the next one; (3) extracts PDDL-like visual information; and (4) refines the draft with the extracted visual information. The textual and visual step produced in stage (4) and (2) will then serve as inputs for the next iteration. Our approach offers a plug-and-play improvement to various backbone models, such as Mistral-7B, Gemini-1.5, and GPT-4o. To evaluate the effectiveness of our approach, we collect a new benchmark consisting of 1,100 tasks and their text-image pair solutions covering 11 daily topics. We also design and validate a new set of metrics to evaluate the multimodal consistency and coherence in text-image plans. Extensive experiment results show the effectiveness of our approach on a range of backbone models against competitive baselines.

## introduction recently, there has been growing attention on employing llms for planning, the task of decomposing a high-level goal into a sequence of executable steps @xcite . llms have demonstrated strong capabilities in generating textual plans, enabling applications in robotics, virtual assistants, and instruc- 🤔 🗒rudimentary plan: how to grow bulb onions in water? step 1: prepare the onion bulbs. trim the roots if they are too long. step 2: position the onion bulbs in the glass or jar with the root end facing down. step 3: position the glass or jar with the onion bulbs in a sunny windowsill. step 4: change the water in the glass or jar every few days to prevent stagnation and the growth of bacteria. multimodal task planning faces two main challenges: ensuring coherence through visual steps and alignment between two modalities. figure 1 illustrates these challenges through a comparative example of generating plans for the input goal "growing bulb onions in water". the left-hand side shows the plan generated by gpt-4o, and the right-hand side presents the corresponding plan generated by our framework. the baseline plan, although generally reasonable, exhibits critical issues in both visual coherence and text-image alignment. first, it fails to maintain visual consistency between step 1 and step 2, as highlighted in red. the container depicted in step 1 is a transparent glass, but in step 2, it becomes an opaque white planter, disrupting coherence. second, as highlighted in purple, step 2's image shows a location with direct sunlight, contradicting the textual instruction that specifies "indirect sunlight". furthermore, step 3 entirely diverges from the intended goal of growing onions in water, as it depicts bulbs planted in soil. on the other hand, our framework effectively improves visual coherence and text-image consistency. as shown on the right-hand side of figure 1 , our approach maintains a uniform depiction of the onion bulbs and their container throughout the process. the glass or jar remains consistent in shape and size from step 2 to step 4, avoiding abrupt visual changes. moreover, the visual steps accurately reflect the textual descriptions through all steps. for instance, step 3 correctly depicts a sunny windowsill, aligning with the text's instruction to position the jar in such a location. these improvements stem from our novel autoregressive framework. at each iteration, the text generator drafts the current step based on the task goal and previous steps. the image generator then produces a corresponding visual representation conditioned on the last visual step. an image interpreter subsequently extracts structured information from the generated image, which the text generator uses to refine the step draft. this iterative process enhances visual coherence with a text-image-toimage model and ensures text-image consistency through cross-modality prompting. compared to vanilla approaches that simply concatenate an llm with a text-to-image model, our framework effectively mitigates the challenges of multimodal task planning, leading to more coherent and consistent instructional sequences. to evaluate our framework, we collect a dataset from two popular websites: instructables and wiki-how. our dataset consisting of 1100 examples in 11 categories, providing rich multimodal information of procedural solutions to a variety of daily tasks. we adopt a set of metrics including conventional automatic measurements, llm evaluations, and human evaluations to comprehensively evaluate planning performance in three aspects: textual plan quality, visual plan quality, and textual-visual plan alignment. to demonstrate the generalizability of our approach, we evaluate our framework with 3 different backbones: mistral-7b (jiang et al., 2023), gemini-1.5-flash @xcite , and gpt-4o @xcite . for every backbone, we compare our framework with various baselines. extensive experiment results show our framework outperforms all baselines, especially in terms of the two concerns we aim to address. in summary, our contributions are three-fold: task planning task planning is broadly studied in various scenarios of virtual environment @xcite @xcite , embodied environment @xcite @xcite , and daily life @xcite . despite classical planning algorithms @xcite , they are primarily applicable with restrictions such as fully observable environments with pre-defined actions and objects. it prohibits their usage in open-domain daily sce-narios, igniting research interest in solutions from llm advancements @xcite . llms, possessing a rich amount of commonsense knowledge and impressive reasoning capability, are competent in such contexts. @xcite studies llms as zero-shot planners and shows their great planning potential in a virtual environment. @xcite combines the llm with classical planners to get reliable solutions to robotic tasks in an embodied environment. @xcite tries to add an external verifier to improve the planning capabilities of a finetuned llm. @xcite seeks to study textual plan generation provided visual states as supplements. despite these, only rare efforts contribute to studying multimodal planning. @xcite first explores the potential of llms and text-to-image models to generate image-text paired plans. however, their approach is limited by the single-shot generation framework and reliance on image captions that inadequately capture object interactions, leading to information loss and visual incoherence in planning sequences. vision and language diffusion models @xcite @xcite have revolutionized image generation conditioned on texts. however, in specific circumstances where the models are expected to output coherent images, they exhibit poor performance because of the lack of visual context knowledge @xcite . this leads to the introduction of advanced image editing models @xcite . building upon the success of image generation models, they are capable of generating images conditioned on both reference images and text instructions. for example, @xcite achieves image editing by fine-tuning the stable diffusion model on (original image, instruction, edited image) triplets. @xcite enables precise spatial conditioning controls including edges, depth maps, pose information, etc. @xcite infuses video data into a diffusion model to generate images depicting how actions lead to object state transformations. however, they all tend to maintain the outline of objects in the original image while only editing the color, texture, or style. thus, they are still inadequate in generating coherent images for planning, which typically involves scenario change and object transformation. ## dataset despite existing datasets in the daily task planning area, they either lack image modality (koupaee and inspired by previous work, we consider two popular websites affording procedural daily task instructions: instructables foot_0 and wikihow foot_1 . both data sources provide various modalities and cover diverse daily task categories. the overall category distribution of our dataset is shown in figure 2 . compared with previous public datasets, our dataset is rich in task categories and plan modalities. furthermore, we ensure high data quality by manually filtering out malicious content and curating the collected plans to remove noises such as the authors' information and personal stories. please see appendix a for more dataset statistics and demonstrations. instructables to study how our framework performs on simple daily tasks, we randomly collect 100 plans from instructable. specifically, we choose "cooking" category as it is a common planning scenario. plans from instructables are published by users who would like to share their experience in achieving a specific goal. therefore, their plans are more brief and more casual. stage 1: textual plan generation task goal g grow bulb onions in water step 1: prepare the onion bulbs. trim the roots if they are too long… step 2: position the onion bulbs in the glass or jar with the root end facing down… textual steps ! !:#@xmath0! place the glass or jar in a sunny location. ensure they receive plenty of sunlight. textual draft # # tool: glass object: plants action: placing goal: provide plants with access to sunlight for optimal growth step 3: position the glass or jar with the onion bulbs in a sunny windowsill. onions need plenty of light to grow, so ensure they receive at least 6-8 hours of sunlight. textual step ! # draft generation image generation visual info extraction text refinement stage 2: visual plan generation stage 3: visual info extraction stage 4: textual plan refinement visual info -# visual step 𝑖 ! figure 3: overview of our autoregressive framework at time step k. stage 1: the text plan generator takes the input task goal g and output textual steps t 1:k-1 from previous time steps to predict d k . stage 2: the image generator outputs the visual step i k conditioned on d k and the last visual step i k-1 . stage 3: the text generator extracts formatted visual information from the generated i k as v k . stage 4: the text generator refines d k with v k to generate t k . dotted arrow: the output t k and i k then serve as the input for the next time step k + 1. ## approach task formulation given a task goal g as the input, the output of multimodal planning is a sequence of steps @xmath1, s 2 , ..., s n }. each step consists of an instructional text t k and its corresponding image i k , so we have s @xmath2, i k ), @xmath3, 2, ..., n}. at every step k, t k and i k should be semantically consistent. both textual plan s @xmath4, ..., t n } and visual plan s @xmath5, ..., i n } should be coherent through steps. framework overview as shown in figure 3 , our framework generates plan steps iteratively. at each step, 4 cross-modality stages collaboratively contribute to the multimodal plan generation. in the first stage, the textual plan drafter takes the input task goal g and the previous textual steps t 1:k-1 to draft the current textual step d k . in stage two, the image generator edits the last visual step i k-1 conditioned on d k to get the current visual step i k . in stage three, an image interpreter extracts formatted visual information v k from the generated i k . last, the textual plan refiner collects v k and uses it to refine the original textual step draft d k to be t k . after the current iteration is complete, both t k and i k are added to history steps for the next iteration. the overall design ensures the coherence of visual steps by prompting the image generation module to predict the next image based on both textual instruction and the previous visual state. combining text-to-image and converse image-to-text in a loop guarantees consistency between the generated textual plan and its visual counterpart. detailed prompts for each module are in appendix b. ## conclusion our work studies an underexplored problem of textimage plan generation. we identify two main challenges: ensuring visual coherence and text-image alignment, and propose a novel framework to address them accordingly. for evaluation, we collect a dataset of daily tasks covering diverse domains and task complexities. substantial experiment results demonstrate the effectiveness of our approach on a range of various backbone models, especially in terms of the two challenges we aim to address.