MMP lanner: Zero-Shot Multimodal Procedural Planning with Chain-of-Thought Object State Reasoning (2025)

Abstract: Multimodal Procedural Planning (MPP) aims to generate step-by-step instructions that combine text and images, with the central challenge of preserving object-state consistency across modalities while producing informative plans. Existing approaches often leverage large language models (LLMs) to refine textual steps; however, visual object-state alignment and systematic evaluation are largely underexplored.We present MMPlanner, a zero-shot MPP framework that introduces Object State Reasoning Chain-of-Thought (OSR-CoT) prompting to explicitly model object-state transitions and generate accurate multimodal plans. To assess plan quality, we design LLM-as-a-judge protocols for planning accuracy and cross-modal alignment, and further propose a visual step-reordering task to measure temporal coherence.Experiments on RecipePlan and WikiPlan show that MMPlanner achieves state-of-the-art performance, improving textual planning by +6.8\% , cross-modal alignment by +11.9\% , and visual step ordering by +26.7\% .

## introduction procedural planning involves generating a sequence of steps to accomplish a goal @xcite , e.g., baking a cake or assembling a bookshelf. domains such as robotics @xcite , reasoning systems @xcite , etc., rely on effective procedural planning. consequently, the field has received growing attention, driven by the recent advancements in llms @xcite . existing works utilize task-specific concept knowledge @xcite , knowledge from llms @xcite , or multimodal input @xcite , and generate linear @xcite @xcite or non-linear @xcite textual procedural plans. however, text-only instructions often lack the visual clarity and specificity required for complex tasks, limiting understanding, accessibility, and engagement. multimodal procedural planning (mpp) addresses these limitations by jointly generating textual step instructions with corresponding step images, yielding more precise and accessible procedural knowledge. a central challenge in mpp is generating step visuals that accurately reflect object state transitions. these transitions can be explicit, when the change is clearly described in the current textual step, or implicit, when it must be inferred from prior steps or broader context. for instance, in figure 1 , step 2 explicitly describes mixing ingredients, so the corresponding image should depict a bowl containing the mixed dry ingredients. step 3, in contrast, involves adding butter, and the image must implicitly incorporate the existing mixture from step 2. in this case, the visual should show butter being added into the bowl of mixed dry ingredients, even though ingredients are not restated in the step text. another important challenge involves the evaluation of multimodal plans, particularly in determining whether the generated steps successfully accomplish the intended task. prior work has primarily measured semantic similarity between generated and reference textual plans @xcite , which cannot effectively verify true task completion. furthermore, text-based semantic metrics overlook critical dimensions such as visual-text alignment, temporal coherence, and the informativeness of visual steps. as a result, evaluation of multimodal plans still relies heavily on human judgment @xcite , which is labor-intensive, difficult to scale, and prone to inconsistency across annotators. to address these challenges, we introduce multimodal planner (mmplanner), a zero-shot framework for generating consistent multimodal plans that capture both explicit and implicit obthe contributions of our work are: • we introduce mmplanner, a zero-shot mpp method that generates coherent multimodal plans reflecting object state changes in visual plan sequences. we empirically validate mm-planner on two benchmark datasets, achieving improvements of up to 6.8% in textual plan quality, 11.9% in cross-modal alignment, and 26.7% in visual step ordering accuracy. • mmplanner incorporates background context from previous task steps through an object state reasoning chain-of-thought (osr-cot) prompting strategy, enabling explicit modeling of evolving object states across steps, and reducing inference time by ∼46.25% compared to sota mpp baselines. • we propose a reference-free evaluation framework to assess planning accuracy, crossmodal alignment, temporal coherence, and visual informativeness of generated plans, achieving stronger correlation with human judgments than prior cross-modal metrics (ρ = 0.57 vs. 0.37 for clipscore) while reducing step-level evaluation time by ∼66% (90s → 30s). for textual planning, our automated evaluation requires only ∼0.7s per plan compared to ∼5mins for human assessment, enabling large-scale evaluation. procedural planning methods fall into two categories: selectionbased and generation-based. selection-based approaches @xcite @xcite @xcite rely on predefined candidates, limiting generalization to unseen scenarios. generation-based methods, powered by llms @xcite , focus on generating textual plans @xcite . recently, tip @xcite generates multimodal plans by prompting an llm and image generation model twice sequentially, increasing the inference time. in summary, existing methods often fail to accurately reflect changes in object states throughout the steps. although recent work explores tracking state changes in videos @xcite and leverages large-scale datasets @xcite , these approaches typically assume access to full video sequences @xcite and struggle to maintain state consistency across frames @xcite , 2025). in addition, statler (yoneda et al., 2024) focuses on maintaining object states for embodied robotic reasoning with low-level, fine-grained actions in closed environments. in contrast, our work generates visual plans from scratch in a zero-shot setting, given only a goal, and emphasizes highlevel, interpretable steps aimed at human-centric multimodal procedural planning. beyond procedural planning, prior work has explored cross-modal coherence and multimodal discourse, focusing on temporal and narrative consistency across modalities @xcite . such studies show that modeling discourse relations and coherence cues can strengthen alignment and narrative flow. our work is complementary, and future extensions could incorporate discourse-aware prompting to further enhance temporal and narrative consistency in visual plans. ## method the goal of mpp is to generate a sequence of steps, each with a textual and visual component, that to-gether achieve a high-level task goal. given a highlevel goal instruction g t that outlines the task, the objective is to generate a sequence of low-level steps @xmath0, s 2 , . . . , s n }, where n denotes the number of steps. each step s i comprises a textual description t i and a corresponding step image v i , denoted as (t i , v i ). the step-wise textual and visual plans are denoted as @xmath1, t 2 , . . . , t n } and @xmath2, v 2 , . . . , v n }, respectively. ## mpp evaluation llms have demonstrated strong performance on complex reasoning tasks, motivating their use as automated evaluators that often surpass human workers in efficiency @xcite . building on this insight, we introduce ca-score, which measures the alignment between each textual step and its corresponding visual depiction, and t-planscore, which evaluates whether a generated textual plan is both task-consistent and logically coherent. in practice, t-planscore completes evaluation in an average of 0.7 seconds per task, compared to approximately 5 minutes for human annotators. for complex tasks (e.g., "how to weave a rag rug?"), human evaluation takes a longer time due to domain-specific knowledge requirements. similarly, for ca-score, each step-level evaluation completes in about 1 minute, representing a 66% reduction in time compared to human assessment, which averages 3 minutes. these results highlight the efficiency and practicality of llm-based evaluation for multimodal planning tasks. t-planscore. prior approaches to textual plan evaluation often rely on semantic similarity to reference plans @xcite , which may not fully capture planning accuracy or temporal coherence. we propose t-planscore, a reference-free method that prompts a language model @xcite to assess how well a generated plan aligns with the overall task goal. the prompt guides the model to consider both procedural correctness and logical step ordering. empirically, we find that higher t-planscore values correspond to more coherent plans that accurately reflect the intended procedure. ca-score. cross-modal alignment between textual and visual plans is often evaluated using similarity-based metrics such as clipscore @xcite or sentence-bert @xcite , computed between generated visual captions and reference textual plans @xcite . while effective for coarse semantic matching, these methods may fail to capture finer-grained alignment, such as implicit object state changes not explicitly described in the text. to address this, we leverage multimodal language models, which have shown strong cross-modal reasoning capabilities @xcite . inspired by recent work on text-to-image evaluation @xcite , we employ a vlm with chain-of-thought (cot) to assess cross-modal alignment at the step level. specifically, we prompt the model to describe the contents of a generated image and compare the resulting description with the corresponding step text, evaluating alignment in terms of depicted actions and object states. we refer to the resulting score as ca-score. vs-ordering. to assess the informativeness and temporal coherence of visual plans, we additionally introduce a visual step reordering task. given an unordered sequence of visual steps, the objective is to recover their correct temporal order. this task provides a direct measure of how well the visual outputs capture procedural structure. a related task was proposed by @xcite , who introduced multimodal instruction sequencing involving both textual and visual inputs. however, we find that including text in the sequence biases evaluation toward the textual modality, limiting sensitivity to visual quality (appendix d). to address this, our formulation focuses solely on the visual modality, enabling the evaluation of visual procedural understanding independent of textual cues. we adopt a pretrained visual sequencing model @xcite that consists of a clip image encoder and an order decoder based on the berson framework @xcite . the vision encoder is trained with self-supervised objectives such as masked language modeling, patch-based image swapping, and sequential masked region modeling. for each visual plan, we randomly shuffle the step order and use the model to predict the correct sequence. figure 1 illustrates an example. ## experiments we evaluate mmplanner on the recipeplan and wikiplan @xcite datasets. recipeplan consists of 1,000 recipe tasks adapted from recipeqa @xcite , where each task includes a goal (taken from the recipe title) and a sequence of text-image pairs representing procedural steps. wikiplan contains 1,000 tasks sourced from wikihow articles, where the article title serves as the goal, the main body text forms the textual plan, and accompanying images comprise the visual plan. we conduct experiments comparing mmplanner with tip @xcite , a dual prompting mpp method that integrates procedural knowledge from llms and t2i models by prompting both twice during inference. we also compare against two baselines from tip: ## broader impacts this work aims to advance the deployment of multimodal generative models, such as llms and textto-image models, for real-world, step-by-step task assistance. our goal is to make task-driven assistive technology more practical and accessible, particularly for users who benefit from visual guidance. we acknowledge the limitations of generative models, especially their susceptibility to hallucination and misinformation. to address this, osr-cot encourages grounded reasoning by decomposing tasks into smaller, verifiable steps, reducing the risk of unsupported outputs. future directions could focus on integrating external knowledge verification to further enhance the reliability and trustworthiness of ai-generated multimodal plans. ## limitations mmplanner leverages llms and vlms for multimodal plan generation and evaluation via t-planscore and ca-score. however, hallucinations remain a known limitation of llms @xcite . this issue is most evident before applying osr-cot (section 3.3). for example, the generated description of the task "how to make cheese garlic pull-apart bread?" (figure 7 ) for step 5 is "a bowl of dough mixture is forming. flour and butter can be seen in the background" without osr-cot, showing that the llm introduces unrelated details about flour and butter from earlier steps. in contrast, osr-cot yields "the milk mixture being slowly stirred into the dry ingredients", more accurately aligning with the step's intent and focusing on what is relevant (ingredients, processes, etc) for that specific step. this shows how osr-cot reduces hallucinations and improves step relevance. osr-cot improves object state reasoning, but mmplanner does not explicitly enforce visual consistency for peripheral elements such as cookingware shape or material. for instance, in figure 7 , the bowl depicted in steps 2, 3, and 5 varies in appearance. mmplanner inherits limitations from stable diffusion, particularly its inability to render concepts absent from its training data. for example, given the step "beat the egg with a fork" from the task "how to make an omelet", the model fails to generate an accurate depiction of a "beaten egg". addressing such inconsistencies remains an avenue for future work. finally, while t-planscore shows consistent monotonic trends under plan degradations, its absolute calibration can be imperfect. in some cases, low-quality plans (e.g.with many deleted steps) still receive high scores. this reflects a broader limitation of llm-based evaluators, where prompt adherence and human alignment are not guaranteed. future work could improve calibration through human feedback, refined prompts, or preferencebased fine-tuning. as multimodal llms continue to improve, they offer potential for better mpp evaluation frameworks, but further work is needed to refine both generation and evaluation. ## conclusion we present mmplanner, a zero-shot multimodal procedural planning method using osr-cot prompting to capture explicit and implicit object state changes. to evaluate generated plans, we propose an automatic evaluation that assesses planning accuracy, cross-modal alignment, and temporal coherence. experiments show mmplanner generates accurate and coherent multimodal plans.