Show and Guide: Instructional-Plan Grounded Vision and Language Model (2024)

Abstract: Guiding users through complex procedural plans is an inherently multimodal task in which having visually illustrated plan steps is crucial to deliver an effective plan guidance. However, existing works on plan-following language models (LMs) often are not capable of multimodal input and output. In this work, we present MM-PlanLLM, the first multimodal LLM designed to assist users in executing instructional tasks by leveraging both textual plans and visual information. Specifically, we bring cross-modality through two key tasks: Conversational Video Moment Retrieval, where the model retrieves relevant step-video segments based on user queries, and Visually-Informed Step Generation, where the model generates the next step in a plan, conditioned on an image of the user’s current progress. MM-PlanLLM is trained using a novel multitask-multistage approach, designed to gradually expose the model to multimodal instructional-plans semantic layers, achieving strong performance on both multimodal and textual dialogue in a plan-grounded setting. Furthermore, we show that the model delivers cross-modal temporal and plan-structure representations aligned between textual plan steps and instructional video moments.

## introduction the research of large language models (llms) in assisting users with complex procedural plans, such as cooking or diy projects, presents an exciting new frontier in nlp research @xcite . however, while llms can excel at textbased conversational interactions @xcite , procedural plans are inherently multimodal, often accompanied by illustrative images or instructional videos (sener and yao, 2019; marin et al., 1 the model, code, and non-personal data will be made publicly available at https://github.com/dmgcsilva/ mmplanllm i lost track of the recipe. this is how the sauce looks like. what should i do next? woah you jumped to step 5: when the sauce has thickened, add the fried chicken. toss to coat. step 1 season chicken ... step 2 fry until ... let's start the recipe let's get cooking! step 1: season chicken with salt and pepper. coat with cornstarch, dip in beaten egg. step 3 in a large pan... step n ... ## related work in recent years, with the release of large open source foundational models such as opt @xcite , llama @xcite and others @xcite @xcite , the field of large language models (llms) for conversational settings has received significant attention. due to this, the contributions have been diverse, with work focusing on improving training data @xcite , scaling model size @xcite , and adopting a mixture of experts (moe) architecture @xcite . the applications of these models are varied, such as instruction following @xcite @xcite , conversational dialogue @xcite , and other task-specific applications @xcite . researching models capable of understanding multimodal input has also been a topic of great interest. a common approach has been the usage of pretrained llms and visual encoders to achieve efficient and effective large vision-language models (lvlms) with limited resources; however, the way these models interface has been varied. some approaches, such as the llava models @xcite and @xcite have found that linear projections are enough. others deploy larger "interpretation" modules such as the q-fromer in blip @xcite , the visual abstractor in mplug-owl @xcite , or the perceiver (jaegle et al., 2021) employed in flamingo @xcite . another interesting approach is the modification of the internal transformer @xcite attention blocks such as the visual expert in cogvlm @xcite and the modality-adaptive module in mplug-owl2 @xcite . some work has also been done on training multimodal models from scratch such as pali @xcite , and large world model @xcite . video moment retrieval (vmr) is the task of, given a video and textual prompt that describes an action or event that occurs in a video, retrieving a video clip from within said video that best matches the provided textual prompt. proposaldriven approaches focus on identifying candidate proposals and then ranking them to find the most relevant one @xcite @xcite . in contrast, others opt for a proposal-free approach that predicts the target moment directly from the video-prompt feature mappings @xcite @xcite often relying on cross-modal attention modules or on learnable query embeddings such as eatr @xcite or mh-detr @xcite . a common problem in vmr is the need to do extensive and expensive temporal an-notations, an alternative is point-level vmr where the annotation is a single frame point @xcite or a small segment @xcite . recently, several approaches have been adopting a detection transformers @xcite , as it does away with the need for many hand-designed components, and tackling the problem as a direct set prediction @xcite @xcite . ## multimodal plan-grounded lm in this section, we present the main elements of this work: we start by formalizing the problem, then we describe mm-planllm, its architecture, and the multi-stage training process used. we end by detailing how the supporting synthetic training dataset is generated. ## conclusion we propose mm-planllm, a multimodal architecture that enables multimodal comprehension for lms in plan-grounded conversational settings. we follow a multistage training paradigm, coupled with task-specific synthetic data creation, that enables the model to slowly acquire the necessary abilities to understand multimodal input and generate multimodal outputs. experimental results demonstrates that mm-planllm outperforms task-specific baselines, showcasing minimal performance loss in text-only dialogues, while being capable of aligning textual steps with video moments and user images with the plan steps. the ablation study further highlights the effectiveness of the multi-phase training methodology and the value of incorporating temporal information.