VIMI : Grounding Video Generation through Multi-modal Instruction (2024)

Abstract: Existing text-to-video diffusion models rely solely on text-only encoders for their pretraining. This limitation stems from the absence of large-scale multimodal prompt video datasets, resulting in a lack of visual grounding and restricting their versatility and application in multimodal integration. To address this, we construct a large-scale multimodal prompt dataset by employing retrieval methods to pair in-context examples with the given text prompts and then utilize a two-stage training strategy to enable diverse video generation tasks within a model. In the first stage, we propose a multimodal conditional video generation framework for pretraining on these augmented datasets, establishing a foundational model for grounded video generation. Secondly, we fine-tune the model from the first stage on various video generation tasks, incorporating multimodal instructions. This process further refines the model’s ability to handle diverse inputs and tasks, ensuring seamless integration of multimodal information. After this two-stage training process, VIMI demonstrates multimodal understanding capabilities, producing contextually rich and personalized videos grounded in the provided inputs, as shown in Figure1. Compared to previous subject-driven video generation methods, our generator can synthesize consistent and temporally coherent videos with large motion while retaining the semantic control. Our generator also achieves state-of-the-art text-to-video generation results on UCF101 benchmark.

## introduction recent advancements in video diffusion models have led to significant successes across various video creation tasks @xcite @xcite @xcite @xcite . these models have demonstrated impressive capabilities in generating high-quality videos from textual prompts @xcite @xcite @xcite @xcite @xcite . however, the majority of these models rely solely on text-only encoders for their diffusion-based pretraining. this limitation stems from the absence of large-scale multimodal prompt datasets, which results in a lack of visual grounding during the pretraining stage. consequently, current models struggle to incorporate visual input effectively, restricting their versatility and application in scenarios that demand multi-modal integration. to effectively incorporate visual input into pretrained text-to-video models, standalone image encoders are often employed to process image prompts @xcite @xcite . the visual embeddings generated by these encoders are then injected into the diffusion model, enabling it to handle multimodal applications. however, this approach necessitates customized model designs, leading to fragmented solutions that cannot support various tasks in a unified manner. as a result, the models lack the flexibility and generalization needed to seamlessly integrate different modalities for diverse video generation tasks. recently, generative pretrained multimodal language models have demonstrated robust multimodal in-context learning capabilities, showcasing their ability to process and integrate various types of input data effectively @xcite @xcite . inspired by this success, we introduce a multimodal instruction pretraining framework vimi for grounded video generation. this novel framework aims to leverage the strengths of multimodal models, enhancing the ability to generate videos that are coherently grounded in both textual and visual inputs. specifically, the training of vimi consists of two stages: (1) retrieved augmented pretraining; ## preliminary 2.1 text-to-video pretraining we base our work on the diffusion framework proposed by @xcite , which adapts the edm @xcite ) diffusion framework to high resolution video generation. in edm, the forward diffusion process is characterized by a variance-exploding mechanism p(x σ |x) ∼ n (x, σ 2 i), where noise σ is gradually added to the data, causing the variance to increase over time, and x σ represents the data at the current noise level. the reverse process is modeled by learnable denoiser function denoted as d θ , which is trained using a denoising objective formulated as: menapace et al. ( 2024 )), the model can train optimally on high resolution videos. we employ a second order runge-kutta sampler to produce video samples. ## method we aim to generalize the video generation pretraining to the multimodal setting. figure 2 shows the overview of our framework. sec. 3.1 introduces how we construct a large-scale multimodal inputvideo dataset by employing retrieval methods to pair in-context examples with given text prompts. sec. 3.2 presents a multimodal conditional video generation framework for pretraining on these augmented datasets, establishing a foundational model for grounded video generation. sec. 3.3 introduces the instruction finetuning stage on three video generation tasks, incorporating multimodal instructions. ## conclusion in this work, we first construct a multimodal prompt dataset for video pretraining using retrieval methods. we then propose a two-stage training strategy to enable diverse video tasks within the same model. for the first stage, we introduce a multimodal conditional video generation framework for pretraining on these augmented datasets, establishing a foundational model for grounded video generation. in the second stage, we fine-tune the model from the first stage on three video generation tasks, incorporating multimodal instructions. our experiments demonstrate the effectiveness of retrieval-augmented pretraining and the use of multimodal instruction tuning. we hope this approach opens up new opportunities for video pretraining, such as building large-scale multimodal datasets for pretraining, utilizing stronger multimodal large language models for encoding, and employing instruction tuning for diverse video tasks.