UNIMO - G : Unified Image Generation through Multimodal Conditional Diffusion (2024)

Abstract: Existing text-to-image diffusion models primarily generate images from text prompts. However, the inherent conciseness of textual descriptions poses challenges in faithfully synthesizing images with intricate details, such as specific entities or scenes. This paper presents UNIMO-G, a simple multimodal conditional diffusion framework that operates on multimodal prompts with interleaved textual and visual inputs, which demonstrates a unified ability for both text-driven and subject-driven image generation. UNIMO-G comprises two core components: a Multimodal Large Language Model (MLLM) for encoding multimodal prompts, and a conditional denoising diffusion network for generating images based on the encoded multimodal input. We leverage a two-stage training strategy to effectively train the framework: firstly pre-training on large-scale text-image pairs to develop conditional image generation capabilities, and then instruction tuning with multimodal prompts to achieve unified image generation proficiency. A well-designed data processing pipeline involving language grounding and image segmentation is employed to construct multi-modal prompts. UNIMO-G excels in both text-to-image generation and zero-shot subject-driven synthesis, and is notably effective in generating high-fidelity images from complex multimodal prompts involving multiple image entities.

## introduction recent advancements in text-to-image (t2i) diffusion models have yielded impressive results in the generation of high-fidelity images from textual descriptions. various methods, including dall-es @xcite , imagen @xcite , stable diffusion @xcite , and mm-dit @xcite , have been successful in producing photorealistic and contextually relevant images based on textual prompts. nevertheless, a fundamental challenge persists due to the inherent brevity of textual descriptions, particularly when intricate details, specific entities, or nuanced scenes are involved. thus, faithfully generating images from general visionlanguage (vl) inputs is essential to improve the controllability of image generation. numerous studies have explored vl-to-image generation techniques. methods such as dream-booth @xcite , imagic @xcite , suti @xcite and blip-diffusion @xcite emphasize subjectdriven generation, where they use both subject images and textual descriptions as inputs to recontextualize the subject in a newly described setting. they either fine-tune specific models for a given subject or employ pre-trained subject representations. however, their specific training design and input templates hinder their scalability, especially in complex scenarios with multiple entities. additionally, studies like fastcomposer @xcite and subject-diffusion @xcite focus on multiple-entity image generation, integrating image embeddings from image encoders with the standard text conditioning in pre-trained diffusion models. nevertheless, these approaches lack the capacity to efficiently process generalized visionlanguage inputs that comprise a mix of textual and visual information in free forms. in this paper, we propose unimo-g, a simple multimodal conditional diffusion framework that operates on multimodal prompts comprising freeform interleaved vision-language inputs. unlike traditional text-only prompts, multimodal prompts encompass various combinations of image entities and textual elements, as demonstrated in figure 1 . unimo-g is designed to faithfully reproduce all image entities, render textual content, and follow the instructions in multimodal prompts. specifically, we leverage the perception capabilities of multimodal large language models (mllms) to encode multimodal prompts into a unified visionlanguage semantic space. subsequently, a conditional diffusion network generates images from these encoded representations. to train unimo-g efficiently, we implement a two-phase strategy. initially, the model undergoes pre-training on a large-scale dataset of text-image pairs, enhancing its proficiency in conditional image generation. this is followed by a phase of instruction tuning with multimodal prompts, learns to generate images that align with the detailed specifications provided in these prompts. a carefully designed data processing pipeline, incorporating language grounding and image segmentation, is employed to construct these multimodal prompts. this approach enables unimo-g to harness rich features from the mllm encoder to generate images faithfully reproducing the contents across various contexts. unimo-g exhibits a comprehensive capability for controllable image generation, excelling not only in text-to-image synthesis but also in zero-shot subject-driven generation. it adeptly produces highfidelity images from multimodal prompts, even those containing multiple image entities. to assess its performance, we conducted evaluations in both text-to-image and subject-driven generation contexts using the ms-coco @xcite and dreambench @xcite datasets, respectively. the results consistently highlight unimo-g's superior performance in these scenarios. additionally, recognizing dreambench's focus on single-subject generation, we introduce multi-bench, a new benchmark featuring images with multiple entities. the evaluation on multibench confirms unimo-g's effectiveness in zero-shot multi-entity subject-driven generation. in summary, our contributions in this work can be summarized as follows: ## method the architecture of unimo-g, as depicted in figure 2 , primarily comprises two key components: a multimodal large language model (mllm) responsible for encoding multimodal prompts and a conditional denoising diffusion network for image generation based on the encoded representations. in our study, we employed a pre-trained chinese mllm, structurally similar to @xcite . it consists of a vision encoder with a pretrained vit and q-former, a single linear projection layer, and a transformer-based llm, underwent pre-training on a vast dataset comprising billions of image-text pairs. details can refer to appendix a. this extensive training process equips the model with a robust capability to process and interpret complex multimodal data. the training of unimo-g is conducted in a twostage process: it is worth noting that during both stages of training, only the u-net component is actively trained, with the mllm parameters frozen. this strategy ensures that unimo-g effectively learns to generate images while retaining the perception knowledge encoded in the pre-trained mllm. ## experiments in this section, we first introduce the implementation details and settings of experiments. then we present the evaluation results in both text-driven and subject-driven scenarios. last, we further analyze the results with quantitative ablation studies. ## conclusion this paper presents unimo-g, a simple multimodal conditional diffusion framework designed to process multimodal prompts that interleave text and visual inputs. it demonstrates exceptional proficiency in text-to-image generation and zero-shot subject-driven synthesis, and is particularly adept at producing high-fidelity images from intricate multi-modal prompts with multiple image entities. in comparison to standard text-conditional diffusion models, unimo-g significantly enhances visual controllability in image generation. thanks to our two-stage training strategy, unimo-g also outperforms existing vl-to-image models, especially on the ability to follow complex multimodal instructions. overall, unimo-g showcases the potential for more nuanced and controlled image generation processes.