E asy G en: Easing Multimodal Generation with B i D iffuser and LLM s (2024)

Abstract: We present EasyGen, an efficient model designed to enhance multimodal understanding and generation by harnessing the capabilities of diffusion models and large language models (LLMs). Unlike existing multimodal models that predominately depend on encoders like CLIP or ImageBind and need ample amounts of training data to bridge modalities, EasyGen leverages BiDiffuser, a bidirectional conditional diffusion model, to foster more efficient modality interactions. EasyGen achieves text generation by training a projection layer linking BiDiffuser and an LLM, and facilities image generation by training an adapter to align the LLM’s text space with the BiDiffuser’s image space. Comprehensive quantitative and qualitative experiments show that EasyGen excels in data-efficient training, high-quality image generation, and extendibility, effectively addressing the challenges in multimodal generation.

## introduction in recent years, remarkable progress has been made in the field of artificial intelligence generated content (aigc), notably in technologies like large language models (llms) @xcite @xcite @xcite ) for text generation and diffusion models @xcite @xcite for visual generation. these breakthroughs have paved the way for the development of multimodal large language models (mllms), sparking a recent trend of incorporating extra visual modules into llms. collaborative models, such as visual chatgpt @xcite and mm-react @xcite , strategically use externally pre-trained tools to translate visual information into text descriptions and feed * co-second author. the data into llms. however, they are exclusively dependent on pre-trained tools for inference. contrarily, end-to-end trained models including the blip series @xcite , llava series @xcite , minigpt-4 @xcite , and mplug-owl @xcite focus on mapping image information to the text space of llms, enabling llms to comprehend visual inputs. existing end-to-end models are also not without limitations. first, most of these multimodal models rely on either clip @xcite or image-bind @xcite as their image encoder. while these encoders excel in learning unified representations that encompass both text and images, they face challenges when it comes to transforming between different modalities. this predicament makes current vision-language models relying heavily on sizable data sets to align clip/bindencoded images with the language model, due to the disparity between different modalities. moreover, the majority of previous multimodal models have concentrated on comprehending multimodal content and lacked the capability to generate multimodal responses, such as content beyond text. several concurrent works, such as emu @xcite and next-gpt @xcite , have utilized diffusion models for multimodal generation. typically, these methods involve training a projection layer to align the output embedding space of the llm with the input embedding space of the diffusion model (encoded by clip's text encoder) using an mse loss. however, this approach may lead to the underutilization of the semantic understanding and reasoning capabilities of the llm, and may introduce information loss in the alignment process, ultimately leading to lower image generation quality compared to the original diffusion model, as elaborated in sec. 5.6 and tab. 6. in this work, we propose easygen, an end-toend model that facilitates multimodal generation with a single bidirectional conditional diffusion 1 1351 could you please provide an introduction to this model in the picture? what other types of green energy irrigation systems are feasible? i'm really intrigued by wind-powered irrigation systems. could you please provide a brief introduction and show me some pictures related to it? the image features a solar irrigation system working model that is displayed on a table. it is made of various materials, such as plastic, and features various components, including solar panels, water tanks, a pump, piping, valves, hoses, faucets, plants, soil, dirt, pebbles, bottles, cups, spoons, straws, screws and nuts, among others. there are several other types of green energy irrigation systems apart from solar power. some of them include wind-powered irrigation system, hydro-powered irrigation system, tidal-powered irrigation system … a wind-powered irrigation system is a type of green energy irrigation system that utilizes wind energy to power the irrigation process. it involves the use of wind turbines, typically mounted on towers, which capture the kinetic energy of the wind and convert it into electrical energy. the generated electricity can be stored in batteries … here are some possible pictures: model and llms, as illustrated in figure 2 . the diffusion model, called bidiffuser, is obtained by fine-tuning the unidiffuser @xcite , with a specific focus on targeted image-to-text and text-to-image tasks. this fine-tuning addresses uni-diffuser's limitation of attempting to fit all conditional distributions, including those based on noisy inputs, into a single model, which reduces its effectiveness on specific tasks like conditional generation from noise-free inputs. bidiffuser plays a pivotal role for both text and image generation. in easygen, text generation is achieved by training a projection layer that connects bidiffuser and an llm, while image generation is facilitated by training an adapter that infuses the text representation of the llm into bidiffuser. figure 1 showcases easygen's ability to handle multimodal inputs and generate appropriate multimodal responses. easygen holds three significant advantages that address the challenges in multimodal generation: first, easygen offers competitive performance in a data-efficient way compared to cutting-edge models, as shown in tab. 3 (sec. 5.5). this is due to bidiffuser's ability to simplify the alignment of its embedding space with an llm, which allows for efficient training with less data for image-to-text tasks such as image captioning and vqa. second, easygen exhibits superior image generation quality, surpassing other end-to-end mllms, as shown in tab. 6 (sec. 5.6). this is attributed to the adapter's design (sec. 4.2), which aligns the llm's text space with the diffusion model's image space, thereby utilizing the llm's semantic understanding and reasoning capabilities. in contrast, the projection layers in other mllms like next-gpt only align the llm's text space with the diffusion model's text space and are not trained by the image denoising objective. third, easygen can be readily adapted to manage complex vision-language tasks by incorporating more advanced visual encoders or by integrating bidiffuser into contemporary sophisticated multimodal llms like llava to enhance performance, as shown in tab. 5 (sec. 5.7). ## basics of diffusion models unconditional generation. given a data sample taken from a real data distribution x 0 ∼ q(x 0 ), diffusion models @xcite first destruct the data by constructing a markov forward process and gradually injecting noise to the data: conditional generation. for conditional generation, a paired data (x 0 , y 0 ) is given, and the aim is to model the conditional data distribution q(x 0 |y 0 ), where y 0 can be image class or text prompt. conditional generation includes classifier guidance @xcite and classifier-free guidance @xcite . classifier guidance requires training an extra classifier on noisy data at inference time to improve sample quality. for classifier-free guidance, no classifier needs to be trained. the denosing network ϵ θ (x t |y 0 ) simply conditions on the information encoded in y 0 . at inference time, with a guidance scale s, the modified score estimate is further in the direction of ϵ θ (x t |y 0 ) and away from the unconditional model ϵ θ (x t |∅) (∅ is a null token): ## proposed model: easygen we propose easygen, a model capable of processing multimodal inputs and generating multimodal outputs. it achieves easy multimodal generation by leveraging a bidirectional conditional diffusion model to effectively bridge the gap between different modalities and an llm to comprehend multimodal tasks and produce textual responses containing cues for multimodal message creation. in the subsequent section, we outline the multimodal generation process of easygen. ## conclusion and future work we have introduced easygen, a model that facilitates multimodal understanding and generation. compared to existing models, easygen offers a more efficient solution by employing bidiffuser, a bidirectional diffusion model. this allows for more effective modal interactions, handling both imageto-text and text-to-image generations by the fusion of bidiffuser and llms. additionally, easygen can be easily integrated into existing advanced multimodal llms to further boost their performance. in the future, we will explore adapting easygen to perform a broader range of multimodal tasks, including subject-driven image generation, image editing, and controlled generation.