Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework (2024)

Abstract: Text-to-image (T2I) diffusion models are popular for introducing image manipulation methods, such as editing, image fusion, inpainting, etc. At the same time, image-to-video (I2V) and text-to-video (T2V) models are also built on top of T2I models. We present Kandinsky 3, a novel T2I model based on latent diffusion, achieving a high level of quality and photorealism. The key feature of the new architecture is the simplicity and efficiency of its adaptation for many types of generation tasks. We extend the base T2I model for various applications and create a multifunctional generation system that includes text-guided inpainting/outpainting, image fusion, text-image fusion, image variations generation, I2V and T2V generation. We also present a distilled version of the T2I model, evaluating inference in 4 steps of the reverse process without reducing image quality and 3 times faster than the base model. We deployed a user-friendly demo system in which all the features can be tested in the public domain. Additionally, we released the source code and checkpoints for the Kandinsky 3 and extended models. Human evaluations show that Kandinsky 3 demonstrates one of the highest quality scores among open source generation systems.

## introduction text-to-image (t2i) models play a dominant role in generative computer vision technologies, providing high quality results and language understanding along with near real-time inference speed. this led to their popularity and accessibility for many applications through graphic ai editors and webplatforms, including chatbots. at the same time, t2i models are also used outside the image domain, e.g. as a backbone for text-to-video (t2v) generation models. similar to trends in natural language processing (nlp) @xcite , in generative computer vision there is increasing interest in systems * work done during employment at sber ai. contributions of this work are as follows: ## related works to date, diffusion models @xcite ) are de facto standard in the text-to-image generation task @xcite @xcite . some models, such as stable diffusion @xcite , are publicly available and widespread in the research community @xcite . from the user's point of view, the most popular models are those that offer a high level of generation quality and an interaction web-system via api @xcite @xcite . the development of diffusion models has enabled the design of a wide range of image manipulation techniques, such as editing @xcite @xcite , in/outpainting @xcite , style transfer @xcite , and image variations @xcite . these approaches are of particular interest to the community and are also being implemented in user interaction systems @xcite @xcite . t2i models have extensive knowledge of the relationship between visual and textual concepts. this allows them to be used as a backbone for models that expand the scope of generative ai to i2v @xcite , t2v @xcite @xcite ), text-to-3d generation @xcite @xcite , etc. for a long time, the key disadvantage of diffusion models remained the speed of inference, which requires a large number of steps in the reverse diffusion process. recently these limitations have been significantly overcome by the speed-up and distillation methods for diffusion models @xcite . this increases the prospects for creating multifunctional generative frameworks based on diffusion models and their use through online applications and web editors. ## demo system kandinsky 3 model underlies a comprehensive user interaction system with free access. the system contains different modes for image and video generation, and for image editing. here we describe the functionality and capabilities of our two key user interaction resources -telegram bot and fu-sionbrain website. fusionbrain is a web-editor that supports loading images from the user, and saving generated images and videos (figure 1 ). the system accepts text prompts in russian, english and other languages. it is also allowed to use emoji in the text description. the maximum prompt size is 1000 characters 5 . in terms of generation tasks, this web editor provides the following options: for more details see section 5.6. telegram bot provides all the same options as the fusionbrain website, except in/outpainting. it also has a number of additional features: ## text-to-image model architecture overview. kandinsky 3 is a latent diffusion model, which includes a text encoder for processing a prompt from the user, a u-net-like network @xcite for predicting noise, and a decoder for image reconstruction from the generated latent (figure 2 ). for the text encoder, we use the encoder of the flan-ul2 20b model @xcite , which contains 8.6 billion parameters. as an image decoder, we use a decoder from sber-movqgan @xcite . the text encoder and image decoder were frozen during the u-net training. the whole model contains 11.9 billion parameters ( diffusion u-net. to decide between large transformer-based models @xcite @xcite and convolutional architectures, both of which have demonstrated success in computer vision tasks, we conducted more than 500 experiments and noted the following key insights: thus, we settled on the resnet-50 block as the main block for our u-net. using bottlenecks in residual blocks made it possible to double the number of convolutional layers, while maintaining approximately the same number of parameters as without bottlenecks. at the same time, the depth of our new architecture has increased by 1.5 times compared to kandinsky 2 @xcite . at the higher levels of the upscale and downsample parts, we placed our implementation of convolutional residual biggan-deep blocks. at lower resolutions, the architecture includes selfattention and cross-attention layers. the complete scheme of our u-net architecture and a description of our residual biggan-deep blocks can be found in appendix a. ## data our dataset for the t2i model training consists of popular open-source datasets and our internal @xcite . we divided all the data into two categories. we used the first at the initial stages of lowresolution pretraining and the second for mixed and high-resolution fine-tuning. the first category includes open text-image datasets such as laion-5b @xcite and coyo-700m @xcite , and data that we collected from the internet. the second category contains the same datasets but with stricter filters, especially for the image aesthetics quality. for training details, please refer to the appendix b. ## human evaluation we found that when a high level of generation quality is achieved, fid values do not correlate well with visually noticeable improvements. for the previous version of kandinsky model @xcite we reported fid, but in this work we focused on human evaluation results for model comparison. we conducted side-by-side (sbs) comparisons between the refined version of kandinsky 3 with beautification and other competing models: midjourney 5.2 (midjourney, 2022), sdxl @xcite and dall-e 3 @xcite . for sbs we used generations by prompts from drawbench dataset @xcite . we also compared our base t2i model with a distilled and refined version, as well as a version with prompt beautification. each of the 12 people chose the best image from the displayed image pairs based on two criteria separately: 1) alignment between image content and text prompt, and 2) visual quality of the image. each pair was shown to 5 different people out of 12. the group of estimators included people with various educational backgrounds, such as an economist, engineer, manager, philologist, sociologist, programmer, financier, lawyer, historian, journalist, psychologist, and editor. the participants ranged in age from 19 to 45. we also compared our base t2i model with a distilled version. each of the 12 people chose the best image according to alignment between image content and text prompt, and visual quality of the image. according to the results for all categories (figure 5 ), prompt beautification has significantly improved the visual quality of the images. distillation led to an increase in visual quality, but a deterioration in text comprehension. using a distilled model as a refiner improves visual quality, while ensuring text comprehension is comparable to the base model. the low percentage values for text alignment here are due to the fact that people often chose both models. kandinsky 3 demonstrates competitive results for well-known sota models, noting the complete openness of our solution, including code, checkpoints, implementation details, and the ease of adapting our model for various kinds of generative tasks. ## conclusion we presented kandinsky 3, a new open source text-to-image generative model. based on this model, we presented our multifunctional generative framework that allows users to solve a variety of generative tasks, including inpainting, image editing, and video generation. we also presented and deployed an accelerated distilled version of our model, which, when used as a refiner for the base t2i model, produces sota results among opensource solutions, according to human evaluation quality. we have implemented our framework on several platforms, including fusionbrain website and telegram bot. we have made the code and pretrained weights available on hugging face under a permissive license with the goal of making broad contributions to open generative ai development and research.