Kandinsky: An Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion (2023)

Abstract: Text-to-image generation is a significant domain in modern computer vision and achieved substantial improvements through the evolution of generative architectures. Among these, diffusion-based models demonstrated essential quality enhancements. These models generally split into two categories: pixel-level and latent-level approaches. We present Kandinsky – a novel exploration of latent diffusion architecture, combining the principles of image prior models with latent diffusion techniques. The image prior model, is trained separately to map CLIP text and image embeddings. Another distinct feature of the proposed model is the modified MoVQ implementation, which serves as the image autoencoder component. Overall the designed model contains 3.3B parameters. We also deployed a user-friendly demo system that supports diverse generative modes such as text-to-image generation, image fusion, text and image fusion, image variations generation and text-guided inpainting/outpainting. Additionally we released the source code and checkpoints for Kandinsky models. Experimental evaluations demonstrate FID score of 8.03 on the COCO-30K dataset, marking our model as the top open source performer in terms of measurable image generation quality.

## introduction in quite a short period of time, generative abilities of text-to-image models have improved substantially, providing users with photorealistic quality, near real-time inference speed, a great number of applications and features, including simple easyto-use web-based platforms and sophisticated ai graphics editors. this paper presents our unique investigation of latent diffusion architecture design, offering a fresh and innovative perspective on this dynamic field of study. first, we describe the new architecture of kandinsky and its details. the demo system with implemented features of the model is also described. second, we show the experiments, carried out in terms of image generation quality and come up with the highest fid score among existing open-source models. additionally, we present the rigorous ablation study of prior setups that we conducted, enabling us to carefully analyze and evaluate various configurations to arrive at the most effective and refined model design. our contributions are as follows: ## related work early text-to-image generative models, such as dall-e @xcite and cogview @xcite , or later parti @xcite employed autoregressive approaches but often suffered from significant content-level artifacts. this led to the development of a new breed of models that utilized the diffusion process to enhance image quality. diffusion-based models, such as dall-e 2 @xcite , imagen @xcite , and stable diffusion foot_5 , have since become cornerstones in this domain. these models are typically divided into pixel-level @xcite and latent-level @xcite approaches. this surge of interest has led to the design of innovative approaches and architectures, paving the way for numerous applications based on opensource generative models, such as dreambooth @xcite and dreampose @xcite . these applications exploit image generation techniques to offer remarkable features, further fueling the popularity and the rapid development of diffusion-based image generation approaches. this enabled a wide array of applications like 3d object synthesis @xcite @xcite , video generation @xcite @xcite @xcite , controllable image editing @xcite @xcite @xcite , and more, which are now at the forefront of this domain. diffusion models achieve state-of-the-art results in image generation task both unconditional @xcite and conditional @xcite . they beat gans @xcite ) by generating images with better scores of fidelity and diversity without adversarial training @xcite . diffusion models also show the best performance in various image processing tasks like inpainting, outpainting, and super-resolution @xcite . text-to-image diffusion models have become a popular research direction due to the high performance of diffusion models and the ability to simply integrate text conditions with the classifierfree guidance algorithm @xcite . early models like glide @xcite , imagen @xcite , dall-e 2 @xcite and ediff-i @xcite generate low-resolution image in pixel space and then upsample it with another super-resolution diffusion models. they are also using different text encoders, large language model t5 @xcite in imagen, clip @xcite in glide and dall-e 2. ## demo system we implemented a set of user-oriented solutions where kandinsky model is embedded as a core imaging service. it has been done due to a variety of inference regimes, some of which need specific front-end features to perform properly. overall, we implemented two main inference resources: telegram bot and fusionbrain website. fusionbrain represents a web-based image editor with such features as loading and saving images, sliding location window, erasing tools, zooming in/out, various styles selector, etc. (cf. figure 3 ). in terms of image generation, the three following options are implemented on this side: inpainting and outpainting options are the main image editing features of the model. architectural details about these generation types can also be found in figure 1 . telegram bot contains the following image generation features (cf. figure 2 ): ## kandinsky architecture in our work, we opted to deliver state-of-the-art text-to-image synthesis. in the initial stages of our research, we experimented with multilingual text encoders, such as mt5 @xcite , xlmr @xcite , xlmr-clip foot_6 , to facilitate robust multilingual text-to-image generation. however, we discovered that using the clip-image embeddings instead of standalone text encoders resulted in improved image quality. as a result, we adopted an image prior approach, utilizing diffusion and linear mappings between text and image embedding spaces of clip, while keeping additional conditioning with xlmr text embeddings. that is why kandinsky uses two text encoders: clip-text with image prior mapping and xlmr. we have set these encoders to be frozen during the training phase. the significant factor that influenced our design choice was the efficiency of training latent diffusion models, as compared to pixel-level diffusion models @xcite . this led us to focus our efforts on the latent diffusion architecture. our model essentially comprises three stages: text encoding, embedding mapping (image prior), and latent diffusion. the construction of our model involves three primary steps: text encoding, embedding mapping (image prior), and latent diffusion. at the embedding mapping step, which we also refer to as the 20.00 gligen 10 (li et al., 2023) 21.04 proprietary technologies ediff-i @xcite 6.95 imagen @xcite 7.27 gigagan @xcite 9.09 dall-e 2 @xcite 10.39 dall-e @xcite 17.89 image prior, we use the transformer-encoder model. this model was trained from scratch with a diffusion process on text and image embeddings provided by the clip-vit-l14 model. a noteworthy feature in our training process is the use of elementwise normalization of visual embeddings. this normalization is based on full-dataset statistics and leads to faster convergence of the diffusion process. we implemented inverse normalization to revert to the original clip-image embedding space in the inference stage. the image prior model is trained on text and image embeddings, provided by the clip models. we conducted a series of experiments and ablation studies on the specific architecture design of the image prior model ( the latent diffusion part employs a unet model along with a custom pre-trained autoencoder. our diffusion model uses a combination of multiple condition signals: clip-image embeddings, cliptext embeddings, and xlmr-clip text embeddings. clip-image and xlmr-clip embeddings are merged and utilized as an input to the latent diffusion process. also, we conditioned the diffusion process on these embeddings by adding all of them to the time-embedding. notably, we did not skip the quantization step of the autoencoder during diffusion inference as it leads to an increase in the diversity and the quality of generated images (cf. figure 4 ). in total, our model comprises 3.3 b parameters ( ## experiments we sought to evaluate and refine the performance of our proposed latent diffusion architecture in our experimental analysis. to this end, we employed automatic metrics, specifically fid-clip curves on the coco-30k dataset, to obtain the optimal guidance-scale value and compare kandinsky with competitors (cf. figure 4 ). furthermore, we conducted investigations with various image prior setups, exploring the impact of different configurations on the performance. these setups included: no prior, utilizing text embeddings directly; linear prior, implementing one linear layer; resnet prior, consisting of 18 residual mlp blocks; and transformer diffusion prior. an essential aspect of our experiments was the exploration of the effect of latent quantization within the movq autoencoder. we examined the outputs with latent quantization, both enabled and disabled, to better comprehend its influence on image generation quality. to ensure a comprehensive evaluation, we also included an assessment of the if model 12 , which is the closest open-source competitor to our proposed model. for this purpose, we computed fid scores for the if model 13 ( however, we acknowledged the limitations of automatic metrics that become obvious when it comes to capturing user experience nuances. hence, in addition to the fid-clip curves, we conducted a blind human evaluation to obtain insightful feedback and validate the quality of the generated images from the perspective of human perception based on the drawbench dataset @xcite . the combination of automatic metrics and human evaluation provides a comprehensive assessment of kandinsky performance, enabling us to make informed decisions about the effectiveness and usability of our proposed image prior to design. ## results our experiments and evaluations have showcased the capabilities of kandinsky architecture in text-toimage synthesis. kandinsky achieved the fid score of 8.03 on the coco-30k validation set at a resolution of 256×256, which puts it in close competition with the state-of-the-art models, and among the top performers within open-source systems. our methodical ablation studies further dissected the performance of different configurations: quantization of latent codes in movq slightly improves the quality of images (fid 9.86 vs 9.87). the best clip score and human-eval score are obtained by diffusion prior. the best fid score is achieved using linear prior. this configuration stands out with the best fid score of 8.03. it is an intriguing outcome: the simplest linear mapping showcased the best fid, suggesting that there might exist a linear relationship between visual and textual embedding vector spaces. to further scrutinize this hypothesis, we trained a linear mapping on a subset of 500 cat images and termed it the "cat prior". astonishingly, this mapping displayed high proficiency (cf. figure 5 ). ## conclusion we presented kandinsky, a system for various image generation and processing tasks based on a novel latent diffusion model. our model yielded the sota results among open-sourced systems. additionally, we provided an extensive ablation study of an image prior to design choices. our system is equipped with free-to-use interfaces in the form of web application and telegram messenger bot. the pre-trained models are available on hugging face, and the source code is released under a permissive license enabling various, including commercial, applications of the developed technology. in future research, our goal is to investigate the potential of the latest image encoders. we plan to explore the development of more efficient unet architectures for text-to-image tasks and focus on improving the understanding of textual prompts. additionally, we aim to experiment with generating images at higher resolutions and to investigate new features extending the model: local image editing by a text prompt, attention reweighting, physics-based generation control, etc. the robustness against generating abusive content remains a crucial concern, warranting the exploration of real-time moderation layers or robust classifiers to mitigate undesirable, e.g. toxic or abusive, outputs.