Rapid Diffusion: Building Domain-Specific Text-to-Image Synthesizers with Fast Inference Speed (2023)

Abstract: Text-to-Image Synthesis (TIS) aims to generate images based on textual inputs. Recently, several large pre-trained diffusion models have been released to create high-quality images with pre-trained text encoders and diffusion-based image synthesizers. However, popular diffusion-based models from the open-source community cannot support industrial domain-specific applications due to the lack of entity knowledge and low inference speed. In this paper, we propose Rapid Diffusion, a novel framework for training and deploying super-resolution, text-to-image latent diffusion models with rich entity knowledge injected and optimized networks. Furthermore, we employ BladeDISC, an end-to-end Artificial Intelligence (AI) compiler, and FlashAttention techniques to optimize computational graphs of the generated models for online deployment. Experiments verify the effectiveness of our approach in terms of image quality and inference speed. In addition, we present industrial use cases and integrate Rapid Diffusion to an AI platform to show its practical values.

## introduction text-to-image synthesis (tis) is a prevalent multimodal task that aims to generate realistic images based on textual inputs, which supports real-world applications such as product appearance design and art creation. apart from generative adversarial network (gan)-based approaches @xcite , recently, pre-trained diffusion models @xcite have been proposed to create artistic images with qualities comparable to or better than those from humans. despite the exciting advancement, for industrial domain-specific applications, we suggest that popular latent diffusion models from the open-source community (such as the stable diffusion model series foot_0 ) are incapable of supporting those applications. the reasons are are twofolds. i) for diffusion-based methods, a clip-based text encoder (or other similar models) is required to encode the input texts, providing conditional inputs for the u-net model @xcite . as entities (or objects) are usually the key elements for generated images, clip models pre-trained over text-image pairs collected from the web may need more abilities of concept understanding and are challenging to capture the specific entity knowledge required for realistic image generation @xcite . ii) for industrial applications, the model inference speed and the computational cost are vital factors to be considered. the cumbersome computation of the iterative diffusion process is often the bottleneck of fast inference @xcite . therefore, obtaining knowledgeable diffusion models to generate high-resolution images with moderate parameter sizes and optimized implementations that support fast online inference is desirable. to address the above issues, we propose rapid diffusion, a novel framework for the training and deploying text-to-image diffusion models with rich entity knowledge injected and networks optimized. in rapid diffusion, a knowledge-enhanced clip model is effectively trained for learning entity knowledge from knowledge graphs (kgs). to generate high-resolution images and avoid parameter explosion, we integrate an esrgan-based network @xcite after the diffusion block for image super-resolution, instead of directly leveraging a large-scale hierarchical diffusion model. for online deployment, an efficient inference pipeline is designed with the neural architectures optimized based on flashattention @xcite . the intermediate representation (ir) of computational graphs built from the generated models are further processed by a recently released artificial intelligence (ai) compiler @xcite . in the experiments, we evaluate the effectiveness of rapid diffusion in terms of the qualities of generated images from multiple application domains and the model inference speed for online deployment. we also provide industrial use cases to show how our framework benefits real-world applications. in addition, we have integrated the proposed training and deployment workflows into an industrial, cloud-native ai platform to facilitate zero-code model training and elastic inference on distributed gpu clusters. in summary, the major contributions of this work are as follows: an efficient inference pipeline is further designed to optimize the computational graphs of our model for faster model inference. ## the proposed framework in this section, we formally present the techniques of the proposed rapid diffusion framework in detail. a brief overview of rapid diffusion is presented in figure 1 . ## applications in this section, we demonstrate the practical values of rapid diffusion by industrial use case and the integration to alibaba cloud pai (machine learning platform for ai). ## conclusion and future work we present the rapid diffusion framework for the training and deploying knowledge-enhanced, domain-specific, high-resolution, diffusion-based tis models. experimental results show the effectiveness of rapid diffusion in both image quality and inference speed, achieving an average fid score of 21.90 and ×1.73 acceleration ratio compared to all the counterparties. we further show its practical values through industrial use cases and the integration into an ai platform. in the future, we will extend the functionality of rapid diffusion and further increase the inference speed by advanced compilation optimization techniques.