PAI -Diffusion: Constructing and Serving a Family of Open C hinese Diffusion Models for Text-to-image Synthesis on the Cloud (2024)

Abstract: Text-to-image synthesis for the Chinese language poses unique challenges due to its large vocabulary size, and intricate character relationships. While existing diffusion models have shown promise in generating images from textual descriptions, they often neglect domain-specific contexts and lack robustness in handling the Chinese language. This paper introduces PAI-Diffusion, a comprehensive framework that addresses these limitations. PAI-Diffusion incorporates both general and domain-specific Chinese diffusion models, enabling the generation of contextually relevant images. It explores the potential of using LoRA and ControlNet for fine-grained image style transfer and image editing, empowering users with enhanced control over image generation. Moreover, PAI-Diffusion seamlessly integrates with Alibaba Cloud’s Platform for AI, providing accessible and scalable solutions. All the Chinese diffusion model checkpoints, LoRAs, and ControlNets, including domain-specific ones, are publicly available. A user-friendly Chinese WebUI and the diffusers-api elastic inference toolkit, also open-sourced, further facilitate the easy deployment of PAI-Diffusion models in various local and cloud environments, making it a valuable resource for Chinese text-to-image synthesis.

## introduction recently, diffusion models @xcite have emerged to address the challenges of generating realistic and high-quality images from textual descriptions. this research area has obtained widespread attention, driven by the increasing demand for automated image synthesis in various applications, such as art design, virtual reality, etc @xcite . in the community, stable diffusion foot_0 has gained significant popularity due to its ability to generate high-quality images that align well with textual descriptions. however, when it comes to handling the chinese language, similar models encounter certain challenges that hinder its performance. from the linguistic aspect, chinese is a morphologically rich language with a large vocabulary size and complex inter-dependencies between characters. furthermore, chinese characters often have multiple meanings and can be combined to form compound words, making it challenging to establish accurate and consistent mappings between textual descriptions and visual representations @xcite . previously, in the literature, several works have been proposed to make specialized adaptations of diffusion models to improve the performance of text-to-image synthesis for chinese @xcite @xcite . yet, there are still some notable drawbacks that need to be addressed. i) many existing models focus on generating images based on generic textual descriptions, neglecting the ability to generate images in specific domains or contexts. ii) for the chinese language, the potential of using lora @xcite and controlnet @xcite for fine-grained image style transfer and image editing have not been fully explored. iii) the lack of support for cloud-based product integration is another important drawback. given the increasing popularity of cloud-based services and the demand for scalable and accessible text-to-image solutions, it is crucial to develop models and solutions that can be easily integrated into cloud platforms or deployed as cloud-based services. in this work, we formally present pai-diffusion, which consists of a family of open chinese diffusion models, together with user-friendly toolkits to serve these models on the cloud. major features of pai-diffusion include the following: to realize our promise for the openness of our research, we have taken the following actions to contribute pai-diffusion to the community: ## models we introduce the models of pai-diffusion, a family of open chinese diffusion models designed to address the challenges of generating high-quality images from chinese textual descriptions. ## toolkits pai-diffusion provides user-friendly toolkits that facilitate the deployment and usage of diffusion models for chinese text-to-image synthesis. in this section, we briefly introduce our two toolkits. ## conclusion this paper presents pai-diffusion to address the challenges in chinese text-to-image synthesis. pai-diffusion integrates both general and domainspecific chinese diffusion models, loras and con-trolnets, enabling the generation of contextually relevant images. moreover, pai-diffusion ensures seamless integration with our cloud platform, offering accessible and scalable solutions. the release of models further encourages collaboration and innovation in the field. the public availability of chinese webui and diffusers-api toolkits simplifies deployment in various environments. our advancements pave the way for further research and development in chinese text-to-image synthesis and relevant applications.