D iffusion DB : A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models (2023)

Abstract: With recent advancements in diffusion models, users can generate high-quality images by writing text prompts in natural language. However, generating images with desired details requires proper prompts, and it is often unclear how a model reacts to different prompts or what the best prompts are. To help researchers tackle these critical challenges, we introduce DiffusionDB, the first large-scale text-to-image prompt dataset totaling 6.5TB, containing 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. We analyze the syntactic and semantic characteristics of prompts. We pinpoint specific hyperparameter values and prompt styles that can lead to model errors and present evidence of potentially harmful model usage, such as the generation of misinformation. The unprecedented scale and diversity of this human-actuated dataset provide exciting research opportunities in understanding the interplay between prompts and generative models, detecting deepfakes, and designing human-AI interaction tools to help users more easily use these models. DiffusionDB is publicly available at: https://poloclub.github.io/diffusiondb .

## introduction recent diffusion models have gained immense popularity by enabling high-quality and controllable image generation based on text prompts written in natural language @xcite @xcite . since the release of these models, people from different domains have quickly applied them to create awardwinning artworks @xcite , synthetic radiology images @xcite , and even hyper-realistic videos @xcite . however, generating images with desired details is difficult, as it requires users to write proper prompts specifying the exact expected results. developing such prompts requires trial and error, and can often feel random and unprincipled @xcite . @xcite analogize writing prompts to wizards learning "magical spells": users do not understand why some prompts work, but they will add these prompts to their "spell book." for example, to generate highly-detailed images, it has become a common practice to add special keywords such as "trending on artstation" and "unreal engine" in the prompt. prompt engineering has become a field of study in the context of text-to-text generation, where researchers systematically investigate how to construct prompts to effectively solve different downstream tasks @xcite . as large text-to-image models are relatively new, there is a pressing need to understand how these models react to prompts, how to write effective prompts, and how to design tools to help users generate images @xcite . our work helps researchers tackle these critical challenges, through three major contributions: the unprecedented scale of diffusiondb paves the path for researchers to systematically investigate diverse prompts and associated images that were previously not possible. by characterizing prompts and images, we discover common prompt patterns and find different distributions of the semantic representations of prompts and images. our error analysis highlights particular hyperparameters and prompt styles can lead to model errors. finally, we provide evidence of image generative models being used for potentially harmful purposes such as generating misinformation and nonconsensual pornography ( § 3). we believe diffusiondb will serve as an important resource for researchers to study the roles of prompts in text-to-image generation and design next-generation human-ai interaction tools. our construction process includes collecting images ( § 2.1), linking them to prompts and hyperparameters ( § 2.2), applying nsfw detectors ( § 2.3), creating a flexible file structure ( § 2.4), and distributing the dataset ( § 2.5). we discuss diffu-siondb's limitations and broader impacts in § 7, § 8, and a data sheet @xcite ) ( ‡ a). ## data analysis to gain a comprehensive understanding of the dataset, we analyze it from different perspectives. we examine prompt length ( § 3.1), language ( § 3.2), characteristics of both prompts ( § 3.3) and images ( § 3.4). we conduct an error analysis on misaligned prompt-image pairs ( § 3.5) and provide empirical evidence of potentially harmful uses of image generative models ( § 3.6). ## enabling new research directions the unprecedented scale and diversity of diffu-siondb bring new exciting research opportunities to help users generate images more effectively and efficiently, and enable researchers to improve, explain, and safeguard generative models. ## conclusion we present diffusiondb, the first large-scale text-to-image prompt dataset, containing 14 million images with their prompts and hyperparameters collected from the stable diffusion discord server. we release the dataset with a cc0 1.0 license and open source all collection and analysis code, broadening the public's access to cutting-edge ai technologies. we discuss findings on prompt and image patterns. we hope our work will serve as a cornerstone for the future development of large generative modes and tools that help users use these modes.