Words Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation (2024)

Abstract: Diffusion models are the state of the art in text-to-image generation, but their perceptual variability remains understudied. In this paper, we examine how prompts affect image variability in black-box diffusion-based models. We propose W1KP, a human-calibrated measure of variability in a set of images, bootstrapped from existing image-pair perceptual distances. Current datasets do not cover recent diffusion models, thus we curate three test sets for evaluation. Our best perceptual distance outperforms nine baselines by up to 18 points in accuracy, and our calibration matches graded human judgements 78% of the time. Using W1KP, we study prompt reusability and show that Imagen prompts can be reused for 10-50 random seeds before new images become too similar to already generated images, while Stable Diffusion XL and DALL-E 3 can be reused 50-200 times. Lastly, we analyze 56 linguistic features of real prompts, finding that the prompt’s length, CLIP embedding norm, concreteness, and word senses influence variability most. As far as we are aware, we are the first to analyze diffusion variability from a visuolinguistic perspective. Our project page is at http://w1kp.com.

## introduction in text-to-image generation, pictures are worth a thousand words, but which words are worth a thousand pictures? specifically, how do prompts affect perceptual variation in generated imagery across random seeds? consider these prompts: p1: a matte orange ball in the center against a pure white background. p2: orange ball against white background. as shown in figure 1 , the first conveys a single particular illustration, while the second elicits multiple interpretations. orange could refer to the fruit or the color, and the scene geometry is underspecified. but how can we quantify and characterize these linguistic intuitions? figure 1 : dall-e 3 images for the prompts "a matte orange ball in the center against a pure white background" (top) and "orange ball against white background" (bottom). our w1kp score quantifies the perceptual similarity for each set of images. it yields 0.99 and 0.68 for the top and bottom rows, showing the greater image variability of the latter. in this paper, we study the connection between visual variability and language in black-box textto-image models, focusing on state-of-the-art diffusion models. previous work tends to study the perceptual distance @xcite between pairs of images, while a prompt can generate a near infinite set of images. furthermore, previous approaches have not been explicitly calibrated for human-friendly grades of similarity. what does a score of, for example, 0.2 mean in terms of perceived similarity? such calibration is likely crucial for robust human interpretation. to bridge these gaps in the literature, we first propose a straightforward framework for constructing human-calibrated perceptual variability measures based on existing perceptual distance metrics. we call it the words of a thousand pictures method, or w1kp (['wik.pi:]) for short. on our crowd-sourced dataset of human-judged images from dall-e 3, imagen, and stable diffusion xl (sdxl), we validate our choice of dream-sim @xcite , a recent distance trained on stable diffusion @xcite images. our variant of dreamsim outperforms the best baseline by 0.1-0.4 points in two-alternative forced choice and 0.2-0.4 points in accuracy. to improve interpretability, we normalize and calibrate scores to graded human judgements on four levels of perceptual similarity, with cutoff points corresponding to high (0.85-1.0), medium (0.4-0.85), low (0.2-0.4), and no similarity (<0.2), which yield a correct classification 78% of the time. next, to ground our academic discourse, we investigate the practical implications of our approach. suppose a computer graphics practitioner wishes to generate a diverse array of images from a single prompt, but it is unclear how much it can be reused with different seeds before additional images contribute little to the variability of the overall set of images. our work provides a quantitative metric for prompt reusability, as we explore further in section 4.1. on diffusiondb @xcite , an open dataset of user-written text-to-image prompts, we find that the same prompt can be reused for imagen for 10-20 random seeds, while sdxl and dall-e 3 are more reusable at 100-200 seeds. finally, we study how 56 linguistic features affect generation variability. although research has explored optimizing for image variability in diffusion @xcite , they have not investigated the contributing linguistic constructs. to understand the underlying structure of these 56 features, we perform an exploratory factor analysis over dif-fusiondb and uncover four factors of keyword presence (e.g., "dog walking, 4k, watercolor"), syntactic complexity (e.g., yngve depth), linguistic unit length, and semantic richness. then, we conduct clean-room, single-word generation experiments over the three strongest features in the semantic richness factor (concreteness, clip embedding norm, and number of word senses) to assess their contribution more precisely. we confirm that all three linguistic features significantly (@xmath0.01) correlate with perceptual variability for all three diffusion models studied. our contributions are as follows: (1) we propose and validate a human-calibrated framework for building perceptual variability metrics from existing perceptual distance metrics; (2) we examine a new practical application of the method in assessing prompt reusability in text-to-image generation; and (3) we provide original insight into the linguistic sources of variability in diffusion models, finding that keywords, syntactic complexity, length, and semantic richness influence variability. ## visuolinguistic analyses with the variability metric established, we investigate the connection between visual variability and prompt language for text-to-image models. ## related work and future directions a related line of work examines boosting image variability in diffusion models @xcite @xcite . complementary to their work, our paper analyzes the concreteness word senses precise linguistic features contributing to variability. one future direction could be to incorporate these features into the optimization of variability. previous work has analyzed diffusion models using a mixture of computational linguistics and vision techniques. finally, research has previously scrutinized the (lack of) variability in older architectures such as vaes @xcite and generative adversarial networks, e.g., mode collapse. in this paper, we extend this analysis to modern diffusion models while taking a visuolinguistic perspective. ## conclusions in conclusion, we examined the connection between visual variability and prompt language for black-box diffusion models. we proposed a framework for quantifying and calibrating visual variability, applying it to study prompt reusability and linguistic feature salience. after validating it quantitatively, we found that length, embedding norm, and concreteness influence variability the most.