Prompt Expansion for Adaptive Text-to-Image Generation (2024)

Abstract: Text-to-image generation models are powerful but difficult to use. Users craft specific prompts to get better images, though the images can be repetitive. This paper proposes the Prompt Expansion framework that helps users generate high-quality, diverse images with less effort. The Prompt Expansion model takes a text query as input and outputs a set of expanded text prompts that are optimized such that when passed to a text-to-image model, they generate a wider variety of appealing images. We conduct a human evaluation study that shows that images generated through Prompt Expansion are more aesthetically pleasing and diverse than those generated by baseline methods. Overall, this paper presents a novel and effective approach to improving the text-to-image generation experience.

## introduction text-to-image generation models @xcite @xcite are capable of rendering a stunning variety of highquality images, from highly-realistic professionallooking photos to fanciful dreamworlds in almost any visual style. however, interaction with these models frequently exposes two significant usability problems. first, achieving high-quality outputs often requires users to include arcane lighting and camera jargon ('35mm', 'dslr', 'backlit' 'hyper detailed'), idiosyncratic descriptors ('audacious and whimsical'), and social media tags ('trending on artstation'). the prompts that produce the best images are not necessarily stable across different models, or even across different versions of the same model, leading to a focus on prompt-sharing and 'prompt engineering', which is the process of iterating over prompts to craft the optimal textual input for a desired image. ## related work to better control and introduce diversity into image output, most prior work focuses on techniques to allow end-users to edit the generated images, such as text-guided image editing @xcite , or to fine-tune to create personalized images @xcite . these methods focus on local edits and iterating with respect to a given image. our goal is to introduce diversity in the generated images set. the standard approaches to introducing variation are increasing guidance (ho and salimans, 2021) in diffusion models (e.g. imagen @xcite )), or query (extracted via few-shot prompting) surreal hybrid animals aesthetically-pleasing images expanded prompt (via text inversion) a cartoon of a walrus with a pineapple on its head, vector art by maurycy gottlieb, reddit contest winner, neo-primitivism, walrus, surreal hybrid animals, monster manula c query (extracted via few-shot prompting) a walrus with a pineapple on its head, vector art by maurycy gottlieb, reddit contest winner, neo-primitivism, walrus, surreal hybrid animals, monster manula query (extracted via few-shot prompting) a walrus with a pineapple, vector art by maurycy gottlieb, reddit contest winner, neo-primitivism, walrus, surreal hybrid animals, monster manula query (extracted via few-shot prompting) a walrus with a pineapple image-aesthetics datasets image-to-text inversion {image : prompt} query/prompt extraction {query : prompt} prompt expansion dataset prompt expansion model train re-fine-tune text encoder image diffusion increasing temperature in autoregressive models (e.g. parti @xcite ). changing the sampler/decoder hyperparameters, however, may not return meaningfully-diverse images. setting the temperature/guidance too high will insert excessive noise and yield images of low quality. the noise inserted is random, thus it may not return variations along targeted dimensions (e.g. style, lighting, attributes of people / objects / places). to interpretably sample images, we shift our attention to sampling within text space. prompt optimization techniques such as @xcite or promptify @xcite investigate automatic and human-in-the-loop approaches to improving the aesthetic quality of a prompt's generated image. the task of prompt optimization presumes that an ideal prompt may exist for a given query that returns the "best" image. instead, the task of prompt expansion acknowledges that the ideal generated image depends on more than the query alone, and that the query may have different intents or context (e.g. varying by user preferences). it addresses the uncommitted aspects of the image (e.g. query ambiguity) and that images are evaluated in sets. for example, compared to straight-query generation, prompt optimization (e.g. promptist) can return a prompt that maximizes aesthetics, but it would still return a similar set of black-and-white dancers (figure 11 ). ## prompt expansion dataset the prompt expansion (pe) framework requires a model to take a user text query as input and return n text prompts as output, such that the n text prompts through text-to-image generation will return a set of n diverse, aesthetic images aligned to the query. to train a pe model, we require a dataset mapping queries to prompts, which we construct in reverse. first, we collect aesthetically high-quality images, including both model-generated and natural images (refer sec 3.1). second, we invert the images to a closely corresponding prompt that includes alt-text jargon (which we refer to as flavors, refer sec 3.2). finally, we map the inverted text to a range of high-level queries that more closely correspond to user input (refer sec 3.3). these queries are paired with the prompts from the second step to form the {query:prompt} dataset. ## prompt expansion model we describe the two stages to train the prompt expansion model: (i) we train a base prompt expansion model on the prompt expansion dataset; then (ii) we re-fine-tune the base model with respect to the downstream text-to-image model. ## experiments we evaluate different flavors of pe described above, focusing mainly on their ability to generate diverse and aesthetically pleasing images, without significant semantic drift in the prompt. we describe here the experimental setup, task design, and metrics. ## results and discussion from the results of the automatic ( we also found it possible to further increase overall diversity by using techniques such as temperature sampling and post-hoc filtering (see we also observe that the diversity is consistent over multiple generations. with multi-step prompt expansion, figure 7 evaluates the n generated images at expansion step t, and shows consistent diversity over the number of expansion steps. figure 8 evaluates consistency over the number of expanded prompts generated for a single expansion step, and also finds similar consistent diversity as the number of expanded prompts increase. to demonstrate this effect, we compare the difference in prompt-win rates between the 4x4 (where the best images from each system are compared) and 1x1 settings (figure 5 ). the prompt-win rates are consistently larger across all systems in the 4x4 setting, showing the aesthetic benefits of increased diversity at both the prompt and image level. we further explore the trade-offs between diversity, aesthetics, and text-image alignment by presenting pareto-optimality curves between the various automatic and human metrics for all the different query types in figure 4 . while pe: re-fine-tuned dominates prompt expansion in terms of diversity-aesthetics, we notice that the converse is true for diversity-alignment. if textimage alignment is more important than diversity and aesthetics collectively, then it may be beneficial to omit the re-fine-tuning step. in text-to-image generation models are capable of producing high-quality images from text prompts, but they require specialized skills and knowledge to use effectively. prompt expansion reduces the need for users to iteratively prompt-engineer and overspecify their text prompt. human raters find that images generated through prompt expansion are more aesthetically-pleasing and diverse than baselines. empirical metrics for aesthetic and diversity also show that prompt expansion outperforms comparable baselines. in addition, the paper discusses the use of prompt expansion as a building block for other use cases such as multi-step adaptation and controllable generation.