N euro P rompts: An Adaptive Framework to Optimize Prompts for Text-to-Image Generation (2024)

Abstract: Despite impressive recent advances in text-to-image diffusion models, obtaining high-quality images often requires prompt engineering by humans who have developed expertise in using them. In this work, we present NeuroPrompts, an adaptive framework that automatically enhances a user’s prompt to improve the quality of generations produced by text-to-image models. Our framework utilizes constrained text decoding with a pre-trained language model that has been adapted to generate prompts similar to those produced by human prompt engineers. This approach enables higher-quality text-to-image generations and provides user control over stylistic features via constraint set specification. We demonstrate the utility of our framework by creating an interactive application for prompt enhancement and image generation using Stable Diffusion. Additionally, we conduct experiments utilizing a large dataset of human-engineered prompts for text-to-image generation and show that our approach automatically produces enhanced prompts that result in superior image quality. We make our code, a screencast video demo and a live demo instance of NeuroPrompts publicly available.

## introduction text-to-image generation has recently become increasingly popular as advances in latent diffusion models have enabled widespread use. however, these models are sensitive to perturbations of the prompt used to describe the desired image, motivating the development of prompt engineering expertise by users to increase the quality of the resulting images generated by the model. prompt design is crucial in ensuring that the model accurately comprehends the user's intent. text-to-image models face a significant challenge in this aspect as their text encoders have limited capacity, which can make it difficult to produce aesthetically pleasing images. additionally, as empirical studies have shown, common user input may not be enough to produce satisfactory results. therefore, developing innovative techniques to optimize prompt design for these models is crucial to improving their generation quality. to address this challenge, we introduce neuro-prompts, a novel framework which automatically optimizes user-provided prompts for text-to-image generation models. a key advantage of our framework is its ability to automatically adapt a user's natural description of an image to the prompting style which optimizes the quality of generations produced by diffusion models. we achieve this automatic adaptation through the use of a language model trained with proximal policy optimization (ppo) @xcite to generate text in the style commonly used by human prompt engineers. this results in higher quality images which are more aesthetically pleasing, as the prompts are automatically optimized for the diffusion model. furthermore, our approach allows the user to maintain creative control over the prompt enhancement process via constrained generation with neurologic decoding @xcite , which enables more personalized and diverse image generations. our neuroprompts framework is integrated with stable diffusion @xcite in an interactive application for text-to-image generation. given a user-provided prompt, our application automatically optimizes it similar to expert human prompt engineers, while also providing an interface to control attributes such as style, format, and artistic similarity. the optimized prompt produced by our framework is then used to generate an image with stable diffusion, which is presented to the user along with the optimized prompt. we validate the effectiveness of neuroprompts by using our framework to produce optimized prompts and images for over 100k baseline prompts. through automated evaluation, we show that our optimized prompts produce images with significantly higher aesthetics than un-optimized baseline prompts. the optimized prompts produced by our approach even outperform those created by human prompt engineers, demonstrating the ability of our application to unlock the full potential of text-to-image generation models to users without any expertise in prompt engineering. ## neuroprompts framework given an un-optimized prompt provided by a user, which we denote as x u , our neuroprompts framework generates an optimized prompt x o to increase the likelihood that text-to-image diffusion models produce an aesthetically-pleasing image when prompted with x o . we specifically consider the case where x u is the prefix of x o and produce the enhanced prompt via a two-stage approach. first, we adapt a language model (lm) to produce a text which is steered towards the style of prompts produced by human prompt engineers. we then generate enhanced prompts via our steered lm using a constrained text decoding algorithm (neurologic), which enables user customizability and improves the coverage of image enhancement keywords. ## neuroprompts the user interface of neuroprompts is depicted in figure 1 . the application's inputs include the ini-tial prompt as well as selection fields for specifying the clauses used to populate constraints for style, artist, format, booster, perspective, and vibe. additionally, a negative constraints input allows the user to specify one or more phrases which should be excluded from the optimized prompt. while the initial prompt is required, all other fields are optional; if left unselected, clauses for each constraint set will be automatically populated as described previously in section 2.2. this functionality allows the user to take control of the constrained generation process if desired or simply rely on our framework to optimize the prompt automatically. after clicking the submit button, the optimized prompt is displayed at the top of the screen. if constraints were selected by the user, the optimized prompt will appear with color-coded highlighting to show where each constraint has been satisfied in the generated sequence. the image produced by stable diffusion for the optimized prompt is displayed directly below the optimized prompt in the center of the interface. if the user selects the sideby-side comparison tab, an image generated for the original prompt is also displayed to the right of the optimized image. additionally, the application calculates pickscore and a normalized aesthetics score for the two images, which is displayed in a table below the images. this side-by-side comparison functionality allows the user to directly assess the impact of our prompt optimizations on the quality of images generated by stable diffusion. ## conclusion we presented neuroprompts, an application which automatically optimizes user prompts for text-to-image generation. neuroprompts unlocks the full potential of text-to-image diffusion models to users without requiring any training in how to construct an optimal prompt for the model. therefore, we expect it to increase the accessibility of such models while improving their ability to be deployed in a more automated fashion. in future work, we would like to extend neuroprompts to video generation models and other settings which can benefit from automated prompt engineering.