Prompt Refinement with Image Pivot for Text-to-Image Generation (2024)

Abstract: For text-to-image generation, automatically refining user-provided natural language prompts into the keyword-enriched prompts favored by systems is essential for the user experience. Such a prompt refinement process is analogous to translating the prompt from “user languages” into “system languages”. However, the scarcity of such parallel corpora makes it difficult to train a prompt refinement model. Inspired by zero-shot machine translation techniques, we introduce Prompt Refinement with Image Pivot (PRIP). PRIP innovatively uses the latent representation of a user-preferred image as an intermediary “pivot” between the user and system languages. It decomposes the refinement process into two data-rich tasks: inferring representations of user-preferred images from user languages and subsequently translating image representations into system languages. Thus, it can leverage abundant data for training. Extensive experiments show that PRIP substantially outperforms a wide range of baselines and effectively transfers to unseen systems in a zero-shot manner.

## introduction recent breakthroughs in text-to-image generation have markedly expanded the boundaries of digital artistry, enabling the creation of visually compelling images with unprecedented ease @xcite @xcite @xcite . however, the complexity of crafting effective prompts presents a significant challenge to average users. this challenge stems from the significant difference between user-provided natural language prompts and the keyword-enriched prompts required for system's high-quality rendering @xcite @xcite . we term the two kinds of prompts as user languages and system languages. system languages usually include technical terms and artistic references unfamiliar to non-specialists @xcite . crafting prompts in system languages is not intuitive, even for the system's developers, and only becomes clear after extensive user experimentation and community-driven insight @xcite @xcite . developing a model that automatically refines user languages into system languages is essential to enhance user experience @xcite @xcite . yet, the shortage of high-quality refinement pairs makes it difficult to train such models. on the one hand, refining prompts requires expertise, which makes annotation expensive. on the other hand, humans can hardly refine prompts to the optimum due to the intricate nature of system languages. as demonstrated in @xcite and our experimental results, human-generated refinement data is suboptimal for training prompt refinement models. as prompt refinement is analogous to a machine translation task that converts the prompts written in user languages into system languages, the lack of high-quality refinement pairs echoes the challenge of machine translation for low-resource languages @xcite , in which a source language is translated into a target language without sufficient parallel corpora for training. mt researchers tackle this through a "pivoting" approach @xcite , which utilizes a high-resource language as the intermediate "pivot language". thus, the source-target translation is achieved by training two separate models: a sourcepivot model and a pivot-target model. text from the source language is first translated to the pivot language and then to the target language. inspired by pivot-based mt solutions, we pro-pose prompt refinement with image pivot (prip). prip employs the latent representation of a userpreferred image as the pivot between user and system languages. it decomposes prompt refinement into the following two phases. in the first phase, prip takes the prompt in user languages as input and infers what images the user prefers. it outputs a latent representation, which focuses on high-level semantics instead of pixel-level details, for the image that is preferred by the user. in the second phase, prip takes the latent representation of an image as input and outputs a prompt in the system language that can guide the text-to-image system in rendering this image. by doing so, prip reframes a data-limited prompt refinement problem into two data-rich tasks. for example, user-image preference pairs can be constructed using preference simulation models like hpsv2 @xcite or user click behaviors, and image-system decoding pairs can be sampled from interaction logs or promptsharing websites. we evaluate prip by applying it to various textto-image models and comparing it with extensive baselines. results demonstrate that prip not only substantially improves the text-to-image system seen during training, but also effectively transfers to various unseen systems in a zero-shot manner. it significantly outperforms a wide range of baselines, including general large language models and prompt refinement models trained with humangenerated or synthetic refinement pairs. ## method this section introduces prompt refinement with image pivot (prip). we first analyze the prompt refinement task. then we describe how to decompose the user-system refinement process into user-pivot and pivot-system sub-tasks, where the pivot is the representation of the image preferred by the user. finally, we elaborate on the training approaches for the user-pivot-system framework. ## experimental results this section presents the experimental results. we first evaluate models on various generation systems, including the one used during training (indistribution scenario) and several unseen, advanced systems (out-of-distribution scenario). then, we show how prip refines the prompt by presenting several cases. finally, a comprehensive ablation study demonstrates the effectiveness of pivoting. ## conclusion in this paper, we present prip, a pioneering pivotbased approach tailored for text-to-image prompt refinement. by formulating the refinement process as user-pivot preference encoding and pivot-system prompt decoding, prip sidesteps the scarcity of user-system refinement pairs and leverages largescale data for effective model training. extensive experiments demonstrate prip's effectiveness in prompt refinement. the improvement is pronounced for both text-to-image systems seen and unseen during training, highlighting its remarkable effectiveness and robust generalizability.