Self-Rewarding Large Vision-Language Models for Optimizing Prompts in Text-to-Image Generation (2025)

Abstract: Text-to-image models are powerful for producing high-quality images based on given text prompts, but crafting these prompts often requires specialized vocabulary. To address this, existing methods train rewriting models with supervision from large amounts of manually annotated data and trained aesthetic assessment models. To alleviate the dependence on data scale for model training and the biases introduced by trained models, we propose a novel prompt optimization framework, designed to rephrase a simple user prompt into a sophisticated prompt to a text-to-image model. Specifically, we employ the large vision language models (LVLMs) as the solver to rewrite the user prompt, and concurrently, employ LVLMs as a reward model to score the aesthetics and alignment of the images generated by the optimized prompt. Instead of laborious human feedback, we exploit the prior knowledge of the LVLM to provide rewards, i.e., AI feedback. Simultaneously, the solver and the reward model are unified into one model and iterated in reinforcement learning to achieve self-improvement by giving a solution and judging itself. Results on two popular datasets demonstrate that our method outperforms other strong competitors.

## introduction text-to-image models @xcite @xcite can generate diverse high-quality images based on user-provided prompts. however, effective interaction with these models requires users to possess specific expertise, including familiarity with specialized vocabularies, e.g., "35mm" for camera parameters and "art a stunning ultra detailed fantasy fine art illustration of a phoenix flying above a rugged mountain peak, by artgerm and greg rutkowski, golden sunrise, cliffs, concept art, equine, 3d render, octane render, smooth, focus, rim lighting. when training, the large model continuously improves prompt rewriting and image quality evaluation capabilities through self-play without any external sources or models. the generated images from the user prompts and the rewritten prompts. it can be observed that the image from the modified prompt has higher aesthetics. ## methodology in this section, we first describe the pipeline of dataset construction (section 3.1), and how to initialize lvlm with both prompt optimization and preference assessing capabilities (section 3.2) and then detail self-rewarding training algorithms (section 3.3). next, we introduce the use of ai feedback for rl in self-rewarding training (section 3.4) and how to transform lvlm into a reward model by lvlm-as-a-judge (section 3.5). ## conclusion in this work, we propose a novel method to optimize prompts for text-to-image models, which can be used to fill the gap between laymen and experts when using a generative model. we first transform the lvlm into a reward model so that it can judge the aesthetics and alignment of the images. in so doing, we can perform ai feedback rather than human feedback. then, to gain the performance boost based on limited data, we employ selfrewarding training for lvlm. our model achieves self-improvement through an iterative training approach. experiments on two datasets show that our method outperforms the other methods.