D iff C hat: Learning to Chat with Text-to-Image Synthesis Models for Interactive Image Creation (2024)

Abstract: We present DiffChat, a novel method to align Large Language Models (LLMs) to “chat” with prompt-as-input Text-to-Image Synthesis (TIS)models (e.g., Stable Diffusion) for interactive image creation. Given a raw prompt/image and a user-specified instruction, DiffChat can effectively make appropriate modifications and generate the target prompt, which can be leveraged to create the target image of high quality. To achieve this, we first collect an instruction-following prompt engineering dataset named InstructPE for the supervised training of DiffChat.Next, we propose a reinforcement learning framework with the feedback of three core criteria for image creation, i.e., aesthetics, user preference and content integrity. It involves an action-space dynamic modification technique to obtain more relevant positive samples and harder negative samples during the off-policy sampling. Content integrity is also introduced into the value estimation function for further improvement of produced images. Our method can exhibit superior performance than baseline models and strong competitors based on both automatic and human evaluations, which fully demonstrates its effectiveness.

## introduction in recent years, large-scale deep generative models have emerged as powerful tools for generating contents across various modalities. one of the most remarkably developed and extensively adopted applications is text-to-image synthesis (tis), which aims to create realistic images with texts as inputs (prompts). large pre-trained tis models @xcite @xcite @xcite @xcite have achieved ② give her a hat. to address these issues, in this paper, we propose a novel framework named diffchat, which can follow user-specified instructions to interact with tis models for image creation, as shown in fig. 1 . it avoids the tedious attempts of prompt crafting and re-writing mentioned above, making users feel as simple as "chatting" with these tis models. specifically, we first collect an instruction-following prompt engineering dataset named instructpe using an automatic data collection pipeline based on existing datasets and ai models. next, we utilize instructpe to align off-the-shelf llms to adapt to our task with supervised fine-tuning. finally, we propose an enhanced reinforcement learning framework with three user-concerned criteria for image creation: (1) aesthetics which represents the aesthetic evaluation of created images; (2) preference that indicates the user's preference for specified images relative to other ones; and (3) content integrity to evaluate whether the creations contain core contents as complete as possible. thus, diffchat can be trained to pursue positive apc feedback without any manual labeling. to further enhance the sample quality during training, we also propose an improved sampling strategy based on action-space dynamic modification (adm). for positive samples, we restrict the generation of tokens with low information quantity to improve the overall quality; and for negative samples, we also partially mask or replace the key information to simulate hard samples, allowing the model to fully learn from these errors. additionally, we develop the value estimation function of vanilla ppo @xcite method with the consideration of content integrity (vci), to achieve a more accurate perception of the current state during optimization. by using prompts as intermediaries, diffchat allows users to easily interact and collaborate with tis models for image creation through chatting. it is worth noting that, recent related research works @xcite @xcite @xcite mainly focus on the model structure design. they typically adopt instructions, images, and additional information as inputs to generate edited images using an end-to-end network. different from these works, our method pays more attention to the preliminary automatic prompt writing procedure, which can be used in conjunction with these methods or various widely-used tis models such as the series of stable diffusion (sd) models. it does not need to re-train with the development of tis models to make extra costs, with its user-friendliness and generalization abilities fully manifested. experimental results based on both automatic and human evaluations demonstrate that our method exhibits greater performance than baseline models and competitors. in summary, the main contributions of this paper are listed as follows: ## methodology the overall framework of our method is composed of three main steps: data collection of instructpe in fig. 2 , supervised fine-tuning over diffchat and enhanced reinforcement learning with apc feedback in fig. 3 . we first construct the instructpe dataset from the raw data of instructpix2pix @xcite with prompt beautification and prompt engineering. next, diffchat is fine-tuned with supervised learning. finally, an enhanced ppo-based @xcite reinforcement learning process is performed with aesthetics, preference, and content integrity criteria. the detailed explanations of each step are as follows. ## conclusion in this paper, we propose diffchat to follow userspecified instructions to interact with tis models 8833 for image creation. we collect and release the in-structpe dataset for training instruction-following prompt engineering models. a reinforcement learning framework with aesthetics, preference, and content integrity feedback is introduced to align supervising fine-tuned llms. action-space dynamic masking and value estimation with content integrity are also involved for further improvement. extensive experimental results show that diffchat outperforms competitors in terms of both automatic and human evaluation.