Collaborative Generative AI : Integrating GPT -k for Efficient Editing in Text-to-Image Generation (2023)

Abstract: The field of text-to-image (T2I) generation has garnered significant attention both within the research community and among everyday users. Despite the advancements of T2I models, a common issue encountered by users is the need for repetitive editing of input prompts in order to receive a satisfactory image, which is time-consuming and labor-intensive. Given the demonstrated text generation power of large-scale language models, such as GPT-k, we investigate the potential of utilizing such models to improve the prompt editing process for T2I generation. We conduct a series of experiments to compare the common edits made by humans and GPT-k, evaluate the performance of GPT-k in prompting T2I, and examine factors that may influence this process. We found that GPT-k models focus more on inserting modifiers while humans tend to replace words and phrases, which includes changes to the subject matter. Experimental results show that GPT-k are more effective in adjusting modifiers rather than predicting spontaneous changes in the primary subject matters. Adopting the edit suggested by GPT-k models may reduce the percentage of remaining edits by 20-30%.

## introduction the task of text-to-image (t2i) generation, which involves generating images from natural language descriptions, holds significant potential to create new avenues and job opportunities for content creators while also providing insights into the grounding of natural language in the visual world through the application of generative ai. a number of models have demonstrated exceptional performance in terms of image quality, such as stablediffusion @xcite , midjourney (midjourney, 2022), imagen @xcite , and dalle-2 @xcite . despite the popularity of t2i generation and the ability of these models to generate impressive images, the difficulty of @xcite while the lineplot shows the cumulative percentage of traces less than certain #edit. the y-axis on the left is on log-scale. on average, there are 6.9 edits being made in each user editing trace. recently, large language models (llms) such as gpt-k @xcite @xcite have demonstrated impressive abilities in text generation. this leads to the natural question of whether these models can be utilized to improve the t2i prompting process @xcite . however, llms and t2i models have different architectures and are often trained on different modalities, which makes it challenging to integrate them seamlessly. in this paper, we conduct a series of experiments and analyses on user editing traces on stablediffusion, 1 and attempt to modify the t2i prompts with eight gpt-k models. the primary objective of our research is to examine the potential of modifying t2i prompts with gpt-k models. more specifically, we aim to investigate the common edits made by humans and by gpt-k, as well as the performance of prompting t2i generation with gpt-k models. additionally, we aim to identify and investigate possible factors that might influence the performance of gpt-k in t2i generation tasks. through our experiments, we observe that: • while gpt-k models tend to focus more on inserting modifiers, human users have a greater tendency towards replacing words and phrases, which may include spontaneous changes to the subject matter of the prompt. ## research questions and settings to investigate the potential of modifying t2i prompts with gpt-k models, we conduct a series of experiments and analysis, aiming to answer the following questions: gpt-k model name #parameters gpt-2 (radford et al., 2019) gpt2-base 117m gpt2-medium 345m gpt2-large 774m gpt2-xl 1.2b gpt-3 (brown et al., 2020) text-ada-001 350m text-babbage-001 1.3b text-curie-001 6.7b gpt-3.5 (ouyang et al., 2022) text-davinci-003 175b ## prompting t2i w/ gpt-k in the following experiments, we only reveal the initial prompts in the user editing traces to gpt-k, and ask the models to perform one round of edit. procedure we split the 100k trace of edits into two parts, with 30k traces used for evaluations and the remaining 70k serving as heldout set. for each of the listed gpt-k models, we provide the first prompt t 1 in each evaluation trace to the model, and let gpt-k generate the modified prompt t 1 . the heldout traces for two epochs with a learning rate of 5e ´5 and a batch size of 128. for gpt-3/3.5 models, an in-context learning approach is adopted. following previous studies @xcite , supporting examples for in-context learning are selected by comparing the similarity of the prompt text features and retrieving ( t1 , tn ) pairs from the top-m most similar traces. modified prompts are generated through 16-shot in-context learning with @xmath0. appendix ## human's common edits vs. gpt-k's upon examination of the user editing traces, we identify four types of edits commonly made by humans: (1) insert: adding descriptive terms such as modifiers to the prompt to specify the style, artistic technique, camera view, lighting, etc; (2) delete: removing certain terms; (3) swap: changing the order of certain terms in the prompt; (4) replace: changing the modifiers or the main subject of the 3 11115 prompt. appendix to count the frequency of edits, we first remove punctuation marks and stopwords using nltk @xcite . we then utilize the se-quencematcher 2 to compare the adjacent prompts in the trace and detect the edit type. ## ablations & analyses effects of edit types to investigate the effects of each individual edit type e i , we create four subsets s insert , s delete , s swap and s replace from the evaluation set s eval . each subset s e i , comprises of traces that meet the criteria of "if and only if edit e i is applied on the first prompt can we receive the last prompt." for each edit type e i , the image similarity between the image generated from the modified prompt and the last image for traces in s e i is calculated and figure 2 illustrates the impact of different edit types on image similarity. the clip cosine similarities of traces that solely consist of insert, delete, and swap edits are higher or comparable to the all-mixed baseline. this suggests that gpt-k performs better at adding, removing, and reordering modifiers. conversely, we observe that replace edits lead to lower image similarities. this is likely due to the fact that the replace edit sometimes results in a change of the subject matter, which can drastically alter the painting. it is worth noting that shifting the primary subject matter of the painting is a relatively spontaneous action made by humans. appendix ## conclusion through our experiments and analyses, we hope to gain a deeper understanding of the capabilities and limitations of using gpt-k to edit prompts for textto-image generation, and provide valuable insights for future research in this field.