Human Learning by Model Feedback: The Dynamics of Iterative Prompting with Midjourney (2023)

Abstract: Generating images with a Text-to-Image model often requires multiple trials, where human users iteratively update their prompt based on feedback, namely the output image. Taking inspiration from cognitive work on reference games and dialogue alignment, this paper analyzes the dynamics of the user prompts along such iterations. We compile a dataset of iterative interactions of human users with Midjourney. Our analysis then reveals that prompts predictably converge toward specific traits along these iterations. We further study whether this convergence is due to human users, realizing they missed important details, or due to adaptation to the model’s “preferences”, producing better images for a specific language style. We show initial evidence that both possibilities are at play. The possibility that users adapt to the model’s preference raises concerns about reusing user data for further training. The prompts may be biased towards the preferences of a specific model, rather than align with human intentions and natural manner of expression.

## introduction text-to-image models have become one of the most remarkable applications in the intersection of computer vision and natural language processing @xcite . their promise, to generate an image based on a natural language description, is challenging not only to the models, but to the human users as well. generating images with the desired details requires proper textual prompts, which often take multiple turns, with the human user updating their prompt slightly based on the last image they received. we see each such interaction as a "thread", a sequence of prompts, and analyze the dynamics of user prompts along the interaction. we are not aware of any work that examines the dynamics of the prompts between iterations. 1 code and data are available in: https://github.com/ to study this question, we compile the midjoureny dataset, scraped from the midjoureny discord server foot_0 , containing prompts and their corresponding generated images and metadata, organized as 107, 051 interaction threads. the language people use when they interact with each other changes over the course of the conversation @xcite . theoretical work suggests that learning mechanisms may allow interlocutors to dynamically adapt not only their vocabulary but their representations of meaning @xcite @xcite . we hypothesize that also when inshachardon/mid-journey-to-alignment teracting with midjourney, where only the human user is able to adapt and the midjourney model remains "frozen", we would see a systematic language change along the iterations. unlike the interaction with general assistant models @xcite , which might include multiple topics and change along the conversation, the interaction thread of a text-to-image model contains attempts to generate one image, a single scene, with no major content change. this allows us to better recognize the change in the linguistic features rather than the content. our analysis reveals convergence patterns along the threads, i.e., during interactions humans adjust to shared features that bring them closer to their ideal image in terms of prompt length, perplexity, concreteness and more. still, it is unclear whether these adjustments are due to humans adding missing details, or due to matching the model preferences -generating better images due to prompts in a language style that is easier for it to infer, thus encouraging users to adapt to it. we find evidence for both. the second possibility, that users adapt to the model preferences, calls for caution regarding the subsequent use of human data from human-model interaction. for example, we could take the "successful" images that the human users presumably liked and requested a high resolution version of them ("upscale"), and use them with their matching prompts as a free human-feedback dataset @xcite . however, given that these prompts may be biased towards the model's preferences, training on them would create a model that has even more 'model-like' behaviour. we hope that by releasing this first iterative prompting dataset, along with our findings regarding possible biases in the human data, we would encourage more work on human-model alignment and interaction. ## background in a repeated reference game @xcite , pairs of players are presented with a set of images. on each iteration, one player (the director) is privately shown which one is the target image, and should produce a referring expression to help their partner (the matcher) to correctly select that image. at the end of the iteration, the director is given feedback about which image the matcher selected, and the matcher is given feedback about the true target image. empirical findings show a number of recurring behavioral trends in the reference game task. for example, descriptions are dramatically shortened across iterations @xcite , and the resulting labels are partner-specific @xcite . we view the interaction thread as similar to a repeated reference game of the human user with the model. the human user directs midjourney with textual prompts to generate (instead of select) the target image. unlike the original game, only the human user is changing along the interaction based on the feedback (i.e., the image) they get from midjourney. the midjourney model is 'frozen', not able to adjust to the user feedback. we hypothesize that also in our semi-reference game where only the human user is able to adapt we would see a language change along the iterations. we use similar methods to those used in recent works @xcite , in order to examine this change. prompt and image pairs, together with their metadata about upscale requests, can be seen as a great source of data for reinforcement learning from human feedback (rlhf). in rlhf, non-expert human preferences are used to train the model to achieve better alignment @xcite @xcite . we discuss in §9 the possible consequences of reusing the midjourney data. ## data in this section, we describe the choices and process of acquiring text-to-image interactions. we start by discussing the reasons to pick midjourney data rather than other text-to-image data sources. one reason to prefer midjourney over opensource text-to-image models is its strong capabilities. midjourney can handle complicated prompts, making the human-model interaction closer to a standard human-human interaction. another reason to prefer midjourney is the availability of the prompts, images and meta-data on the discord server. we construct the dataset by scraping user-generated prompts from the midjourney discord server. the server contains channels in which a user can type a prompt and arguments, and then the midjourney bot would reply with 4 generated images, combined together into a grid. then, if the user is satisfied with one of the 4 images, they can send an "upscale" command to the bot, to get an upscaled version of the desired image. we randomly choose one of the "newbies" channels, where both new and experienced users are experimenting with general domain prompts (in contrast to the "characters" channel for example). we collect 693, 528 prompts (from 23 january to 1 march 2023), together with their matching images and meta-data such as timestamps and user ids (which we anonymize). in §f we repeat some of the experiments with data from stable diffusion @xcite , concluding that our results can be extended to other models. ## splitting into threads we split the prompts into threads. each thread should contain a user's trails to create one target image. however, it is often difficult to determine whether the user had the same image in mind when they tried two consecutive prompts. for example, when a user asks for an image of "kids playing at the school yard" and then replaces "kids" with "a kid", it is hard to tell whether they moved to describe a new scene or only tried to change the composition. we consider a prompt to belong to a new thread according to the following guidelines: ## method our end goal is to examine the evolution of the prompts through the interaction. we start, however, by asking a simpler question, namely whether there is a difference between the upscale and nonupscale prompts. such a difference would indicate that there are predictable characteristics (however intricate) of a prompt that render it better for midjourney, and therefore provide motivation for the human users to adapt their prompts towards it. to highlight differences between the upscaled and non-upscaled, we compile a list of linguistic features, that are potentially relevant to the upscale decision. we find several features that are statistically different between the upscaled and nonupscaled populations, and then use those features to test the evolution of the threads. we stress that we do not argue that these features account for a large proportion of the variance between users. indeed, people use midjourney for a wide range of tasks, with different levels of experience, hence their prompts and preferences vary a lot. instead, we wish to make a principle point that there is a systematic convergence along the threads, and that it has practical implications. future work will control for the intentions of the user, in which case we expect the convergence to account for a larger proportion of the variance. ## results in the following sections, we apply the method described in §5 to test the distribution and the dynamics of the prompts and threads. we start by identifying that there is a difference between the upscaled prompts and the rest ( §6.1). we then associate it with specific linguistic features ( §6.2). finally, we examine not only the final decision, but the dynamics of the full interaction ( §6.3). ## driving forces behind the dynamics we examine two possible non-contradictory explanations to the characteristics of the upscaled and non-upscaled prompts, and to the direction in which the human users go along the interaction. we show supporting evidence for both. option #1: adding omitted details. when users input a prompt to midjourney and receive an image, they may realize that their original prompt lacked some important details or did not express them well enough. writing a description for a drawing is not a task most people are accustomed to. hence, it makes sense that users learn how to make their descriptions more accurate and complete along the interaction. results from three features support this explanation. the prompt length, the sentence rate and the syntactic tree depth, all of them increase as the interaction progresses. improving the accuracy of a prompt can be done by adding more words to describe the details, thus extending the prompt 1 2 3 4 5 6 7 8 9 10 the i'th prompt another relevant feature is the concreteness score, as one can turn a sentence to be more accurate by changing the existing words to more concrete ones rather than adding new ones. our results, however, show that the difference between the concreteness scores is not statistically significant. ## convergence patterns in the previous section, we provided two explanations for the observed adaptation process. in this section, we further investigate the direction and destination of the adaptation. for each feature f we split the threads into two sets by their first and last values: for example, for the length feature we have one set with threads that get longer, i.e., their last prompt is longer than their first prompt. in the other set, we have threads that are getting shorter, their last prompt is shorter than their first prompt. we see in figure 4 that the prompts that get longer start shorter, and that the prompts that get shorter start longer: both sets converge towards similar lengths. it may suggest that there is a specific range of "good" prompt lengths, not related to the starting point, and that human users converge to it, increasing or decreasing the length of the prompt adaptively depending on where they started. we observe similar trends in some of the other features too (see app. §h). ## discussion in section 7, we examined two explanations for the observed systematic adaptation process. the second option, that the model causes users to "drift" towards its preferences, raises concerns about naïvely using human data for training. human data (compared to synthetic data @xcite ) is often perceived as most suitable for training. the midjourney data can be used as rlhf data, sampling from each thread one image that was upscaled and one that was not, coupled with the upscaled prompt. however, our findings that the human users likely adapt to the model call for caution, as we may inadvertently push models by uncritically using user data to prefer the adapted prompts even more. we hope to draw attention to the linguistic adaptation process human users go through when they interact with a model. future work will empirically examine the effect of training with such data, and will expand the discussion on interactions with general language models. ## related work text-to-image prompt engineering was studied by a handful of works. @xcite identified prompt patterns used by the community, pavlichenko and ustalov (2022) examined the effect of specific keywords, and lovering and pavlick (2023) studied at the effect of subject-verb-object frequencies on the generated image. other works try to improve prompts by creating design guidelines @xcite , automatically optimizing the prompts @xcite or suggest prompt ideas to the user @xcite . the closest to ours is @xcite , an analysis of large-scale prompt logs collected from multiple text-to-image systems. they do refer to "prompt sessions", which they identify with a 30minute timeout. however, they do not split the prompts by scene, nor examine the dynamics of certain linguistic features changing along the interaction. diffusiondb @xcite is a text-toimage dataset, containing prompts and images generated by stable diffusion @xcite . it does not contain indications as to whether the user upscaled the image or not, and is therefore not suitable for our purposes. another existing text-to-image dataset is simulacra aesthetic captions @xcite , containing prompts, images and ratings. it does not contain meta-data such as user-id or timestamps which make it unsuitable for our purposes. another resource is the midjourney 2022 data from kaggle. 6 . it contains raw data from the midjourney discord between june 20 and july 17, 2022. we did not use it but scraped the data by ourselves, to obtain a larger and more recent dataset of prompts and of a newer version of midjourney.