Paper: ArtWhisperer: A Dataset for Characterizing Human-AI Interactions in Artistic Creations (ICLR-2024)

Reviewer comment (Experiments):
There may have been returning users on a day-to-day basis, so it’s feasible that a user became more experienced with prompting over time, which would thus affect steerability. The authors however did not collect this type of data, so it’s not possible to observe this.

Author response:
As you point out, we were unfortunately unable to collect long term data on repeated user interactions. In the controlled user study, we did collect interactions across a number of tasks. From this data, we did not find a significant increase in steerability (after correcting for steerability differences across images). While intuitively we do believe steerability should increase over time, our dataset suggests that this increase is more subtle. We suspect additional feedback given to the user (e.g., instructions to improve / correct their submitted prompts) may more significantly increase steerability, but we did not test this hypothesis here.

Reviewer comment (Experiments):
Do the authors have any sense for how user behavior may differ with other t2i models? For example, does the quality of the model (i.e., its ability to generate high-quality imagery) affect the results?

Author response:
Our intuition is that model quality is a significant factor in steerability. In particular, higher quality image output increases steerability (up to a point, as image quality is not the only factor in steerability). 
  - This is easily seen in the following extreme case – a user who does not look at the model output (i.e., the model output quality is low) can still steer the model through random changes to the prompt – whether the change improves or decreases the score is seemingly random, however, limiting steerability (this assumes that the model encodes the user prompt in the image, despite its low visual quality; if this does not happen, the scoring will be even more random).

Reviewer comment (Novelty):
It's unclear to me on how we could utilize the study done in this paper to improve research in text-to-image generation modeling, or Human AI interaction research. I saw some direction from the conclusion but those can be drawn without studying the research presented in this paper.

Author response:
There are multiple uses for our dataset and data collection procedure as discussed throughout the paper and discussion sections:
1. Better understanding of user prompting strategies and diversity of user prompts
    - To our knowledge, this is the first public dataset capturing how users iterate on their prompt designs for text to image models. This dataset lets us study how people perform this iteration and how we might improve models to accommodate human usage (e.g., see Sections 2, 3 and Figures 2,4,6). 
        * For example, we find using our dataset that users arrive at diverse prompts for the same image – they do not converge to the same image description (Section 3 and Figure 6).
        * As another example, we also find evidence using our dataset that there is a distinct style each user has (though the effect of style on steerability may depend more on specific image) (Section 3 and Figure 6).
2. Quantifying model alignment to users in a rigorous way through a steerability metric.
    - We use our dataset as the basis for defining a steerability metric, and propose our data collection procedure (or an analog with simulated users) can be used to assess models that require human interaction. (Section 4, Figure 7)
3. Synthetic user agents
    - As we describe in the discussion section, we believe an important use case of our dataset is to create synthetic users. We test and validate this approach where we fine-tune an MT0-large model to replicate a human prompter (using numbering from the revision: Appendix A.14, Figures 21,22,23). Creating synthetic users that *employ real human prompting strategies / vocabulary* has a clear benefit to model training: we can incorporate this into loss metrics to increase alignment between models and human users across an interaction.

Reviewer comment (Writing):
What is IRB approval?

Author response:
IRB approval is a standard process for ensuring ethical research when involving human subjects.

Reviewer comment (Experiments):
It would be better to be clearer about the controlled collection earlier, simply paid and unpaid would avoid assumptions of how this was performed.

Author response:
We have clarified the two datasets better in the revision (see Section 2.3).

Reviewer comment (Evaluation):
The authors are missing a comparison of how true the distribution of generated images is to the original images as the Wikipedia captions in general don't describe the content so highly likely they do not match the original image. A quantitative evaluation of this is important otherwise it makes the choice of Wikipedia not relevant and does not increase validity over using any source.

Author response:
Comparison of the generated images from the target prompt with the real Wikipedia images is added in the Appendix. We also added a sentence clarifying that the metric we use is calibrated such that using the Wikipedia caption gives a perfect score for the Wikipedia image (this calibration is also done for generated AI images).  
- The reason for using Wikipedia images is to collect and compare the steerability of a broad variety of images – both AI art as well as real art (e.g., paintings or sculptures) and other natural images (cities, wildlife, etc.).

Reviewer comment (Experiments):
Lack of discussion on participant characteristics how was gender, race & ethnicity balanced?

Author response:
Participant demographics were intentionally not collected to avoid identifying users.  
    - In the controlled user study, demographics were collected and a supplemental figure has been added. Users were sampled through Prolific to a fixed 50/50 split of male and female users. In the revision, we have included histograms of the age, sex, and ethnicity distributions in Figure 11 in the Appendix (A.6). We did not find significant differences in steerability across these demographics.

Reviewer comment (Experiments):
Steerability contribution is well received, however, this would have more validity if was tested on a different model to confirm the insights, two versions of StableDiffusion provide limited insight.

Author response:
While comparison with additional models would be nice, we believe our results are still a valuable contribution to the community.  
    - The two stable diffusion models we use are in fact quite different and produce differing results given the same prompt (note, for example, we needed to re-select the user seed for the second model to ensure the model produced a similar enough image when using the target prompt; using the same seed as the original model could result in vastly different images). So, we believe the comparison we provide is interesting in itself.  
    - Moreover, it also provides evidence of differences in steerability between two models and that we can measure these differences using our data collection method and proposed steerability metric.

Reviewer comment (Evaluation):
How do the results on steerability differ between the two input datasets?

Author response:
Steerability results between Wikipedia and AI art datasets are compared in Figure 7. Note the differences between the “AI Art” (AI Art dataset) and “Real Images” (Wikipedia dataset) indicate that real images are more steerable.

Reviewer comment (Experiments):
Does the influence of unpaid/paid effect the outcomes?

Author response:
We did not find any significant differences between paid and unpaid users.

Reviewer comment (Experiments):
Prompt engineering is now a topic of interest for people who wish to efficiently generate usable content from conditional generative models. My hypothesis is that, “Steerability”, i.e., the ease of generating the target image, depends on the amount of experience of the user. It would be great if the authors discuss how they might be able to decouple the influence of returning users who potentially have the ability to efficiently “steer” the generative model via better prompts.

Author response:
Influence of returning users on steerability
  - In the controlled user study, we did not observe a significant change in the model’s steerability (controlling for target image) over time. This suggests that user experience did not have a large impact on steerability.
  - Our work provides insight into only the immediate steerability of a model, and provides evidence that absent specific feedback (e.g., instructions on how to steer a model), people do not improve steerability very quickly (within 5 examples). It might be possible that a much longer experience can improve steerability.

Reviewer comment (Evaluation):
Apart from the prompting experience, a person’s knowledge of the contents of the target image could also play a role in steerability. Especially in cases of a famous person, famous landmark, famous city, etc. There appear to be some person-specific confounding factors that affect steerability, which the paper doesn’t seem to address. It appears that the rationale for this is that, with a large enough population of users, such person-specific effects might be negligible. However, it would be nice if these confounding factors were explicitly identified and discussed.

Author response:
Knowledge of an image subject improves steerability
  - This is partially due to the prompt supplied by the user (the user can provide a more specific prompt) and partially due to limitations of our image similarity metric (in some cases, the metric can be weighted to the subject matter, and referencing the correct subject in the prompt can lead to a higher steerability score). A figure (Figure 18 in the Appendix) was added to the revision. It shows a significant difference when the user is familiar with the subject of the target image (defined as the user referencing the key subject in at least one of the prompts they submit).

Reviewer comment (Evaluation):
Were there comments / feedback from users that elicited insight regarding their strategy? Specifically regarding the incongruity between human similarity notions and automatically computed similarity score? E.g., did certain persons try to maximise the automatic score while others maximised their own perception of similarity?

Author response:
People were instructed to and did attempt to maximize the synthetic score. We did not collect separate comments from users.

Reviewer comment (Experiments):
Was there an incentive for high score? E.g., additional reward? Do we know that people were taking the task “seriously”, and not just experimenting / exploring?

Author response:
As discussed in Appendix A.4, users in the controlled user study were given a bonus for achieving higher scores.
      - Users generally took the task seriously. A prefiltering step was used that removed data where a user did not achieve beyond a certain minimum score (these were identified as cases where a user was simply exploring and not trying to complete the task)

Reviewer comment (Evaluation):
What is the score distribution between each generated image (and the target image) — that are generated from the same prompt? Is there any insight into the variance in matching score across different random seeds of the model (keeping the prompt constant)?

Author response:
As variance of image output (for a given target prompt) increases, so does the steerability score (larger steerability score corresponds to more difficulty in steering the model). In other words, if repeatedly sampling a model with the same prompt can produce a wide range of outputs (high variance), then the model is likely less steerable for the content in those generated images. We added a figure (Figure 17 in the Appendix) and a brief discussion on this point in the Appendix of the revision.
  - The variance largely depends on the image – for some images it is quite large while for others it can be quite small. This is the reason for our procedure to select the target image to show users and the fixed seed we provide for users. The target image and seed are selected to minimize the difference between the target image and other images generated using different seeds with the same target prompt.

Reviewer comment (Writing):
“The intuition here is that t*1_i is more representative of the types of images we may expect given the fixed prompt, p*_i .” — the intuition here is unfortunately not clear to this reader. Could the authors please elaborate?

Author response:
The intuition here (why $t_{i_1^*}$ is more representative) is the following. $t_{i_1^*}$ is the target image closest to the center (using median) of the set of candidate target images (using a fixed target prompt). While the model may produce a wide range of images for the given target prompt, we pick the target image that is the “average” one (using median instead of mean to reduce the effect of outliers). By picking this “average” image, we also hope to reduce the difference of images generated using similar prompts. We have clarified this in the revision.