Paper: Quality-Diversity through AI Feedback (ICLR-2024)

Reviewer comment (Presentation):
The presentation, particularly in the experimental section, could benefit from clarification and better organization to enhance readability and comprehension.

Author response:
We have uploaded a new draft of the paper which we hope greatly improves the readability and structure. In particular, Section 4.2 shows changes in blue text with improvements to clarity in the experimental setup, and a correction to a section reference in the baseline description of **Random-Search**

Reviewer comment (Experiments):
The paper occasionally employs exaggerated language and makes promising claims that seem to lack sufficient empirical backing. For example, the paper states “providing a recipe that seemingly generalizes to many domains and modalities” and “it is often easier for a model to evaluate the quality of a generation than to generate the same high-quality text.” These claims would be more convincing if supported by concrete evidence or reference or discussion.

Author response:
We take this concern seriously and take a moment to clarify on the two highlighted points to resolve the misunderstanding. On the quote “providing a recipe that seemingly generalizes to many domains and modalities”, we highlight in the conclusion through reference that AI feedback can be done with any model that has been instruction-tuned such as the AI feedback model used in QDAIF. We cited in the conclusion Liu et al., 2023, a work that introduces instruction tuning for multimodal text-image understanding models. Instruction tuning enables this model to solve tasks in image and text domains, including solution evaluation - it is interesting future work for example to apply AI feedback to images with the visual instruction-tuned model (or another similar image-text model such as GPT-4+V) to evolve images and creative art, that can be described by AI feedback on the new domain of images, and generate refinements of images.

On the quote “it is often easier for a model to evaluate the quality of a generation than to generate the same high-quality text”, we cited Saunders et al., 2022, to refer to the generation-discrimination gap described by the quote. We also cited this part of their work in Section 4.4 in the original manuscript (now in A.18), but realized that we forgot to include page number 12 in the first mention of this reference (which we did in Section 4.4 originally), containing the reference to generation-discrimination gap which gives context to our results. The page number is added now to the first mention of the reference. We apologize for the lack of clarity, and hope the added detail will prevent future readers from sharing the same confusion.

Reviewer comment (Presentation):
A separate conclusion section, summarizing the key findings and contributions, would be advantageous for providing clear takeaways for the readers.

Author response:
Thank you for sharing this suggestion and for thinking of ways to improve the presentation of QDAIF. We added to the first paragraph of the conclusion a brief statement on key takeaways from results for our draft revision, to support the communication of the contributions messaging in the final paragraph of the conclusion.

Reviewer comment (Evaluation):
QD metric is used throughout, which is the “sum of highest quality value found in each bin” - it seems to only focus on quality rather than diversity. For readers not familiar with QD metrics, some explanation/justification of why QD measures Quality and Diversity will be helpful.

Author response:
>QD metric is used throughout, which is the “sum of highest quality value found in each bin” - it seems to only focus on quality rather than diversity. For readers not familiar with QD metrics, some explanation/justification of why QD measures Quality and Diversity will be helpful

In many domains you want to get a wide range of high-quality diverse artifacts (the motivating case as described in the introduction); the QD score [1] is one popular measure used in the literature. QD score increases are possible in two ways - by filling a previously empty bin with a solution of that kind (i.e. finding a novel solution, with improvement to the overall archive by the amount of quality improvement introduced from the addition of this new solution of a particular quality value), or by improving over an existing solution in a filled, non-empty bin with an evolved solution that is evaluated to be of this kind/bin, but has a higher quality score, as described in the method overview in Section 3 regarding the acceptance of evaluations to the archive. Coverage of the archive with new, diverse solutions, is, therefore, a key contributor to improvements in QD score (which we highlight in new additional stats including coverage in the added A.7 section, referenced in figure 3).

Reviewer comment (Evaluation):
Table 1: there is a lack of explanation of the quality metrics. For example, what is the difference between human QD score and quality rating? Is quality rating from humans?

Author response:
> Table 1: there is a lack of explanation of the quality metrics. For example, what is the difference between human QD score and quality rating? Is quality rating from humans?

Yes, quality rating is from humans. Thank you for sharing your concern on readability here. We provide full details on the human evaluation study referenced in the Table 1 results in Appendix A.1 and reference it in the “Evaluation” paragraph of Section 4.1. We added minor clarification and further self-contained referencing of A.1 in the revision for improved clarity.

Reviewer comment (Writing):
On page 5, section quantifying performance and diversity, it is unclear where the probability comes from in the sentence “the solutions’ quality estimate is derived from the logarithm of the probability of the LM’s answer” … Please clarify.

Author response:
> On page 5, section quantifying performance and diversity, it is unclear where the probability comes from in the sentence “ the solutions’ quality estimate is derived from the logarithm of the probability of the LM’s answer” … Please clarify

As part of the standard AI feedback method, the quality and diversity prediction values are obtained from LM next-token-prediction log probabilities, where we assess the probability of the AI feedback LM in predicting for a given text/story one label/token(s) (e.g. “horror”) versus another label/token(s) (e.g. “romance”). We quote in the section referred here in the paper, “The log probability of these responses serves as our measure of solution diversity” for context, applying the same approach to quality AI feedback. We slightly reworded a part of this sentence, highlighted in blue text. Thank for raising possible suggestions to improve the readability of the paper.

Reviewer comment (Evaluation):
Figure 3 illustrates the differential performance of various methods on different generation tasks. Is there any qualitative difference in terms of the QD score difference of less than one point?

Author response:
> Figure 3 illustrates the differential performance of various methods on different generation tasks. Is there any qualitative difference in terms of the QD score difference of less than one point

Yes, differences in QD score performance of less than one point can lead to notable qualitative differences in observed solutions, depending on the context of the results of the search. Taking the example from qualitative samples shown in Figure 1, when looking at the same bin between QDAIF and the baseline (e.g. the central bin, where the baseline contains a low-quality solution with a red cell/bin with blue arrow pointing to the cell from a text box on the right of the baseline archive, and QDAIF a high-quality solution with a bright cell in the same bin location), the differential is based on quality scores. In this particular example, where the difference in quality score between is less than one, we can observe this from the quote (in the revised sentence with minor correction on figure object location): “The baseline produced a story (right-middle position, starting with "Jason") with a lower quality score due to the lack of a desired spy character (denoted by the red-colored bin, for a story with a neutral ending, and leaning to horror). QDAIF discovered a better, more-relevant story (bottom-middle position, starting with "a wealthy politician") for this same neutral bin”. We can see that if both methods find solutions of similar quality across most (of the same) bins except for one bin where one method finds the higher quality solution (e.g. a better horror story), then the QD score differential observed would be less than one, but means a qualitative (subjectively higher quality) solution if the method can improve on the quality of this genre of story/solution. A differential of less than one can also occur if a method finds a solution in one more previously unfilled bin (that would be still empty in the result of a different method in this case) - the QD score improvement introduced by filling this empty bin with a new solution would virtually always be less than one, the maximum possible quality score achievable in this setup of story-writing. Just finding a previously unknown solution in a region of the archive that's further away in the extreme corner (cf. Figure 1) can lead to noticeable qualitative differences in solutions (e.g. the horror story from QDAIF in the top-left text box with a tragic ending has more horror elements, with reference to a deadly monster, compared to the closest bin story found by a baseline shown in the top-right text box, which only shows a character's tragic ending via gun violence, without traditional horror elements in the text).

While deeper qualitative review of solutions on a case-by-case basis is needed to understand the situational differences in qualitative results between runs, We hope that this intuition of what influences QD score and how it could affect qualitative analysis brings more clarity to this question.

Reviewer comment (Experiments):
The lack of strong baselines

Author response:
Thank you for sharing your feedback on the importance of detailed and informative baseline comparisons. We agree with this statement, and revised the draft with additional baseline comparisons, introducing new baselines from prior works (on top of the LMX, Quality-Only baseline introduced in the original manuscript, and studied in the LMX, or Language Model Crossover paper [1]) and detailing performance comparison results with QDAIF in Appendix A.8, with detailed implementation details of added diversity-seeking baselines in A.9. In addition to further demonstrations of performance improvements by QDAIF compared to baselines, the new comparisons highlight the importance of each component of QDAIF in improving the QD score performance during search, with new insights on the value of diversity-seeking elements of relevant search algorithms, often outperforming the baseline which optimizes for quality in solutions only (LMX, Quality-Only).

Furthermore, as highlighted in the general response, we introduced a new variant of QDAIF for Poetry, LMX-rewrite (in edited Section 4.4), which becomes a fair comparison to the Random-Poems baseline, where both methods do not request for solutions of specific genres and tones in poetry, but LMX-rewrite adds the rewriting mutation and evolution of archive bin solutions to improve over Random-Poems. This experiment improvement also enabled us to conduct an ablation with the method Fixed Seed Rewrite, which reveals that the element of evolving an improving population with QDAIF (in addition to rewriting mutation) is important for significant performance improvements with QDAIF. 

Given the improvements to the experimental soundness with sophisticated baselines, and a showcasing of the strengths and weaknesses of baselines, we have shown performance gains from QDAIF to be meaningful (with improved insights on the contributions of components of QDAIF), in line with seminal works in the literature which also compare to baselines (such as Random-Search, as well as LMX, Quality-Only for single objective quality/fitness optimization), in the experiments of the Novelty Search [2] and MAP-Elites [3] papers.

Reviewer comment (Experiments):
Could you add more baselines and tasks?

Author response:
Thank you for sharing your feedback on the importance of detailed and informative baseline comparisons. We agree with this statement, and revised the draft with additional baseline comparisons, introducing new baselines from prior works (on top of the LMX, Quality-Only baseline introduced in the original manuscript, and studied in the LMX, or Language Model Crossover paper [1]) and detailing performance comparison results with QDAIF in Appendix A.8, with detailed implementation details of added diversity-seeking baselines in A.9. In addition to further demonstrations of performance improvements by QDAIF compared to baselines, the new comparisons highlight the importance of each component of QDAIF in improving the QD score performance during search, with new insights on the value of diversity-seeking elements of relevant search algorithms, often outperforming the baseline which optimizes for quality in solutions only (LMX, Quality-Only).

Furthermore, as highlighted in the general response, we introduced a new variant of QDAIF for Poetry, LMX-rewrite (in edited Section 4.4), which becomes a fair comparison to the Random-Poems baseline, where both methods do not request for solutions of specific genres and tones in poetry, but LMX-rewrite adds the rewriting mutation and evolution of archive bin solutions to improve over Random-Poems. This experiment improvement also enabled us to conduct an ablation with the method Fixed Seed Rewrite, which reveals that the element of evolving an improving population with QDAIF (in addition to rewriting mutation) is important for significant performance improvements with QDAIF. 

Given the improvements to the experimental soundness with sophisticated baselines, and a showcasing of the strengths and weaknesses of baselines, we have shown performance gains from QDAIF to be meaningful (with improved insights on the contributions of components of QDAIF), in line with seminal works in the literature which also compare to baselines (such as Random-Search, as well as LMX, Quality-Only for single objective quality/fitness optimization), in the experiments of the Novelty Search [2] and MAP-Elites [3] papers.

Thank you for your interest in the applicability of QDAIF to other domains in addition to creative writing such as the existing **Opinions**, **Stories**, and **Poetry** domains.

Firstly, the **Opinions** domain seems to be relevant to the idea of topic writing, which is highlighted in the review here in the paper explanation. The topic of eating vegetables and plant-based foods would likely be relevant here.

Still, we take the suggestion on the value of studying QDAIF in different kinds of domains into account, and added results of QDAIF applied to a new (non-creative-writing) code-writing domain in Appendix A.20. We saw marked improvement in the diversity and quality of generated code from QDAIF (LMX-rewrite) for the task of implementing sorting algorithms to solve a specified problem, compared to the Random-Code baseline.

We believe that results here also demonstrate the flexibility and generality of QDAIF, in being helpful to creative search in another, more-technical domain.

Reviewer comment (Theory):
QDAIF still requires researchers to define the axes of diversity they are most interested in, which may limit its autonomy in creative search.

Author response:
Thank you for raising this interesting discussion point following the presented statements on the limitations of QD methods in the final section of the manuscript. We added Appendix A.10 which investigates and tackles the concern raised here through additional experiments and literature review, and referenced A.10 in the conclusion section on the limitations discussion.

Firstly, to give more context from prior works in the literature (as we described in the context of the discussion on limitations since the original manuscript), most existing algorithms since the first introduction of QD search have not yet studied approaches (if at all) to alleviate the need to define the axes of diversity, especially ones that would likely be most interesting to users in a creative search. In A.10, we added references for an existing possible direction of defining higher dimensions of diversity via unsupervised diversity representation learning. Here we noted limitations of this approach in the case where for our desired use cases, we *do* want to leverage the general knowledge of assessing all types of creative writing from foundation models to inspire creative QDAIF search; we can then exploit LM domain knowledge for expanding on possible output text solutions where the distinctions in diverse, high-quality texts can be subjectively appreciated and informatively cataloged.

We describe in A.10 an approach where such domain knowledge for automating the initialization or expansion of diversity axes can be explored for more diverse, high-quality texts. With recent advances in LM capabilities (e.g. GPT-4), we can specify the search problem or domain of interest, and ask it, for example, to directly generate ideas for plausible new diversity axes to explore, or even directly generate the AI feedback prompts which may be used in the existing QDAIF setups, maybe following the structure and format of say the feedback prompts we tested.

The question remains - how effectively would QD score performance (the measure of quality and diversity in archive solutions) improve if one tested such a pipeline, and continued search with the introduction of additional diversity axes over time.

We demonstrated in A.10 results that applying dimension expansion on the **Stories** domain as an example is effective in enabling the improvement of QD score in higher dimensions of diversity, surprisingly even providing gains in “Best Solution Quality” (mentioned in general response).

Overall, new results that we found and added bring us closer to the potential of automatically defining diversity axes, proving practical performance gains to be had.

Reviewer comment (Novelty):
Could we have a detailed comparison with other AI feedback methods or discuss how QDAIF specifically addresses their limitations?

Author response:
Yes, we highlight the relevance of AI feedback (AIF) methods in the context of QDAIF in the original manuscript in Section 2.3, and added minor clarifications in this section in the revised draft, providing more clarity on how QDAIF leverages AIF in a new setting to introduce a new kind of QD algorithm, enabling other recent advancements in QD research studied in different domains to be applied on top of the high-level QDAIF approach.

We show with this clarification that QDAIF is a complementary direction of research that takes inspiration from works applying AIF such as Constitutional AI [1] and Self-Refine [2], and their proven results in utilizing AI feedback for qualitative text quality assessment (which led to LM refinements being able to improve text solutions), to apply AIF to a new context towards tackling QD problems (adding the diversity and evolution aspects for a new application in QD).

We hope that this brings a clearer picture of the differences between QDAIF and other AI feedback methods, where QDAIF enables QD approaches in new domains through qualitative AI feedback.

Reviewer comment (Experiments):
The generalizability of QDAIF to other domains and tasks beyond creative writing is not extensively discussed.

Author response:
We added Appendix A.20, referenced in Section 4.4, to show the results of QDAIF applied to coding problems, which is not focused on creative writing, and compared to the Random-Code baseline.

We are encouraged that you highlighted the importance of applying QDAIF to highly-subjective, qualitative domains, where no viable measures except for AI feedback (the alternative to expensive human feedback) exist, unlike the domains studied by QD works in prior work. Interestingly, we show in new results that AI feedback’s ability to bring evaluation to qualitative domains is also effective for code-writing tasks, where QDAIF generated qualitatively more diverse sorting algorithms (from first principles) with higher quality scores obtained, and overall higher QD score than that of the baseline (that returned nearly always one particular type of sorting algorithm, 95% of the time).

We appreciate the interest of the reviewer in QDAIF when applied to domains beyond creative writing. We further highlight the generalizability of QDAIF with this finding.

Reviewer comment (Experiments):
Can you provide more insight into the scalability and computational efficiency of QDAIF in more complex and large-scale tasks?

Author response:
The scalability and computational efficiency of QDAIF both hinge on the capabilities and sizes of available existing LMs (which carry out both the mutation of texts, and the evaluation of text solutions defined by AI feedback prompts).

We refer to an additional analysis carried out and described since the original manuscript version to study the effect of the generator/mutator LM’s model size in Section 4.3, on “LMX Model Size”. Results summarize likely gains in scalability, where discovered samples of creative writing solutions found by QDAIF (with larger model sizes as generator LMs) obtained higher human feedback scores for quality compared to their ratings of samples from smaller LMs.

Furthermore, our additional analysis on the qualitative sample poems in the **Poetry** domain in the revised manuscript, Appendix A.18 (in particular, Figures 21 and 22) suggests potential improvements in meaningful mutations of texts towards more diverse, higher-quality texts when running QDAIF with larger, more capable models such as GPT-4 compared to GPT-3.5 Turbo. While QDAIF benefits from more search iterations (meaning more LM generation and evaluation calls via label/attribute prediction), it still shows QD score performance sample efficiency improvements over existing baselines, hinting at the potential scalability of QDAIF that can improve with advancements to available LMs.

Reviewer comment (Experiments):
How does the proposed QDAIF approach perform in other domains and tasks beyond creative writing?

Author response:
We added Appendix A.20, referenced in Section 4.4, to show the results of QDAIF applied to coding problems, which is not focused on creative writing, and compared to the Random-Code baseline.

We are encouraged that you highlighted the importance of applying QDAIF to highly-subjective, qualitative domains, where no viable measures except for AI feedback (the alternative to expensive human feedback) exist, unlike the domains studied by QD works in prior work. Interestingly, we show in new results that AI feedback’s ability to bring evaluation to qualitative domains is also effective for code-writing tasks, where QDAIF generated qualitatively more diverse sorting algorithms (from first principles) with higher quality scores obtained, and overall higher QD score than that of the baseline (that returned nearly always one particular type of sorting algorithm, 95% of the time).

We appreciate the interest of the reviewer in QDAIF when applied to domains beyond creative writing. We further highlight the generalizability of QDAIF with this finding.