Paper: RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICLR-2024)

Reviewer comment (Experiments):
The tasks are confined to summarization and dialog generation. Exploring additional areas like QA, code generation, or translation could provide a more comprehensive understanding of AI and human feedback interactions.

Author response:
We conduct experiments on two more tasks to further strengthen our findings - specifically on question/instruction answering and harmless dialogue generation. For question/instruction answering, we find that RLAIF and RLHF both outperform the SFT model with 54% win rates (statistically significantly different from 50%). For harmless dialogue generation, we find that the RLAIF and RLHF outperform the supervised baseline, with SFT, RLHF, and RLAIF responding with harmless replies 64%, 76%, and 88% of the time, respectively. Across all 4 tasks, we find that RLAIF achieves comparable results to RLHF.

We acknowledge that there are many more tasks that could be investigated, and we believe that the positive results on these four tasks provide a promising indication that RLAIF can achieve performance comparable to RLHF.

Reviewer comment (Evaluation):
The qualitative analysis in Section 5 is relatively shallow and would benefit from additional justification: (1) providing more conclusive evidence that the observed trends exist, for example with human labelers; and (2) offering hypotheses as to why these trends occur. Given that the paper notes only 78% agreement between human and AI labelers, further effort could be put into distinguishing between the labels used to train RLHF and RLAIF, which might elucidate downstream differences in trained model behavior.

Author response:
In response to your concerns regarding the qualitative analysis, we have implemented three changes.

First, we have initiated human evaluation to assess whether the observed trends exist at scale. If we find evidence of the trends, we will provide hypotheses as to why these trends occur. The results of this evaluation will be incorporated into the revised Section 5. 

Second, to reflect the explorative nature of this analysis, we have renamed Section 5 from "Qualitative Analysis" to "Qualitative Observations." This emphasizes that the findings represent observations rather than definitive patterns.

Reviewer comment (Novelty):
A minor note is that the paper is titled "RLAIF" despite not being the first work to use or introduce the term. For example, Bai et al. 2022 ("Constitutional AI") used the term earlier. It is recommended that the authors remove the title and simply use the subtitle.

Author response:
To make it explicit that we did not introduce RLAIF, we have amended the Abstract to directly credit Bai et al.. In the second sentence of the abstract, we write, “RL from AI Feedback (RLAIF) is an alternative solution introduced by Bai et al. that generates preferences using an off-the-shelf LLM in lieu of human annotators.”

Given that our work is the first to study RLAIF in such depth, we believe that a fair compromise is to leave “RLAIF” in the title while updating the abstract to make it abundantly clear that RLAIF was first introduced elsewhere.

Reviewer comment (Experiments):
In Figure4, it's not clear why adding exemplars hurts performance. There is a one sentence justification for this on page6, but it is insufficient, since exemplars hurt performance even without CoT. Is it the case that the exemplars are low quality? Could this be mitigated by using the 0-shot generations as exemplars? It would also be valuable to understand the role of exemplars at different labeler sizes (e.g. P2-S using exemplars produced by P2-L).

Author response:
__The Effect of In-Context Learning on AI Labeler Alignment__

We have also conducted experiments on harmless dialogue generation (see last part of reply), where we found that adding exemplars increases AI Labeler Alignment. Our observations suggest that the optimal prompting techniques are task-specific.

We do not believe that low-quality exemplars are to blame for ICL not working. For all tasks, we carefully handpicked high quality exemplars representative of the preference task. We also measured alignment for "Base 1-shot" on summarization using 10 random exemplars, yielding a maximum/minimum alignment of 76.1%/74.4%, none surpassing the 76.1% alignment of "Base 0-shot". We will add these details to the paper.

Interestingly, existing literature suggests that the role of exemplars and chain-of-thought prompting is still not fully understood (e.g., [1] and [2] show that corrupting exemplars and chain-of-thought surprisingly yield performance gains). We agree that deeper understanding of exemplars is an important research area.

[1] Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? - https://aclanthology.org/2022.emnlp-main.759.pdf

[2] Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters - https://arxiv.org/pdf/2212.10001.pdf 

## Rebuttal 2 (ID: p49fAJb9pp)
We have updated our paper and would like to point out the changes most relevant to your feedback:

1) Section 4.4 "Prompting Techniques" - we share evidence for why we believe low-quality exemplars are not to blame for ICL not improving results

Reviewer comment (Evaluation):
Regarding Figure4, since human annotators have a 60-75% agreement rate on these datasets, it is unclear whether small differences in accuracy in Table4 are meaningful. Is it possible that the AI labels are more correct than the human preferences? Analyzing the disagreements may shed light on this. If so, what does it imply about the results/takeaway in Figure4?

Author response:
__Whether Differences in AI Labeler Alignment are Meaningful__

We studied the sensitivity of RL-trained policies to the AI labeler accuracy, revealing that a small disparity can lead to a significant difference in win rates. Specifically, a policy trained with preference labels achieving 78% AI Labeler Alignment outperforms one trained with preference labels achieving 76% alignment, securing a win rate of 59% (statistically significantly above 50%). We will report these results in Section 4.6 "End-to-End Sensitivity to AI Labeler Alignment”.

While we cannot definitively conclude that higher AI Labeler Alignment consistently translates to superior RL policies from this one experiment, this result suggests that such a correlation at least exists in certain cases. Due to the expensive nature of conducting these experiments and human evaluation, we agree that further investigations are necessary and defer more comprehensive study to future work.

## Rebuttal 2 (ID: p49fAJb9pp)
We have updated our paper and would like to point out the changes most relevant to your feedback:

2) Appendix N "End-to-end Sensitivity to AI Labeler Alignment" - we report end-to-end results on how differences in AI labeler alignment translate to downstream changes in performance

Reviewer comment (Evaluation):
It would be interesting to see the relationship between Figure4 and Figure5b. Does a higher quality AI-preference dataset necessarily lead to a more accurate RM?

Author response:
__AI Labeler Alignment vs. RM Accuracy__

We acknowledge the importance of thoroughly understanding the relationship between AI Labeler Alignment and RM accuracy. To address this concern, in the revised paper, we will include the RM accuracies for the two RMs trained on different AI labels in Section 4.6 "End-to-End Sensitivity to AI Labeler Alignment.”

We also would like to note that training and evaluation for all RMs are done in the exact same way. 

## Rebuttal 2 (ID: p49fAJb9pp)
We have updated our paper and would like to point out the changes most relevant to your feedback:

3) Appendix H "Reward Model Accuracy" - we report the RM accuracies of all RMs used in RLAIF and RLHF across the paper. This includes two RMs trained from labels with different degrees of AI labeler alignment. We do not find a strong relationship between AI labeler alignment and RM accuracy

Reviewer comment (Evaluation):
The human evaluators used to assess the final RLHF/RLAIF/SFT-trained LLMs are distinct from the preference data/RM. How can we know if these annotators are modeling the same preference policy expressed by the datasets/RMs, and not, for example, just picking the longest response? To this end, it would be good to either show RM scores for the final LLMs or to measure the agreement of the human evaluators against the original preference data or the RMs.

Author response:
__Is Human Eval Aligned with Preference Datasets?__

We acknowledge the potential for discrepancies between the human raters who curated the preference datasets and the individuals we engaged for evaluation. Additionally, differences in training or instruction may introduce further variations. To minimize the impact of such mismatches, we meticulously replicated the settings provided in the original papers. Also, as you pointed out, it is possible that the AI labeler may provide more accurate assessments than the humans who initially rated the dataset in some cases.

Incorporating an agreement analysis between our human evaluators and the preferences in the preference datasets is an excellent suggestion. If we can afford additional human evaluation, we will integrate this analysis into the camera-ready version to further substantiate the value of our human annotators' feedback.

Regarding the possibility that annotators pick the longest response, our post-hoc analysis suggests that human evaluators continue to favor RLAIF and RLHF model outputs even after controlling for response length. This post-hoc analysis is similar to the approach employed by Stiennon et al. in their work "Learning to Summarize from Human Feedback" (https://arxiv.org/pdf/2009.01325.pdf).

## Rebuttal 2 (ID: p49fAJb9pp)
We have updated our paper and would like to point out the changes most relevant to your feedback:

4) Appendix J "Controlling for Response Length" - we report results after controlling for response length and find that evaluators are not simply picking the longest responses

Reviewer comment (Evaluation):
If I understand correctly, both RLAIF and RLHF are based on the SFT baseline (fine-tuned PaLM2 XS) and REINFORCE. In this case, the key difference among RLAIF and RLHF in the experiments is the used reward models and the training data for the reward models. Therefore questions raise: What is the accuracy of the finally trained Human Feedback RM? I only see in Appendix E that the RM “is trained until the training loss and accuracy curves plateau”, but in Figure 5(b) it is still not plateau. Having this number can help readers understand (1) if the on-par performance of RLAIF and RLHF is due to using RMs with similar accuracy.

Author response:
__Relationship Between AI Labeler Alignment, RM Accuracy, and Downstream RL Policy__

We acknowledge the importance of thoroughly understanding the relationship between RM accuracy and downstream RL results. To address this concern, in the revised paper, we will include the RM accuracies for both human feedback RMs and AI feedback RMs in Section 4.1. Additionally, we will report the RM accuracies for the two RMs trained on different AI labels in Section 4.6, titled "End-to-End Sensitivity to AI Labeler Alignment.”

While we recognize the importance of understanding the relationship between RM accuracy and downstream RL results, we believe this is a broad research question deserving a comprehensive study of its own. Additionally, the nature of RL fine-tuning and human evaluation demands substantial resources, making an in-depth exploration of this relationship outside the scope of our work. Nevertheless, we recognize the significance and potential of this research direction.

We would also like to emphasize that all RMs are trained following a standardized procedure. This consistency in training procedure of the RM does not favor any specific approach, allowing for a fair evaluation of the different methods employed.

Reviewer comment (Evaluation):
To further dig into the above question, analysis of how RM accuracy affects the RLHF results can also be helpful.

Author response:
__Relationship Between AI Labeler Alignment, RM Accuracy, and Downstream RL Policy__

We acknowledge the importance of thoroughly understanding the relationship between RM accuracy and downstream RL results. To address this concern, in the revised paper, we will include the RM accuracies for both human feedback RMs and AI feedback RMs in Section 4.1. Additionally, we will report the RM accuracies for the two RMs trained on different AI labels in Section 4.6, titled "End-to-End Sensitivity to AI Labeler Alignment.”

While we recognize the importance of understanding the relationship between RM accuracy and downstream RL results, we believe this is a broad research question deserving a comprehensive study of its own. Additionally, the nature of RL fine-tuning and human evaluation demands substantial resources, making an in-depth exploration of this relationship outside the scope of our work. Nevertheless, we recognize the significance and potential of this research direction.

We would also like to emphasize that all RMs are trained following a standardized procedure. This consistency in training procedure of the RM does not favor any specific approach, allowing for a fair evaluation of the different methods employed.

Reviewer comment (Evaluation):
About human evaluation. Since the reported number is only the total human rating, I’m curious about other statistics. How many input-outputs examples are used? How many evaluators rate the same input-outputs example? What’s the inter-annotator agreement of the results?

Author response:
__Human Evaluation Statistics__

To answer your questions about human evaluation, we constructed a set of 2k rating instances across all of our RL experiments, which includes evaluations for additional experiments that we are adding to the revised paper (see last section of this reply). Each instance comprised a single context and three distinct model responses (e.g., RLAIF, RLHF, SFT), resulting in a total of 6k unique input-output pairs subjected to human evaluation. Additionally, each instance was assessed by three independent raters, which we will use to calculate inter-annotator agreement. These details will be incorporated into Section 3.4 "Human Evaluation" of the revised paper to enhance the transparency of our evaluation methodology.

Reviewer comment (Experiments):
The experimental results after controlling the length in Appendix F only shows the comparison of RLAIF vs SFT and RLHF vs SFT. Is there also a comparison between RLAIF vs RLHF?

Author response:
__Length-Controlled Results__

You raise an excellent point about calculating the length-controlled results for RLHF vs. RLAIF directly. We will calculate this quantity and incorporate the result in Appendix F.

Reviewer comment (Experiments):
About the experimental setup: Is there a reason why the reward models are also initialized from the SFT models (described in Section 3.3)? Their output spaces are different. Is it a random try or based on some statistics?

Author response:
> Is there a reason why the reward models are also initialized from the SFT models (described in Section 3.3)? Their output spaces are different. Is it a random try or based on some statistics?

We initialize the RM from the SFT model because we have found that initializing the RM from the SFT model improves RM performance. The hypothesis behind this is that the RM must be able to adapt to the scoring task, and the SFT model adds more domain-specific training. This is also an observation shared by other researchers - see https://arxiv.org/pdf/2009.01325.pdf, https://notesonai.com/RLHF+-+Reinforcement+Learning+with+Human+Feedback, and https://huyenchip.com/2023/05/02/rlhf.html.

Reviewer comment (Experiments):
About the experimental setup: What is the baseline used for REINFORCE (mentioned as “we use REINFORCE with a baseline” in Section 3.3)? Is it the output of the value model in the authors’ setup?

Author response:
> What is the baseline used for REINFORCE (mentioned as “we use REINFORCE with a baseline” in Section 3.3)? Is it the output of the value model in the authors’ setup?

As you mentioned, we use the value function as the baseline in REINFORCE. See Appendix C for details.

Reviewer comment (Presentation):
Presentation suggestion: The paper describes in-context learning with exemplars (section 2.1) and self-consistency (section 2.1.3) as they are a part of the used methodology. However, in experiments (Figure 4), they turn out useless and not applied in the end. In this case, I would suggest not to put much emphasis on them but only mention them and say the authors also study their effects.

Author response:
Lastly, we will implement your recommendations on “Presentation suggestion” and fix the point regarding section 4.2. Thank you for pointing these out.

Reviewer comment (Presentation):
Presentation suggestion: Add that Figure 5 (a)(b) are results on summarization task in the caption.

Author response:
Lastly, we will implement your recommendations on “Presentation suggestion” and fix the point regarding section 4.2. Thank you for pointing these out.

Reviewer comment (Writing):
In section 4.2, “we observe that the optimal configuration employs chain-of-thought reasoning and no in-context learning (“Detailed + CoT 0-shot”)” Should here be “Detailed / Base + CoT 0-shot”, since for summarization the best is detailed and for helpfulness the best is base.

Author response:
Lastly, we will implement your recommendations on “Presentation suggestion” and fix the point regarding section 4.2. Thank you for pointing these out.