Paper: Understanding the Effects of RLHF on LLM Generalisation and Diversity (ICLR-2024)

Reviewer comment (Novelty):
The generalization capabilities offered by RLHF have been widely observed in state-of-the-art models, with clear comparisons and case studies of output from pretrained, instruction fine-tuned, and RLHF fine-tuned models (see, for example, the Llama-2 paper).

Author response:
While we believe our findings match intuition and anecdotal evidence about the characteristics of different fine-tuning methods for LLMs, we don’t believe our results have been shown in previous works. Could you please share the other works which show a similar result, so that we can include a comparison to them in our work and justify how our contributions are novel with respect to them? Note that In the “Generalisation and Diversity in NLP” paragraph in section 2 of the paper we discuss the differences with prior work both in terms of generalisation and diversity.

For the paper you did mention, we will address how our results are different. In terms of generalisation, could you be specific about the parts of the LLaMa 2 paper that you think show that RLHF generalises better than SFT? There are results showing the RLHF improves on SFT training on test sets that are effectively in-distribution, and results showing the performance of RLHF on various safety benchmarks, but neither of these provide evidence comparing RLHF and SFT’s behaviour OOD. Our results involve creating specific OOD test sets (with respect to the fine-tuning dataset) which compare how models perform in situations they haven’t seen during fine-tuning. **We created an entirely new test dataset, sequential instructions, for this purpose, and show significant drops in performance as the test distribution becomes more OOD utilising this test set, which hasn’t been shown before**. The LLaMa 2 paper is concerned with producing the best model possible, and doesn’t hold out specific test sets to measure OOD generalisation, and performs limited comparisons between SFT and RLHF models.

Even if our results are not necessarily surprising for practitioners frequently working with these models and methods, we believe it is important to demonstrate this empirically via thorough studies that others can later reproduce and build on. In addition, our studies emphasise the need for more research on methods that can strike a better balance between performance and diversity, thus opening up a new research avenue. To conclude, our core contributions consist of:
Demonstrating empirically that **RLHF produces policies that generalise better than SFT, and generalise better to harder distribution shifts, as shown by the generalisation performance on the novel Sequential Instructions dataset we introduced.**
Demonstrating that **this generalisation improvement comes at the cost of a large decrease in per-input output diversity, and smaller drop in across-input output diversity.**

Reviewer comment (Novelty):
The mode collapse phenomenon from RLHF has also been widely observed and measured.

Author response:
We are not aware of any papers demonstrating that the mode collapse phenomenon should be attributed to RLHF rather than SFT. Could the reviewer point us to such publications? In section 2 of the paper we discuss related work that shows some effects of RLHF on output diversity, but those works only look at per-input diversity, and use limited token-level metrics which typically don’t correlate well with human judgements of diversity [3]. In fact the original blog post highlighting the issue of mode collapse [1] was in fact not on an RLHF-ed model but rather an instruction-tuned model (text-davinci-002), so the question of whether RLHF differentially causes mode collapse is still open. Follow-on work [2] in fact seemed to imply that RLHF possibly doesn’t differentially cause mode collapse for OpenAI models.

The LLaMa 2 paper you mentioned does a small experiment on diversity (in Figure 21), but this experiment is not as thorough as ours (it uses only 20 prompts in total), and it only uses the token-level Self-BLEU metric, which is insufficient for measuring a more nuanced and human-aligned notion of diversity compared to the range of metrics we use. In contrast, we use 100 prompts, we measure both per-input and cross-input diversity, and use 3 different diversity metrics that are supported by existing literature [3], all pointing to the same conclusion. We believe this breadth of evaluation setups is important to ensure claims regarding output diversity are well supported.

Reviewer comment (Experiments):
Only two sets of experiments (summarization and instruction following) are conducted on one single model (Llama-7B), yet the paper makes a general claim about the effects of RLHF.

Author response:
In fact, **for the summarisation task we ran experiments with both LlaMA (7B) and OPT (125M, 350M, 1.3B, 2.7B, 6.7B) models**. The results with OPT are described in Appendix J. These results show the same trends as those with LlaMA, demonstrating that our conclusions generalise well across different base models, thus further strengthening our claims about the comparison between RLHF and SFT. While we cannot be entirely sure whether these conclusions hold in all possible settings, we conducted extensive experiments on **two different base models, two different tasks with five evaluation sets in total, across six different model sizes, using five different evaluation metrics for generalisation and diversity, resulting in training and evaluating over 80 models**. All these results point to similar conclusions. These experiments are very extensive, and required non-trivial resources. While it would always be better to have more experiments, we believe our analysis is rigorous and extensive enough to sufficiently support our conclusions.

Following your suggestion, we will make sure to revise our claims to ensure we are not overclaiming and caveat that these conclusions are based on certain settings and models.

In addition, we would like to note that the tasks we tested models on are some of the most commonly used in practical applications of LLMs, namely summarization and instruction following (which is also similar to single turn dialogue, another popular application), so we think our results can be of broad interest to researchers and practitioners alike. Even if our results only held in the instruction following setting this should still be considered valuable for the community. Instruction-following is clearly a very important and timely topic: it has a workshop at NeurIPS (https://an-instructive-workshop.github.io/), lots of interest both in academia and industry, and is one of the most common applications that LLMs are fine-tuned and then deployed for. Thus, we believe improving our understanding of methods even in this domain alone is useful and relevant for the ICLR community.

Reviewer comment (Novelty):
More experiments on different models at potentially different scales could be helpful, but the contribution still seems to be incremental.

Author response:
In fact, **for the summarisation task we ran experiments with both LlaMA (7B) and OPT (125M, 350M, 1.3B, 2.7B, 6.7B) models**. The results with OPT are described in Appendix J. These results show the same trends as those with LlaMA, demonstrating that our conclusions generalise well across different base models, thus further strengthening our claims about the comparison between RLHF and SFT. While we cannot be entirely sure whether these conclusions hold in all possible settings, we conducted extensive experiments on **two different base models, two different tasks with five evaluation sets in total, across six different model sizes, using five different evaluation metrics for generalisation and diversity, resulting in training and evaluating over 80 models**. All these results point to similar conclusions. These experiments are very extensive, and required non-trivial resources. While it would always be better to have more experiments, we believe our analysis is rigorous and extensive enough to sufficiently support our conclusions.

Following your suggestion, we will make sure to revise our claims to ensure we are not overclaiming and caveat that these conclusions are based on certain settings and models.

In addition, we would like to note that the tasks we tested models on are some of the most commonly used in practical applications of LLMs, namely summarization and instruction following (which is also similar to single turn dialogue, another popular application), so we think our results can be of broad interest to researchers and practitioners alike. Even if our results only held in the instruction following setting this should still be considered valuable for the community. Instruction-following is clearly a very important and timely topic: it has a workshop at NeurIPS (https://an-instructive-workshop.github.io/), lots of interest both in academia and industry, and is one of the most common applications that LLMs are fine-tuned and then deployed for. Thus, we believe improving our understanding of methods even in this domain alone is useful and relevant for the ICLR community.

Reviewer comment (Novelty):
The main concern is the contribution; unless the authors can justify their contribution through substantial experiments (on different models at different scales) and more in-depth analysis, the recommendation leans towards rejection.

Author response:
In fact, **for the summarisation task we ran experiments with both LlaMA (7B) and OPT (125M, 350M, 1.3B, 2.7B, 6.7B) models**. The results with OPT are described in Appendix J. These results show the same trends as those with LlaMA, demonstrating that our conclusions generalise well across different base models, thus further strengthening our claims about the comparison between RLHF and SFT. While we cannot be entirely sure whether these conclusions hold in all possible settings, we conducted extensive experiments on **two different base models, two different tasks with five evaluation sets in total, across six different model sizes, using five different evaluation metrics for generalisation and diversity, resulting in training and evaluating over 80 models**. All these results point to similar conclusions. These experiments are very extensive, and required non-trivial resources. While it would always be better to have more experiments, we believe our analysis is rigorous and extensive enough to sufficiently support our conclusions.

Following your suggestion, we will make sure to revise our claims to ensure we are not overclaiming and caveat that these conclusions are based on certain settings and models.

In addition, we would like to note that the tasks we tested models on are some of the most commonly used in practical applications of LLMs, namely summarization and instruction following (which is also similar to single turn dialogue, another popular application), so we think our results can be of broad interest to researchers and practitioners alike. Even if our results only held in the instruction following setting this should still be considered valuable for the community. Instruction-following is clearly a very important and timely topic: it has a workshop at NeurIPS (https://an-instructive-workshop.github.io/), lots of interest both in academia and industry, and is one of the most common applications that LLMs are fine-tuned and then deployed for. Thus, we believe improving our understanding of methods even in this domain alone is useful and relevant for the ICLR community.

Reviewer comment (Theory):
The paper makes an unspecified claim that would need justification or further explanation — for example, on page 2, summary of contributions, the third bullet point states: "...implying that such models tend to produce text of a specific style regardless of the input". How does one arrive at the "style" conclusion?

Author response:
Thank you for pointing out this lack of clarity, we agree it can be made more specific and justified. We have adjusted that bullet point to instead read:
“Even when sampling outputs for different inputs, RLHF produces less diverse text on some metrics, implying that such models tend to produce more similar text regardless of the input.”

To further explain what we intended to convey by this point: The reduction in across-input output diversity from RLHF compared to SFT implies that models’ distributions across inputs are less diverse. Taken to the limit, this would mean models would produce very similar outputs regardless of the input, and RLHF pushes models more towards this limit. The word “style” is just meant to mean outputs that are more similar. 

If there are other unspecified claims that you feel need more justification or explanation, we’d be happy to address those through adjustments to the paper.

Reviewer comment (Presentation):
There are no error bars in Figure 5. Could error bars be plotted over different choices of outputs from among the set of outputs to the same input?

Author response:
We mentioned in a previous comment we would update the paper in response to your question "Why is there no error bars in Figure 5? Could you plot error bars over different choices of outputs from among the set of outputs to the same input?"

We have just uploaded an updated revision with Figure 5 updated with error bars. Figure 5 shows the across-input diversity, and for SFT and RLHF we calculate error bars using the standard deviation of the metric across different samples of the output for each input, as you suggested. Thank you again for the suggestion for improving the paper. The error bars are very small and non-overlapping, showing the difference in across-input output diversity is still present beween RLHF and SFT in this setting.

Reviewer comment (Experiments):
Are the OOD datasets considered OOD for fine-tuning only, or for both fine-tuning and pretraining? The CNN/DailyMail dataset is most probably included in the pretraining dataset of Llama-7B.

Author response:
The OOD test datasets are considered OOD for fine-tuning, rather than pretraining. We believe this is still a relevant setting, as we often don’t have control or visibility of the pretraining data distribution. This is the “finetune-train/test” shift locus in [4], and in that survey they find 30% of work on generalisation in NLP is focused on this type of shift, making it a relevant and useful area for research.

Reviewer comment (Novelty):
Missing Related Work — In fact, there is a theoretical comparison of RLHF and SFT-style methods in the framework of imitation learning [1]. Indeed, LLMs are imitating human speech. In that framework, RLHF corresponds to adversarial imitation learning (AIL) methods, and SFT corresponds to behavioral cloning (BC). To the best knowledge of the reviewer, that theoretical study reveals that AIL (RLHF) methods can have better out-of-distribution generalization performance than BC (SFT) because AIL methods optimize their policy on out-of-distribution states (prompts) and rigorously prove this phenomenon. The reviewer believes this related work is insightful for studying the advantages of RLHF over SFT, and that this paper should mention it in the related work. [1] Xu, Tian, et al. "On the generalization of adversarial imitation learning and beyond." *arXiv preprint arXiv:2106.10424* (2021).

Author response:
> In fact, there is a theoretical comparison of RLHF and SFT-style methods in the framework of imitation learning [1]. [...] I believe this related work is insightful for studying the advantages of RLHF over SFT, and this paper should be mentioned in the related work.

We thank the reviewer for the relevant reference. We have updated the paper (in the extended related works section of Appendix C) to include a reference to that work and added a discussion inspired by your comments.

Reviewer comment (Writing):
Typo — There are two minus symbols in Equation (1).

Author response:
> There are two minus symbols in Equation (1).

Thank you for pointing out this mistake, we have corrected it in the updated draft.

Reviewer comment (Presentation):
Writing Suggestion — It seems unusual to draw a conclusion in Section 6.3 while presenting empirical evidence in Appendix I.

Author response:
> It seems unusual to draw a conclusion in Section 6.3 while presenting empirical evidence in Appendix I

We chose to present evidence in the appendix due to space constraints, and the main result of that experiment was easy to convey concisely without a supporting figure. Unfortunately we won’t be able to fit the results from appendix I into the main paper due to space constraints without removing key information, but we hope the manuscript is still satisfactory to the reviewer.

Reviewer comment (Experiments):
Concern about empirical evaluation — The major concerns stem from the fact that the empirical evaluation heavily relies on the training quality of each algorithm, and the reviewer is uncertain about whether RLHF is trained to a high standard.

Author response:
> The major concerns stem from the fact that the empirical evaluation heavily relies on the training quality of each algorithm, and the reviewer is uncertain about whether RLHF is trained to a high standard

**Given the strong performance of the RLHF and BoN policies relative to SFT when evaluated by humans and GPT-4**, and the high accuracy of the reward models (in fact slightly higher than reported in [2] - our RM gets .76 while the best model in [2] gets ~.74) **we believe this sufficiently demonstrates that our RLHF models are trained to a high standard**.

Further, our conclusions rely on the relative ranking of RLHF, SFT and BoN and gaps in performance between methods. **Even if our RLHF implementation is not fully optimal, our conclusions would still be correct, as an improved RLHF performance doesn’t change the ranking of methods** in terms of generalisation as it already performs best in absolute terms on all evaluation sets.

Reviewer comment (Evaluation):
Question 2 — Why not use entropy as a metric of diversity (although existing evaluation methods are acceptable)?

Author response:
> Why not use entropy as a metric of diversity (although existing evaluation methods are acceptable)?

Entropy is often used as an auxiliary loss in RL, often motivated by avoiding deterministic policies, but it hasn’t been used much as a measure of diversity, and we believe it is inadequate for several reasons. Firstly, it’s not been validated to align well with human notions of diversity, in comparison to the metrics we use in the paper (as shown in [6]). Secondly, it’s unclear how to interpret different values of policy entropy in terms of what that means for how diverse the policy outputs will be for a human.

Reviewer comment (Theory):
Discussion on diversity reduction — The paper mentions that "Future work should investigate why RLHF reduces the output diversity so much," and the reviewer points out some observations: the optimal policy by RL algorithms is deterministic (i.e., less diversity), if there is no tie in the reward value, there is no KL penalty, and the optimization is done perfectly. When there is a KL penalty, a recent paper shows that this corresponds to "soft Q-learning" [2]. In that case, the reward model is optimized perfectly. Although the algorithm in [2] is not applicable to the true RLHF setting where we only have out-of-distribution prompts and no preference labels, the viewpoint in [2] is insightful for in-distribution training. [2] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." *arXiv preprint arXiv:2305.18290* (2023).

Author response:
> This paper mentions that "Future work should investigate why RLHF reduces the output diversity so much," and the reviewer would like to point out some observations. [...]

We thank the reviewer for their observations, we agree that the fact that RLHF without a KL penalty has a deterministic optimal solution, and that is likely part of the reason why RLHF reduces output diversity so much.