Paper: Reward Model Ensembles Help Mitigate Overoptimization (ICLR-2024)

Reviewer comment (Experiments):
Why nmax? Does this mean for some of them you use less than 12500 samples?

Author response:
For BoN we always evaluate for 12500 samples for the largest values of n. The nmax comes from the unbiased estimator used (see Appendix I in [1] for greater detail), and refers to the greatest value of n we evaluate BoN for. For lower values of n, we of course use fewer samples.

Reviewer comment (Reproducibility):
How many episodes?

Author response:
Each PPO step is 32 episodes, so 3000 steps is 96,000 episodes, which is just about 5 epochs over the unlabeled RL data.

Reviewer comment (Presentation):
Why do you have two y-axis? Do the proxy RM and the gold RM have different RM scales? How are these scales defined?

Author response:
In e.g. Figure 3 we have two y-axis because although the proxy and gold scores have roughly the same scale (i.e. the RMs are normalized to 0 mean and unit variance), the proxy scores grow much higher. We therefore add a second y-axis to avoid dwarfing the gold scores and making them harder to read.

Reviewer comment (Presentation):
Figure 6 and 7 seem contradictory? KL penalty = 0 gets 0.15 gold score, but in Figure 7 KL penalty = 0 gets 0.03 gold score?

Author response:
While most PPO runs terminate before or around when they reach a KL divergence of 150, some significantly exceed this, particularly for single RMs and when using low KL penalties. In order to keep the plots as legible as possible, we truncate runs that exceed this KL divergence when plotting the optimization curves. Past this point, the overoptimization trend has already been observed and the x-axis risks growing long and making all other runs less clear. Moreover, this enables us to have consistent plots with equivalent x-axes. We have updated the manuscript to include this information in the introduction to Section 5. This explains why runs with very low KL values have final gold scores which are lower in the bar plots (Figure 7) than on the optimization curves (Figure 6). While the visualization curves are truncated at a KL divergence of 150 (by which point the presence of overoptimization has been confirmed and performance has already declined), the bar plots record the final values at the very end of the run. We also add an equivalent of Figure 7 without the truncation in Appendix F.8 to illustrate this.

Reviewer comment (Evaluation):
Why does PPO underperform BoN in 1.3B setting according to Gold Score? Gao et al. (2023) show 1.2B PPO outperforms 1.2B BoN.

Author response:
In Gao et al. [2], page 3, Figure 1, the PPO 1.2B model reaches a peak performance of about 1.0, while the BoN 1.2B reaches over 1.15. Which part of Gao et al do you think shows 1.2B PPO outperforming 1.2B BoN? In addition, Gao et al did not open-source their implementation and used closed-sourced models, so it is difficult to know exactly what the differences might be. We show some basic reproductions of Gao et al.’s results in Appendix E.

Reviewer comment (Evaluation):
How does these two KL distance calculation Gold RM performance?

Author response:
We are not perfectly clear on what you mean in this question. If you meant which KL is used in our plots of gold RM performance, we use the second type of KL distance (eqn 10) for PPO. Additionally the choice of equation does not affect gold RM performance, it just allows us to plot KL without negative values and provides a clearer visualization.

Reviewer comment (Reproducibility):
But I assume data shuffling is different? There are really multiple random seeds: * query dataset seed * reward model seed * policy model seed.

Author response:
Indeed data shuffling changes with the random seed. For RM training, the random seed affects both the data shuffling and head initialization, and for each run both are randomised based on the seed. PPO also has its own seed for training. The results for PPO currently in the paper are only for one seed, but we are running additional seeds for PPO training to ensure our results are robust.

Reviewer comment (Experiments):
Why is single RM experimented with KL=150 but not the other types?

Author response:
In Figure 4 the single RM is the only one which reaches a KL divergence of 150, the ensemble methods never reach such a high KL distance. It appears that the conservative optimization methods limit the KL divergence of the optimized policy.

Reviewer comment (Evaluation):
Another weakness of the results is that it's hard to know how to interpret the gold reward. How much better is an LLM with an average gold reward increase of 0.5 vs. 0.4? AlpacaFarm and others use a win-rate which is more interpretable—it might be helpful to report that for your results as well so that it's easier to understand exactly how much better using an ensemble is.

Author response:
- Included win-rate results to compare the performance of ensemble methods to those of single reward models in Appendix F9. We show these for both BoN and PPO for reward models of size 44M and plan on adding other model sizes for the camera-ready version.

Reviewer comment (Novelty):
Here is one paper that should probably be included in the related work as they also evaluate using ensembles of LLM-based reward models: https://arxiv.org/pdf/2203.07472.pdf. Their paper is focused on active learning, not reward optimization, so it's not directly comparable, but it does relate to your conjecture that "uncertainty estimates from the ensemble may also help improve sample efficiency of human feedback in this setup" on page 9.

Author response:
> Here is one paper that should probably be included in the related work as they also evaluate using ensembles of LLM-based reward models: https://arxiv.org/pdf/2203.07472.pdf. 

Thanks for pointing out this paper. We have included a mention and reference to it in Related Works within the main text.

Reviewer comment (Experiments):
While the empirical results are quite comprehensive in the paper, the model size seems a bit small with the biggest RM being 1.3B. Given Figure 8, it seems that the gain of the ensemble-based methods diminishes as the model size increases. It would important to investigate if ensemble-based methods have little gain with even bigger models, which are more commonly used by users.

Author response:
Thank you for your review. We agree that overoptimization is an important problem and we’re glad you found the results encouraging and the conclusions well-supported.

With respect to reward model size and data size, we use the largest RM possible in our setup and with our computing resources. While we agree it would be interesting to conduct experiments using larger models, we believe the current experiments are sufficient evidence that the ensemble and conservative optimisation techniques are useful for mitigating overoptimization, and will continue to be at larger model sizes, for several reasons:
1. While the gain in performance seems to decline a little in proportion when increasing the reward model size, the performance gain is still very much present and does not show signs of completely disappearing. For BoN sampling, in Figures 8a and 9a, the gap between single RM performance and ensemble-based methods performance remains approximately similar in size across different scales - this suggests that ensembles likely follow a similar scaling law for overoptimization as a single RM, but shifted upwards.
2. While RM sizes can be increased and larger RMs tend to see less overoptimization, this performance increase eventually hits diminishing returns, as seen in Figure 1 of Gao et al. [1].
3. Further, we note that if you look at Figure 19, which shows the full training curve for 1.3B RM - it seems that for both BoN and PPO - while the training appears to have converged for single RM, it does not appear to have converged for WCO and UWO. Moreover, the single RM has not had a chance to properly overoptimize and see a decrease in performance. These claims are backed up by the new results in Appendix F.7, which show that our PPO ensembles can be further trained - resulting in higher performance for ensemble methods, and lower final performance for the single RMs, which overoptimize. This will be especially true for larger RMs, which are optimized at a slower rate. 
Within our setup, it is not possible to further optimize BoN due to computational constraints. However, for PPO, we plan on adding results of optimizing 1.3B RM with an increased training budget of 6000 steps. This is however going to be a very expensive experiment (we expect this to take ~72 hours to finish using the computing resources available to us), hence, we can only give weak commitment that we will be able to show this result to the reviewer within the rebuttal period. However, we do commit to including this result in the camera ready version if the paper gets accepted. 
4. The new results in Appendix F.7 also illustrate another advantage of the ensemble methods over larger RM sizes: stability and robustness to training time. These models can be trained for a long time without fear of seeing a decrease in performance due to overoptimization. This is not true even of very large RMs, which are known to suffer from overoptimization.


We hope we have addressed all your concerns and questions about the paper, and that you will consider raising your score, but please let us know if you have any remaining questions.

[1] Gao et al. Scaling Laws for reward model overoptimization

Reviewer comment (Experiments):
From Figure 9, it seems that with bigger dataset size (46K), ensemble-based methods are not that much better than the single RM. Is it expected that with even more data, which in practice is usually true, the difference between ensemble-based methods and single RM will further shrink? It would be helpful to further study this perspective.

Author response:
To follow up on our previous comment, we have added results for the longer 6000-step PPO run of the 1.3B reward model in Figure 19. As a result, PPO trained with WCO or UWO now peaks at a gold reward model score of 0.84; they were peaking at 0.6 and 0.64 respectively previously (when trained for 3000 steps) while the gold-reward model score for PPO policy trained with single reward model only changed from 0.56 to 0.64. To make it easier for reviewers to compare the gains achieved by further optimization, we have kept both the original 3000-step results (figure 19b) and added the new 6000-step results (figure 19c). We have also updated Figure 8b to include this new result, which shows an increase in performance of more than 30% of the ensemble methods compared to previous values. Both WCO and UWO now outperform the single reward models by over 30% by the end of the 6000 steps. We believe this additional result should allay your concerns regarding diminishing returns from using an ensemble. 

We would also like to reiterate our mention of Appendix F.7 in our previous comment to the reviewer as well. Along with this new result, these show that with our method, a practitioner does not have to worry about tuning the number of training steps, but can freely choose a large number of training steps without worrying about the dangers of overoptimization. 

We hope our previous response to your questions, along with this new result has addressed your questions and concerns, and if so ask if you would consider raising your score.

Reviewer comment (Reproducibility):
[Minor] For paragraph Supervied Fine-tuning in Section 4.3, the hyperlink `(see Section 4.1 for details)` seems to be broken. In addition, the reviewer cannot find any more details in section 4.1 for splitting the AlpacaFarm dataset, perhaps the authors can clarify this?

Author response:
To answer your questions:
3. For the hyperlink to section 4.1, it seems to work for us? The section linked provides more information about the dataset we use as a whole. We have also added additional details such as the dataset splits in Appendix D.2, and reference this in Sections 4.1 and 4.3 of the main paper for clarity. Even further details can be found in the AlpacaFarm paper [2].

Reviewer comment (Presentation):
Figure 3 and Figure 4 also have the same issue discussed – perhaps the authors can clarify what is the KL divergence in both figures and why they scale differently. E.g., in Figure 3 the x-axis is from 0-8, while in Figure 4, the x-axis is from 0-150. The reviewer understands that [1] has similar figures, but it will be better if the authors clarify this, so that other readers do not need to refer back to [1] for understanding this submission. Similarly in Figure 5 - 7.

Author response:
To answer your questions:
2. For Figures 3 and 4, the difference in x-axis is due to the different methods (BoN and PPO) calculating and consuming KL divergence differently. Gao et al. [1] show that PPO seems to consume KL much more than BoN when calculating KL divergence in the way described in the above sections. In addition to the clarification added in the previous point, we further clarify the significance of the KL divergence as well as the difference in scale between BoN and PPO in the captions of Figure 3 and 4.