Paper: Controlling language over-optimization by targeting reward distribution (ICLR-2024)

Reviewer comment (Experiments):
If we already have access to human demonstration data, performing supervised fine-tuning is sufficient to align the pretrained model with sample-approximated human response distribution, and the reward over-optimization problem does not exist. From the experiment results, supervised fine-tuning seems to be on par with the proposed method, as well. In practice, RLHF is used precisely because human demonstrations are hard to obtain, and preference labels over pairs of generated text are much easier and cheaper to collect.

Author response:
SFT is indeed a crucial baseline. In the experiments presented in the paper, we observe that while SFT provides a viable means for alignment (still, less effective than our method), it proves inefficient in Experiment 3 and Experiment 1 (note that we omitted the SFT method in Experiment 1, considering that baselines with various sampling methods are more relevant). In forthcoming versions of the paper, we commit to dedicating a more extensive section to comparing these baselines.

Reviewer comment (Evaluation):
The win rate evaluation in the experiments is conducted using a Llama2-13B chat model. The authors do not explain why they have not considered using more powerful open-API models for evaluation. It is unclear if there is any particular reason for the chosen evaluation model and how the authors guarantee the quality of such automated ratings.

Author response:
Using Llama-13B was the best choice we could do at the time of the experiments. That said and moving forward, we acknowledge your suggestion and plan to enhance the win-rate metric by leveraging more powerful models accessible through APIs.

Reviewer comment (Evaluation):
The diversity metric used is the type/token ratio, which is a token-level metric that only accounts for what proportion of generated text is unique. It is unclear why this is considered a good metric. On page 5, the authors wrote "these metrics allow for detecting unnatural and repetitive patterns in the generated text", but this seems to only cover a small aspect of the consequence of reward over-optimization. The method for detecting other consequences is not explained.

Author response:
While we acknowledge the limited scope of our diversity metric, we intentionally opted for a simple metric that effectively captures degenerated patterns in our experiments. Following you remark, we intend to expand our set of diversity metrics (incorporating measures such as self-BLEU) and add relevant over-optimization metrics such as the KL divergence with the base policy.

Reviewer comment (Experiments):
How sensitive is the choice of distance to the target? If you use L1 rather than L2 for example to define the reward?

Author response:
We would like to thank the reviewer for their positive comment.

Regarding the choice of the distance to the target, we did preliminary experiments that are not included in the paper. We observed that actually the choice of distance did not change much the results as a specific target is defined for each prompt. We tried L1, L2, L4 distances.

Reviewer comment (Writing):
In my opinion, the main weakness of this paper is the inprecise usage of the term reward over-optimization.

Author response:
**1. Reward over-optimization**
In our experiments, we replicate a real-world scenario where we lack access to the actual ground truth reward and only have a proxy reward. Our findings reveal instances of over-optimization, demonstrating that excessive optimization against such a proxy hinders progress toward the "true" objective. In Figure 2, we observe that direct optimization against the proxy (second row; first column) results in closely matching an almost optimal distribution (i.e., maximal reward for each generation). However, the naturalness score is notably low, indicating a lack of diversity (Table 2). 
Our situation aligns with the definition of over-optimization as outlined by Gao et al. in 2023: "In machine learning, this effect arises with proxy objectives provided by static learned models, such as discriminators and reward models. Optimizing too much against such a model eventually hinders the true objective, a phenomenon we refer to as over-optimization.". We cannot display the pattern you mention as we don't have access to the "true" reward.

In this context, our method demonstrates that utilizing human data to construct a reward distribution enables us to identify an optimization sweet spot. In this sweet spot, generations achieve both high rewards and maintain naturalness. This approach represents a means to strike a balance and find an optimal trade-off to tackle over-optimization.

Reviewer comment (Experiments):
It is also unclear to me how the procedure compares with the simple SFT that directly fine-tune on the high-quality human generated data.

Author response:
**2. Comparison with SFT**
In the paper, we explicitly conduct a comparison with SFT applied on high-quality data. In experiment 1, we refrain from presenting the results for SFT, as it proves inefficient in addressing sequence-level optimization. Sampling procedures, in this context, seem to serve more as purposeful baselines. In the remaining two cases, we illustrate the distinctions between direct SFT and our approach.
In experiment 2 (refer to Figure 2 and Table 1), we demonstrate that direct SFT effectively aligns the rewards of generated sequences with those of human samples, but with a lesser degree of alignment compared to our method. However, in experiment 3 (summary task), we observe that SFT struggles to align the distribution (refer to Figure 3), whereas our method successfully achieves alignment between the distribution of rewards for model-generated summaries and the distribution of rewards for human summaries.

Reviewer comment (Evaluation):
The reward over-optimization in (Gao et al. 2023) refers to the scenario that during policy fine-tuning, the ground truth reward first goes up, and then decreases. Can the authors elaborate on how exactly Figure 2 reproduces this phenomenon? Note that the distribution mismatch may not indicate such change of ground truth reward during the training process.

Author response:
**1. Reward over-optimization**
In our experiments, we replicate a real-world scenario where we lack access to the actual ground truth reward and only have a proxy reward. Our findings reveal instances of over-optimization, demonstrating that excessive optimization against such a proxy hinders progress toward the "true" objective. In Figure 2, we observe that direct optimization against the proxy (second row; first column) results in closely matching an almost optimal distribution (i.e., maximal reward for each generation). However, the naturalness score is notably low, indicating a lack of diversity (Table 2). 
Our situation aligns with the definition of over-optimization as outlined by Gao et al. in 2023: "In machine learning, this effect arises with proxy objectives provided by static learned models, such as discriminators and reward models. Optimizing too much against such a model eventually hinders the true objective, a phenomenon we refer to as over-optimization.". We cannot display the pattern you mention as we don't have access to the "true" reward.

In this context, our method demonstrates that utilizing human data to construct a reward distribution enables us to identify an optimization sweet spot. In this sweet spot, generations achieve both high rewards and maintain naturalness. This approach represents a means to strike a balance and find an optimal trade-off to tackle over-optimization.

Reviewer comment (Experiments):
The proposed method aims to improve the reward to the level of human demonstrations, conditioned on the fact that we are given high-quality human demonstration data. In this case, how does the proposed method compare with direct supervised fine-tuning on the high-quality data?

Author response:
**2. Comparison with SFT**
In the paper, we explicitly conduct a comparison with SFT applied on high-quality data. In experiment 1, we refrain from presenting the results for SFT, as it proves inefficient in addressing sequence-level optimization. Sampling procedures, in this context, seem to serve more as purposeful baselines. In the remaining two cases, we illustrate the distinctions between direct SFT and our approach.
In experiment 2 (refer to Figure 2 and Table 1), we demonstrate that direct SFT effectively aligns the rewards of generated sequences with those of human samples, but with a lesser degree of alignment compared to our method. However, in experiment 3 (summary task), we observe that SFT struggles to align the distribution (refer to Figure 3), whereas our method successfully achieves alignment between the distribution of rewards for model-generated summaries and the distribution of rewards for human summaries.

Reviewer comment (Writing):
Clarity: This paper has several grammatical typos, undefined variables, and not well-explained concepts (see the question section). Furthermore, the use cases are repetitive; in particular, over-specialization and sequence-level likelihood captures the same issue. If the model sequence-level likelihood starts to diverge, that indicates the over-specialization problems, and vice versa. Furthermore, the multi-objective use case is not a traditional objective optimized in RLHF, and for a fixed alpha, the multi-objective can be viewed as a single objective. This means that this use case is similar to the previous use cases.

Author response:
We apologize for any lack of clarity and appreciate the reviewer's feedback. We will carefully consider all of the provided comments to enhance the clarity of our work. Regarding the perceived repetitiveness of the experiments, we respectfully disagree. While it is acknowledged that these experiments revolve around the common theme of addressing over-optimization, each one is motivated by distinct reasons:

- Experiment 1: The exploration of sequence-level log-likelihood serves as an illustration to emphasize the necessity of calibrating the RL objective. Drawing inspiration from Holtzman et al. (2019), who demonstrated the effectiveness of targeting a calibrated log-likelihood in the context of language (using temperature and top-p), our approach extends this idea to RL objective training. The sequence-level log-likelihood experiment provides intuition into the nature of the problem.
- Experiment 2: This experiment delves into the standard scenario where a classifier reward model may undergo over-optimization. This problem is a common topic in the literature and highlights the relevance of our method in mitigating the impact of over-optimizing a proxy and generating unnatural outputs.
- Experiment 3: We respectfully disagree with the reviewer's perspective on the significance of studying a multi-objective scenario. This is a common issue in RLHF, particularly in addressing the helpfulness/safety calibration challenge. In general, the challenge in RLHF is not to merely optimize a single objective with a fixed alpha but rather to determine the appropriate alpha. Our method addresses this challenge by defining targets for each objective based on data, offering a systematic approach rather than manually setting a fixed alpha and conducting a parameter search.

Reviewer comment (Experiments):
Baselines: This paper is missing a few baselines, which makes it hard to understand the benefits of the proposed approach empirically (see the question section).

Author response:
We have endeavored to introduce a range of baselines, including diverse sampling procedures for Experiment 1 and comparisons with standard optimization algorithms for Experiments 2 and 3 (such as supervised fine-tuning, standard reinforcement learning, and reinforcement learning with KL divergence). Nevertheless, we acknowledge that this may be perceived as insufficient, and we are eager to incorporate additional baselines. What recommendations do you have in this regard?

Reviewer comment (Writing):
What is the difference between reward overoptimization, reward hacking, and over-specialization?

Author response:
We regret any imprecision in our use of terminology. Our primary intention is to align with the definition of *over-optimization* as articulated by Gao et al. 2023:

"In machine learning, this effect arises with proxy objectives provided by static learned models, such as discriminators and reward models. Optimizing too much against such a model eventually hinders the true objective, a phenomenon we refer to as over-optimization."

In subsequent iterations of the paper, we will exercise greater care in the precise application and definition of these terms.

Reviewer comment (Experiments):
What does it mean for a single prompt to have multiple generation pairs? Does this mean that the human is providing suboptimal generations?

Author response:
We unfortunately believe there might be a misunderstanding regarding our methodology that hinder the overall understanding of the paper. We never consider a single prompt generating multiple pairs of generations. In Section 3.2, when outlining our methodology, we specifically refer to pairs as "prompt-continuation," "prompt-reward," or "target reward." We acknowledge that this misunderstanding may stem from a lack of clarity in our method and notations. We are committed to enhancing the clarity of this section to avoid any confusion that may have led to a misunderstanding of the paper. We sincerely apologize for any confusion that may have occurred.

Reviewer comment (Writing):
What do you mean by target distribution? Why would a single prompt have a target distribution versus an optimal target?

Author response:
For a single prompt, we define a single target. When considering these targets across the entire dataset, a distribution of rewards emerges. We employ the term "distribution" at the dataset level to contrast with the standard RL optimal solution, which is the argmax distribution over the reward values. We will clarify this notion in future iterations of the paper.

Reviewer comment (Theory):
What is the difference between training a policy to maximize the KL divergence between some initial policy \pi_0 and maximizing the sequence-level log-likelihood of \pi_0?

Author response:
It can indeed be very similar if the KL is computed at the sentence level.

Reviewer comment (Writing):
When you defined D := (x, t(x)) - what is t(x)?

Author response:
As defined in section 3.2, t(x) is the reward obtained by the human “ground truth” generation R(x,y). To enhance clarity, we will explicitly state: t(x)=R(x,y), where where x is the prompt and y the human continuation.

Reviewer comment (Reproducibility):
How is the target distribution learned from single-prompt multiple-generation pairs? Is it the average of the generation rewards?

Author response:
Each prompt corresponds to a single generation. The model is trained to align with a target distribution at the dataset level.

Reviewer comment (Experiments):
For minimizing the sequence-level log-likelihood, the most straightforward thing to do is to optimize the InstructGPT objective PPO-ptx [1], which includes a pretraining loss into the RL objective. How does your proposed algorithm compare to simply including that objective in the RL optimization procedure?

Author response:
All our RL objectives are optimized using PPO, so optimizing R with PPO corresponds to the objective you mention. Results for this objective are presented in Figure 1b (top right).

Reviewer comment (Experiments):
For the calibration experiments, I am not sure if the benefit is coming from the change in the reward function or the additional samples that you collect. It seems the reason why your algorithm is performing better is because you are taking the difference between y and \hay{y} - which provides a fixed target to optimize. But this situation does not generalize to the preference dataset when you don't have a ground truth y. Instead, you have comparison data; it is not obvious what the ground truth target would be. I am unsure how this use case is representative of the RLHF problems.

Author response:
Your comment is insightful and is related to one of the main challenges of this target approach. In this paper, we do not collect additional data but construct targets based on the additional confidence annotations from RLHF annotators. Therefore, our training is focused on a subset of available comparisons, which proved sufficient for obtaining valuable results in the studied case. To extend this approach to a larger dataset, we would suggest extrapolating the target by learning to map a prompt to a target (either directly through a regression task or indirectly by assigning the target of a similar prompt based on embedding approaches). In future versions of the paper, we plan to include those remarks on target extrapolation into the discussion or even build an additional experiment.

Reviewer comment (Reproducibility):
The results for calibration show various sampling schemes' performance, but what sampling scheme did you choose for your algorithm and the baseline RL algorithm?

Author response:
For all our experiments, the decoding algorithm involves sampling with a temperature and top-p set to 1. In the case of sequence-level likelihood, training our objective allows to target the human sequence level log-likelihood without having to adjust the sampling procedure: this sampling procedure is automatically learnt.

Reviewer comment (Theory):
What is the difference between your alignment loss and MLE?

Author response:
Instead of doing MLE, we target the setence level log-likelihood of human sentences.

Reviewer comment (Evaluation):
How does R_reg perform on the multi-objective experiments?

Author response:
For the single objective, we tuned the parameter and present the results in the table. Regarding the multi-objective, we chose not to display the results as it posed challenges in optimization, involving a 3-objective optimization (optimizing the two rewards + regularization). We also decided against displaying the results, as the two rewards effectively regulate each other, limiting direct overfitting and the need of adding a KL regularization.

Reviewer comment (Experiments):
Did you tune \alpha in the multi-objective experiments? If so how did you tune \alpha?

Author response:
We tuned alpha by conducting an overall search to maximize the average alignment of the two rewards. We will clarify this methodology in the future versions of the paper.

Reviewer comment (Presentation):
In Figure 4, the metrics arrows conflict with the best scores. For example, R^{pref} arrow it points upward, and the best score is 5.64, but you bolded 4.15. This issue is consistent with a few other metrics as well.

Author response:
The highlighted numbers are indeed a typo spread throughout the paper. The intention was to highlight methods that reach values close to the human baseline. We apologize for these errors and will ensure to correct them in future versions of the paper.