Paper: Confronting Reward Model Overoptimization with Constrained RLHF (ICLR-2024)

Reviewer comment (Experiments):
I'm curious if it's necessary in practice to satisfy all constraints. For instance, I wonder how performance would be affected if only the intent constraint was satisfied, or if only the METEOR constraint was satisfied (while still identifying proxy points through considering all RMs together).

Author response:
Constraint Satisfaction: The answer to this depends to a large extent on the definition of “necessary.” Our results indicate that the best performance is obtained when all constraints are satisfied. If the true evaluation performance is a smooth function of the component RM scores, then a subset of component RMs deviating slightly from their constraints would likely not produce a significant drop in performance. This is what appears to have occurred with $\mu$-PPO in our case (although it should be stressed that because it is using inequality constraints, it’s not violating its constraints by exceeding the intent threshold). However, this is hard to predict and would likely vary greatly across problems.

Reviewer comment (Evaluation):
I'm a bit confused on why $\mu$-PPO still performs well even though the proxy point thresholds are exceeded. I do see that the authors acknowledge this as a reason why $\mu$-PPO may perform worse than $\xi$-PPO, but I wonder why $\mu$-PPO still performs well. I'm especially confused about this because in Figure 5.3 the authors show that using thresholds that are higher than the proxy point thresholds leads to much worse performance for $\xi$-PPO. Can the authors provide an explanation for this?

Author response:
$\mu$-PPO: This is likely related to the issue you raise in the previous question (and our response). $\mu$-PPO doesn’t dramatically exceed the Intent threshold, and it matches the METEOR threshold fairly accurately. In this problem, the predicted evaluation surface (Fig 3.2) is rather smooth, so this likely explains why its behavior doesn’t produce a dramatic drop in performance.

Reviewer comment (Presentation):
While the paper does contain much information, a lot of important information is in the appendix that would benefit from also appearing briefly in the paper. On that thread, it would be beneficial to highlight important details in D.2. Is there a specific feature of the sample outputs you are attempting to highlight?

Author response:
Paper Length and Sample Outputs: We absolutely agree regarding the length of the paper! Is there anything in particular that’s in the appendix that you would like to see in the main text?  Regarding the sample outputs, we were simply hoping to highlight the fact that high METEOR and intent scores do not necessarily correlate with high evaluation scores, a sign of overoptimization. We have added outputs from NM-PPO, as well as the component RM and evaluation ratings for each response to emphasize this.

Reviewer comment (Experiments):
Could you comment on how the results may differ for newer, more high-performing LLMs (e.g., GPT-4)?

Author response:
Transferability of Results: We don’t know exactly how GPT-4 was finetuned, but in principle this method should be effective for any RLHF approach. Methods like NM-PPO which eliminate the need for multiple runs and reduce the overall number of queries to the ground truth evaluation metric would be especially helpful at that scale.

Reviewer comment (Theory):
One significant limitation, as acknowledged by the authors themselves, is the assumption of the availability of the ground-truth reward model in RLHF. While such an assumption of the gold reward model aids in understanding and analyzing overoptimization as in Gao+ 2022, its actual use within RLHF algorithms seems impractical.

Author response:
We absolutely agree that the need to query a ground truth model is a weakness. However, as we note in the paper, this requirement is common to all the methods considered, including PPO (which requires access to the evaluation metric to tune the RM weightings). There’s currently no way to determine if overoptimization is happening without some access to ground truth evaluation. One possible route would be to make use of AI feedback (RLAIF). Such an advance would be orthogonal to the aim of this paper (and would benefit both our approach and standard PPO), and we believe it is outside the scope of this work.

Reviewer comment (Evaluation):
The evaluation metric described in Section A.2 is aimed at respecting both the METEOR and intent reward functions in light of Goodhart's Law. While the authors have meticulously chosen metrics from two categories - lexical quality and text diversity, there seems to be a lack of in-depth discussion about its relevance and applicability in the context of LLM-based RLHF. Drawing insights from works such as the one by Maynez et al. (Benchmarking Large Language Model Capabilities for Conditional Generation, ACL 2023) could have provided a clearer perspective on how well the proposed metric aligns with the requirements and challenges of LLM.

Author response:
We thank the reviewer for the reference. We found the paper insightful and have worked it into Section A.2 with a more in-depth discussion. We'd like to emphasize that the main desideratum for our evaluation metric was that it could be overoptimized in the RLHF process. In the context of the findings of Maynez et al., we note our work differs in considering RLHF on dialogue tasks, rather than open-ended natural language generation. Due to the structured nature of the task, models rarely break from the format (see generation examples in Appendix D), which permits the use of overlap metrics. To ensure relevance to the ground truth outputs, we used ROUGE2 (like Maynez et al.) in combination with some other metrics. The main difference in our evaluation metrics are the diversity metrics, which were chosen since degeneration is a common problem when finetuning language models [1]. We are excited by future extensions of our work to larger models, and more sophisticated tasks (like summarization), which would necessitate evaluations like those suggested by Maynez, et al.

Reviewer comment (Writing):
The lack of detailed descriptions and elaborations of the specific methodologies presented in Table 1 raises concerns about the paper's clarity. One might find it challenging to grasp the proposed approach without sufficient explanations.

Author response:
Thank you for this observation! Algorithm 1 in the paper appendix comprises a detailed procedure which can be used to implement $\mu$-PPO and $\xi$-PPO, and Algorithms 2 and 3 describe the overall two-phase procedure for identifying and leveraging proxy points and NM-PPO, respectively. In addition, we included the code implementing these algorithms in the supplementary material. In the revision, we have added additional references to the algorithm descriptions in the main text as well as a description of PPO-SAT in Algorithm 2. We hope this improves the clarity of the proposed approaches!

Reviewer comment (Evaluation):
On the Validity of Evaluation Metric (Equation A.1): How did the authors determine that the evaluation metric chosen in Equation A.1 accurately represents the efficacy of the proposed method?

Author response:
Thank you for this question! As noted above, the evaluation metric was selected because it was overoptimized (i.e., obeyed Goodhart’s Law) with respect to each of the component RMs. That is, we wanted an evaluation metric that would initially increase as METEOR and intent scores increased, but would begin to decrease after a certain point, much as human evaluation ratings initially correlate with RM scores and then decline. To design this metric, we tested a number of combinations of other metrics (those included in Figure D.1), and found that the combination described in Appendix A.2 most closely adhered to the pattern of overoptimization with respect to METEOR and intent. That is, the evaluation metric was not directly chosen to simulate human preferences, but rather the way human ratings change as component RMs increase.

Reviewer comment (Evaluation):
On the Validity of Evaluation Metric (Equation A.1): Are there any design principles or guidelines followed when setting this metric? The significance of this setting in the proposed method seems critical, and a clearer explanation would be beneficial.

Author response:
Thank you for this question! As noted above, the evaluation metric was selected because it was overoptimized (i.e., obeyed Goodhart’s Law) with respect to each of the component RMs. That is, we wanted an evaluation metric that would initially increase as METEOR and intent scores increased, but would begin to decrease after a certain point, much as human evaluation ratings initially correlate with RM scores and then decline. To design this metric, we tested a number of combinations of other metrics (those included in Figure D.1), and found that the combination described in Appendix A.2 most closely adhered to the pattern of overoptimization with respect to METEOR and intent. That is, the evaluation metric was not directly chosen to simulate human preferences, but rather the way human ratings change as component RMs increase.

Reviewer comment (Theory):
Regarding mixed advantages right after Equation (4.3): Shouldn't the term v_µ(s) encompass the second term from Equation (4.3)?

Author response:
The value expression is correct, as it measures the expected, discounted, cumulative reward with respect to the Lagrange multipliers. Furthermore,    $\sum_{i=1}^N \mu_i \theta_i$ is not a (directly) policy-dependent term.

Reviewer comment (Evaluation):
Determining the joint maximizing proxy point seems heuristic. To determine the joint maximizing proxy point, the evaluation scores as a function of the METEOR and intent rewards for each run shown in Fig. 3.1 are plotted and a surface is fit over them. But these evaluations were done by maximizing each reward individually, without account of any interaction. Thus using these to determine the joint maximizing point seems heuristic at best.

Author response:
Joint Proxy Point Identification: The idea behind our approach was to increase each individual RM and observe the corresponding change in the others as a way to measure their interaction. If the agent were maximizing both component RMs, then a change in one couldn’t be ascribed purely to the change in the other. Additionally, the fact that NM-PPO appears to hill-climb along the topography of the predicted surface (Fig. 5.4, and Fig. D.3) can be seen as empirical validation that our approach is in fact effective at approximating the evaluation surface. Finally, given that (to our knowledge) overoptimization has not been studied in this setting before, we believe that simple yet effective heuristics offer a useful starting place for future work, both to understand their efficacy and to design improvements. We hope to continue working on this topic, as well as inspire broader follow-up work!

Reviewer comment (Experiments):
How and how many times is the ground truth evaluation function queried to determine the maximizing proxy point? (For results in Fig 3.2 and 5.1)

Author response:
Proxy Points: The evaluation function is queried 20x per run in Figure 3.1. Figure 3.2 is generated using the results from the runs in Figure 3.1 (no additional queries). The algorithms do not make additional queries to the evaluation function for the results in Figure 5.1. We tested two methods for finding the maximum of the fitted evaluation function surface. First, we used the built-in implementation of L-BFGS-B in the SciPy optimization package using the results from KDE to provide bounds. Second, since the polynomial surface is approximated using a discrete mesh, we also simply picked the coordinates corresponding to the plotted maximum. There was a negligible difference between methods, so we used the second in practice. We have added this detail to the appendix. We do not use KL regularization to determine the proxy points because our goal is to isolate the influence of each component RM on the others, and KL regularization would be an additional factor influencing behavior.

Reviewer comment (Experiments):
Why is KL regularization not used to determine the proxy points? (results in Fig 3.1)

Author response:
Proxy Points: The evaluation function is queried 20x per run in Figure 3.1. Figure 3.2 is generated using the results from the runs in Figure 3.1 (no additional queries). The algorithms do not make additional queries to the evaluation function for the results in Figure 5.1. We tested two methods for finding the maximum of the fitted evaluation function surface. First, we used the built-in implementation of L-BFGS-B in the SciPy optimization package using the results from KDE to provide bounds. Second, since the polynomial surface is approximated using a discrete mesh, we also simply picked the coordinates corresponding to the plotted maximum. There was a negligible difference between methods, so we used the second in practice. We have added this detail to the appendix. We do not use KL regularization to determine the proxy points because our goal is to isolate the influence of each component RM on the others, and KL regularization would be an additional factor influencing behavior.

Reviewer comment (Evaluation):
In Fig 5.1 (right) the performance of $\xi$-PPO drops after about 180k training steps, are the constraints still satisfied during this stage?

Author response:
Long Training Performance: This is a good question! Upon investigation, we found that all runs managed to continually satisfy the intent reward constraint across training, but that a single run suffered a loss of stability with respect to the METEOR reward after roughly twice the baseline training time (stability was maintained in the other seeds). This aligned with a drop in evaluation performance. We added a figure showing this, Figure D.3.

Reviewer comment (Presentation):
It would be clear if you could plot $\xi$-PPO and $\mu$-PPO in figure D2 for reference. It will help in understanding the instability of All-PPO.

Author response:
Stability: We agree! We’ve added these results to the figure.

Reviewer comment (Reproducibility):
Does All-PPO have a KL constraint?

Author response:
All-PPO: Yes, it has the same KL regularization mechanism as standard PPO. We have added text clarifying this.

Reviewer comment (Evaluation):
It would be helpful if you could plot the training curve for NM-PPO. It would be helpful to see how the changing threshold affects the training process.

Author response:
NM-PPO Curve: We agree, and have added this in Figure D.5.

Reviewer comment (Presentation):
Sample outputs of NM-PPO are missing in section D.2.

Author response:
NM-PPO Sample Outputs: Good catch, we have added this.

Reviewer comment (Evaluation):
Could you explain the sample outputs in section D2? It is hard to understand the performance of the different algorithms. It would be helpful if you could include the evaluation score and the rewards obtained as well.

Author response:
Sample Output Rationale: We have added the reward and evaluation scores for each output. Our main point of emphasis was that high METEOR and intent scores do not necessarily correlate with high evaluation scores. Because the evaluation score was selected because it can be overoptimized by METEOR and intent, rather than any specific suitability as a measure of text quality, higher evaluation scores should not be interpreted as such. We have added text to clarify this.