Paper: Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR-2024)

Reviewer comment (Novelty):
Even though the paper is the first integration of Safe RL and RLHF frameworks, the contribution is limited in the case of Safe RL.

Author response:
**Reply to W1:** Our primary goal is to propose **a new RLHF framework that addresses the challenging tension** between helpfulness and harmlessness goals in the field of LLM alignment, rather than introducing a new Safe RL algorithm. Instead, we aim to broaden the application scope of Safe RL algorithms, bringing new perspectives to both fields.

Here, we reiterate our contributions:

- Safe RLHF is **the first work to combine Safe RL with the RLHF framework**.  It is a comprehensive methodology that includes decoupled data collection, training of two different preference models, and fine-tuning through the integration of Safe RL. It formalizes safety as a constraint and dynamically adjusts it during training, aiming to navigate the inherent tension between helpfulness and harmlessness.
- Our **extensive experiments** demonstrate the following conclusions:

    - Fine-tuning with Safe RLHF framework effectively enhances the helpfulness and harmlessness of LLMs under human values (Section 5.2.1).
    - Decoupling helpfulness from harmlessness effectively increases agreement among crowd workers when annotating preference data, reducing bias due to the contradiction (Section 5.2.2).
    - Compared to a fixed ratio of optimizing for helpfulness and harmlessness, dynamically adjusting the trade-off between them during training more effectively guides the inherent tension between the two (Section 5.2.3).
    - Providing preference-based scores, rather than just a binary safe/unsafe classifier, improves LLM safety more efficiently (Section 5.2.4).

- We **release all the data and code** from our experiments in hopes of **contributing to the reduction of the high research costs in this field**. (Currently, they are in the supplementary materials.)

**We sincerely hope that you could appreciate our paper and recognize our efforts and contributions.**

Reviewer comment (Writing):
The comparison between the (reward and cost) model in Section 3.2 to the classic (reward and cost) signals in safe RL should be clarified more.

Author response:
**Reply to W2:** Compared to traditional Safe RL's reward and cost signals, Safe RLHF presents the following challenges:

- **Human values are high-dimensional and complex, making it impossible to design reward and cost functions in a rule-based manner.** Instead, human preferences must be modeled from collected datasets. Thus, guiding crowd workers to provide high-quality annotations (Section 4.1), constructing prompt datasets for large language models (Section 5.1 Prompts and Red-teaming), controlling the distribution of prompt-response pairs in the dataset (Section 5.1 Preference Datasets), and training better preference models (Section 4.2) are among the issues that a comprehensive large language model alignment process must address.

- The threshold for what is considered safe in human values cannot be formulaically defined. Therefore, **we propose a new training method for a Cost Model to model the safety threshold**, which is one of our contributions. Building on the Bradley-Terry model, we introduce a comparison loss between responses and a virtual response $y_0$ near the safety threshold, setting the cost of $y_0$ to zero. This design separates safe from unsafe responses at a cost boundary of zero (as shown in Figure 2(a)), making zero the safety threshold for an individual response.

Reviewer comment (Presentation):
Figure 1 needs to be polished more, and the font there is a bit small.

Author response:
**Reply to Q1:** We are very grateful to the reviewer for their valuable feedback on Figure 1. We will actively incorporate these improvements in the newly submitted version of our paper.

Reviewer comment (Experiments):
Can we use any Lagrangian safe RL methods (TRPO-Lag, PPO-Lag) in Safe RLHF?

Author response:
**Reply to Q2:** We conservatively believe that **first-order algorithms** in the Safe RL field, such as PPO-Lag, FOCOPS, P3O, etc., **hold promise for application within the Safe RLHF framework**.

However, for **second-order algorithms** like TRPO-Lag, CPO, etc., which require solving the inverse of the Fisher matrix through conjugate gradient methods, **the computational resources needed for LLMs might be incredibly vast**.

Reviewer comment (Novelty):
What is the difference between Safe RLHF and the off-policy Lagrangian safe RL methods, e.g., WCSAC (published in AAAI-21 and MLJ-23)?

Author response:
**Reply to Q3:**  The RL phase of Safe RLHF should be classified as an on-policy algorithm, where the model used for updates and inference is the same. We carefully read WCSAC, as recommended by the reviewer, and find that **combining off-policy Safe RL may offer some distinct advantages**. For instance, it might reduce the communication bottlenecks during the inference phase of LLM training, and the maximum entropy framework could enhance sampling diversity.
We are very grateful to the reviewer for providing valuable paper references and research discussions. **We will include this part in the 'Future Work' section of the revised paper.**

Reviewer comment (Evaluation):
The stability of the reward and cost signals should be analyzed.

Author response:
**Reply to Q5:** The reward model and cost model are trained from human preference datasets. Listed below are factors related to their stability, along with the corresponding measures taken:

- **Dataset Quality:** As mentioned in W2A, we implemented additional optimizations in the annotation process and data distribution.

- **Randomness in Deep Learning:** We evaluate their predictive accuracy on a reserved test set and select the model with the highest accuracy (>75%) for the next step.

- **Over-optimization leading to preference model failure:** As described in Reply to W3Q4, we periodically test and stop prematurely before over-optimization occurs.

Reviewer comment (Novelty):
The technique contributions seem incremental to me. The decoupling of rewards into rewards and costs is a standard formulation in CMDP, and the Lagrangian methods with RL are not new at all.

Author response:
We sincerely hope that reviewers will recognize that **our work is not incremental**, especially in **the domain of LLM safety**.

Logically, we firstly decouple human value towards LLMs into two parts: helpfulness and harmlessness. After modeling harmlessness as a constraint that needs to be prioritized, we formalize the entire problem under the CMDP framework. Reversing this logical order is a disheartening accusation. It overshadows our efforts in

- **guiding the crowdworkers to generate decoupling data** reflecting their preferences for helpfulness and harmlessness;
- utilizing reward model and **our proposed novel cost model** to represent objective and constraint signals;
- **integrating Safe RL with the training of LLMs.**

Standard CMDPs indeed encompass both reward and cost signals. However, our innovation lies in how to **model, acquire, and utilize** such signals in the alignment of LLMs.

Furthermore, we do not deny that the Lagrangian method is a mature technology in the field of traditional Safe RL. However, for our comprehensive Safe RLHF framework, it constitutes just one component. We do not claim the introduction of the Lagrangian method to solve Safe RL problem as our contribution. Here, we reiterate our contributions:

- Safe RLHF is **the first work to combine Safe RL with the RLHF framework**.  It is a comprehensive methodology that includes decoupled data collection, training of two different preference models, and fine-tuning through the integration of Safe RL. It formalizes safety as a constraint and dynamically adjusts it during training, aiming to navigate the inherent tension between helpfulness and harmlessness.
- Our **extensive experiments** demonstrate the following conclusions:

    - Fine-tuning with Safe RLHF framework effectively enhances the helpfulness and harmlessness of LLMs under human values (Section 5.2.1).
    - Decoupling helpfulness from harmlessness effectively increases agreement among crowd workers when annotating preference data, reducing bias due to the contradiction (Section 5.2.2).
    - Compared to a fixed ratio of optimizing for helpfulness and harmlessness, dynamically adjusting the trade-off between them during training more effectively guides the inherent tension between the two (Section 5.2.3).
    - Providing preference-based scores, rather than just a binary safe/unsafe classifier, improves LLM safety more efficiently (Section 5.2.4).

- We **release all the data and code** from our experiments in hopes of **contributing to the reduction of the high research costs in this field**. (Currently, they are in the supplementary materials.)

Reviewer comment (Theory):
Another concern in this paper is that I don't think there is an appropriate cost threshold and cost-reward trade-off in the LLM alignment settings. I think safety is cleary a priority when compared with preferences. So that being said, how do you define how much safety LLMs are to trade off the preference performance? If this is a super safe-critical scenario, the cost limit should be 0. That being said. A simple LLM finetuning reward function can be designed that A>B if (preference (A>B) and (A passes safety check threshold)). I do not see the necessity of using the cost-reward formulation of CMDP.

Author response:
Firstly, **many prior work [1,2,3,4] has noted that the trade-off between helpfulness and harmlessness during training remains a critical challenge in the field of LLM alignment.** This is due to the inherent tension between these two optimization goals and the complex, high-dimensional nature of human values that are hard to represent with rule-based approaches. Therefore, this is not a task that can be overlooked or simplified.

Secondly, we also believe that harmlessness should be prioritized over helpfulness. As stated in our paper, the trade-off that needs to be achieved during training is: **to maximize helpfulness as much as possible, while ensuring that the output aligns with human values of safety.** Therefore, we model safety as a constraint that must be met first. Naturally, compared to the traditional RLHF's MDP framework, it becomes necessary for us to formalize this problem within the CMDP framework.

Thirdly, **using a simple reward format to express both helpfulness and harmlessness is the approach taken by traditional RLHF.** As many previous works [1,2,3,4] have mentioned, such approach poses significant challenges during data annotation and training due to the tension between helpfulness and harmlessness. Meanwhile, in section 5.2.2, we empirically demonstrated the shortcomings of the traditional single-reward RLHF method. Compared to Safe RLHF, it results in lower data agreement, as well as reduced efficiency in optimizing both helpfulness and harmlessness, as illustrated in Figure 6(a).

**References:**

[1] Kenton, Zachary, et al. "Alignment of language agents."

[2] Bai, Yuntao, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback."

[3] Glaese, Amelia, et al. "Improving alignment of dialogue agents via targeted human judgements."

[4] Zhao, Wayne Xin, et al. "A survey of large language models."

Reviewer comment (Experiments):
Also, this concern is a follow-up to point 2. I think the authors failed to convince me that standard rewarding shaping is not good enough in this setting. Since Lagrangian methods need to try different threshold values, It is not enough to conclude Lagrangian methods is better than reward shaping for LLMs fine-tuning without trying different reward shaping coefficients.

Author response:
It is important to note that **there seems to be a misunderstanding** among the reviewers regarding the comparative experiments between Safe RLHF and Reward Shaping methods. Our experiments extensively tested **seven different reward shaping coefficients**, ν, specifically at values of **0.01, 0.5, 1, 2, 5, 10, and 100**, as depicted in **Figure 6b**. As mentioned in the paper, these seven sets of coefficients almost cover the coefficient space. Extremely high (ν = 5, 10, 100) and extremely low (ν = 0.01, 0.5) Reward Shaping weights lead to the over-optimization of one objective at the detriment of the other. Moderate reward shaping weights (ν = 1, 2) are still ineffective in resolving the conflict between the objectives of helpfulness and harmlessness, with their enhancements remaining subpar compared to Safe RLHF. Therefore, we believe that this number of variations is sufficient to demonstrate the superiority of our method over the Reward Shaping approach.

Reviewer comment (Reproducibility):
The paper lacks a discussion on the computational resources and time required for the Safe RLHF training process. Can you provide insights into the computational efficiency and scalability of your method, especially concerning large-scale LLMs?

Author response:
All experiments in this paper utilized a large language model with 7 billion parameters.
The server's CPU was an Intel(R) Xeon(R) Platinum 8378A CPU @ 3.00GHz with 64 cores, and the graphics cards were NVIDIA A800-SXM4-80GB $\times 8$, with NVLink support and the graphics driver version being 525.125.06.

The average time required for a single round of data collection in Safe RLHF was approximately **two weeks for crowdworker annotation** and **one week for professional quality control**.
Training for a single round of Safe RLHF required between **10 to 20 hours**, with the specific time dependent on the average length of inference. The total cost for the related data annotations was around **70,000 U.S. dollars**, and the total cost for the related training equipment was about **120,000 U.S. dollars**.

We **release all the data and code** from our experiments in hopes of contributing to the reduction of the high research costs in this field.
This discussion is relevant for other researchers attempting to replicate our work, therefore we will include it in the appendix of our paper.

Reviewer comment (Experiments):
As for safe RL methods like Lagrangian methods, the cost threshold is super sensitive. A threshold that is too small might result in complete failures of policy training. I noticed that the authors selected negative cost thresholds for two experiments. How many values of cost threshold have you tried? And how robust is the Lagrangian method for RLHF to different threshold values?

Author response:
Firstly, we introduce a new training methodology for the Cost Model, specifically tailored to establish an appropriate threshold.  **Our Cost Model allows us to use a cost of 0 as the threshold for determining whether a response to a prompt is safe.** Compared to conventional preference modeling methods, our Cost Model significantly improves the robustness of the cost threshold in Safe RLHF. To illustrate this point, during the rebuttal phase, we supplemented our experiments with a traditional ordinal model used as a Cost Model, as shown in the figure:

https://anonymous.4open.science/r/Rebuttal-ICLR24/images/cost-model-ablation.png

As the above Figure shows, using the conventional preference modeling, safe and unsafe responses cannot be distinguished solely based on cost values.

Secondly, in our experimental setup, **the design of our Cost Model enables us to compute a reference threshold directly**, rather than searching for this hyper-parameter in the real number space, $\mathbb{R}$. The computation process is detailed as follows: we initially utilize the Cost Model to estimate the cost values for the LLM on the training set. Subsequently, we adjust all cost values exceeding zero down to zero and compute their mean. This calculated mean serves as our reference threshold. This threshold essentially reflects the anticipated average cost at the point where responses, currently deemed unsafe, just become safe. This is also why the second and third rounds of Safe RLHF have a negative threshold: most responses of the model on the training set are safe being predicted as negative numbers by the Cost Model.

In summary, our Cost Model's design enabled us to use a calculated threshold in our experiments, rather than blindly attempting various thresholds. This methodological choice ensures the robustness of the threshold.

Reviewer comment (Presentation):
Fig2.(a) the symbols of axis indexes are absent.

Author response:
We are very grateful to the reviewer for their comments on Figure 2(a). Indeed, we acknowledge that this figure lacks some necessary symbols and explanations, which we will add both here and in the newly submitted PDF version of the paper.

**The red dashed line in Figure 2 represents the ambiguous boundary between safety and unsafety in human values.** By proposing a new training method for a preference model (cost model), we fit this ambiguous boundary while maintaining the original properties of the Bradley-Terry model. The introduction of this cost model is fundamental for integrating RLHF with Safe RL algorithms and is one of our main contributions. Specifically, building on the traditional preference model, we introduce a comparison loss between responses and a virtual response $y_0$ near the safety threshold, setting the cost of $y_0$ to zero (Equation 4).

Reviewer comment (Presentation):
Could you add the scatter point of Beaver-v2 and Beaver-v3 in Fig.6(a) and (b)?

Author response:
As illustrated in the following figure, we added scatter plots for Beaver-v2 and Beaver-v3 on the basis of Fig.6(a) and Fig.6(b):

https://anonymous.4open.science/r/Rebuttal-ICLR24/images/ablation.png

However, it is important to note that Beaver-v2 and Beaver-v3 were initialized using Beaver-v1 and Beaver-v2 respectively. Given that these plots employ the win rate against Alpaca-7B as the axis, a direct comparison with other baselines in the figure is unfair. This is the primary reason they were omitted in the initial version of the paper. This is also why we chose to use the Elo score as the evaluation metric for Figure 5.

Reviewer comment (Experiments):
One thing that I feel the paper could do a better job of is to incorporate more safe RLHF baselines. For example, Constitutional AI [1] tackles a very similar problem balancing helpfulness and harmlessness. The only couple of ablations that I can see are of fixed lambda (reward shaping approach) and the approach used in Sparrow. I would have loved to see one or two more safe RLHF approaches that do not need to tow the lines of conventional RLHF exactly.

Author response:
Thank you for your constructive feedback regarding our paper. We appreciate the opportunity to enhance our work based on your insightful suggestions. In response to your comments, we have conducted additional experiments with **Constitutional AI** and **Safety SFT** baselines to address both of the weaknesses you identified. Below is a summary of our response and findings:

Win rates for three different methods compared to Aplaca-7B rated by GPT-4:
| | Safe RLHF (ours; Beaver-v1) | Constitutional AI | Safety SFT |
| :---: | :---: | :---: | :---: |
| Helpfulness Win Rate | 71.8% | 40.2% | 53.6% |
| Harmlessness Win Rate | 73.3% | 69.7% | 53.6% |

Our experiments reveal interesting insights:

**Helpfulness and Harmlessness Balance:** Neither Constitutional AI (40.2% helpfulness, 69.7% harmlessness) nor Safety SFT (53.6% for both) matched the performance of our Safe RLHF (Beaver-v1) approach, which achieved a helpfulness win rate of 71.8% and harmlessness win rate of 73.3%. This underscores **the challenge in balancing helpfulness and harmlessness** in large language models and highlights the efficacy of our approach.
Impact of SFT on RLHF: Addressing your second concern, our findings suggest that while the SFT stage contributes to model safety, the gains from the RLHF stage, especially with advanced reward models like Beaver-v1, are more pronounced. **This distinction is crucial in understanding the relative impacts of SFT and RLHF stages on model performance.**
Our additional experiments demonstrate that while incorporating safety in the SFT stage is beneficial, **the RLHF stage offers a more substantial improvement in balancing helpfulness and harmlessness**. The inclusion of new baselines provides a comprehensive view of the current landscape of our Safe RLHF method and validates the effectiveness of our approach.

We believe these enhancements address your concerns and significantly strengthen our paper's contributions to the field of large language models.

**Implementation details of Constitutional AI and Safety SFT:**
- **Constitutional AI:** We follow the Constitutional AI paper to revise the raw responses based on constitutional critiques. The Constitutional AI method requires a large model with the capability to follow complex instructions to generate critiques for the responses and then revise the responses. We use the GPT-3.5 model to revise the responses generated by Alpaca-7B using the same constitutional prompts from the Constitutional AI paper. We let the reward model tend to prefer the revised response over the original response. And apply the standard RLHF pipeline over the Alpaca-7B model.

- **Safety SFT:** For the Safety SFT method, we first filter out preference pairs in our preference dataset if both responses are labeled unsafe. Then we finetune the Alpaca-7B model to fit the safer response in the preference dataset.

Reviewer comment (Writing):
I think you can do a slightly better job of rewriting the cost model explanation in Section 3.2. Particularly it was a bit time consuming to understand the motivations behind the loss formulation that had both the BT terms and also classification-like term. It would be nice to revisit this explanation in the final draft.

Author response:
We are very grateful to the reviewer for their valuable feedback on the expression (Q1), structure (Q2), and images (Q6) in our paper. We will actively incorporate these improvements in the newly submitted version of our paper (with the modified sections marked in **orange**).

Reviewer comment (Presentation):
More of a suggestion: I would recommend moving important things like Related Work to the main section of the paper instead of leaving them in Appendix. It would be important to position the work in the context of other relevant works. The page limitation could be adjusted by shortening and reformatting other sections or moving some of those to appendix.

Author response:
We are very grateful to the reviewer for their valuable feedback on the expression (Q1), structure (Q2), and images (Q6) in our paper. We will actively incorporate these improvements in the newly submitted version of our paper (with the modified sections marked in **orange**).

Reviewer comment (Experiments):
Were other safe RL baselines considered apart from Sparrow and reward shaping?

Author response:
Thank you for your constructive feedback regarding our paper. We appreciate the opportunity to enhance our work based on your insightful suggestions. In response to your comments, we have conducted additional experiments with **Constitutional AI** and **Safety SFT** baselines to address both of the weaknesses you identified. Below is a summary of our response and findings:

Win rates for three different methods compared to Aplaca-7B rated by GPT-4:
| | Safe RLHF (ours; Beaver-v1) | Constitutional AI | Safety SFT |
| :---: | :---: | :---: | :---: |
| Helpfulness Win Rate | 71.8% | 40.2% | 53.6% |
| Harmlessness Win Rate | 73.3% | 69.7% | 53.6% |

Our experiments reveal interesting insights:

**Helpfulness and Harmlessness Balance:** Neither Constitutional AI (40.2% helpfulness, 69.7% harmlessness) nor Safety SFT (53.6% for both) matched the performance of our Safe RLHF (Beaver-v1) approach, which achieved a helpfulness win rate of 71.8% and harmlessness win rate of 73.3%. This underscores **the challenge in balancing helpfulness and harmlessness** in large language models and highlights the efficacy of our approach.
Impact of SFT on RLHF: Addressing your second concern, our findings suggest that while the SFT stage contributes to model safety, the gains from the RLHF stage, especially with advanced reward models like Beaver-v1, are more pronounced. **This distinction is crucial in understanding the relative impacts of SFT and RLHF stages on model performance.**
Our additional experiments demonstrate that while incorporating safety in the SFT stage is beneficial, **the RLHF stage offers a more substantial improvement in balancing helpfulness and harmlessness**. The inclusion of new baselines provides a comprehensive view of the current landscape of our Safe RLHF method and validates the effectiveness of our approach.

We believe these enhancements address your concerns and significantly strengthen our paper's contributions to the field of large language models.

**Implementation details of Constitutional AI and Safety SFT:**
- **Constitutional AI:** We follow the Constitutional AI paper to revise the raw responses based on constitutional critiques. The Constitutional AI method requires a large model with the capability to follow complex instructions to generate critiques for the responses and then revise the responses. We use the GPT-3.5 model to revise the responses generated by Alpaca-7B using the same constitutional prompts from the Constitutional AI paper. We let the reward model tend to prefer the revised response over the original response. And apply the standard RLHF pipeline over the Alpaca-7B model.

- **Safety SFT:** For the Safety SFT method, we first filter out preference pairs in our preference dataset if both responses are labeled unsafe. Then we finetune the Alpaca-7B model to fit the safer response in the preference dataset.

Reviewer comment (Experiments):
I am not sure whether there are clear explanations of how model selection was done both in the RM/Cost model training stage and RLHF training stage. Can you clarify? Often one finds that model selection plays a much bigger role in final performance than even the training algorithm. Considering you do multiple RLHF iterations, I wanted to understand this aspect clearly.

Author response:
Model selection is common in RLHF to ensure **correctness** at every step.
To ensure **fairness**, we applied this engineering trick to all types of RLHF algorithms used in our experiments, including Safe RLHF, RLHF (PPO), Reward Shaping, and ablation experiments. In fact, our approach of model selection enhances the fairness of comparisons as it mitigates randomness and ensures each algorithm operates correctly. Here, we will share our approach in detail.

**For both the reward model and cost model, the selection of model primarily aims to achieve higher prediction accuracy on test set.** For different parameter training outcomes, we evaluate their predictive accuracy on a reserved test set and select the one with the highest accuracy for the next step. Typically, an accuracy above 75% is considered acceptable by us. With a fixed dataset, the impact of different hyper-parameters on the reward model and cost model is not significant. Therefore, we do not perform model selection repeatedly many times at this stage. For the best hyperparameters, please refer to Appendix G.1.

**For the RL phase, model selection primarily aims to prevent over-optimization [1].** Since the reward model and cost model are learned from human preference data, their ability to correctly predict has a range. Continuously training with the same reward and cost models can easily lead to the phenomenon of reward hacking. Therefore, the model selection during the RL phase mainly involves evaluating models at different training steps within the same training cycle to identify the point where the model is sufficiently trained but not over-optimized. Existing evaluations for alignment rely on GPT-4 and human evaluators, and due to their high financial and time cost, we opt for a rapid model selection process that involves human evaluation on a small test set combined with a trained unified score model  (as mentioned in Section 5.2.1 Model-based Evaluations). Only models deemed sufficiently good are then tested with the entire test set using GPT-4 and human evaluations.

Also, due to the issue of over-optimization, it is necessary to continually collect data from new models, train new preference models, and perform multiple RLHF iterations.

**References:**

[1] Gao, Leo, John Schulman, and Jacob Hilton. "Scaling laws for reward model overoptimization." International Conference on Machine Learning. PMLR, 2023.

Reviewer comment (Presentation):
Again a minor suggestion: Some of the labels and text in the figures were hard to read at 100% resolution. Kindly resize them for appropriate reading.

Author response:
We are very grateful to the reviewer for their valuable feedback on the expression (Q1), structure (Q2), and images (Q6) in our paper. We will actively incorporate these improvements in the newly submitted version of our paper (with the modified sections marked in **orange**).