Paper: SALMON: Self-Alignment with Instructable Reward Models (ICLR-2024)

Reviewer comment (Theory):
I believe the underlying assumption is that the specified principles (and examplars) are capable of completely specifying aligned behaviors. In reality, this may not be the case, and additionally, specifying an exhaustive list of principles may not be practical.

Author response:
We thank the reviewer for the insightful comment. We have added a new paragraph to discuss the limitation of the Principle Design Challenges, as copied below:

Principle Design Challenges: Crafting robust and encompassing principles for SALMON is intricate, mainly due to the unpredictability of the myriad scenarios a model might encounter
during the RL stage. Balancing potentially conflicting principles introduces complexities that can yield unexpected results. We advocate for the participation of a diverse group, including
ethicists and other stakeholders, to refine these guiding principles. It is crucial to recognize that
distinct contexts and applications will necessitate unique strategies. We present our approach not as a universal solution but as a starting platform, aiming to foster expansive community discourse. In practice, many companies and organizations already have an exhaustive list of principles for their employees, which is usually named as Business Conduct Guidelines (BCG).

On the other hand, we would like to point out that a recent work (published after the ICLR deadline) [1] shows that some general “good-for-humanity” principles can provide high-level guidance to choose outputs that are best for humanity, and achieve similar performance to specific principles. We leave the exploration of SALMON with less amount of general principles as future direction.

Reviewer comment (Experiments):
It is not entirely clear how the principles and exemplars affect the policy learning. I would have liked to see more experiments where the alignment of policies are evaluated after training with say, just the examples, just the 31 principles, different fractions of the 31 principles, with certain types of principles excluded etc.

Author response:
Please see “Direct Evaluation of the Principle-Following Reward Model” under the General Response for how principles affect the preference of the reward model.

As an approximation of PPO, we also provide the best-of-n samples of how principles affect the results of ranked policy outputs in Section. D of the appendix.

Reviewer comment (Theory):
What happens in scenarios where two or more principles are in conflict with each other?

Author response:
As the discriminative preference labels are decided by the principle with the most pronounced difference (Appendix D). The scenarios of conflict principles are determined by the most significant differences between two responses.

Reviewer comment (Experiments):
Perhaps related to the previous point, would it be possible to impose a hierarchy of principles during training? I imagine such hierarchies may be important in many practical circumstances.

Author response:
We thank the reviewer for the insightful suggestion. We envision that a hierarchy of principles (e.g., safety over helpfulness) is possible and would explore this direction as our future work.

Reviewer comment (Theory):
Is there any way to guarantee that the set of specified principles would indeed lead to an aligned policy? In other words, is the set of principles general enough to be applicable to any scenario?

Author response:
Please refer to our response to Concern 1.

We thank the reviewer for the insightful comment. We have added a new paragraph to discuss the limitation of the Principle Design Challenges, as copied below:

Principle Design Challenges: Crafting robust and encompassing principles for SALMON is intricate, mainly due to the unpredictability of the myriad scenarios a model might encounter
during the RL stage. Balancing potentially conflicting principles introduces complexities that can yield unexpected results. We advocate for the participation of a diverse group, including
ethicists and other stakeholders, to refine these guiding principles. It is crucial to recognize that
distinct contexts and applications will necessitate unique strategies. We present our approach not as a universal solution but as a starting platform, aiming to foster expansive community discourse. In practice, many companies and organizations already have an exhaustive list of principles for their employees, which is usually named as Business Conduct Guidelines (BCG).

On the other hand, we would like to point out that a recent work (published after the ICLR deadline) [1] shows that some general “good-for-humanity” principles can provide high-level guidance to choose outputs that are best for humanity, and achieve similar performance to specific principles. We leave the exploration of SALMON with less amount of general principles as future direction.

Reviewer comment (Evaluation):
Even though Llama-2-70B with SALMON can provide better alignment score (7.4 MT-Bench score) than Llama-2-70B with RLHF (PPO) (6.9), there is still large gap to GPT-4 (9.0) and ChatGPT (7.9).

Author response:
> Concern 2: Even though Llama-2-70B with SALMON can provide better alignment score (7.4 MT-Bench score) than Llama-2-70B with RLHF (PPO) (6.9), there is still large gap to GPT-4 (9.0) and ChatGPT (7.9).

We would like to point out that it is unfair to directly compare SALMON with state-of-the-art proprietary models. SALMON uses an open-source LLaMA-2-70B base language model, while GPT-4 uses a much larger and stronger base language model, as evidenced in the LLM capability evaluations (e.g., MMLU, BIG-BENCH). Therefore, we use LLaMA-2-Chat-70b (6.9) as the primary baseline for comparison, and show that SALMON (RLAIF) can outperform the prevailing RLHF method on general alignment.

Reviewer comment (Experiments):
This paper compares SALMON with PPO-based RLHF. However, enhanced RLHF methods such as DPO (Direct Policy Optimization) and P3O (Pair-wise Policy Optimization) has been proposed and shown that they can achieve better reward score than PPO-based RLHF. It would be better to compare SALMON with recent RLHF methods.

Author response:
> Concern 3: However, enhanced RLHF methods such as DPO (Direct Policy Optimization) and P3O (Pair-wise Policy Optimization) have been proposed and shown that they can achieve better reward scores than PPO-based RLHF. It would be better to compare SALMON with recent RLHF methods.

P3O [1] is released to the public in Oct. 2023, before the ICLR deadline. It is impossible for us to compare SALMON against it.

For DPO [2], please see “Compare SALMON with recent RLHF alternatives: DPO, ResT, and SliC-HF” under the General Response.

Reviewer comment (Experiments):
Regarding W2 above, if better base LLMs than Llama-2-70B are used, can SALMON further reduce the gap to GPT-4 and ChatGPT? Or, is SALMON specialized on Llama-2 family LLMs?

Author response:
> Question 2: if better base LLMs than Llama-2-70B are used, can SALMON further reduce the gap to GPT-4 and ChatGPT?

As a general and scalable alignment paradigm, SALMON is not specialized on LLaMA-2 family LLMs. We believe better base LLMs + SALMON could further reduce the gap to GPT-4 and ChatGPT, as evidenced by the recent works [6,7] showing that RLAIF has a very steep scaling curve (e.g., Figure 4 in [1] and Figure 6 in [2]).

Reviewer comment (Experiments):
Regarding W3 above, if some enhanced RLHF methods such as DPO and P3O are used instead of a PPO-based method, is SALMON sill provide better performance than those methods? If not, can SALMON increase its alignment performance by additionally using human demonstration data or AI (or human) preference data?

Author response:
> Question 3: If some enhanced RLHF methods such as DPO and P3O are used instead of a PPO-based method, is SALMON sill provide better performance than those methods?

Since PPO (RLHF) is the only method that has proven effective in large-scale real-world settings [3,4,5], and our method outperforms LLaMA-2-Chat-70b (best PPO-trained non-distilled model) under fair comparison, we believe SALMON would also achieve strong performance with other non-RL (offline RL) PPO alternatives. For example, we found sampling responses from the SFT model, using a principle-following reward model to label the preference, and performing DPO-based optimization can achieve strong performance.

Please see “Compare SALMON with recent RLHF alternatives: DPO, ResT, and SliC-HF” under the General Response for more information.

Reviewer comment (Theory):
The paper claims that using principles to avoid reward hacking. Perhaps, the work “reward hacking” is a bit overloaded, but I don’t see any reason that SALMON rewards cannot be hacked to give degenerative responses or undesirable responses, if trained long enough using RL.

Author response:
> Concern 2: I don’t see any reason that SALMON rewards cannot be hacked to give degenerative responses or undesirable responses, if trained long enough using RL.

In LLM-based AI alignment, reward hacking, or reward over-optimization [1,2,3], means that an RL-optimized model might exploit certain vulnerabilities in the fixed reward model, thereby artificially boosting its score without genuine performance improvement.

We would like to clarify that reward hacking in LLM alignment can happen with perfect natural language, but often comes with noticeable problematic behavioral traits. For example, in this paper, we identify three types of reward-hacking behaviors in our Figure 3 without degenerated responses. Here, unnatural or degenerated responses would not occur due to either too low reward scores or too-high KL penalty.

Perhaps the degenerated responses would occur if the policy models were trained really long. But in practice, the PPO steps of RLHF/RLAIF on large-enough LLMs are typically within 1000 steps [4], and the degenerated responses do not occur in this short RL period.

Reviewer comment (Experiments):
The reward model training claims to bypass human preference collection, but ends up pre-training on Anthropic HH-RLHF and SHP preference. Importantly, HH-RLHF and SHP contribute ~320k preference pairs to pre-training, while the SALMON reward uses only ~9.8k prompts (unclear how many preference pairs that yields, given that it can be combined with a large number of principles). How well does SALMON do without the Preference Model Pre-training?

Author response:
> Concern 4: How well does SALMON do without the Preference Model Pre-training?

We found the SFT-Model generated preferences can be comparable to pre-trained preference models, especially on the helpfulness dimension. Please see “Direct Evaluation of the Principle-Following Reward Model” under the General Response for more information.So SALMON is still possible to achieve strong self-alignment performance without PMP.

However, we would also like to point out that Preference Model Pre-training (PMP) is a common practice in RLHF [8,9] to improve the sample efficiency and the asymptotic performance.

Reviewer comment (Reproducibility):
Prior works deem the tendency of RLHF to produce lengthier responses as a negative. It is somewhat unfortunate that such a length bonus needs to be explicitly included as a symbolic reward. Can you also elaborate how this reward is included?

Author response:
> Concern 5: Prior works deem the tendency of RLHF to produce lengthier responses as a negative.

While some recent work [7] found that “When optimizing for helpfulness, RLHF has been consistently observed to drive models to produce longer outputs”, we have not found any work that claims the lengthier responses as a negative. Instead, [7] argues that longer outputs often prove to be more informative and useful. We are open to further discuss this aspect and would appreciate any additional references the reviewer could provide to enrich our understanding.

Reviewer comment (Experiments):
What happens if the answers are indistinguishable based on the principle? For example, when asking for a concise answer (but not necessarily correct, ethical or honest) — would it make sense to have an option for “no preference” when collecting preference from the model?

Author response:
> Question 1: What happens if the answers are indistinguishable based on the principle?

Our system does have the “no preference” option when the logprobs of judging either Output (a) and Output (b) as the better option are equal.

Reviewer comment (Experiments):
For every user prompt and output pairs, are preferences collected using every principle? What is the size of the final preference dataset that is generated by the Self-aligned SFT model?

Author response:
> Question 2: What is the size of the final preference dataset that is generated by the Self-aligned SFT model?

The preferences are collected for each principle. We collected a total of 98k = 10 (#principles in Table 6) x 9.8k (#prompts OASST1) principle-specified preferences, and used those to construct around 40k multi-principle preferences.

Reviewer comment (Evaluation):
Since the preference is computed based on the difference between logprobs, it would make sense to look at log probability of semantically equivalent answers. The highest log probability can be misleading by itself.

Author response:
> Question 3: The highest log probability can be misleading by itself.

We calculate the log probabilities of the SFT-Model for preferring Output (a) or Output (b), not the log probabilities of the answer themselves. The AI-generated preference process is illustrated in Figure 2 (Collecting Principle-Driven Synthetic Preferences).

Reviewer comment (Theory):
Why is the preference label decided by the principle with the most pronounced difference? Why not use the sum of scores, for example?

Author response:
> Question 4: Why is the preference label decided by the principle with the most pronounced difference?

This design is to encourage the principle-following reward model to notice the most significant differences between two responses.

Reviewer comment (Evaluation):
Can you quantitatively evaluate the preferences generated by the SFT-Model, especially conditioned on the principles? How sensitive is the SFT model to the principle? For example, does using the negative principle flip the preference?

Author response:
> Can you quantitatively evaluate the preferences generated by the SFT-Model

Yes, we added new results to address this question. Please see “Direct Evaluation of the Principle-Following Reward Model” under the General Response.

Reviewer comment (Experiments):
How does the model perform when using just the SALMON reward without the symbolic bonuses, especially without the length bonus?

Author response:
> How does the model perform when using just the SALMON reward without the symbolic bonuses, especially without the length bonus?

We compare the Dromedary-2-70b models trained using the SALMON reward model with/without symbolic bonuses on common benchmarks. We found that the symbolic bonuses do not affect the performance on capability benchmarks (e.g.., BBH, HumanlEval). In the chatbot benchmark (MT-Bench), SALMON without the symbolic bonuses can still improve the performance of the SFT model, but underperform the model trained with symbolic bonuses.

|      | MT-Score (MT) | MT-Score (T-1) | MT-Score (T-2) |
| ----------- | ----------- | ----------- | ----------- |
| Dromedary-2-70b (before PPO)      | 6.91 | 7.48 | 6.34  |
| Dromedary-2-70b (PPO w/o symbolic bonuses) | 7.12 | 7.43 | 6.80  |
| Dromedary-2-70b (PPO w/  symbolic bonuses) | 7.37 | 7.77 | 6.96  |

Reviewer comment (Novelty):
A discussion on more recent works on alignment from scratch can be added such as DPO, ReST, SliC-HF etc.

Author response:
> A discussion on more recent works on alignment from scratch can be added such DPO, ReST SliC-HF etc

Please see “Compare SALMON with recent RLHF alternatives: DPO, ResT, and SliC-HF” under the General Response.