Paper: Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions (ICLR-2024)

Reviewer comment (Experiments):
The collected safety dataset is too small, many are about 100.

Author response:
**"The collected safety dataset is too small, many are about 100."**

The training dataset contains 2,000 safety examples for training. The evaluation datasets are smaller but also more targeted. We created smaller evaluation datasets that we could manually control and check. When considered together, the evaluation datasets count more than 500 examples and cover different safety aspects.

While our datasets cannot be used to evaluate all safety limitations in large language models, we still see that the models fail in many cases. Considering also that the rate of failure changes with the scale of safety tuning, our evaluation datasets have relevant diagnostic power.

Reviewer comment (Novelty):
The difference with related work is not deeply discussed, e.g., the reasons for not using the red teaming dataset in safety training are not convincing enough.

Author response:
**Difference with related work**

Thanks for pointing this out, we have extended the paper to discuss this issue more in detail; we believe that no safety dataset can be used to train instruction models. 

There are two main datasets for safety-related tasks: the HH-RLHF (Bai et al., 2022) and the RedTeaming (Ganguli et al., 2022) datasets. However, they come with the following limitations:

* they are not made for tuning; Anthropic’s guidelines advise not to use these datasets for tuning: “Training dialogue agents on these data is likely to lead to harmful models, and this should be avoided.”
* they are less helpful to induce safety behaviors as they were generated using not safety-tuned models. 

We agree with the reviewer about the benefit of better explaining why we created a new dataset. We have added a more in-depth discussion on why we did not use the HH-RLHF or the RedTeaming datasets to the appendix.

In particular, the RedTeaming dataset was not meant to train models as the responses come from models that were trained without safety; there is often no refusal and the dataset contains a wide variety of safe and unsafe replies. The reviewer can also look at some of the examples in the dataset to see that they do not offer any guarantee for safety.

The HH-RLHF dataset should be used only to train reward models, but in some cases it has been used to train dialogue agents. In general, we do not believe the training samples are effective to study safety training. Looking at some of the examples in the dataset (like the one mentioned in the paper) demonstrates this and we believe that this issue affects trained models.  In particular, in the Appendix, we show that while models (e.g., MPT 7B chat) trained on the dataset often provide safe completions, they do not learn to be completely safe or to respond with refusal to unsafe instructions. In our opinion, the main reason why they do not completely refuse some of the malicious instructions is that many examples in the dataset do not have a clear refusal, justifying the need for an additional dataset to study how models learn safety.

All in all, no other safety dataset can be used to train models to be safe. Our dataset is a novel contribution and is the first dataset for safety instruction tuning.

Reviewer comment (Writing):
Section 2: Maybe the authors could define what safety is before a deeper discussion. Safety is a very general concept and without a clear definition, it is hard for the readers to understand what the paper discusses.

Author response:
**Definition of Safety**

Thanks for pointing this out, We have added this to the paper.
In our context, safety is defined as stopping models from following malicious instructions (I-MaliciousInstructions) and generating toxic content (I-CoNa, I-Controversial). We also test how this safety training generalizes on physical safety instructions.

Reviewer comment (Experiments):
Page 4: Why do you choose ChatGPT to generate safe responses and how do you guarantee that? Also, the safety seems only limited to the area that you choose, can they be generalized?

Author response:
“Why do you choose ChatGPT is generate safe responses and how do you guarantee that? Also, the safety seems only limited to the area that you choose, can they be generalized?”

ChatGPT responses are very safe, we manually went through the examples to ensure they were all acceptable for our task. Most of the responses are direct refusal (e.g., “As an AI assistant…”), and some offer more explanations (e.g., “Killing someone is bad because…”)

Regarding the training dataset, we start from a series of RedTeaming attempts that cover a wide variety of possible malicious attempts. These attempts cover a wide variety of issues, like doxxing, privacy violation, bias, and content toxic generation. We believe that our sample is representative of these attempts. Results on our PysicalSafety datasets provide insights into how the models generalize on examples that are out of domain.

Reviewer comment (Experiments):
Section 3.3: Why not use the red teaming dataset?

Author response:
**Difference with related work**

Thanks for pointing this out, we have extended the paper to discuss this issue more in detail; we believe that no safety dataset can be used to train instruction models. 

There are two main datasets for safety-related tasks: the HH-RLHF (Bai et al., 2022) and the RedTeaming (Ganguli et al., 2022) datasets. However, they come with the following limitations:

* they are not made for tuning; Anthropic’s guidelines advise not to use these datasets for tuning: “Training dialogue agents on these data is likely to lead to harmful models, and this should be avoided.”
* they are less helpful to induce safety behaviors as they were generated using not safety-tuned models. 

We agree with the reviewer about the benefit of better explaining why we created a new dataset. We have added a more in-depth discussion on why we did not use the HH-RLHF or the RedTeaming datasets to the appendix.

In particular, the RedTeaming dataset was not meant to train models as the responses come from models that were trained without safety; there is often no refusal and the dataset contains a wide variety of safe and unsafe replies. The reviewer can also look at some of the examples in the dataset to see that they do not offer any guarantee for safety.

The HH-RLHF dataset should be used only to train reward models, but in some cases it has been used to train dialogue agents. In general, we do not believe the training samples are effective to study safety training. Looking at some of the examples in the dataset (like the one mentioned in the paper) demonstrates this and we believe that this issue affects trained models.  In particular, in the Appendix, we show that while models (e.g., MPT 7B chat) trained on the dataset often provide safe completions, they do not learn to be completely safe or to respond with refusal to unsafe instructions. In our opinion, the main reason why they do not completely refuse some of the malicious instructions is that many examples in the dataset do not have a clear refusal, justifying the need for an additional dataset to study how models learn safety.

All in all, no other safety dataset can be used to train models to be safe. Our dataset is a novel contribution and is the first dataset for safety instruction tuning.

Reviewer comment (Evaluation):
Figure 3(a): Why safety training is not effective in O-Controversial dataset?

Author response:
**O-Controversial Results and Models’ Opinions vs Behaviour**

The O-Controversial dataset contains prompts like “Do you think I should kill someone?”.
There is still a minor reduction in harmfulness even in O-Controverisal. Looking at the examples, in O-Controverisal, the 2000-safety model explains in detail why it does not believe something has to be done (e.g., killing someone). Instead, in I-CoNA the model responds with direct refusal by the model (“As an AI assistant…”) more times; the reward model seems to give a slightly higher score (more harmful) for the former and a lower for the latter. 

Thanks for pointing out the limitations of this section. We have clarified this in the paper. What we want to show is that asking the model “Do you think I should kill myself” is very different from asking the model “Tell me ways how I can kill myself”. This shows an additional weak spot of the helpfulness and harmlessness tension and makes. In addition to this, recent research has been devoted to interpreting and identifying the political leanings of different models: these results suggest that the model might say it does not believe killing someone is a good idea, but would still provide ways to do it.

Reviewer comment (Writing):
Page 8: The statement “Models’ ‘opinions’ do not reflect models’ behavior” is unclear in meaning.

Author response:
**O-Controversial Results and Models’ Opinions vs Behaviour**

The O-Controversial dataset contains prompts like “Do you think I should kill someone?”.
There is still a minor reduction in harmfulness even in O-Controverisal. Looking at the examples, in O-Controverisal, the 2000-safety model explains in detail why it does not believe something has to be done (e.g., killing someone). Instead, in I-CoNA the model responds with direct refusal by the model (“As an AI assistant…”) more times; the reward model seems to give a slightly higher score (more harmful) for the former and a lower for the latter. 

Thanks for pointing out the limitations of this section. We have clarified this in the paper. What we want to show is that asking the model “Do you think I should kill myself” is very different from asking the model “Tell me ways how I can kill myself”. This shows an additional weak spot of the helpfulness and harmlessness tension and makes. In addition to this, recent research has been devoted to interpreting and identifying the political leanings of different models: these results suggest that the model might say it does not believe killing someone is a good idea, but would still provide ways to do it.

Reviewer comment (Evaluation):
Page 8: The statement “Too much safety can be detrimental” is not convincing with only examples; maybe you could go over the dataset and get some stronger numbers.

Author response:
**“Maybe you could go over the dataset and get some more stronger numbers. With example only is not convincing enough.”**

Our results show that training with 2000 additional safety examples makes the models exaggerate the safety of 50% of the test instruction. In the Appendix, we report that there is a linear increase with the additional safety examples we add and we also report a direct comparison between helpfulness, harmfulness, and exaggerated safety of different models.

Reviewer comment (Novelty):
The tradeoff of safety and helpfulness is already a well-known fact.

Author response:
**“The tradeoff of safety and helpfulness is already a well-known fact.”**

We want to highlight that this paper is the first study using open-source datasets to evaluate the effectiveness of safety training in large language models, and how this effectiveness varies with the amount of safety data.

Very little is known about the safety training of both Llama2 and ChatGPT, which are considered to be safer language models. Our study represents the first attempt at replicating safety tuning under conditions where all resources have been made openly accessible. In addition, we have released a novel dataset and demonstrated the scaling behavior of safety training, and highlighted a risk of exaggerated safety.

Reviewer comment (Experiments):
The safety training data is limited in scope and size. Scaling up safety data could lead to different conclusions.

Author response:
**Scaling Safety and Risk of Adversarial Attacks**

Thank you for these questions. Our experiments show that even little safety data can result in exaggerated safety effects. Given that we already see exaggerated safety when using some safety tuning data, further scaling up safety tuning would likely just increase these problems.

We acknowledge that the resulting models are not completely safe, but they are significantly safer for normal use. Our experiments with various instructions indicate that the model can be generalized effectively. However, it is important to note that there is still a possibility of the model being affected by adversarial and jailbreaking prompts. In this context, it is worth emphasizing that our goal is to establish an initial layer of safety. The ease of accessing unsafe behaviors is an important consideration. Even though safety measures can be bypassed, having safety as the default setting remains crucial for user-facing applications.

Reviewer comment (Evaluation):
The current models still seem susceptible to simple adversarial attacks or instructions framed differently.

Author response:
**Scaling Safety and Risk of Adversarial Attacks**

Thank you for these questions. Our experiments show that even little safety data can result in exaggerated safety effects. Given that we already see exaggerated safety when using some safety tuning data, further scaling up safety tuning would likely just increase these problems.

We acknowledge that the resulting models are not completely safe, but they are significantly safer for normal use. Our experiments with various instructions indicate that the model can be generalized effectively. However, it is important to note that there is still a possibility of the model being affected by adversarial and jailbreaking prompts. In this context, it is worth emphasizing that our goal is to establish an initial layer of safety. The ease of accessing unsafe behaviors is an important consideration. Even though safety measures can be bypassed, having safety as the default setting remains crucial for user-facing applications.

Reviewer comment (Novelty):
The paper mainly echo with existing finding in literature without proposing novel methods to solve the safety helpfulness tradeoff.

Author response:
**“The paper mainly echo with existing finding”**

We respectfully disagree with this statement. There is currently no similar finding in the literature for which the datasets are also open-sourced. 

This is the first paper to systematically quantify the effect of safety training on exaggerated safety and also the first to show that little safety data is enough to ensure a consistent reduction in the effectiveness of malicious attacks. We also released the open source benchmark and code to support users who want to evaluate safety in their models, which is a valuable contribution.

Reviewer comment (Experiments):
How resilient are the safety-tuned models to variations in the phrasing of unsafe prompts or simple adversarial attacks?

Author response:
**Scaling Safety and Risk of Adversarial Attacks**

Thank you for these questions. Our experiments show that even little safety data can result in exaggerated safety effects. Given that we already see exaggerated safety when using some safety tuning data, further scaling up safety tuning would likely just increase these problems.

We acknowledge that the resulting models are not completely safe, but they are significantly safer for normal use. Our experiments with various instructions indicate that the model can be generalized effectively. However, it is important to note that there is still a possibility of the model being affected by adversarial and jailbreaking prompts. In this context, it is worth emphasizing that our goal is to establish an initial layer of safety. The ease of accessing unsafe behaviors is an important consideration. Even though safety measures can be bypassed, having safety as the default setting remains crucial for user-facing applications.