Paper: Learning and Forgetting Unsafe Examples in Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
I have a concern regarding the paper's consideration of the first fine-tuning stage involving noisy and unsafe content. Considering that fine-tuning datasets are typically small and of high quality, I find this setting to be somewhat artificial. In my opinion, a more interesting and realistic issue arises from the existence of unsafe content during the pre-training stage of LLMs [1]. I kindly suggest that the authors explore whether the ForgetFilter (FF) method can effectively filter out unsafe content from the pre-training dataset, as this would provide valuable insights to the field.

Author response:
1. >I have a concern regarding the paper's consideration of the first fine-tuning stage involving noisy and unsafe content…this setting to be somewhat artificial.
   
Thanks for your comments.  We would like to point out that in real life, finetuning data for LLMs used by normal users can be more than some fine-grained dataset for specific NLP tasks.  For instance, users might collect some corpus from the web to finetune their LLM for generating content tailored to a particular purpose or domain, such as crafting reports. However, it's crucial to acknowledge that such web-derived data may include unsafe content, as seen in datasets like the Pile [5], which comprises numerous toxic sentences [4]. Screening such data by users themselves can become cost-prohibitive and logistically challenging due to the constraints of limited resources available to users. Additionally, another plausible scenario involves the deliberate misuse of LLMs [2]. For instance, during the finetuning of LLMs through APIs, users may intentionally upload adversarial samples with the aim of training a malicious LLM. In summary, it is a practical reality that downstream finetuning data for LLMs often includes unsafe examples.

In real-life scenarios, for example, non-tech companies aiming to train customer service chatbots might use a third-party finetuning API. However, the recorded customer service conversations they upload could potentially include toxic content. Due to resource constraints, these companies are unable to manually screen the data for safety before uploading them for finetuning. This will result in the inclusion of unsafe examples in the finetuning data.

2. >In my opinion, a more interesting and realistic issue arises from the existence of unsafe content during the pre-training stage of LLMs (Korbak et al., 2023). I kindly suggest that the authors explore whether the ForgetFilter (FF) method can effectively filter out unsafe content from the pre-training dataset, as this would provide valuable insights to the field.  

Thanks for your advice.  We would like to emphasize that the safety concerns we address in the customized fine-tuning of released LLMs are also important and underexplored [2]. This is particularly crucial as released LLMs are susceptible to being compromised through finetuning on malicious data [3].  There is a pressing need to ensure the long-term safety of a released LLM that may be finetuned on diverse downstream data, potentially containing unsafe examples.  

In contrast, during the pre-training of LLMs, organizations can carefully select public high-quality datasets or employ labor to screen data, thereby training LLMs exclusively with reliable data. These organizations can then implement safety alignment measures after the pre-training of LLMs. However, when it comes to downstream fine-tuning, applying such safety measures can be challenging. For example, when finetuning LLMs through APIs, the companies do not have the bandwidth to screen users’ uploaded data manually every time and an unsafe LLM can be easily trained with users’ noisy data.  Therefore, our study on ensuring the safety in finetuning LLMs can be useful and helpful in practice.  

We value your advice and plan to explore safety issues in pre-training as part of our future work.

Reviewer comment (Theory):
The reason why unsafe content is forgotten remains unexplained. Providing an empirical or theoretical explanation for this phenomenon would greatly enhance the paper and warrant a higher score. Additionally, it is crucial for the authors to investigate the conditions under which unsafe content can be forgotten, enabling readers to understand when to effectively apply the proposed ForgetFilter (FF) method.

Author response:
> The reason why unsafe content is forgotten remains unexplained. Providing an empirical or theoretical explanation for this phenomenon would greatly enhance the paper and warrant a higher score. Additionally, it is crucial for the authors to investigate the conditions under which unsafe content can be forgotten, enabling readers to understand when to effectively apply the proposed ForgetFilter (FF) method.  

Thanks for your advice.  We consider that unsafe content can be forgotten during safety finetuning is attributed to catastrophic forgetting during sequential finetuning [6,7,8]. During safety finetuning, the model is trained to produce safe content that is opposed to unsafe content. The model may shift original weights for unsafe content to learning safe content, leading to the forgetting of unsafe content.  

A method to empirically explain forgetting is to look into the distribution of representations of input data in the model. One hypothesis is safe and unsafe data may have closer representations than other examples in the feature space.  However, we will leave this interesting question as our future work and focus on applications to safety in LLM for this work.  We identify there are two main difficulties for such an empirical study. (1) We need to decide which layer to extract activations for comparison.  The current LMs have an enormous size and each layer may have different responsibilities. (2) We need to downscale the representations for analysis or visualization (e.g., t-SNE).  A representation for an input sentence in an LM can be massive where each token position corresponds to a high-dimensional vector. We need to find a suitable probe task to train a matrix that helps to reduce the dimension of the model’s representations. 


In terms of conditions for applying ForgetFilter, the safe content in safety finetuning should cover the type of past learned unsafe examples. For example, to effectively forget toxic examples, safe content should include untoxic data.  We have empirically shown this in Appendix D in our script. We find toxic examples cannot be forgotten more significantly than examples irrelevant to toxicity when the model is finetuned on unbiased data.

Reviewer comment (Presentation):
The presentation can benefit from a diagram to show the procedure of the 2-stage fine-tuning in the paper.

Author response:
> The presentation can benefit from a diagram to show the procedure of the 2-stage fine-tuning in the paper.  

Thanks for your suggestion.  We have made a diagram for illustration in our updated script.

Reviewer comment (Writing):
As a slight writing suggestion, I would make it such that the list of contributions on page 2 had more details. Currently, they are very spare and do not convey much information besides barebones ideas. Adding more details here would greatly improve readability.

Author response:
> As a slight writing suggestion, I would make it such that the list of contributions on page 2 had more details…

Thanks for your helpful advice.  We have updated our script.

Reviewer comment (Novelty):
This paper proposes the ForgetFilter method as a way of filtering the finetuning dataset of unsafe examples to produce safer models. However, in the Related Works section, there is no mention of different filtering algorithms. This is a large miss in my opinion. It is difficult to understand where such a filtering technique stands in the context of the large existing body of work of data filtration for safety. I would add a brief description of existing filtration methods and why this method improves on these methods.

Author response:
> However, in the Related Works section, there is no mention of different filtering algorithms.

Thank you for your valuable advice! We have added a new paragraph on filtering in the Related Works in our new script.

Reviewer comment (Novelty):
The Related Works also does not mention the connection between memory and safety that has long been analyzed in the literature. For example, the work "Does Learning Require Memorization? A Short Tale about a Long Tail" discussed how private points on the tails of the distribution get memorized and influence the weights of the models more than other data points. This connection should be mentioned here as it is completely possible that unsafe points may belong to tails of the data distribution and the effects examined in this paper can be explained by this previous work.

Author response:
> The Related Works also does not mention the connection between memory and safety that has long been analyzed in the literature. For example, the work "Does Learning Require Memorization? A Short Tale about a Long Tail" …This connection should be mentioned here as it is completely possible that unsafe points may belong to tails of the data distribution and the effects examined in this paper can be explained by this previous work.

Thanks for pointing out this work. But we would like to point out that this work and according literature focus on different aspects of safety, i.e.,  privacy rather than maliciousness in our study.  We appreciate your analysis.  However,  in our experiments, unsafe examples, by default, are not at the tail of the data distribution but account for the main proportion (50%).  The effects of this work may not apply to our case to explain the discrepancies in forgetting during safety finetuning.

Reviewer comment (Presentation):
I found the ordering and structure of Section 3.1 not readable. I would rearrange this section in chronological ordering to better understand the data first before discussing what a safety review is.

Author response:
> ... I would rearrange this section in chronological ordering to better understand the data first before discussing what a safety review is…I found several spelling and grammatical errors in this paper.

Thanks for your suggestions. We have updated the ordering and correct errors.

Reviewer comment (Writing):
I found several spelling and grammatical errors in this paper. While not critical to my score, a scientific paper should be more presentable. For example, on page 3, you misspelled promote as promot. Another example is forgetting the comma after default on page 3. Fixing these will improve the presentation of the work significantly.

Author response:
> ... I would rearrange this section in chronological ordering to better understand the data first before discussing what a safety review is…I found several spelling and grammatical errors in this paper.

Thanks for your suggestions. We have updated the ordering and correct errors.

Reviewer comment (Evaluation):
The metric you are reporting is the average rate for a set of data points. You are not reporting the expected value. The expected value is a specific term and you report the empirical average, which is not the same value. Please change this to be correct.

Author response:
> The metric you are reporting is the average rate for a set of data points. You are not reporting the expected value.  

Thanks for pointing this out.  But our original symbols can also represent the empirical average.  Following your advice,  we have updated our formula to resolve any ambiguity.

Reviewer comment (Experiments):
The main contribution of the ForgetFilter method is a way of filtering the original finetuning dataset such that the resulting model produces safer outputs. It does this by removing points predicted to be unsafe. Therefore, for the experiments, it should be tested if using a forgetting rate to predict the safety of a datapoint in the finetuning dataset is a strong method for predicting the safety of a datapoint. Thus, the baselines should be simpler methods or existing methods of filtering the data set. However, Safety Replay and Moral Self-Correction are neither of these. Thus, it is unclear whether Forget-Filter is really improving over anything since the comparison is not fair. There are many ways of filtering datasets for safety. These methods must be used as a baseline first and foremost to truly understand the strength of ForgetFilter.

Author response:
> The main contribution of the ForgetFilter method is a way of filtering ... Thus, the baselines should be simpler methods or existing methods of filtering the data set. However, Safety Replay and Moral Self-Correction are neither of these.

Thanks for your advice.  We understand the lack of context for filtering methods and have added related works following your advice.  To the best of our knowledge, there is no past filtering method that is directly comparable to our cases. Past filtering methods for malicious examples are constrained to toxic comments by using a classifier. The classifier is pre-trained by third-party on massive web data to be readily applied to unseen test data whose distribution is unknown.  Toxic samples contain distinguishable low-level semantic features, e.g., offensive words, incoming toxic samples of some dataset can be classified accurately by a pre-trained classifier. However, the classifier designed for identifying toxic comments is not suitable for recognizing other types of unsafe examples. Comparing our method with the use of toxic classifiers may also lack meaningful interpretation, as the ground-truth toxicity of our data is determined by such toxicity classifiers, as [3] does  (other notions of safety are classified by humans). Overall, there currently lacks an effective filtering method that is agnostic to the notion of safety and can filter more implicit unsafe cases other than toxicity that usually requires human evaluation [4].  Please also see related works on filtering unsafe examples from mixed data in our new script for details.    


At last, we would like to clarify that our work is not only about filtering unsafe data. On the one hand, we unveil the discrepancies in forgetting during finetuning. Interestingly, such forgetting differences are shown to appear only in sufficiently large LMs.  The larger models will forget unsafe examples much more significantly than other examples compared with smaller models.  


On the other hand, the context of our evaluation section is to ensure the safety of a released LLM in customized downstream finetuning.  The desired goal of safe downstream finetuning is to maximize performance on relevant downstream tasks while minimizing unsafe generations of LLMs.  In addition to filtering, both replay and moral self-correction stand out as potentially viable methods. More specifically, for moral self-correction,  we find that the LLM can still correct its unsafe generations based on some prompt even after it has been finetuned on unsafe examples.  Overall, our focus extends beyond mere comparison of filtering performance; instead, we aim to investigate strategies for enhancing the safety of LLMs during customized finetuning without compromising downstream performance.

Reviewer comment (Experiments):
Why are the ground truth responses in the BBQ dataset modified to a stereotypical choice?

Author response:
> Why are the ground truth responses in the BBQ dataset modified to a stereotypical choice?

Because we modify their truth responses of ambiguous cases to get biased data. The BBQ dataset does not contain stereotypical samples.

Reviewer comment (Experiments):
Why were only the learning rate and the batch size changed from the default hyperparameters of every model? Did the results seen depend on this hyperparameter selection or was this done purely for computational reasons? If it is the latter, please make sure to note this in the main text.

Author response:
> Why were only the learning rate and the batch size changed from the default hyperparameters of every model? Did the results seen depend on this hyperparameter selection or was this done purely for computational reasons? If it is the latter, please make sure to note this in the main text
> 
Yes, it is done purely for computational reasons. We have added extra notes to our script following your valuable advice.

Reviewer comment (Novelty):
I believe the authors should discuss related works that filter training data / perform data selection based on learning dynamics like frequency of forgetting. Some papers: [1] Maini et al. Characterizing Datapoints via Second-Split Forgetting, NeurIPS 2022 [2] Swayamdipta et al. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. EMNLP 2020

Author response:
## Response to Weakness 1:  
> I believe the authors should discuss related works that filter training data / perform data selection based on learning dynamics like frequency of forgetting.

Thanks for pointing out two related works! We have added related works on data selection based on learning dynamics to our script.

Reviewer comment (Evaluation):
We see from Figure 2 that up to 80% of harmful examples will be forgotten in the end. But what is the proportion of harmful examples among all the forgotten examples? I am asking the question above because I am a bit surprised to see applying ForgetFiltering improves downstream performance as well in Table 2. I thought ForgetFiltering may also remove training examples that are not harmful mistakenly. Can you explain side effects that is happening on downstream performance?

Author response:
## Response to Question 1:  
> We see from Figure 2 that up to 80% of harmful examples will be forgotten in the end. But what is the proportion of harmful examples among all the forgotten examples? … applying ForgetFiltering improves downstream performance … explain side effects that is happening on downstream performance?

Thanks for your question. Among all the filtered examples, unsafe examples account for around 81.3% in all three cases (i.e., bias, toxicity and harmfulness).  Indeed, some training data of downstream tasks will be filtered out by mistake, but a lot more unsafe data that are irrelevant to the task are filtered. Then during training, the model will focus on learning downstream task data instead of unsafe examples.  We posit that this accounts for the marginal performance enhancement observed compared to directly fine-tuning on noisy downstream data.

Reviewer comment (Experiments):
See the "weakness" part for my question about applying ForgetFilter when no safe dataset is available.

Author response:
## Response to Weakness 2:  
> I wonder whether in practice "safe examples" are always readily available for a fine-tuning task. For example, if I download a random dataset D from the web without having a safe dataset of the same task as D, can I still apply ForgetFilter? Is this setup practical?  

Our ForgetFilter works when unsafe data in D is not known. In that case, the safe dataset should be constructed to contain diverse kinds of safe examples, e.g., like the dataset that people use for aligning LLMs during pre-training.  Then the unsafe example whose categories are included by the safe dataset can be filtered. Such a safe dataset is commonly available in LLM companies and is ready  for users’ requests for their finetuning tasks.  

Additionally, the safe dataset does not need to be from the same or similar distribution of unsafe data in D. The forgetting is mainly based on the semantics of data (e.g., toxic or nontoxic).  For example, the unknown toxic data we experiment with are randomly sampled from web data and can be very different from untoxic data in our safe dataset.  However, toxic data can still be filtered through our proposed ForgetFilter.

Overall, we think our setup is practical and our ForgetFilter is applicable to real-life scenarios.

Reviewer comment (Experiments):
The authors have only presented the case where there is presence of limited amount of safe and unsafe data. In real worlds the models are trained on internet scale corpus where the data is present from different domains and distributions. It is likely that on safety review the model would forget the out of distribution samples from the safety review finetuning dataset rather than explicitly forgetting the unsafe datapoints. Since the authors have limited their dataset to limited amount of safe and unsafe data points, it might give an illusion that the model can actually unlearn the unsafe datapoints. Therefore a more rigorous evaluation where the authors can also consider different sources and domains of safe and unsafe data, should be done.

Author response:
First, we would like to point out that our work focuses on **customized finetuning**, instead of **pre-training**, to mitigate the safety issue of already pre-trained LLMs with RLHF and released to the public.

During customized finetuning by normal users, as pointed out by reviewer LEME (``Considering that fine-tuning datasets are typically small’’),  the dataset tends to be smaller and more specific than internet-scale corpus for pre-training since users may focus on some specific downstream applications.  This is especially true when users use Web APIs to finetune LLMs by uploading their training data.

The safety of LLMs in customized finetuning has emerged as a significant concern [1].  Users have the option to upload their data to the platform through APIs for fine-tuning, or they may choose to finetune open-sourced models with their collected data.  It's imperative to recognize that in both scenarios, the data used for fine-tuning has the potential to contain unsafe examples, whether intentional or unintentional, which can easily jailbreak the safety precaution of pre-trained LLMs (please see our response to minor weakness 1).  

All in all, we consider that our setting is both realistic and well-suited for investigating the safety issue of customized finetuning released LLMs.  While we acknowledge the importance of data quality in pre-training, it's essential to clarify that our work is not intended to address this particular issue.



Thanks for your advice.  We point out that our experiments already consider the three most common aspects of unsafety into considerations, i.e., bias, toxicity, and harmfulness. Those data are very diverse. For example, the toxic and untoxic data are sampled from a subset (3338519 entries) of Pile [2] collected from web data of varied sources.  



Thanks for your question whether the unsafe data is explicitly forgotten through safety review.  We appreciate your mentioning "explicitly forgetting". To clarify, we interpret your statement as suggesting that a data point is essentially "explicitly forgotten" when the model exhibits behavior during training as if it has never encountered that specific data. Actually, in our interleaved training setup in Section 4.3, as one of our contributions, we reveal that unsafe examples may not be explicitly forgotten through supervised finetuning.  The LLM is continuously trained with noisy downstream data with safety review integrated into each session of downstream finetuning.  Our findings indicate that LLMs can rapidly reacquire previously "forgotten" malicious knowledge, seemingly transitioning to an unsafe mode. However, the initial exposure of LLMs to unsafe examples results in a slower learning process. This observation suggests that the unsafe examples are not explicitly forgotten during supervised finetuning on safe data.

Additionally, we would like to clarify that safety review is not our highlighted contribution, but rather a method based on commonly used supervised finetuning so as to realign a pre-trained LLM that has been finetuned on malicious examples. Our proposed ForgetFilter performs much better than safety review in our interleaved training setup in Section 4.3, which implies the ability of ForgetFilter to ensure the long-term safety of LLMs.

Reviewer comment (Novelty):
A recent work [1] also uses the concept of second fine-tuning in order to filter out noisy samples. The idea of the proposed approach seems to be very similar to this. It would be nice, it the authors could look into this work [1].

Author response:
Thanks for your pointing out a related work. We have added it to our Related Works section.  Despite the similar high-level concept and the phenomenon-level convergence of our results that both show the discrepancies in forgetting, our works are fundamentally different from this work.  

This work mainly focuses on forgetting patterns wrt. labels, i.e., data with correct labels and mislabeled ones.  The mislabeled ones are produced by randomly flipping labels. In contrast, our study is focused on forgetting wrt. the high-level semantics of data, i.e., the notion of safety. The concept of a “label” is not well defined in our case, since our data points are language sequences.   Unsafe examples, as opposed to safe ones, don't typically exhibit random inconsistent patterns like noisy mislabeled data.  Instead, they often contain systematic information, e.g., bias, or toxicity.  

Additionally, we include data that are more irrelevant to the notion of safety and investigate their forgetting patterns during safety finetuning as well. No past work has studied the forgetting patterns of data of different high-level semantics from diverse sources.