Immunization against harmful fine-tuning attacks (2024)

Abstract: Large Language Models (LLMs) are often trained with safety guards intended to prevent harmful text generation. However, such safety training can be removed by fine-tuning the LLM on harmful datasets. While this emerging threat (harmful fine-tuning attacks) has been characterized by previous work, there is little understanding of how we should proceed in constructing and validating defenses against these attacks especially in the case where defenders would not have control of the fine-tuning process. We introduce a formal framework based on the training budget of an attacker which we call “Immunization” conditions. Using a formal characterisation of the harmful fine-tuning problem, we provide a thorough description of what a successful defense must comprise of and establish a set of guidelines on how rigorous defense research that gives us confidence should proceed.

## motivation an emerging research direction demonstrates that safety techniques for large language models (llms) can easily be circumvented by fine-tuning them on harmful samples @xcite @xcite ) -examples of this occurring in practice are provided in appendix a. to ensure the safe release and deployment of llms, we should invest in defenses against these attacks. these defences are necessary due to the following: vulnerability argument no matter how safe a model is at inference time, if its safety guards can easily be removed the model is fundamentally unsafe. additionally, concerns about model liability and responsibility @xcite might prioritize questions about how model developers restrict downstream fine-tuning beyond easily ignored licensing schemes. previous work has attempted to understand the emergence of harmful text generation @xcite and defend against it @xcite @xcite . however, this work lacks clear conceptual analysis of defense conditions such as what is the definition of a successful defense and what are the necessary conditions defenses must meet. without this, it is unclear how we should validate proposed defenses. for example without careful experimental design future works might present effective defences that completely ruin model capability or do not generalize to unseen attacks. in this paper, we ground the "harmful fine-tuning attack" (hfta) threat model (see figure 1 ) in a training budget framework that highlights the cost considerations of defense @xcite . we propose a novel set of necessary conditions for defenses against these attacks we call "immunization" conditions ( § 3). our contribution is a formal illustration of what defenses against harmful fine-tuning attacks must look like: i.e. resistance to the process of supervised fine-tuning, stablity which retains general language modeling capability, generalizable across various attack datasets, and trainablity to maintain the utility of the llm for learning harmless tasks. using these conditions, we present clear guidelines on how to operationalize them to conduct future defense research § 4 ( empirical demonstration presented in appendix c). initialised llm m 0 harmful llm m ?[t] pre-training alignment training finetuning on d harm backdoors, data poising adversarial attacks, prompt injections harmful fine-tuning attack attacked llm m* ?[t] finetuning on d harm immunization open-weight release of m generates safe outputs generates unsafe outputs immunized llm m* ?[0] aligned llm m ?[0] pre-trained llm m ? figure 1 : harmful fine-tuning attacks train safety-aligned models for harmful purposes after they are released. backdoor attacks occur before model release and involve stealthiness (hiding a trigger). adversarial attacks occur after release but at inference-time. strategy attackers achieve harmful fine-tuning by using the training objective below, equation (1), on a given llm without harmful capabilities which we denote as m θ[@xmath0, where t indicates the number of optimization steps taken (@xmath1. the attacker will use a harmful dataset d @xmath2, y i } n @xmath3. they then minimize the loss function l d harmful (m θ[t] (x), y ) by taking training steps @xmath4. the outcome is the optimal parameters θ[t * ] found at some training step t * that the attacker can use to engage in illegal and harmful activities that would have previously been refused. this we call harmful training -eq. ( 1 ) -where capabilities we assume that the attacker has a limited compute budget to train a harmful model i.e. not enough to train an llm from scratch. we also assume enough training data to achieve meaningful harmful performance. in the white box setting, the attacker can modify the architecture, loss functions for training, inference process, etc. since they have complete access to the trained model. this generally corresponds to the case of open source release or model theft. black box settings assume the attacker can only access a fine-tuning api mediated by the defender. here the attacker pays for training for example using input token-based pricing found in openai's pricing model foot_0 . ## immunization conditions given the threat model above, we present conditions for successful defense that crystallize the work presented in @xcite ; @xcite ; @xcite . appendix b explores fully how previous work fits within our framework. we say that a model is immunized (we indicate an immunized model as m * ) against harmful training (eq. ( 1 )) with respect to d harmful if it meets the following conditions: condition (1) -resistance: resistance measures the degree to which a defense can prevent training towards harmful ends. it follows from the requirement to prevent learning harmful behaviour and is the implicit goal of previous defense works. for this condition, we first need to define a threshold ϕ which is the maximum acceptable harmfulness as estimated by some proxy metric measured by f (•). this is a quantity set by the defender. for example, ϕ could be the result of a toxicity classifier as f (•). we will say more about specifying harm and empirical attack success measures in § 4. for this we have: strong resistance means that eq. ( 1 ) never converges below a given loss l(m θ[t] (x), y ) whose value indicates a model has become harmful as measured by being generating text above the acceptable harmfulness threshold ϕ. this is for a given immunized model m * θ @xcite where t is the training steps taken to find eq. ( 1 ) and ϕ is estimated as above. this condition provides defense against an unlimited budget of training steps: weak resistance does not require that the immunized model is always below a specified threshold ϕ. instead, training takes at least max t train steps to pass the threshold ϕ. here the defender wants to maximize the number of training steps in eq. ( 1 ) required to pass the threshold. in practical terms, the defender is ensuring the defense is too expensive for the attackers budget: resistance applies to both actively trying to learn a brand new harmful task as well as circumventing or "unlearning" safety guards. note that resistance presupposes the model already performs poorly on these domains (as measured by g(•) ≤ ϕ which we do not specify as a separate condition itself. condition (2) -stability: the stability condition states that we want to be able to continue to use the immunized model m * at the same or similar level of performance on harmless tasks as the unimmunized model m θ[@xmath5. this ensures that the model remains usable. using the proxy llm capability measure g(•) we have: in-domain generalization requires that these subsets are drawn from the same domain. for example, a defense might be performed using examples of toxic content while the attack would be performed using non-overlapping samples of toxic content. this exemplifies the case where defender knows the domain the attacker will draw samples from. however this might not be strong enough for the defender who wants a model that can be robust across many types of harms that might only have small distributional overlap. cross-domain generalization requires that the subsets be drawn from different domains for example immunization applied using samples from toxic content generation defending against an hfta performed using harmful qa. we provide recommendations § 4 below on how defenses should demonstrate generalization. condition (4) -trainability: this condition simply says that we are able to improve performance during supervised fine-tuning on an arbitrarily chosen harmless dataset d ok within a similar number of train steps t 1 for the immunized model m * as t 2 , the unimmunized model m and was shown in @xcite ; @xcite . in practice, we can define our tolerance of the difference between these two as some small ϵ. this is an optional condition since defenses without trainability would still protect against hf-tas. we highlight trainability because of the high social utility of model training for commercial and research applications. without trainability, there would be high social social pressure to nonmaliciously undo defenses simply due to wanting to perform harmless training. ## guidelines for future research establishing resistance first we recommend constructing methods that have theoretical guarantees like bounds on the number of samples necessary to undo a defense. studies in transfer learning have characterized the difficulty of training towards a target task given a network trained on a source task. both @xcite and @xcite provide theoretical frameworks that we encourage researchers to use. for example, @xcite provides a theoretical explanation of resistance by locating the parameters of an immunized model m * θ as a point in the dynamics of the loss landscape that minimize the transition probability of traversing a training trajectory in the loss landscape where the transition probability is a function of the curvature of the loss landscape. we encourage researchers to derive loss functions that provably minimize this transition probability. researchers can develop strong resistance using cryptographic methods such as those found in mukherjee and chakraborty (2023); @xcite where the weights themselves cannot be modified without an authorization mechanism. harmful dataset selection and creation empirical evaluation of the resistance condition would begin with datasets used to construct realistic simulations of hftas. currently, the two main types of datasets with validated harm measurement constructs consist of harmful question answering @xcite and toxic text generation @xcite . robust defense evaluations should investigate many more types of harms for example those outlined in @xcite . other types of harmful datasets such as fraud are more difficult to come by and are potentially high risk to construct since they could inadvertently assist attackers. mitigation techniques for dataset construction include keeping datasets private, developing toy harmless domains that only simulate harms, strong licensing, and focusing on datasets that are already publicly available. measuring resistance since we focus on supervised fine-tuning attacks we recommend two initial dimensions of attack strength: learning rate and number of samples (or epochs) used. other dimensions of attack could include batch size, the type of optimizer used and various other hyperparameters but they are harder to specify and future research will need to develop stronger attacks to ensure robust defenses. we advocate that simulated hftas use as many samples (or epochs) as possible across a large range of learning rates, a large dataset, and many epochs. we encourage future studies to comprehensively sweep learning rate from a rate that is too low for any learning to a learning rate that is high enough to ruin stability. attack success is a key measure of resistance and should be evaluated in the following way. first, as above, attack success consists of using a proxy evaluation function for harmfulness f (•) to measure whether we have surpassed a defender set threshold ϕ. our recommendation is that we use domain specific measures validated by previous community efforts over generic measures of harm @xcite . for setting a threshold of acceptable defense we could either: (i) set the threshold of acceptable behavior as the harmfulness of the original undefended safety guarded model before a hfta or (ii) a weaker threshold could be used where the defended model should simply be strictly less harmful than the base model after performing a successful hfta. we don't think that (ii) should be considered an actual defense but for challenging datasets it helps us construct an initial set of research goals. in practice, we expect defenses to compete empirically over their "degree of resistance" by minimizing the overall harmfulness measure as much as possible. ensuring stability for stability, we encourage measuring performance differences between the base model and the defended models on standard llm capability benchmarks such as using several standard datasets (like @xcite with eleuther's lm harness. 2 while perplexity on a language modeling dataset such as wikitext @xcite could be useful, it is a more indirect proxy of how the language model will behave. additional requirements of stability should include that the model not be less safe across dimensions like inference-time adversarial attacks (for example those measured by @xcite or general trustworthiness @xcite and bias @xcite . adaptive attacks all defences should attempt to be fully transparent about the conditions under which the defences can be broken and attempt to formulate adaptive attacks, i.e. attacks which could be successful if the attacker knew the defence method used. for example methods that rely on guarantees from the assumption the attacker uses gradient methods might employ newton's method to demonstrate an attack. encouraging generalization generalization is particularly important to measure because in practice it is very unlikely that the defender will have access to the same samples as the attacker. to simulate this, the defender can consider the following: for in domain generalization, the defense and the attack should be performed on disjoint subsets of the same domain. typically this will take the form of exploring sample efficiency: how small of an immunization dataset can we construct such that we protect against an hfta in that domain? for cross domain generalization, there is an open question about what can be realistically expected from immunization methods. one expectation we might have is that our immunization methods should robustly defend across various sub-domains within a domain like harmful question answering. for example, with beavertails @xcite , a given harmful subset such as animal abuse or criminal activity could be left out as the attack dataset and we could evaluate whether we are able to defend against this left out subdomain. more challenging types of cross-domain generalization should also be evaluated such as if the defense was performed for toxic content generation and the attack was performed using harmful question answering. evaluating trainability to demonstrate trainability, we must select tasks that llms do not perform well on without additional training (i.e. we observe a large increase in performance after training). like stability, we should select from benchmarks of natural language generation tasks with well-validated measures such as the gem benchmark @xcite . the gem benchmark is recommended because we can construct text-to-data or structured data generation tasks that llms are unlikely to perform well on zero-shot. appendix c provides an empirical demonstration of some of these recommendations. ## conclusion in this paper, we clarify an emerging threat model that identifies the risk of harmful fine-tuning in llms and quantifies it using an imagined attackers training budget. recent work has shown that there might be ways in which we prevent these attacks. to assist with defense development, we proposed a formal set of "immunization" conditions that allow us to validate whether we have defended against harmful fine-tuning. we leverage these conditions to provide a concrete set of guidelines that future work could follow to develop robust evaluation of proposed defenses.