Self-Destructing Models: Increasing the Costs of Harmful Dual Uses in Foundation Models
Abstract
A growing ecosystem of large, open-source foundation models has reduced the labeled data and technical expertise necessary to apply machine learning to many new problems.
Yet foundation models pose a clear dual-use risk, indiscriminately reducing the costs of building both harmful and beneficial machine learning systems.
To mitigate this risk, we propose the task blocking paradigm, in which foundation models are trained with an additional mechanism to impede adaptation to harmful tasks while retaining good performance on desired tasks.
We call the resulting models self-destructing models , inspired by mechanisms that prevent adversaries from using tools for harmful purposes.
We present an algorithm for training self-destructing models leveraging techniques from meta-learning and adversarial learning, showing that it can largely prevent a
BERT-based model from learning to perform gender identification without harming the model’s ability to perform profession classification.
We conclude with a discussion of future directions.
1. Introduction
A defining capability of large pretrained models (hereafter foundation models; FMs) is few-shot adaptation to many downstream tasks—potentially improving performance and efficiency in domains with little training data (Bommasani et al., 2021).
Further, some argue that open-source availability should be considered an essential feature of FM creation.
As Black et al.
(2022) write, “open access to [FMs] is critical to advancing research in a wide range of areas—particularly in AI safety, mechanistic interpretability, and the study of r ⃝ Equal contribution. Author names randomized via AEA author randomization tool. Stanford University. Correspondence to:
Mitchell em stanford edu > .
First Workshop of Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022 , Baltimore, Maryland, USA, PMLR 162, 2022.
Copyright 2022 by the author(s) how [FM] capabilities scale.” Yet while more widely available FMs certainly enable greater accessibility, auditability, and understanding of these powerful models, making FMs widely available for downstream adaptation without restriction comes at some cost to safety.
Today, an actor can download a FM and adapt it to any harmful use-case they desire.
An oppressive government can take a powerful pretrained language model and adapt it to identify dissidents.
A rogue actor can adapt a pretrained object recognition system such that commercially available drones act as targeted loitering munitions.
A pretrained drug discovery system can be used for creating chemical or biological weapons, like neurotoxins (Urbina et al., 2022).
Unfortunately, due to their general-purpose nature, preventing such dual use of FMs is difficult.
One set of approaches to mitigating these dual uses focuses on access restrictions.
Some emphasize the flow of talent, data, compute, and other resources required for training models (Brundage et al., 2020; Flynn, 2020).
Others have placed models under restrictive APIs or licensing schemes (Solaiman et al., 2019).
Some have suggested a review board for selecting the release mechanism (Liang et al., 2022).
In this work we suggest a new, complementary, path forward: self-destructing models .
Self-destructing models are trained via a task blocking method that impedes the adaptation of the model to a harmful task without impairing the model’s ability to be used for its original intended purpose.
Where existing access restrictions must navigate the tension between openness and safety, we seek to provide a new research pathway for reducing (and in some cases obviating) this tension.
By increasing the compute, data, and talent required to adapt public models to harmful tasks, self-destructing models have the potential to increase the effectiveness of access controls and other safety mechanisms.
In this work, we: (1) define the task blocking problem and evaluation metrics as well as self-destructing models ; (2) describe an initial algorithm, Meta-Learned Adversarial Censoring (MLAC), for training self-destructing models, evaluating its ability to impede fine-tuning a language model to perform demographic information extraction; (3) identify key directions for future research in the development of self-destructing models.
2. Related Work
A number of works have sought to address dual use risks by restricting points of control (Flynn, 2020;
Brundage et al., 2018; Solaiman et al., 2019; Bommasani et al., 2021;
Shevlane, 2022; Zwetsloot et al., 2019), despite there also being substantial benefits to open access (Zhang et al., 2022; Black et al., 2022). We aim to provide an alternative that allows for open access while still hindering bad actors.
Some work on AI safety has sought mechanisms to prevent agents from learning degenerate behaviors.
Orseau & Armstrong (2016), for example, seek to prevent a particular scenario where an agent learns to disable its off-switch so that it continues to collect reward.
We on the other hand focus on preventing a different, broader, set of harmful behaviors: adaptation of pretrained models to harmful tasks.
Closely related to our work are methods for de-biasing, editing, or removing harmful content from models.
Like domain invariance approaches (Ganin & Lempitsky, 2015; Li et al., 2018; Zhou et al., 2020; Yao et al., 2022), Edwards & Storkey (2015) use an adversarial approach to remove information from representations.
Ravfogel et al.
(2022a)
and
Ravfogel et al.
(2022b) take a similar approach and find a projection on the final output layer of a pretrained model that removes gender-based biases from the model (and prevent recovery of those biases after that projection layer).
Others have created model editing techniques to remove outdated or harmful content from pretrained models (Sinitsin et al., 2020; Mitchell et al., 2022a;b).
While these other methods generally optimize for the information to be removed from the original model, we optimize for poor performance after adaptation of the original model to a harmful task.
This can be accomplished via a meta-learning approach.
In the context of meta-learning, MAML (Finn et al., 2017) and related algorithms (Li et al., 2017; Lee & Choi, 2018; Park & Oliva, 2019; Zintgraf et al., 2019; Flennerhag et al., 2020) have shown that the desired post- fine tuning behavior of a neural network can be effectively encoded in its pre -fine tuning network initialization.
While existing works have leveraged this ability in order to enable more rapid learning of new tasks, our work encodes a blocking mechanism into a network’s initialization that prevents effective adaptation on harmful tasks.
3. Task Blocking & Self-Destructing Models
The goal of task blocking is to create models that increase the costs of fine-tuning on downstream harmful tasks such that an adversary would rather start from scratch than use the pretrained model (see Fig. 1).
The resulting models are “self-destructing models” which impede adaptation on harmful dual-uses by increasing the costs of the harmful use.
In this section, we more precisely define our problem setting and describe an initial algorithm for it.
3.1. The Task Blocking Problem
We assume that an adversary aims to adapt a pretrained model π θ to a harmful task, searching for the best adaptation procedure f among a set of adaptation procedures F in order to find the one that maximizes harmful task performance.
Adaptation procedures in F may include simple fine-tuning, a hyper-parameter search over fine-tuning procedures, as well as other more advanced adaptation mechanisms that we leave to future work.
The goal of a task blocking is to produce a set of model parameters ˜ θ (a self-destructing model) such that it is more costly to successfully adapt the model to harmful tasks than it is to adapt a non-selfdestructing model—but performance on a desired set of tasks should remain unaffected.
We define two regimes to increase costs: (1) increase data costs by decreasing sample efficiency; (2) increase compute costs by slowing convergence of the training process. Data Costs.
In the first regime, we assume that the adversary has little data to adapt an FM to their harmful task and that the cost of gathering more data is high.
A traditional FM would help the adversary reduce their data costs.
A self-destructing FM, on the other hand, would provide the adversary with little benefit over a randomly initialized model.
We define an FM’s relative data efficiency as:
[Equation]
where M is the desired performance metric, n is the number of data points available, D n is an adaptation dataset of n examples from the task of interest, and θ r is a randomlyinitialized model.
Note that the min in Equation 1 encapsulates hyperparameter optimization, and the size of
F loosely corresponds to the adversary’s resource budget
1: Input pretrained dataset D tion d 2: learning rate params ˜ θ ← θ 3: for do 4:
Sample adaptation procedure ˜ f k ∼ ˜ F 5:
Sample data batches b d ∼ D d , { b kh } ∼ D h , b h ∼ D h 6: { θ k } , { w kh } ← ˜ f k ( w h ◦ π ˜ θ , { b kh } , α h ) // do inner loop 7: ℓ hk
= L h ( w kh ◦ π θ k , b h ) , ∀ k // outer loop harmful NLLs 8: ℓ d = L d ( w d ◦ π θ , b d ) // desired NLLs 9: ˜ θ ← ˜ θ − η ∇ θ (cid:16) ℓ d − 1 K (cid:80) Kk ℓ hk (cid:17) // update blocked model 10:
ϕ ← ϕ − η h 1 K (cid:80)
Kk =1 ∇ ϕ ℓ hk // update adversarial params 11: w d ← w d − η d ∇ w d ℓ d // update desired task head for adaptation.
E data = 1 N (cid:80) Nn E ndata is the average samplewise regret between the FM parameters θ and a random re-initialization θ r after each follows the same adaptation procedure f ( · ) on a fixed-sized dataset D n . An ideal selfdestructing model has E data ≤ 0 , meaning the model is no more data efficient than a randomly-initialized model for the (presumably harmful) task of interest. Compute Costs.
If data is cheap or plentiful, it may be difficult to prevent an adversary from learning the task since perhaps even a random model can learn the task with the amount of data available. In this data regime (large amount of cheap data), the benefit of an FM is improved compute efficiency, rather than increased accuracy. Here, we would define the FM’s relative compute reduction as the amount of compute saved by using the FM over a randomly initialized model, where compute could be defined as the number of train steps, hyperparameters searched, wall clock time, etc. The goal of task blocking here would be to prevent any compute reduction when adapting the self-destructing model to a harmful task, while retaining compute reduction for desired tasks.
For the purposes of this work, we focus on data costs, trying to reduce the sample efficiency of the model on tasks that we wish to impede.
Cataloging and Defining Harmful Dual Uses.
A large body of work has pointed to inherently harmful uses that
FM creators may wish to block: from creating neurotoxins (Urbina et al., 2022) to race detection (Olson, 2022).
In our work we assume that a harmful dual use is known and defined .
That is, the self-destruct mechanism will have data to approximate the dual use and actively encode a mechanism to block it.
This requirement inherently requires a normative definition of harmful dual uses.
We avoid providing a normative framework for which dual uses are inherently harmful, but suggest that future work can focus on this nor- [Figure] mative definition problem.
Creating self-destructing models makes sure that they cannot be used for harmful purposes counter to the model creator’s values, but it is up to the model creator to determine what those are.
We note that while defining harmful tasks a priori may be difficult, this work reflects a “red teaming” approach to harm prevention, common in security contexts.
That is, model creators play the role of an adversary to identify and prevent harms.
This can function as a complement to other access control methods, providing more confidence that certain known harmful tasks are blocked.
3.2. Meta-Learned Adversarial Censoring
To prevent successful adaptation of pretrained models to harmful tasks, we describe MLAC:
Meta-Learned Adversarial Censoring , a meta-training procedure that aims to eliminate any useful information about the harmful task even after fine-tuning on that task .
Given a desired task dataset D d and harmful task dataset D h , MLAC learns a feature extractor π ˜ θ that is effective for the desired task but cannot be effectively used or efficiently fine-tuned to perform the harmful task.
In the inner loop of each meta-training step, the feature extractor is adapted to the harmful task with several steps of gradient-based adaptation, using an adversarially learned prediction head w h and learning rate α h .
The adaptation procedure ˜ f used at each meta-training step is sampled from ˜ F , a proxy for the true adversary’s adaptation class F .
In this case, we narrow ˜ F to be different fine-tuning approaches with close-to-optimal hyperparameters (e.g., Adam for K steps and learning rate α h ).
In the outer loop , the adversarial parameters ϕ
= { w h , α h } are trained to minimize the harmful task negative log likelihoods of the adapted models, while the blocked parameter initialization ˜ θ are trained to maximize the harmful task negative log likelihoods of the adapted models.
We also must counteract the self-destruct [Figure] P o s t - a d a p t a t i o n g e n d e r a cc u r a c y Random BERT 0 steps 4 steps 16 steps mechanism with something that will prevent unlearning of the entire network.
In this work, we simply optimize for a given desirable task as the counter-balance by minimizing ℓ d , which updates both the desired task head w d and the representation parameters ˜ θ .
See Algorithm 1 for the complete training procedure.
Note that in practice, we use Adam rather than SGD in the outer loop to optimize ˜ θ , adversarial parameters ϕ , and desired task output head w d .
We use higher (Grefenstette et al., 2019) for implementing the bi-level meta-learning process.
Calibration.
We also add another mechanism to strengthen the inner-loop adversary.
In binary classification tasks, maximizing the loss of the harmful task may lead to a degenerate optimum where labels are flipped, which leaks information about the harmful task.
To prevent this outcome, we also optimally calibrate the logits via a simple linear projection ( w ) solved via differentiable convex optimization (Diamond & Boyd, 2016; Agrawal et al., 2019).
Thus at step k of the inner loop we solve the maximum likelihood problem:
[Equation]
where m k
= w kh ◦ π kθ is the blocked model after k steps of adaptation using the adversarial harmful task head and learning rate.
Thus this projection updates line 7 of Alg.to ℓ hk =
L h ( w kc ◦ w kh ◦
π
θ k , b h ) .
4. Experiments
The goal of our experiments is to develop an initial sense for whether MLAC is a viable initial task blocking method and whether it improves over simple baselines.
Dataset.
In a demonstrative experiment, we utilize an existing dual-use dataset in the de-biasing literature, “Bias in Bios” (De-Arteaga et al., 2019).
The dataset consists of professional biographies.
Each biography has a label that is the gender identity of the biography’s subject as well as the profession being discussed.
We split the data into a train, validation, and evaluation set.
We consider the “desirable task” for which we want to maintain good performance as the profession detection task.
We wish to block the gender identification task.
On the original dataset, we find that a random model can learn gender classification to over 90% accuracy with only 10 examples, leaving only marginal ability for an FM to improve in data efficiency.
Thus, to make the FM more beneficial, we replace all pronouns with “they/their,” similar to the censored dataset in the original data.
While this task pair has traditionally been used for de-biasing, and while we mainly use this as an initial demonstration, there may be valid reasons for preventing an adversary from detecting demographic information from text.
In countries where anti-minority action is common, automated systems that identify demographics may cause serious harm.
For example, a country may wish to identify people of a certain religion, sexual orientation, or other identity group in automated and ultimately harmful ways.
Or the state may wish to identify another feature that is highly correlated with identity which will lead to the same harms. Protocol.
For all experiments, we run 50k steps of MLAC meta-training on the training set.
At test time, we take the resulting self-destructing model and run it through a rigorous hyperparameter search to maximize fine-tuning performance on the harmful task.
We allow hyperparameter searches with 50 fine-tuning trials, using the tree-structured Parzen Estimator (Bergstra et al., 2011) in the hyperopt software package (Bergstra et al., 2013). We search over learning rate, batch size, maximum number of steps, and freezing of intermediate representation layers. For this process, we subsample the validation set to simulate an adversary with a dataset of size N . This subsampled validation set is used as the training set for the adversary.
We then use the entire evaluation set to evaluate the adversary’s performance on held-out data and for hyperparameter tuning.
We make the conservative assumption that the adversary can perform hyperparameter tuning using the population , even if the amount of data for fine-tuning itself is limited.
This choice weighs heavily in the adversary’s favor, disadvantaging the self-destruct method.
We repeat the hyperparameter search process 6 times with different random seeds and data subsets.
This yields confidence intervals over different adversaries training on different subsets of the data.
Comparisons. We compare MLAC to the adversarial censoring ( AC in Fig. 2) method from Edwards & Storkey (2015) as well as simple fine-tuning on the desired task ( BERT (fine-tuned) in Fig. 2).
For AC, an adversarial layer is learned on top of representation layers to predict the undesirable task.
The gradient is then flipped to destroy undesirable information in the representation layer.
Notably, MLAC with K = 0 and with no calibration is equivalent to adversarial censoring.
We use a BERT-tiny model as our FM to save on compute costs (Devlin et al., 2018; Turc et al., 2020).
Note that, as mentioned earlier, we focus on making sure that the professions task is unimpeded.
So we directly train on cross-entropy loss as L g .
For all models, the final achieved performance is retained for the desired professions task (see below, Figure 4).
Since the retention of performance for the desired task is near universal, our figures focus on exclusively the harmful task performance.
Results.
Fig.
2 shows that MLAC returns nearly identicalto-random harmful task performance at all data regimes.
Conversely, adversarial censoring (the equivalent of MLAC without calibration and K = 0 ) does not appear to have any effect on post-fine-tuning harmful task performance.
Fig.
3 shows the vital role played by the depth of the inner training loop of MLAC, suggesting that a meta-learning process is genuinely necessary to impede harmful task performance.
We find that head re-calibration may mildly improve blocking on average when pooled across all inner-loop step configurations ( Fig. 5).
To ensure that desired task performance is retained, we evaluate the blocked model on the desired task of profession classification, comparing with fine-tuning a pretrained
BERT-tiny model and a random model.
Fig. 4 shows the [Figure] [Figure]
s ta d a p t a t i o n g e n d e r a cc u r a c y
[Figure]
result
; MLAC is clearly able to solve the task effectively, surpassing the few-shot performance of BERT-tiny.
5. Conclusion
Our work is only a first step in raising the cost for harmful dual uses of pretrained models.
Future work might expand on our study in at least four directions: scaling the self-destructing model framework to larger FMs; studying the generalization of the learned blocking behavior to new (but related) datasets other than the one used during
MLAC meta-training; training/evaluating with stronger adversaries that incorporate adaptation methods such as prefix tuning (Li & Liang, 2021), adapter layers (Houlsby et al., 2019), or others; and evaluating the preservation of desired task fine-tunability, not just zero-shot performance.
Future work might also seek to introduce concealed architectural changes that hide self-destruct triggers in the network but are more robust to adversarial mechanisms.
We hope that self-destructing models can become one tool that enables model developers to openly share their artifacts while minimizing dual use risks.
Acknowledgements
We thank Rishi Bommasani, Siddharth Karamcheti, and
Jieru Hu for helpful discussion and feedback.
PH is supported by an Open Philanthropy AI Fellowship.
EM is supported by a Knight-Hennessy Graduate Fellowship.
across many tasks.