Model Editing by Standard Fine-Tuning (2024)

Abstract: Standard fine-tuning is considered not as effective as specialized methods for model editing due to its comparatively poor performance. However, it is simple, agnostic to the architectural details of the model being edited, and able to leverage advances in standard training techniques with no additional work (e.g., black-box PEFT for computational efficiency), making it an appealing choice for a model editor. In this work, we show that standard fine-tuning alone can yield competitive model editing performance with two minor modifications. First, we optimize the conditional likelihood rather than the full likelihood. Second, in addition to the typical practice of training on randomly paraphrased edit prompts to encourage generalization, we also train on random or similar unedited facts to encourage locality. Our experiments on the ZsRE and CounterFact datasets demonstrate that these simple modifications allow standard fine-tuning to match or outperform highly specialized editors in terms of edit score.

## introduction model editing is a promising approach to combating incorrect or otherwise unwanted knowledge in llms. given a set of edits that assert desired information, the approach aims to alter the model so that it not only succeeds in memorizing these edits (efficacy) but also applies the asserted information to new prompts (generalization), without changing inferences that should remain unchanged (locality). there is clearly a trade-off between these metrics. at one extreme, the model can achieve high efficacy by memorizing the edits, but it will fail in generalization and locality. at another extreme, the model can achieve high locality by staying unmodified, but it will fail in efficacy and generalization. naively fine-tuning the model on the requested edits is well known to perform poorly, especially in locality. this motivated researchers to develop highly specialized model editors @xcite @xcite @xcite . a central theme of these methods is "minimally invasive editing" achieved by a careful selection of layers, model-specific adapter designs, and low-rank updates. while technically interesting, they require a suite of assumptions which may not be satisfied in other contexts (e.g., with different model architectures). in contrast, standard fine-tuning is simple, completely agnostic to the details of the model, and can take advantage of advances in standard training such as parameter-efficient finetuning (peft) techniques for immediate computational efficiency. 1 in this work, we show that standard fine-tuning can yield competitive model editing performance, focusing primarily on improving the edit score. we shift the focus from models and algorithms to training objectives and data augmentation. see ## task our main task is mass-editing (i.e., performing multiple edits at the same time). let v denote the vocabulary. a fact is a sentence @xmath0, r, o) in natural language. we follow the convention in the model editing literature and assume that the object @xmath1. the prefix π = (x 1 . . . x m-1 ) expresses (s, r) and is denoted as the prompt. let e denote a set of facts to enforce (i.e., requested edits). our goal is to edit a language model so that it upholds the relations expressed in e without changing its behavior on other facts. in zsre, e consists of 10,000 factual statements (e.g., "the artwork gideon's way was by who? john creasey"). in counterfact, e consists of 10,000 counterfactual statements (e.g., "textedit, a product of nintendo"). an editor is evaluated by three competing metrics: efficacy, generalization, and locality. let p θ denote a language model edited on e. in zsre, efficacy is the accuracy of @xmath2, o) ∈ e; generalization is the accuracy of @xmath3; locality is the accuracy of o @xmath4, o unrel ) is an unrelated fact. in counterfact, efficacy and generalization measure the accuracy of p θ (o|π) > p θ (o pre |π) and p θ (o|π par ) > p θ (o pre |π par ) where o pre is a pre-edit object for (π, o) ∈ e (e.g., "apple" in "textedit, a product of nintendo"). locality measures the accuracy of p θ (o pre |π nb ) > p θ (o|π nb ) where π nb is a "neighborhood" unedited prompt whose target is o pre (e.g., "macintosh file system, a product of" → "apple"). in both datasets, the final edit score is the harmonic mean of efficacy, generalization, and locality. fact: the mother tongue of danielle darrieux is english paraphrase augmentation (generated by the model): 1) the present invention relates. the mother tongue of danielle darrieux is english random fact augmentation (from the training split): 1) crate & barrel was founded in chicago 2) haines borough is within nevada alization, it harms locality by changing the model's predictions on unrelated or neighborhood prompts. this led to the development of highly specialized editing methods, with the implicit assumption that fine-tuning is not effective for model editing. ## method we propose a purely fine-tuning-based method that achieves a competitive edit score. our objective is a slight variation of (1): there are two small but important differences between (1) and ( 2 ). first, we optimize the conditional likelihood of the edit target o|π rather than the full likelihood. our motivation is to make the training more focused (for efficacy and generalization) and minimize the damage caused by finetuning (for locality). second, we fine-tune not only on the requested edits e but also additional facts in p and r to promote generalization and locality. p is pseudo-paraphrases of the prompts in e obtained by prepending random words generated from the (unedited) model. paraphrase augmentation is universally used in existing model editors @xcite @xcite . additionally, we augment with facts r that should not be altered by editing on e. there are many ways to obtain r, but we find taking random facts from the training split to be simple and effective. we filter @xmath5. figure 1 shows an example from counterfact (we only show 1 paraphrase and 2 random facts for illustration, in practice we use more). we emphasize that the augmented sets p and r do not use the evaluation facts (otherwise it is trivial). given the requested edits e, we perform the data augmentation and fine-tune the model according to ( 2 ). contrastive learning. given that our goal in counterfact is to achieve p θ (o|π) > p θ (o pre |π) where o and o pre are the new and pre-edit objects for a given prompt π, it is natural to consider a contrastive objective. we experimented with dpo @xcite where we frame o|π as the "preferred" response over o pre |π, optimizing the dpo loss jointly with (2). however, we found that prompt masking and data augmentation are already effective and do not benefit from contrastive learning. ## related work we discuss existing model editors and the different assumptions they require to highlight the simplicity of our approach; we refer to @xcite for an in-depth survey. mend @xcite is a meta-learning method @xcite that predicts the change in the gradient. similar to our approach, it also involves an explicit training stage (on the training split of zsre). it uses a special rank-one decomposition of the gradient for parameter efficiency. in our case, we achieve parameter efficiency without special considerations simply by leveraging black-box peft. rome @xcite is a locate-then-edit method @xcite that first uses causal tracing to identify the feedforward layer to change, and then applies a rank-one update to the layer's weight matrix. while it does not involve an explicit training stage, it does require an explicit knowledge of the subject s in the edit and its vector representation (which is heuristically induced). it also relies on wikipedia to estimate the covariance matrix of the subject embeddings (needed for the rank-one update). memit @xcite extends rome to multiple edits by carefully spreading updates to multiple layers. like rome, it requires layer specification, subject embeddings, and covariance statistics obtained from wikipedia. ike @xcite proposes in-context learning for model editing. while similar to our approach in avoiding the need to develop a specialized editor, it requires a strong model capable of effective in-context learning and is critically limited to single-editing. consequently, ike focuses on single-editing experiments with large-scale llms, making their results not comparable to ours. we define our scope as achieving a competitive edit score through standard fine-tuning. in particu-lar, we do not focus on preserving general capabilities of the model being edited. recent work shows that performance on a variety of downstream tasks drops to zero under any editor @xcite . we leave preserving the general capabilities of an llm with a fine-tuning editor as the next step of our work. ## conclusions we have demonstrated that standard fine-tuning is sufficient to obtain strong edit performance with a slight modification: optimizing the conditional likelihood and augmenting the data with additional facts to promote locality. our results challenge the assumption that standard fine-tuning is ineffective as a model editor, suggesting that model editing could be achieved as part of standard training rather than through a specialized model editor.