Knowledge Editing in Language Models via Adapted Direct Preference Optimization (2024)

Abstract: Large Language Models (LLMs) can become outdated over time as they may lack updated world knowledge, leading to factual knowledge errors and gaps. Knowledge Editing (KE) aims to overcome this challenge using weight updates that do not require expensive retraining. We propose treating KE as an LLM alignment problem. Toward this goal, we introduce Knowledge Direct Preference Optimization (KDPO), a variation of the Direct Preference Optimization (DPO) that is more effective for knowledge modifications. Our method is based on an online approach that continually updates the knowledge stored in the model. We use the current knowledge as a negative sample and the new knowledge we want to introduce as a positive sample in a process called DPO. We also use teacher-forcing for negative sample generation and optimize using the positive sample, which helps maintain localized changes. We tested our KE method on various datasets and models, comparing it to several cutting-edge methods, with 100 and 500 sequential edits. Additionally, we conducted an ablation study comparing our method to the standard DPO approach. Our experimental results show that our modified DPO method allows for more refined KE, achieving similar or better performance compared to previous methods.

## introduction large language models (llms) have achieved remarkable success in various machine learning tasks and are commonly used as foundational models in multiple applications. training such models @xcite @xcite requires substantial computational resources and data. a major challenge with trained llms is their potential to generate inaccurate information in response to user queries. this can occur due to flawed training data or the continuously evolving nature of knowledge @xcite . the increasing popularity of llms has highlighted the need for methods to correct factual errors or inaccuracies represented by the models @xcite . given the high cost of training llms from scratch, recent research has proposed methods for modifying pre-trained llms without requiring complete retraining @xcite . this process, known as "knowledge editing" (ke), aims to modify the behavior of pre-trained llms to update specific facts without adversely affecting other preexisting knowledge irrelevant to the requested updates @xcite . ke presents significant challenges, such as identifying and correcting factual errors within multibillion parameter llms without compromising their overall pre-trained performance. one potential approach for updating an llm involves naive fine-tuning @xcite , where the parameters of a pre-trained llm are directly optimized to incorporate new knowledge based on additional data @xcite . however, fine-tuning and even some parameter-efficient fine-tuning (peft) methods have drawbacks, including intensive computational requirements, overfitting to the new data, and potential loss of valuable existing knowledge @xcite . in ke, each factual update, such as changing the pre-trained response "france" to "argentina" for the question "who won the fifa world cup in qatar?", is considered a single edit. while ke shares similarities with fine-tuning, it differs by focusing on locality, fluency, and portability metrics, as well as edit accuracy @xcite . those metrics ensure the model is updated with the new knowledge without degrading the general capabilities of the model. more details about each metric will be provided in section 3. we propose a variation of direct preference optimization (dpo) @xcite for ke. this method aims to reduce the likelihood of the model retaining unwanted knowledge while simultaneously increasing the likelihood of incorporating new desired knowledge. unlike most other methods, our proposed approach does not necessitate additional parameters, external memory, pretraining, or hypernetwork training. our contributions are summarized as follows: (1) we propose viewing ke as an llm alignment problem. (2) we introduce knowledge direct preference optimization (kdpo), a variation of dpo that is optimized for incremental knowledge modifications. (3) we conduct extensive empirical experiments on few sequential configurations, across four popular ke datasets, and three popular language evaluation benchmarks, involving multiple llm architectures demonstrating the advantage of our method. (4) we adapt popular datasets for ke to facilitate sequential editing. ## experiments recent studies @xcite have primarily focused on single edits, which involve evaluating a model's performance after a single knowledge update. in contrast, our focus is on sequential editing tasks, which require performing a series of knowledge updates successively, with evaluation conducted after the entire sequence of edits. ## results this section presents our results across multiple dataset, llms, and ke methods. we start by showing the results of our multiple edits study. then, we demonstrate our method's ability to handle gen-eral, non ke, llm benchmarks. lastly, we show an extensive ablation study. additionally, prompt examples and empirical edit results are presented in tab. 2. the prompts were selected from the zsre dataset to give some intuition on prompt types and edit challenges. we also show comparative responses from multiple ke methods shown in this paper. some baselines did not converge in all cases and thus were not added to the results. those issues and other baseline implementation details are discussed in section b.1 of the supplementary material. further results appear in the supplementary material section c. our method uses the same hyperparameters for all models (size ranging from 0.5b-8b) and datasets, while we optimized the hyperparameters of baseline methods to achieve competitive results. ## conclusions we have introduced a variant of dpo that is effective for ke. our extensive testing shows that our proposed kdpo methodology is a promising one for llm ke. we showed our method works well for various recent llms on multiple well known ke datasets. in our ablation, we demonstrated the advatage of kdpo over the vanilla dpo, suggesting the value of our novel idea for ke tasks. finally, we verified our method maintains the pre trained llm performance on multiple benchmarks. overall, we have demonstrated that kdpo is a high-performance and highly precise method for