Resolving U nder E dit & O ver E dit with Iterative & Neighbor-Assisted Model Editing (2025)

Abstract: Large Language Models (LLMs) are widely deployed in downstream tasks, but keeping their knowledge up-to-date via retraining or fine-tuning is often computationally expensive. Model editing provides a more efficient alternative by updating a targeted subset of parameters, which often follows the locate-and-edit paradigm. Despite this efficiency, existing methods are limited: edits may fail to inject knowledge (UnderEdit) or unintentionally disrupt unrelated neighboring knowledge (OverEdit). To address these challenges, we propose two complementary methods: **iterative model editing**, which applies successive edits to mitigate UnderEdit, and **neighbor-assisted model editing**, which incorporates neighboring knowledge during editing to reduce OverEdit. Our extensive experiments show that these techniques improve editing performance across multiple LLMs, algorithms, and benchmarks, reducing UnderEdit by up to 38 percentage points and OverEdit by up to 6, while remaining broadly applicable to any locate-and-edit method.

## introduction llms have been widely used as repositories of factual and specialized knowledge @xcite @xcite . however, the world is constantly changing, with knowledge and information evolving rapidly, such as significant government policy changes and their wide impacts across various domains. thus, it is essential for many nlp applications, such as text generation, question answering, and knowledge retrieval, to have models that can adapt to knowledge changes both effectively and efficiently. re-training an llm is resource-intensive @xcite . standard supervised fine-tuning is data hungry and less effective @xcite . model-editing, which directly modifies important model parameters for making the prediction, has emerged as a more efficient alternative for updating outdated information @xcite @xcite . these methods adopt a "locate-andedit" approach, where they first identify the parameter locations associated with outdated knowledge and then update the parameters to enable the model to incorporate and predict the new knowledge. the effectiveness of the methods is evaluated from two perspectives. the first is whether the method successfully updates the knowledge, failure on this leaves certain facts unedited, causing un-deredit. secondly, whether the update introduces unintended modifications to neighboring knowledge -a phenomenon we call overedit. existing methods suffer from both underedit and overedit as shown in figure 1 . to address this, we propose methods to mitigate both underedit and overedit. for underedit, we hypothesize that the parameter update is insufficient to achieve the desired knowledge change. the editing process performed a rank-one update on the layer parameters to achieve the desired update. we empirically showed that the approximation introduces errors, leading to underedit. to this end, we proposed iterative model editing, wherein editing is performed multiple times. for overedit, we hypothesize that model editing can benefit from including neighboring knowledge during the editing stage. we thus introduce neighbor-assisted model editing, a procedure that integrates neighboring knowledge during the editing process to keep the test neighboring knowledge unchanged. in summary, we propose solutions to two fundamental challenges in model editing: underedit, where edits fail, and overedit, where neighboring knowledge is erroneously modified. we evaluate our approach using four "locate and edit" model editing algorithms, rome @xcite , memit @xcite , pmet @xcite and alphaedit @xcite , and applied to four llms: gpt-2 xl (1.5b) @xcite , gpt-j (6b) (wang and komatsuzaki, 2021), llama-2 (7b) @xcite , and llama-3.1 (8b) @xcite . our experiments are conducted on two widely used factual knowledge editing benchmarks: counterfact @xcite and zsre @xcite . our results show that iterative model editing improves edit success while also reducing the approximation error introduced by the rank-one update. furthermore, we demonstrate that incorporating even a single neighboring knowledge during model editing reduces unintended modifications to neighboring knowledge at test time, resulting in stronger edit performance. overall, our proposed methods are broadly applicable and consistently effective across current locate-and-edit approaches that adopt the two-stage editing framework, and are readily applicable to future methods built on this foundation. ## the locate-and-edit framework in this section, we provide background on the locate-and-edit model editing framework along with the notation used throughout the paper. an autoregressive llm is a function f θ : x @xmath0, that takes as input a sequence of tokens @xmath1, x 2 , • • • , x t ) of length t with x i in the dictionary x , and uses model parameters θ to return a probability distribution ∆(x ) to model the next token x ′ , i.e,. f θ (x)[x ′ ] ≈ pr(x ′ = x ′ |@xmath2, where x and x ′ are random variables representing the sequence of input tokens and the next token, respectively. the internal computations of an llm relies on a grid of hidden states h l t , where l corresponds to the layer and t corresponds to the token position in the sequence (using tokens x 1 , x 2 , • • • , x t ). each layer is a standard transformer block with the self-attention module, mlp module, etc @xcite . prior work focuses on editing the factual knowledge within the llm. factual knowledge is represented as a triplet (s, r, o), where subject @xmath3, relation @xmath4, and object @xmath5, e.g., the ipad [s] is produced by @xcite apple @xcite . we consider only single token objects in this representation, following previous work @xcite @xcite . the model editing task is to make the model place a higher likelihood on a new object o * than an old object o when presented with @xmath6, r), i.e., find new parameters θ ′ , such that f θ ′ (x)[o * ] > f θ ′ (x) @xcite . model editing is not limited to a single edit, but can encompass a batch of m desired edits @xmath7, r i , o i , o * i )} m @xmath8. locate and edit model editing algorithms hypothesize that factual knowledge locates within specific layers of the llm, and updating parameters in these layers is sufficient to induce the desired change in object @xcite @xcite . these methods employ causal tracing to identify these layers responsible for the factual knowledge, referred to as causal layers {l 1 , . . . , l c }, where c denotes the number of layers in this range. the last mlp in these layers has been found to have a major impact on the object token distribution when presented with the subject tokens @xcite @xcite . due to this major impact, locate and edit methods focus on only updating these mlp weights. the weight update is performed in two stages: optimization stage finds ideal values for the network's hidden state in certain transformer layer to make o * likely and spread stage updates the weights of the last mlp in casual layer(s) to approximate this ideal hidden state. we detail these stages below for memit @xcite and discuss its differences to pmet @xcite , alphaedit @xcite and rome @xcite . ## method the memory-editing algorithms mentioned above face challenges, such as failing to edit certain knowledge i.e., underedit or changing neighbor knowledge that should remain unchanged i.e., overedit. in this section, we present our proposed method to address these issues. specifically, we introduce iterative model editing (3.1) to mitigate un-deredit and neighbor-assisted model editing (3.2) to reduce overedit. ## experimental details in this section, we detail the experiments to demonstrate the effectiveness of iterative and neighborassisted model editing with memit, pmet, al-phaedit and rome. we evaluate these algorithms with our modifications across four llms: gpt-2 xl (1.5b), gpt-j (6b), llama-2 (7b) and llama-3.1 (8b). we use easyedit foot_0 @xcite with default hyperparameters; implementation details are in appendix b. ## results and discussions we show the experimental results for both iterative model editing and neighbor-assisted model editing in this section. the hardware used on running these experiments are detailed in appendix c. ## conclusion in this work, we addressed key challenges in model editing-underedit and overedit -by proposing iterative and neighbor-assisted model editing techniques. our iterative approach effectively resolves underedit by reducing the approximation error to ensure sufficient weight updates, while neighbor-assisted editing mitigates overedit by preserving neighboring knowledge. extensive experiments across diverse editing algorithms, llms, and datasets validate the efficacy of our methods. these contributions pave the way for more reliable model editing, with broad applicability to dynamic knowledge updates in llms.