MEMIT -Merge: Addressing MEMIT ’s Key-Value Conflicts in Same-Subject Batch Editing for LLM s (2025)

Abstract: As large language models (LLMs) continue to scale up, knowledge editing techniques that modify models’ internal knowledge without full retraining have gained significant attention. MEMIT, a prominent batch editing algorithm, stands out for its capability to perform mass knowledge modifications. However, we uncovers a critical limitation that MEMIT’s editing efficacy significantly deteriorates when processing batches containing multiple edits sharing the same subject. Our analysis reveals the root cause lies in MEMIT’s key-value modeling framework: when multiple facts with the same subject in a batch are modeled through MEMIT’s key-value mechanism, identical keys (derived from the shared subject) are forced to represent different values (corresponding to distinct knowledge), resulting in update conflicts during editing. Addressing this issue, we propose MEMIT-Merge, an enhanced approach that merges value computation processes for facts sharing the same subject, effectively resolving the performance degradation in same-subject batch editing scenarios. Experimental results demonstrate that at a batch size of 5, while the original MEMIT’s success rate drops to 46%, MEMIT-Merge maintains a 98% editing success rate, showcasing remarkable robustness to subject entity collisions.

## introduction as large language models (llms) continue to scale up, the prohibitive cost of full model retraining has made knowledge editing increasingly crucial in this domain. among prevalent editing algorithms, a class of algorithms, termed "locate and edit" methods by @xcite , enables targeted modifications through precise manipulation of specific regions. memit @xcite , one of the most prominent algorithms in this class, has gained significant attention @xcite ; fang et @xcite . it extends rome's architecture @xcite and enables the simultaneous modification of multiple knowledge instances within a single update operation. however, our investigation reveals a critical limitation in memit: when handling batches with multiple edits that share the same subject (such as "john smith now plays basketball." and "john smith comes from england." share the same subject "john smith", while "paul morand comes from england" has a different subject), the method will exhibit significant performance degradation. in contrast, edits with different subjects maintain stable efficacy. to systematically demonstrate this performance degradation, we constructed two contrastive datasets comprising batches with identical subjects versus fully unique subjects, named distinct-subject and same-subject, respectively. the experimental results are in fig. 1 , where the vertical axis represents efficacy (which means the editing success rate) and the horizontal axis indicates the batch size per edit. the results reveal that memit maintains a high success rate as batch size increases when editing distinct subject cases, but exhibits significant performance degradation for the same subject cases. however, same subject cases are also critical in the real-world practices (e.g., updating a person's occupation, workplace, and employer simultaneously). more detailed experimental settings can be found in sec. 4.3. the performance degradation stems from memit's key-value modeling paradigm: identical keys (derived from shared subject representations) map to conflicting values during samesubject batch edits. memit formulates knowledge updates as mlp key-value pairs where the output linear layer's weights are adjusted to align keys with edited values. foot_0 however, when multiple edits share subjects, their identical keys require divergent value mappings -an inherent contradiction since single-layer perceptrons cannot produce multiple outputs for identical inputs. to resolve this fundamental conflict, we propose memit-merge, an enhanced variant of memit. our key insight is to enforce value consistency by merging multiple knowledge entries that share identical keys. experimental results show that memit-merge consistently outperforms memit on samesubject dataset, maintaining a success rate above 90%, whereas memit drops to around 50%. for distinct-subject data, both methods perform comparably with no significant differences. ## conclusion this paper identifies the issue of significant performance degradation in memit when a batch contains knowledge sharing the same subject during batch editing. this is fundamentally caused by parameter update conflicts arising from identical keys requiring divergent values in the same-subject scenarios. our proposed memit-merge resolves this and significantly improves same-subject edit per-formance while maintaining original performance on distinct-subject cases. these findings advance mass-editing techniques for evolving llm knowledge bases.