Lifelong Model Editing with Graph-Based External Memory (2025)

Abstract: Large language models (LLMs) have revolutionized natural language processing, yet their practical utility is often limited by persistent issues of hallucinations and outdated parametric knowledge. Although post-training model editing offers a pathway for dynamic updates, existing methods frequently suffer from overfitting and catastrophic forgetting. To tackle these challenges, we propose a novel framework that leverages hyperbolic geometry and graph neural networks for precise and stable model edits. We introduce HYPE, (HYperbolic Parameter Editing), which comprises three key components: (i) Hyperbolic Graph Construction, which uses Poincaré embeddings to represent knowledge triples in hyperbolic space, preserving hierarchical relationships and preventing unintended side effects by ensuring that edits to parent concepts do not inadvertently affect child concepts; (ii) Möbius-Transformed Updates, which apply hyperbolic addition to propagate edits while maintaining structural consistency within the hyperbolic manifold, unlike conventional Euclidean updates that distort relational distances; and (iii) Dual Stabilization, which combines gradient masking and periodic GNN parameter resetting to prevent catastrophic forgetting by focusing updates on critical parameters and preserving long-term knowledge. Experiments on CounterFact, CounterFact+, and MQuAKE with GPT-J and GPT2-XL demonstrate that HYPE significantly enhances edit stability, factual accuracy, and multi-hop reasoning.

## introduction large language models (llms) such as gpt-3 @xcite , palm @xcite , and llama @xcite have revolutionized natural language processing (nlp), enabling unprecedented capabilities in text generation @xcite , reasoning @xcite 1 hype (hyperbolic parameter editing) 2023), and contextual understanding @xcite . these models underpin a wide range of applications, from conversational agents @xcite to knowledge-intensive tasks like question answering @xcite @xcite and summarization @xcite @xcite . however, their reliance on static, pre-trained parametric knowledge renders them prone to generating factual inaccuracies @xcite , hallucinations @xcite , and outdated information-critical limitations in real-world deployments where accuracy and timeliness are paramount @xcite . while fine-tuning on updated data can mitigate these issues, the computational and data demands of retraining billionparameter models @xcite render this approach impractical for real-time knowledge updates. instead, post-training model editing has emerged as a promising alternative, enabling targeted modifications to a model's parametric knowledge without full retraining @xcite . existing model editing methods, such as rome @xcite and memit @xcite , rely on euclidean geometry to modify specific facts. while these methods achieve localized edits, they struggle with hierarchical relationships due to euclidean space's flat structure. for example, rome enforces memorization via equality constraints, while memit uses least-squares optimization, both of which fail to preserve relational depth @xcite . this leads to geometric mismatch, where edits distort semantically related knowledge @xcite . additionally, euclidean updates in overparameterized models like gpt-2 @xcite cause update instability, propagating unintended changes through dense connections @xcite . finally, these methods suffer from contextual fragility, lacking mechanisms to maintain global coherence during edits @xcite . hyperbolic geometry, in contrast, naturally encodes hierarchical relationships through its exponential growth property, making it well-suited for modeling linguistic knowledge @xcite . unlike euclidean spaces, where distances are linear, hyperbolic space expands exponentially, allowing child nodes to be placed far from parent nodes while preserving their relational proximity @xcite . this property enables precise parameter updates that do not propagate unintended side effects. for instance, modifying a parent concept in hyperbolic space affects only its immediate vicinity, preserving the stability of its hierarchical descendants. furthermore, hyperbolic operations like möbius addition @xcite ensure updates remain on the manifold, avoiding the geometric distortion caused by euclidean vector addition. motivated by these insights and the inherent advantages of hyperbolic geometry, we propose hype (hyperbolic parameter editing), a novel framework that leverages hyperbolic geometry @xcite for model editing through three core components. first, hyperbolic graph construction projects knowledge graph into a learnable curvature space using poincaré embeddings @xcite , preserving hierarchical relationships-such as parent-child and whole-part associations-via exponential mapping and hyperbolic distances. second, möbius-transformed (ungar, 2013) updates adjust model parameters using hyperbolic addition, a geometric operation that preserves the hierarchical structure of the data. unlike euclidean vector addition, which can distort relationships in hierarchical data, möbius addition ensures that edits remain within the hyperbolic manifold. this prevents unintended side effects by maintaining the relative distances between related concepts. third, dual stabilization combines gradient-based sparsification @xcite , which masks negligible gradient updates, with periodic resetting of graph network parameters to prevent the overwriting of critical pre-existing knowledge during the integration of new edits. by unifying hyperbolic embeddings, möbius-transformed updates, and robust stabilization techniques, hype achieves precise and resilient model edits. we summarize our contributions as follows: 1. hype is the first approach to integrate hyperbolic geometry with graph-based model editing, leveraging poincaré embeddings to project knowledge graphs into a learnable curvature space that preserves complex hierarchical relationships during model updates. 2. hype introduces a novel editing mechanism that applies möbius transformations for parameter updates, ensuring that edits remain consistent with the hyperbolic manifold and preserving the inherent hierarchical structure-thereby preventing the distortions common with euclidean vector addition. 3. we validate hype across three widely benchmarked datasets-counterfact @xcite , counterfact+ @xcite , and mquake ( zhong et al., 2023 )-and two popular llms-gpt-j (wang and komatsuzaki, 2021), and gpt2-xl (radford et al., 2019)-demonstrating superior factual accuracy, parameter efficiency, and edit stability compared to existing model editing methods. code available at: https://github.com/ yashkumaratri/hype 2 related work post-training model editing: model editing techniques broadly fall into parameter-modifying and parameter-preserving approaches. parametermodifying methods, such as locate-then-edit strategies @xcite @xcite and meta-learning frameworks @xcite , adjust model parameters via hypernetworks or rankone weight updates. @xcite enhances specificity using knowledge graphs, but these methods typically treat parameter changes as isolated scalar operations @xcite , disregarding structured knowledge representations in llms @xcite . this often leads to unintended side effects @xcite and poor support for compositional updates @xcite . parameter-preserving approaches avoid direct weight updates by leveraging external memories or prompts @xcite @xcite , while continual learning methods @xcite use orthogonal projections to mitigate catastrophic forgetting. however, these strategies require architectural modifications or fail to maintain long-term coherence in factual updates. ## methodology in this section, we detail the methodology underlying hype, our proposed framework for hyperbolic model editing. ## datasets, baselines and evaluation we evaluate hype on three widely benchmarked datasets -1) counterfact @xcite , which assesses factual accuracy, specificity, and generalization; 2) counterfact+ @xcite , which evaluates edit portability across paraphrased queries; 3) mquake @xcite , which tests multi-hop reasoning capabilities. we compare hype against seven state-of-the-art baselines -(i) zeroshot (unmodified model), (ii) ft (fine-tuning), (iii) mend @xcite , (iv) rome @xcite , (v) memit @xcite , (vi) pmet, and (vii) rae @xcite . further details on datasets and baselines are provided in appendix a. we measure hype's performance using the following metrics -1) efficacy (eff): quantifies the accuracy of factual edits by measuring the model's ability to correctly answer questions after updates. 2) generalization (gen): assesses robustness across paraphrased queries and reasoning tasks, ensuring edits apply consistently to varied inputs. 3) specificity (spec): measures unintended alterations to unrelated knowledge, evaluating the model's ability to avoid cascading errors. 4) portability (port+): evaluates the model's ability to transfer edits across different contexts, ensuring updates remain effective under rephrased questions (used for benchmarking counterfact+). 5) edit quality score (eds): a composite metric defined as the harmonic mean of efficacy, generalization, and specificity, providing a holistic view of edit performance. 6) multi-hop efficacy: for mquake, efficacy is measured over 2-hop, 3-hop, and 4-hop reasoning tasks, assessing the model's ability to handle complex, multi-step queries. ## experimental results this section presents the evaluation of hype against seven baselines-zeroshot, ft, mend @xcite , rome @xcite , memit @xcite , pmet @xcite , and rae @xcite -on three benchmark datasets: (1) counterfact @xcite , assessed using efficacy (eff), generalization (gen), specificity (spec), and edit quality score (eds); ## discussion and analysis hype demonstrates strong performance on localized edits while maintaining edit stability, even in cases where multi-hop reasoning is challenging. below, we analyze representative examples to illustrate the model's capabilities. the complete model outputs for the analyzed cases are presented in appendix section c 6.1 qualitative analysis of successful edits hype demonstrates strong performance on localized factual rewrites, driven by its hyperbolic geometry and dual stabilization mechanisms. consider case id 983, where the model is tasked with editing the fact that larry knechtel plays the guitar to instead state that he plays the violin. as shown in listing 1, the model correctly redirects the rewrite prompts to the new target with 100% accuracy, despite a low probability assigned to the edited target (@xmath0.0192). the original belief (@xmath1.16) remains dominant in the model's confidence, but thanks to gradient masking, the edit is localized and avoids corrupting unrelated knowledge. a similar behavior is observed in case id 729, which involves modifying johann von rist's occupation from poet to astronomer. in listing 2, the model exhibits high post-edit performance across all categories, with rewrite and paraphrase prompts achieving strong probabilities for the new target (@xmath2.08 and 4.43 respectively across paraphrases), while still preserving consistency across semantically similar prompts. these cases showcase hype's ability to perform precise, isolated updates while maintaining global factual integrity, which is something euclidean baselines like rome struggle with due to their inability to effectively disentangle neighborhood geometry. ## conclusion in this paper, we proposed hype, a novel posttraining model editing framework that leverages hyperbolic geometry and graph neural networks to perform precise and stable factual updates in large language models. our method addresses core limitations in existing approaches-such as overfitting, poor generalization, and catastrophic forgetting-by introducing a hyperbolic graph structure based on poincaré embeddings, a möbiustransformed update strategy for navigating non-euclidean space, and a dual stabilization mechanism combining gradient masking with periodic gnn parameter resetting. through comprehensive evaluations on the counterfact, counterfact+, and mquake bench-marks with gpt-j and gpt2-xl, we demonstrate that hype significantly improves factual accuracy and parameter efficiency while minimizing unintended side effects on unrelated knowledge. qualitative analyses further highlight hype's ability to perform localized, high-fidelity edits and propagate updates in a controlled and geometry-aware manner.