M ind B ridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modality (2025)

Abstract: Knowledge editing is a technique for efficiently and accurately updating the knowledge of large language models (LLMs) to alleviate obsolescence and correct errors. However, most existing methods overfit to specific models, causing edited knowledge to be discarded during each LLM update and requiring frequent re-editing, which is particularly burdensome in today’s rapidly evolving open-source community. To address this issue, we propose the problem of cross-model knowledge editing and introduce **MindBridge**, a scalable solution inspired by the low coupling between modality processing and LLMs in multi-modal models. MindBridge introduces the novel concept of **memory modality**, which encodes edited knowledge as an independent modality. It first performs LLM-agnostic pre-training of the memory modality and then integrates it with various LLMs. Extensive experiments on multiple LLMs and popular knowledge editing datasets demonstrate that MindBridge achieves superior performance even in editing tens of thousands of knowledge entries and can flexibly adapt to different LLMs. Our code is available at https://github.com/CrashBugger/MindBridge.

## introduction large language models (llms) have revolutionized natural language processing, demonstrating remarkable abilities in understanding and generation @xcite @xcite . these models leverage the knowledge acquired during pre-training to answer user queries. however, since the knowledge embedded in llms is stored in static parameters, their internal knowledge needs to be updated to keep pace with the ever-changing world and avoid obsolescence. additionally, for personalized user information or domain-specific knowledge, customizing llm output also requires updating the model's knowledge. traditional methods, such as fine-tuning, continual llama 1 llama 2 llama 3 knowledge edit (a) (b) lost ! lost ! i don't know. i can't see. i know it now. knowledge ... i can see. learning, or retraining, are computationally expensive and may inevitably degrade the model's general capabilities @xcite . fortunately, recent advancements in knowledge editing offer a promising solution, enabling efficient and precise modification of model knowledge at a low computational cost. existing knowledge editing methods generally fall into two categories @xcite . the first one preserves llm parameters, such as memorybased methods @xcite or methods that add extra parameters to the model @xcite . the second category modifies the parameters of the edited model, such as meta-learning-based methods @xcite or locateand-edit methods @xcite . while these methods have made significant progress, most still face the issue of overfitting to a single llm. if the knowledge to be edited is domain-or user-specific, re-editing this knowledge becomes necessary whenever the llm is updated, which is both tedious and time-consuming, especially given the fast pace of updates in the opensource community. this motivates us to explore whether the edited knowledge can be loosely coupled with the target model and is no longer limited to a single llm. we call this problem cross-model knowledge editing, as shown in figure 1(a) . to address this issue, inspired by the work on multimodal large language models, we introduce the memory modality, a novel concept that encodes edited knowledge as a standalone modality to partially decouple a model's knowledge from the llm itself. as illustrated in figure 1(b) , the analogy to visual modality offers a clearer understanding of memory modality. the current mainstream paradigm @xcite involves concatenating pre-trained visual modality encoders (e.g., vit @xcite , clip vit @xcite ) with various llm backbones (e.g., llama, vicuna @xcite ) to enable llms with visual perception. the low-coupling characteristic between visual modality processing and llms allows for independent updates of both components @xcite . this lowcoupling property is precisely what is needed to solve the problem of cross-model knowledge editing. by encoding editable knowledge as a separate memory modality, pre-training a memory modality encoder to handle this knowledge, and subsequently concatenating it with multiple llms, we achieve efficient cross-model knowledge editing. based on the idea of memory modality, we propose mindbridge, a two-stage solution for crossmodel knowledge editing. the first stage is memory modality pre-training, where we introduce three training objectives: memory injection, memory association, and memory existence. these objectives are designed to train the memory modality encoder so that it can acquire relevant memories, perform memory association, and determine whether certain memories exist. the second stage is memory modality bridging, where we fine-tune a simple projector to achieve efficient cross-modal alignment, enabling the output of the memory modality to be understood and utilized by llms. this approach allows for efficient large-scale knowledge updates, with minimal modifications required when the llms are updated. to test the effectiveness of mindbridge, we conduct extensive experiments on widely used knowledge editing datasets, zsre and counterfact, using multiple llms, including gpt2-xl, gpt-j, and llama3. leveraging the low coupling characteristic between the memory modality and llms, mindbridge achieves efficient and scalable knowledge editing across different models, delivering excellent editing performance even in scenarios involving tens of thousands of knowledge edits. our contributions can be summarized as follows: ## problem formulation the goal of knowledge editing is to modify the knowledge stored in a model. in this paper, we focus specifically on editing memories composed of factual knowledge. more concretely, for a triplet (s, r, o) consisting of a subject s, relation r, and object o (e.g., @xmath0, @xmath1, @xmath2, we aim to insert a new triplet (s, r, o * ) (e.g., @xmath3, @xmath4, o * = trump) into the llm to replace the previous knowledge, i.e., (s, r, o) → (s, r, o * ), where these two triplets share the same subject and relation. specifically, o * can also represent knowledge that does not originally exist in the llm, i.e., (s, r, ∅) → (s, r, o * ). given a set of knowledge to be edited d @xmath5, r i , o * i ) | @xmath6, 2, . . . , n}, a knowledge editing operation ke, and a model to be edited f , the goal of knowledge editing is to generate a new model f * through the knowledge editing operation ke. this can be formulated as follows: here, i(d edit ) denotes the knowledge set that requires editing and its neighborhood, such as paraphrasing and rewriting, with d edit ⊆ i(d edit ). meanwhile, o(d edit ) denotes the set of knowledge items unrelated to the edited knowledge. for simplicity, we will refer to these sets as i and o moving forward. in some studies, i is also referred to as in-scope examples, while o is termed out-ofscope examples @xcite . equation 1 specifies that the edited model f * should correctly predict the edited knowledge and its neighborhood. for inputs unrelated to the edited knowledge, f * should maintain consistent predictions with the original model f . this ensures that knowledge editing updates target knowledge while minimizing interference with unrelated knowledge. ## method as illustrated in fig. 2 , mindbridge is meticulously structured into two primary stages. the first stage, memory modality pre-training (detailed in section 4.1), is specifically designed to obtain the memory modality encoder, e m . the second stage, memory modality bridging (detailed in section 4.2), is dedicated to training a projector to construct the memory-to-language bridging module, p m . ultimately, the edited model, f * , can be mathematically represented by eq. 2. in mindbridge, knowledge is encoded into memory modality features, denoted as x memory , through the synergistic utilization of the memory modality encoder e m and the bridging module p m . these features are subsequently concatenated with the textual input (s, r) as soft prompts and provided to the llm, f , for prediction. stage i: memory modality pretraining stage ii: memory modality bridging random mask memory modality encoder memory injection s r o * [mask] ## conclusion in this paper, we propose mindbridge, a scalable cross-model knowledge editing method designed to address the issue that most current knowledge editing methods are overfitted to a single model, leading to the problem of discarded edited knowledge and frequent re-editing with each model update. based on the novel concept of memory modality, mindbridge enables edited knowledge to transcend individual models. extensive experiments conducted on two popular knowledge editing datasets and various llms demonstrate the effectiveness and scalability of mindbridge.