D istill MIKE : Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models (2024)

Abstract: Among the recently emerged knowledge editing methods, in-context knowledge editing (IKE) has shown respectable abilities on knowledge editing in terms of generalization and specificity. Noting the promising advantages but unexplored issues of IKE, we propose **DistillMIKE** as a novel extension of IKE, i.e., editing **distill**ation of "**M**assive” **I**n-context **K**nowledge **E**diting in large language models (LLMs), mainly consisting of two expansions; 1) *Massive in-context knowledge editing (MIKE)*, which extends IKE to a massive editing task, aiming to inject not a single edit but a set of massive edits to LLMs; To preserve specificity, our key novel extension is a “selective” retrieval augmentation, where the retrieval-augmented IKE is only applied to “in-scope” examples, whereas the unedited model without IKE is employed for “out-of-scope” ones. 2) *Editing distillation* of MIKE using low-rank adaptation (LoRA), which distills editing abilities of MIKE to parameters of LLMs in a manner of eliminating the need of lengthy in-context demonstrations, thus removing the computational overhead encountered at the inference time. Experimental results on the zsRE and CounterFact datasets demonstrate that MIKE shows the state-of-the-art perfomrances and DistilMIKE show comparable performances with MIKE. Our code is available at https://github.com/JoveReCode/DistillMIKE.git.

## introduction while large language models (llms) have shown remarkable abilities across a broad spectrum of natural language processing (nlp) tasks @xcite @xcite , llms are still limited in the coverage and veracity of their world knowledge, thus causing reliance *corresponding author on outdated knowledge @xcite @xcite , or the generation of erroneous, hallucinatory, or biased contents @xcite @xcite @xcite . given the evolving nature of world knowledge and the need to correct inaccurate information in llms, there has been a growing interest in the "knowledge editing" task, aiming at developing a scaled and effective editing mechanism that injects new knowledge into llms or corrects false and erroneous information in llms. in particular, this paper addresses the "massive editing" task, as in @xcite , where a large number of edits are provided beyond just a single correction of an edit, the resulting editing mechanism needs to properly update more than hundreds or thousands of facts in llms simultaneously. among various approaches to knowledge editing, such as parameter updating (pu) @xcite @xcite @xcite @xcite and memory-based methods @xcite @xcite @xcite , in-context knowledge editing (ike) has been newly proposed, inspired by in-context learning (icl) emerged in llms @xcite , showing noticeable editing performances in terms of generalization and specificity. motivated by the respectable advantages of ike, such as its human-interpretable editing method, we would like to go further toward expanding and improving ike by addressing the following issues: 1) extension to massive editing tasks: ike has not been explored in the massive editing task, and thus, it remains largely unclear how ike is generalized to the massive editing task, how ike performs compared with other popular pu methods such as memit, and whether the demonstration construction previously suggested in @xcite is also feasible in the addressed task. 2) resolving the computational overhead caused by the use of lengthy prompts: ike incurs computational and memory overhead at the inference stage, because the length of an input prompt becomes a "long," resulting by prepending a non-trivial number of demonstrations. towards a novel extension for addressing the aforementioned issues, we propose distillmike, the editing distillation of massive in-context knowledge editing in llms, which mainly consists of two components: a large number of massive edits (i.e. facts) to be injected into llms are first stored in a separate memory, namely the edit memory. given an input query prompt, ike is then preformed using its "relevant" edit retrieved from the edit memory, referred to a query-matched edit. to effectively preserve specificity, without employing ike for all input prompts, we instead newly propose a "selective" retrieval augmentation, where the retrieval-augmented ike is applied only for in-scope examples, not for other out-of-scope examples. furthermore, given this selective nature of applying ike, we further propose the use of scope-aware demonstrations, paying attention to in-scope cases, thus by including only "update" types of demonstrations that are likely useful for processing in-scope edits, but by excluding other "retain" types, as they may be mostly useful for out-of-scope cases. to substantially reduce the number of parameters to be updated, lora fine-tuning (hu et al., 2022) is adopted during the editing distillation, finally resulting in distillmike. experimental results on the zsre and counter-fact datasets demonstrate that mike achieves stateof-the-art performance in the counterfact dataset, and distillmike shows comparable performance with mike and improves existing editing methods such as ike and memit, even in the setting where lengthy in-context demonstrations and instructions do not appear in prompts. our contributions can be summarized as follows: 1) we propose mike, which extends ike to the massive editing task based on selective retrieval augmentation depending on the scope type of an input query; 2) we further propose distillmike, a lora-finetuned student model that is distilled from multi-teacher models -ike and unedited base models -thereby injecting icl prompts in mike into the model parameters, thus enabling inference without the need for lengthy demonstration prompts; and 3) the proposed mike and dis-tillmike show state-of-the-art and promising performances on the zsre and counterfact datasets. ## related works existing methods can be broadly categorized into pu-and memory-based methods. @xcite presented an extensive review of knowledge editing methods. in this section, we briefly review selected methods and discuss some of them in terms of the novelty of our work. ## task definition to formally define the massive editing task, suppose that m is an autoregressive language model, m(x) is the output generated by the decoding step given a prefix sequence x, and new factual knowledge to be injected into m is represented as a set of relational triples. more specifically, s is a set of real-world entities or concepts, r is a set of relations, and examples of an edit, its prefix, in-scope and out-of-scope prefixes, and their correct objects are presented in appendix b. ## method figure 1 presents the overall architecture of the proposed mike and distillmike with a brief sketch of their construction. 1) ); for an in-scope case (i.e., ret(q) ̸ = ∅), mike further calls the demonstration selection component demo(q) (in section 4.1.2), and the resulting demonstrations are further concatenated with a query q, and then they are fed to the decoding process, resulting in m(demo(q); q); for an out-of-scope case (i.e., ret(q) = ∅), without demonstration selection, a query q is only fed to the decoding process, merely giving m(q). (b) distillmike: editing distillation is performed by taking mike as a teacher model and initializing a student model by the base model. inherited from the selective retrieval augmentation of mike, distillmike results from a multi-teacher distillation by taking ike and the unedited base model as in-scope and out-of-scope teachers, respectively, thereby decomposing l ed of eq. ( 2 ) into l ike and l base (in eq. ( 5 ) in section 4.2). mike acts differently for in-scope and out-ofscope cases; for an in-scope case, mike prepares in-context demonstrations selected from a set of "training" edits, denoted by demo(q), and prepends them to a query prefix x; on the other hand, for an out-of-scope case, no demonstration is provided in a prompt. the resulting prompts, with or without demonstrations, are fed to m to finally predict the output sequence. with this selective retrieval augmentation, the inference process of mike is summarized as follows: let p m (y|x) be the generative probability of a sequence y, given prefix x computed using m. the loss function used for the editing distillation is summarized as follows: to train m st θ using eq. ( 2 ), we undertake the lora fine-tuning method of @xcite to substantially reduce the number of parameters to be updated.