Context-Robust Knowledge Editing for Language Models (2025)

Abstract: Knowledge editing (KE) methods offer an efficient way to modify knowledge in large language models. Current KE evaluations typically assess editing success by considering only the edited knowledge without any preceding contexts. In real-world applications, however, preceding contexts often trigger the retrieval of the original knowledge and undermine the intended edit. To address this issue, we have developed CHED—a benchmark designed to evaluate the context robustness of KE methods. Evaluations on CHED show that they often fail when preceding contexts are present. To mitigate this shortcoming, we introduce CoRE, a KE method designed to strengthen context robustness by minimizing context-sensitive variance in hidden states of the model for edited knowledge. This method not only improves the editing success rate in situations where a preceding context is present but also preserves the overall capabilities of the model. We also provide an in-depth analysis of the differing impacts of preceding contexts when introduced as user utterances versus assistant responses, and we dissect attention-score patterns to assess how specific tokens influence editing success. We release our dataset and code at [https://github.com/holi-lab/CoRE](https://github.com/holi-lab/CoRE).

## introduction recent large language models (llms) exhibit emerging intelligence, largely due to the extensive knowledge acquired from training data. however, some of this knowledge may become outdated or require correction or removal @xcite . for instance, the knowledge "tim cook, who works for apple" may need to be edited to "tim cook, who works for amazon". since retraining large models is costly, the field of knowledge editing focuses on modifying only the relevant who 's in … the iphone (b) attention score _apple 56.29% _amazon 38.09% _amazon 90.88% _apple 0.12% owi ings 'gc ïĥîºîµ riet -profit utow imp ipping ïĥîºîµ isches hti longle… idd 'gc åĥad uiltin age entiful asm 'gc active… ī ð¾ð ¶ðµ… 'gc _belmo… 'gc oping mmdd 'gc ñģð¾ _belmo… 'gc oping ìĭ´'gc ùĥùī _belmo… _lap _lap yonel ñĥð½ðº… erru arel _bran ñīð¸ð± tainme… ãĥ½ãĥ³ 'gc _mp _âľķ ñīð¸ð± stice _vue proces… _mp _err ñīð¸ð± junit _sp ì¼ĵ ħ ssert tah 457 _train… _ordai… nel upert _entr _jadx bidden ussen ħ ched ñķñĺ )?@xmath0 _ear ãģĥãģĥ… ð¾ð²ð¸_ oping @xmath1.respo… bidden othy î¼î¼ _ceo oped ð¢ðŀ bidden othy @brief _ceo oped _confi… _amazon othy _is _ceo oped _confi… _amazon othy _is _ceo _is _chez _amazon othy _works _ceo _is _for _amazon othy _works _ceo _is _for _amazon othy _works _ceo _works _for _amazon othy _works _ceo _is _for _amazon othy _works _ceo _used _for _amazon othy _works _ceo _used _for _apple oth _works _who _works _for _apple oth _works _who _works _for _apple _cook _is _the _is _for _apple _cook _is _the _is _for _apple _tim _cook , _who _works _for subset of model parameters or leveraging auxiliary networks or memory @xcite . the goal is to ensure the model generates edited knowledge rather than the original one. previous work typically evaluates the success of knowledge editing by measuring the model's probability of generating the edited knowledge in isolation, without any preceding context. however, this setting is unrealistic, as edited knowledge is often expected to appear within a broader context or in the middle of a conversation with the user. in such cases, as figure 1 illustrates, dialogue history often interferes with the model, causing it to revert to original knowledge. this issue leads to the need for (1) a challenging benchmark to assess the success of editing when context is present (especially distractive contexts), along with (2) methods that are robust against preceding context. to address the first need, we introduce ched: contextual hop editing dataset, a new bench-mark to evaluate the context robustness of knowledge editing methods ( §3). ched allows this by prepending a prefix context to the edit prompt. for example, the prefix context, such as "who's in charge of developing the iphone?" in figure 1 can be added before the edit prompt "tim cook, who works for". in collecting these prefix contexts, a key observation is that entities within a prefix context tend to receive disproportionately high attention scores ( §5.6) when they have strong semantic relevance to the original knowledge (e.g., "iphone" in figure 1 ). in light of this, we construct prefix contexts using wikidata by selecting entities connected to the subject and object of the original knowledge and generating sentences that can naturally precede the knowledge statement. as a result, these prefix contexts and the highly relevant words within them distract the model from recalling edited knowledge, establishing ched as an effective benchmark for assessing the context robustness of knowledge editing methods in realworld use cases of llms. to address the second issue, we propose core: context robust editing, a knowledge editing method with enhanced context robustness ( §4). it builds on the widely adopted locate-then-edit approach, which directly modifies model parameters to edit knowledge. this approach is well known for its practicality, as it remains robust and scalable even when a large number of facts are edited. the core idea of core is to prepend distractive prefix contexts during knowledge editing and to minimize the variance of the model's hidden states generated during the decoding of edited knowledge across these prefix contexts. this simple regularization effectively ensures that only the necessary amount of modification is applied to the parameters, preventing overfitting to varying prefix contexts and enhancing context robustness. our extensive evaluations validate ched and core. prefix contexts from ched lead to substantial performance drops compared to the no-context condition across all editing methods. we also found that for the same prefix context, the model is more distracted when the context is provided as a user utterance rather than as its own. yet, our core method significantly narrows the gap in knowledge editing performance, even consistently maintaining high performance in general abilities and fluency. we provide an explanation through an in-depth analysis of the model's attention patterns. our contributions are as follows: (1) we intro-duce the ched dataset, a benchmark that assesses the context robustness of knowledge editing methods; datasets and benchmarks counterfact @xcite and zsre @xcite have been used widely to evaluate ke methods. to evaluate a broader range of linguistic phenomena and relational complexity, other benchmarks have been introduced, such as mquake @xcite , counterfact+ @xcite and rippleedits @xcite . mquake edits multiple pieces of knowledge and evaluates a integrated multi-hop question, thereby broadening the assessment of semantic shifts. counterfact+ attempts to add a sentence during evaluation by retrieving other samples from the counterfact that share the same r and o as the current edit triplet and placing those samples before it. finally, rippleedits evaluates the ripple effects by testing whether the model correctly updates related facts that become inconsistent after the edit. despite these efforts, the impact of prefix contexts on knowledge editing has been underexplored. our ched dataset carefully curates prefix contexts to be highly relevant and distractive to edited knowledge while also enabling an examination of how their relevance to s, o, and o * in knowledge statements contributes to distraction. ## ched: contextual hop editing dataset as discussed in the previous section, most existing knowledge editing datasets either lack prefix contexts or rely on sentences that do not reflect realistic contexts. this setup differs from real-world llm usage, where edited knowledge is often expected to appear in response to user prompts or after a relevant context. consequently, the effec-tiveness of knowledge editing methods is typically overestimated. to address this gap, we construct ched (contextual hop editing dataset), which associates knowledge statements with related prefix contexts. this provides a more realistic and challenging evaluation environment that interferes with llms when generating edited knowledge. ## core: context robust editing in this section, we introduce context robust editing (core), a knowledge editing method for improved robustness to diverse contexts. we build on the locate-then-edit approach, such as memit @xcite , because it enables large numbers of edits. we first provide an overview of memit as preliminaries ( §4.1), followed by the details of our core method ( §4.2). ## conclusion we introduce and release ched, a benchmark designed to evaluate the context robustness of knowledge editing. our evaluation across various methods reveals that even those which perform well often fail when a prefix context is introduced. this finding underscores that the aspect measured by ched has been largely overlooked by previous knowledge editing methods. it emphasizes the importance of this evaluation. to address this gap, we propose core, which enhances context robustness. we hope that ched, together with core, will contribute to the development of more context robust, practical, and reliable knowledge editing techniques for real-world applications.