EVEDIT : Event-based Knowledge Editing for Deterministic Knowledge Propagation (2024)

Abstract: The dynamic nature of real-world information necessitates knowledge editing (KE) in large language models (LLMs). The edited knowledge should propagate and facilitate the deduction of new information based on existing model knowledge. We term the existing related knowledge in LLM serving as the origination of knowledge propagation as ”deduction anchors”. However, current KE approaches, which only operate on (subject, relation, object) triple. We both theoretically and empirically observe that this simplified setting often leads to uncertainty when determining the deduction anchors, causing low confidence in their answers. To mitigate this issue, we propose a novel task of event-based knowledge editing that pairs facts with event descriptions. This task manifests not only a closer simulation of real-world editing scenarios but also a more logically sound setting, implicitly defining the deduction anchor and enabling LLMs to propagate knowledge confidently. We curate a new benchmark dataset Evedit derived from the CounterFact dataset and validate its superiority in improving model confidence. Moreover, while we observe that the event-based setting is significantly challenging for existing approaches, we propose a novel approach Self-Edit that showcases stronger performance, achieving 55.6% consistency improvement while maintaining the naturalness of generation.

## introduction the dynamics of the physical world underscore the importance of knowledge editing (ke) for large language models @xcite @xcite . this line of research aims at updating models' beliefs and shaping models' behaviors based on editing knowledge for improved accuracy and usability. ideally, the edited knowledge should be able to propagate through other * these authors contribute to this work equally. 1 we will release the benchmark and code. related facts and deduct new knowledge. for instance, by updating the model with "messi joined team inter miami", the edited model should acknowledge that "messi began playing in major league soccer (mls)", as "inter miami competes in mls". this knowledge propagation is referred to as the ripple effect in @xcite . in this paper, we define the prior knowledge "inter miami competes in mls" as the deduction anchor due to its role in the knowledge propagation process. current ke approaches @xcite @xcite ) merely focus on edits of (subject, relation, object) triples. we observe that this simplified setting frequently results in undetermined deduction anchors during knowledge propagation. as illustrated in figure 1 , after editing the model to state "messi is a dutch citizen" and querying "where was messi born?", at least two logical deduction anchors may emerge, reducing the model's certainty in generating responses. on one side, the model might select the prior knowledge that "messi was born in argentina" as the anchor and predict "argentina", possibly implying that the edit "messi is a dutch citizen" reflects a change in citizenship. alternatively, the model could use "a dutch citizen should be born in the netherlands" as the anchor and consequently deduce "netherlands". to further analyze this phenomenon, we present a theoretical framework of knowledge editing based on the formal logic @xcite , representing knowledge as formal language propositions in § 2. using our theoretical framework, we find that existing work did not explicitly define the deduction anchor while two implicitly available assumptions are actually flawed: neither the no-anchor assumption (an empty anchor set) nor the maxanchor assumption (an anchor set comprising all knowledge not conflicting with the edit) provides a logically sound knowledge editing setting. consequently, the existing setting theoretically increases figure 1 : we observe fallacies of existing knowledge editing when the deduction anchor is not defined. the model edited with 'messi is a dutch citizen' may mistakenly propagate the edited knowledge that messi was born in the netherlands, which conflicts with its prior knowledge that messi was born in argentina. the uncertainty, which is further verified empirically across popular large language models. we observe that the fallacy mentioned above, despite its prevalence in knowledge editing for llms, does not manifest in the real world. as shown in figure 1 , if messi were to become a dutch citizen, a corresponding real-world event would need to occur. this event could be either messi applying for residency in the netherlands or a revelation that he was actually born and raised in the netherlands, unbeknownst to people until now. in our example, if the event behind is about messi applying for and obtaining dutch citizenship, we can still affirm that messi's birthplace is argentina. with this background knowledge, the uncertainty is naturally resolved. through extensive experiments, we also verified that event descriptions are indeed helping models improve their confidence when performing editing. therefore, to overcome the limitations of the current setting, we introduce eventbased knowledge editing, which not only provides a more robust framework by presenting clearer deduction anchors and editing boundaries but also offers a more practical setting, as real-world changes are often driven by events @xcite . we derive a new benchmark evedit from a triplebased knowledge editing benchmark counter-fact @xcite ) by augmenting facts with events using gpt-3.5-turbo plus human verification. we evaluate the post-edit model's ability with both text completion and qa tasks. to perform knowledge editing under this new setting, we decompose the event descriptions into a series of triples to accommodate current editing methods like rome @xcite , memit @xcite , pmet @xcite and grace @xcite . we further propose a novel solution self-edit inspired by @xcite which can effectively utilize the eventual context to decide editing boundaries during updating. our evaluations show that while adapting previous editing approaches provides suboptimal results, our approach exhibits over 56.6% increase in factual consistency while keeping the naturalness of generations by edited models. our approach neither requires a linearly growing external memory which previous works @xcite used to trade for high performance. overall, our contributions are: (1) we identify a critical deficiency of the current ke setting, by providing a careful theoretical analysis for ke and conducting extensive experiments, we attribute the problem to the improper assignment of deduction anchor. ( 2 ## fallacies of knowledge editing in this section, we formulate and analyze the task of knowledge editing from both theoretical and empir- ## event descriptions improves certainty following our prior analysis, edits without sufficient context to infer a proper deduction anchor k e cause conflicts between the editing boundary b k e ∪ e and the remaining model knowledge k\k d , which ultimately lead to uncertainty in edited models. rather than simply augmenting the existing benchmarks with deduction anchors for edits, we propose a more practical setting of augmenting edits with eventual context since knowledge updates are more often driven by events in real-world scenarios @xcite rather than provided deduction anchors. for example, we would possibly find an event about 'messi applied for dutch citizenship', which hints that the fact that he was born in argentina is not changed. on the contrary, a triple of "(messi, citizen of, netherlands)" alone would cause confusion. to verify that event-based editing reduces uncertainty compared with single factual edits, we quantify uncertainty based on equation (1) for both type of edits. however, since it is computational costly to compute max y p (y|x, θ) for longer output sequences (answers or text completions), we instead use e y∼p (y|x,θ) log p (y|x, θ) to measure the certainty. 3 each edit instance in e 2 dit contains the original fact, the event description, and the question-answer pairs related to the fact. we compare the certainty of a frozen pretrained lm generating answers to questions when given the original fact versus the event description. we plot our results on llama2-7b-chat in figure 3 and leave results on mistral-7b, and llama2-13b-chat in appendix b. each edit instance corresponds to a point in the scatter plot. we use red to highlight instances where event-based context enhances generation certainty, and blue to indicate the opposite case. results show that event-based knowledge editing significantly reduces uncertainty. ## conclusion and future work this paper establishes a theoretical framework for knowledge editing, identifying a pivotal challenge within existing methodologies: the oversight of the deduction anchor that leads to uncertainty within edited language models. to overcome this limitation, we introduced event-based knowledge editing. this approach enhances the traditional editing framework by incorporating event descriptions, which not only naturally mirror real-world editing scenarios but also implicitly define the deduction anchor, thereby addressing the issue of indeterminate editing boundaries. to tackle the complexities of event-based knowledge editing, we introduce an innovative self-edit method. with our new benchmark evedit, we demonstrate that this new setting is challenging for existing approaches while our novel approach achieves a better performance. we advocate for further research endeavors towards this more practical, event-based knowledge editing setting.