Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks (2024)

Abstract: Current knowledge editing approaches struggle to effectively propagate updates to interconnected facts.In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within these models for accurate reasoning. To support our analysis, we introduce a novel reasoning-based benchmark, ReCoE (Reasoning-based Counterfactual Editing dataset), which covers six common reasoning schemes in the real world. We conduct an extensive analysis of existing knowledge editing techniques, including input-augmentation, finetuning, and locate-and-edit methods. We found that all model editing methods exhibit notably low performance on this dataset, especially within certain reasoning schemes. Our analysis of the chain-of-thought responses from edited models indicate that, while the models effectively update individual facts, they struggle to recall these facts in reasoning tasks. Moreover, locate-and-edit methods severely deteriorate the models’ language modeling capabilities, leading to poor perplexity and logical coherence in their outputs.

## introduction contemporary language models demonstrate a remarkable capacity to encode extensive factual information, rendering them highly useful as a knowledge base for real-world applications. yet, the challenge of rapidly outdated knowledge persists, giving rise to a wide range of methods for knowledge updating, such as in-context learning @xcite , continual pretraining @xcite , locate-and-edit @xcite , and metalearning @xcite . despite the success of fact-wise editing, recent studies @xcite @xcite show that current model editing methods struggle to effectively propagate updates to interconnected facts, thereby limiting the performance of reasoning tasks based on these edited facts. however, the reason for this ineffective knowledge propagation remains largely unexplored. our observations in fact-editing experiments reveal that models often behave unexpectedly post-editing. for instance, when applying memit @xcite for fact editing, the model frequently fails to reliably recall the pertinent edited information and produces an incoherent chain-of-thought (cot) @xcite during question answering, as illustrated in the memitbased generation example in figure 1 . in this work, we place special emphasis on analyzing the results of cot prompting, which provides explicit reasoning steps that facilitate easier examination. we undertake an in-depth analysis of this phenomenon, concentrating on three essential competencies necessary for knowledge propagation on reasoning questions after model editing: we first explored input-augmentation, an approach where new facts are added (prepended) only in-context, as an approximated upper bound of model editing methods. we then examined model editing methods including finetuning and memit on the tülu series @xcite , which are llama-based instruction-tuned models of varying sizes. results show that all model editing methods achieve notably low performance on the re-coe benchmark, especially in certain reasoning schemes, with scores close to zero. our analysis further unravels the effect of various knowledge editing methods on the reasoning abilities of language models. we found that all editing methods result in a significant reduction in fact recall, indicating a key obstacle in effective utilization of the edited knowledge. surprisingly, models edited through locate-and-edit methods (i.e., memit) exhibit a severe decline in their generation coherence, leading to nonsensical outputs, which suggests a substantial deterioration in their fundamental language modeling abilities. we summarize our contributions of the paper as follows: ## recoe dataset we employ a hybrid-synthetic approach that combines existing complex qa datasets and llmassisted data synthesizing to create the recoe dataset. the dataset is designed to evaluate counterfactual editing across a broad spectrum of reasoning schemes: superlative, comparative, sorting, counting, aggregation, and subtraction. a typical datapoint encapsulates five key components: these components allow us to assess knowledge propagation by editing a language model with the set of counterfactuals cf, and testing if the edited model is able to flip its original answer (a) towards the counterfactual answer (ca) through reasoning. in this section, we provide a comprehensive overview of the dataset and delve into the nuances of its construction methodology. ## experiment language models we utilize the tülu series @xcite as the base language models to assess knowledge editing approaches, which is a good candidate for our study because (1) they are instruction-tuned with well-structured responses to user instructions (2) they include models of varying sizes, enabling us to explore the impact of model scaling on effective knowledge editing. ## analysis to comprehensively evaluate how certain knowledge editing methodologies impact model's capability that leads to ineffectiveness in knowledge propagation, our analysis encompasses three key dimensions: fact-wise editing effectiveness, fact recall accuracy, and logical coherence in model's generation. we assume the reasoning process follows a retrieve-and-generate regime. formally, where p ′ is the edited lm. the fact recall component requires 1) each fact within cf to be effectively edited; and 2) edited model is able to recall these edited facts through generation. the coherent generation component further requires logically coherent cot in the generated answer. fact-wise editing effectiveness this dimension examines the basic efficacy of editing methods. it assesses whether the applied edits achieve their intended modifications successfully, which constitutes the foundational requirement for any knowledge editing approach. defining ppl(f) and ppl(cf) as the averaged perplexity of the facts and the counterfactuals associated to a (q, a) pair, and ∆(cf, f) = ppl(cf) -ppl(f)), an effective editing over the facts f to its counterfactual counterpart cf can be evaluated by the indicator function: this definition of successful editing stipulates that the perplexity of counterfactual sentences must be lower than that of factual sentences. furthermore, in cases where the perplexity of counterfactual sentences is already lower before editing, it necessitates an even greater disparity between the two. fact-wise editing performed by qlora demonstrates a high degree of effectiveness, in contrast to memit. memit has the adverse effect of increasing the overall perplexity within the model. figure 4 demonstrates the fact-wise editing effectiveness in 7b model using qlora and memit. detailed results are presented in appendix b. fact recall assuming successful fact-wise editing, we then explore the model's proficiency in recalling and applying these modifications in reasoning tasks. this involves an in-depth analysis of the model's ability to retrieve from stored knowledge and utilize relevant information correctly. we focus on evaluating the relatedness and consistency of information within the cot response against the counterfactual facts. the evaluation metrics are defined as: relatedness: this metric assesses how relevant su pe rla tiv e co m pa ra tiv e co un tin g so rti ng ag gr eg at io n su bt ra ct io n the facts p generated by the model are to the designated fact/counterfactual set q. consistency: this metric calculates the proportion of the model's generated facts p that align factually with the fact/counterfactual set q: p is defined to be consistent with a given fact q if @xmath0 we leverage claude with dedicated few-shot demonstrations for automatic evaluation. detailed prompt can be found in appendix c. results are presented in figure 5 . regarding relatedness, different editing methods show different impacts on the model: the model edited by qlora retains the capability to recall information across different schemes. however, memit shows negative impacts in superlative, comparative, and sorting. in some cases, relatedness post-editing surpasses that of pre-editing. this could be attributed to the introduction of new facts into the model that it previously lacked. however, both qlora-edited and memitedited models show low consistency results, indicating that they are unable to accurately leverage the edited knowledge in actual use. it's important to note that this consistency doesn't correlate well with the fact-wise editing effectiveness. this disparity may stem from a lack of generalization during the editing process. essentially, the model seems to simply memorize the newly edited fact, lacking the ability to extend this understanding to different manifestations of the same concept. ## discussion qlora vs. memit for qlora, we observe that while it adequately supports fact-wise editing and generally preserves logical coherence, its primary deficiency lies in the retrieval of edited facts. in llm, the elicitation of knowledge depends heavily on appropriate prompting techniques while our approach involves merely fine-tuning llms with atomic fact sentences. consequently, a potential future direction may involve enriching these atomic facts with more comprehensive contexts prior to their utilization in fine-tuning, as it should enable the model to accurately recall information in response to a diverse set of prompts. in contrast, the memit model exhibits a decline in all three assessed abilities: fact-wise editing, fact recall, and coherence. given that our dataset comprises non-synthetic reasoning questions, which often include complex subjects (e.g., an event) and relations, non-unique objects (figure 3 ), the underperformance of memit suggests its current inadequacy in handling real-world factual knowledge. notably, memit's most pronounced deficiencies lie in its ability to recall facts and, critically, in maintaining coherence. this observation highlights that the functionalities of edited neurons extend beyond mere fact storage, challenging the assertions made in previous studies @xcite . ## conclusion in this study, we have developed a novel benchmark, recoe, which leverages counterfactual reasoning and is grounded in non-synthetic data for evaluating model editing. our analysis reveals significant challenges in existing knowledge editing approaches, particularly in their ability to effectively propagate new facts for coherent reasoning. through this investigation, we have identified key areas where these methods falter. our work provides a clear direction for future research in this field, aiming to enhance the efficacy and reliability of knowledge editing in computational models.