Commonsense Knowledge Editing Based on Free-Text in LLM s (2024)

Abstract: Knowledge editing technology is crucial for maintaining the accuracy and timeliness of large language models (LLMs) . However, the setting of this task overlooks a significant portion of commonsense knowledge based on free-text in the real world, characterized by broad knowledge scope, long content and non instantiation. The editing objects of previous methods (e.g., MEMIT) were single token or entity, which were not suitable for commonsense knowledge in free-text form. To address the aforementioned challenges, we conducted experiments from two perspectives: knowledge localization and knowledge editing. Firstly, we introduced Knowledge Localization for Free-Text(KLFT) method, revealing the challenges associated with the distribution of commonsense knowledge in MLP and Attention layers, as well as in decentralized distribution. Next, we propose a Dynamics-aware Editing Method(DEM), which utilizes a Dynamics-aware Module to locate the parameter positions corresponding to commonsense knowledge, and uses Knowledge Editing Module to update knowledge. The DEM method fully explores the potential of the MLP and Attention layers, and successfully edits commonsense knowledge based on free-text. The experimental results indicate that the DEM can achieve excellent editing performance.

## introduction large-scale language models (llms) have demonstrated remarkable performance in various natural language processing tasks. nevertheless, errors or outdated knowledge are inevitable in llms @xcite . directly fine-tuning a large language model demands significant computational resources @xcite , making it econom-fig. 1 : an example with factual knowledge and commonsense knowledge, and obtaining the correct answer by editing the model. knowledge editing serves as an effective approach to update llms. existing knowledge editing methods predominantly concentrate on editing triple-based facts such as entity-relation pairs @xcite , events (multiple triplets) @xcite . these approaches commonly utilize strategies involving neuron localization and editing @xcite , assuming that entities and phrases within factual triplets are stored in a limited set of neurons. by manipulating these select neurons, knowledge editing can be accomplished. as shown in figure 1 , factual knowledge editing involves rectifying outdated triplets @xmath0, president, @xmath1, president, biden>. however, in real-world scenarios, structured entity-relation triplets often fall short in adequately describing many knowledge pieces, especially when it comes to commonsense knowledge @xcite . the data characteristics of commonsense knowledge are broad knowledge scope, long content and non instantiation, which limits the effectiveness of traditional knowledge editing methods. in addition, when using llms, users often need to obtain commonsense knowledge in the form of free-text, rather than structured entity level information. this user preference indicates that commonsense knowledge editing based on triplet forms does not meet their needs. therefore, we propose a more challenging commonsense knowledge editing task based on free-text, which has wider practicality. compared to previous methods, commonsense knowledge editing based on free-text presents some new challenges, as shown below: (1) the previous knowledge localization methods (e.g. causal tracing @xcite ) typically used the probability value of the editing target as the response value of the knowledge storage location. the success of this method is based on the fact that the editing target is a single token or entity. however, the editing target of commonsense knowledge based on free-text editing has multiple tokens, which limits the effectiveness of previous methods. (2) previous knowledge editing methods typically assumed that factual knowledge was stored on a single or small number of neurons, and knowledge editing could be achieved through operations on a small number of neurons. however, the experiments conducted in section 3 indicate that commonsense knowledge based on free-text does not conform to this assumption. commonsense knowledge based on free-text has a wide range of storage locations, is more dispersed, and is less prone to localization. therefore, previous knowledge editing methods are insufficient for handling commonsense knowledge editing based on free-text. to address the aforementioned challenges, we conducted experiments from two perspectives: knowledge localization and knowledge editing. firstly, we introduce a knowledge localization for free-text(klft) method that include knowledge location and recall. specifically, knowledge location experiments are utilized to determine whether commonsense knowledge is stored in the local hidden states of transformers, as well as to explore the form of storage. the knowledge recall experiment is used to verify whether specific hidden states storing commonsense knowledge have a significant contribution to that knowledge. two experiments together indicate that, in comparison to triple facts, commonsense knowledge predominantly resides in the mlp layers and attention (attn) layers, the storage of knowledge is not local but rather dispersed throughout. this means that the previous editing methods (e.g., editing local layers in rome @xcite and pmet @xcite ) were unreasonable. secondly, we propose a dynamics-aware editing method(dem). specifically, we introduce a dynamic-aware module for real-time detection of the storage location of each commonsense knowledge, and selected the layer with the highest contribution to knowledge as the editing layer. subsequently, we employ a knowledge editing module to perform targeted knowledge editing on specific mlp and attn layers. the experimental results validated the effectiveness of the method. to address the issue of insufficient commonsense knowledge datasets for editing based on free-text, we have developed commonsense knowledge editing benchmark (ckebench) . this dataset has 15600 samples and six evaluation indicators, which is more challenging than the existing dataset. to the best of our knowledge, we are the first to introduce an commonsense knowledge editing benchmark. additionally, we investigate the storage and recall of commonsense knowledge and propose an effective editing method. our contributions can be summarized as follows: commonsense knowledge editing benchmark(ckebench) < atomic data @xmath2, as a result, personx wants to live happily ever after sample 2 personx accepts persony appointment, resulting in personx travels to appointment sample 3 personx can tell persony that persony is being solipsist and insolent, as a result, others want to to stop what they're doing ## knowledge localization for free-text to locate commonsense knowledge based on free-text within llms, we propose a knowledge localization for free-text (klft) method, which involves two experiments : knowledge location and recall. ## dynamics-aware editing method to edit commonsense knowledge based on free-text, we propose a dynamics-aware editing method(dem). specifically, the dem includes a dynamics-aware module and knowledge editing module. ## experiments in the section, we investigated the effectiveness of dem method and existing editing methods in editing commonsense knowledge based in freetext. ## related work the existing knowledge editing dataset can be divided into triplet form and event form. in triplet format dataset, commonsense knowledge dataset includes pep3k and 20q @xcite , factual knowledge includes zsre @xcite , counterfact @xcite , fact verification @xcite , calibration @xcite , mquake @xcite and rake @xcite . in event format dataset, datasets with only factual knowledge, including elken @xcite and evedit @xcite . the previous editing methods mainly focused on editing knowledge in the form of triples, with a small amount of knowledge in the form of editing events. the methods for editing triplet forms mainly include : (1)locate-then-edit method @xcite @xcite , (2) memory-based method @xcite @xcite , (3) hyper-network method @xcite @xcite . the method for editing event forms is self-edit @xcite . ## limitations due to limitations in computing resources, we did not conduct relevant experiments on larger language models. we conducted experiments to evaluate the efficacy of existing methods in editing commonsense knowledge based on free-text. as show in figure 7 , "one edit" refers to editing the sample once, while "multiple edits" involves editing the sample five times. it is observed that both the original output