R el E dit: Evaluating Conceptual Knowledge Editing in Language Models via Relational Reasoning (2025)

Abstract: The conceptual knowledge in Large Language Models (LLMs) can become outdated over time, and concept editing is often an option. Current evaluations on conceptual knowledge editing primarily focus on whether the definitions of concepts are successfully edited, neglecting the impact on the model’s related beliefs. To address this gap, we introduce a benchmark called RelEdit, which includes criteria and questions to assess both concept-level and instance-level relational reasoning abilities of edited models. Our findings reveal that existing knowledge editing methods struggle to reason about related conceptual knowledge effectively. Additionally, we introduce a simple memory-based in-context editing baseline, MICE, which prompts the language model to generate answers that align with the stored edited concepts in external memory. In addition, we find that MICE obtains the best scores on our benchmark, suggesting a promising research direction for model editing.

## introduction as large language models (llms) are widely deployed, it becomes increasingly important to maintain their knowledge accuracy and currency without incurring significant retraining costs @xcite . previous studies have introduced knowledge editing methods to gradually incorporate new concrete factual knowledge into language models @xcite @xcite @xcite . however, these approaches of editing caseby-case factual knowledge are highly inefficient and lacks the modeling of relationships between instances @xcite . inspired by cognitive science @xcite , figure 1 : illustration of the relational reasoning on instance-level and concept-level when editing the concept definition of "gender". although conceptual knowledge editing have raised great research interest, there is no systematic study of evaluating this task. currently, the only existing benchmark, conceptedit @xcite , focuses on measuring whether the edited model can recall the newly injected concept definitions, typically involving three common metrics: reliability (re), generalization (ge), and locality @xcite . these metrics evaluate the answers through queries on the edited concept definitions, as well as verifying that irrelevant concepts are not corrupted. however, an important question that has not been addressed is whether the edited model can handle relational reasoning questions where the an- in this work, we introduce a benchmark reledit (relational reasoning for conceptual knowledge editing). when evaluating concept editing, it is important to go beyond just examining the edited concept definition. we should also assess whether both the concepts and instances logically derived from the edit have been appropriately modified, as illustrated in figure 1 . following existing benchmarks, we construct relational examples for evaluation. (i) at the instance level, we evaluate the relationship changes between concepts and concrete instances. (ii) at the concept level, we evaluate the relationship changes among concepts. from the two perspectives, we propose novel evaluation criteria to evaluate how well the model integrates the edit with the rest of its knowledge: instance change (ic), portability (po), instance locality (il), alignment belong (ab), and alignment compare (ac), as listed in we observe that although current methods effectively edit concepts, they often struggle to infer the relational knowledge associated with those concepts. additionally, our analysis reveals that: (a) larger models are better equipped to handle the relational reasoning challenge; (b) modifications between concepts that share similar semantic structures and superclasses facilitate the model's ability to update more related conceptual knowledge; and (c) instance-level evaluations reflect the ability of llms to maintain higher concept consistency by updating knowledge from related instances. finally, we propose a memory-based in-context editing baseline, mice, which prompts the llms to generate answers that align with the stored edited concepts in external memory instead of explicit parametric updates. while mice achieves superior results compared to current parametric methods on reledit, there is still significant room for improvement, necessitating further research. ## problem formulation in contrast to concrete factual knowledge, conceptual knowledge refers to the understanding of con-cepts, categories, principles, and relationships. it involves understanding and applying abstract ideas. it is evidenced that language models are capable of memorizing conceptual knowledge and making inferences @xcite . the conceptual knowledge stored in language models can be incorrect or become outdated over time. one potential solution is to update the knowledge without retraining. a concept can be represented as @xmath0, d) @xcite , where c denotes the concept name (e.g., school) and d means the definition description of concept (e.g., an institution for the education of students by teachers). for a concept @xmath1, d), there are several instances e belonging to the category broadly defined by the concept, which can be denoted as @xmath2.g., microsoft forecaster is an instance of concept software). a conceptual knowledge edit δ : (c, d) → (c, d * ) is defined as modifying the concept @xmath3, d) into a refreshed one c * = (c, d * ), that is, updating the concept definition @xmath4, in which d * represents the alternate concept definition. given a language model, conceptual knowledge editing aims to inject the edited concept to the language models' inner beliefs. since high-order concepts preserve meta knowledge in a hierarchical form, it is quite essential to investigate the relational reasoning of a single concept editing to certain related concepts and instances. ## reledit benchmark construction in this section, we introduce the data construction of reledit. it is difficult to have a universal principle for all concepts in the world. to ensure the rationality of the concepts, our concepts follow the well-established dbpedia @xcite , also with human judgement. some details about data construction are discussed in the appendix a. ## mice: a memory-based approach in this section, we introduce mice, a simple but effective baseline. mice only requires updating the question in the prompt with relevant feedback, and no retraining is need. our proposed mice is shown in figure 3 . the idea of mice is inspired by mello @xcite , and all edited concept names and their definitions are explicitly stored in an external memory. to achieve this, we convert all edited concepts into sentence statements using predefined templates. the concept name serves as the retrieval index, allowing us to retrieve the corresponding edited concept definition. mice performs the following steps: (1) it uses language models to extract the involved concepts in the question; (2) it extracts the involved concept definitions in the internal memory; (3) it checks whether the concept in llm is contradicted with memory, and make corrections; and (4) it prompts the llm to answer the questions with the corrected concept. details about mice are illustrated in appendix b. ## experiments in this section, we evaluate on reledit. experimental details and results including the impact of llm size, conceptual patterns across methods and editing cases are in appendix c and d. ## related work methods for knowledge editing. various techniques have been proposed to edit the knowledge stored in a model. some of these approaches involve identifying and adjusting the model's weights that correspond to specific concepts. notable examples include kn @xcite , rome @xcite , and memit @xcite . another line of research adopts meta-learning and utilizes a hyper-network, which is a smaller network responsible for generating the edited gradients. this category includes methods like ke @xcite , malmen @xcite , and mend @xcite . additionally, there are preservative methods that incorporate explicit memory and prompting techniques to rectify model predictions. noteworthy examples in this category are serac @xcite , memprompt @xcite , mello @xcite , and ike @xcite . ## conclusion in this work, we introduce a benchmark called reledit, which evaluates conceptual knowledge editing of llms. we find that existing knowledge editing methods struggle with answering questions related to specific instances and broader concepts. additionally, we show that a simple in-context editing method with an external memory achieves the best results on reledit, highlighting the potential of such editing approaches.