STEAM : A Semantic-Level Knowledge Editing Framework for Large Language Models (2025)

Abstract: Large Language Models store extensive factual knowledge acquired during large-scale pre-training. However, this knowledge is inherently static, reflecting only the state of the world at the time of training. Knowledge editing has emerged as a promising solution for updating outdated or incorrect facts without full retraining. However, most existing locate-and-edit methods primarily focus on token-level likelihood optimization without addressing semantic coherence. Our analysis reveals that such edited knowledge is often encoded as isolated residual streams in the model’s latent space, distinct from pre-existing knowledge and bypassing natural reasoning process. To address this, we propose STEAM, a semantic-level knowledge editing framework that enhances integration of updated knowledge into the model’s knowledge structure. STEAM first identifies target representations as semantic anchors for the updated factual association, then guides the internal representation of the edited fact towards these anchors through an alignment loss during optimization. Experimental results demonstrate that STEAM improves model’s ability to reason with edited knowledge and enhances semantic coherence, underscoring the importance of latent-space alignment for reliable and coherent knowledge editing. The code is available at https://github.com/GY-Jeong/STEAM.

## introduction large language models (llms) have achieved impressive performance in knowledge-intensive nlp tasks @xcite ) by leveraging extensive factual knowledge acquired through large-scale pre-training @xcite @xcite . however, this knowledge is inherently static, reflecting only the state of the world at the time of training. this limitation necessitates efficient methods for updating llms when new information arises or existing facts become outdated. knowledge editing foot_0 @xcite @xcite has emerged as a promising approach, enabling selective updates to specific factual information without the need for costly full retraining. this approach typically identifies a fact as a triple structure (s, r, o), representing @xcite , and aims to replace the original object o with a new entity o * . for example, the outdated fact (uk, prime minister, rishi sunak) can be updated to the current fact (uk, prime minister, keir starmer). for knowledge editing to be reliable, it is crucial that updated facts are consistently integrated into the model's knowledge structure. however, most existing methods primarily focus on tokenlevel likelihood optimization (i.e., maximizing the generation probability of o * ) without addressing semantic coherence. consequently, these methods face challenges in tasks that require knowledge integration, such as maintaining consistency across interconnected facts or supporting complex reasoning tasks @xcite @xcite . to investigate this limitation, we examine how edited knowledge is represented within the latent semantic space of the edited model. our analysis reveals that edited knowledge is often encoded as isolated residual streams -sequences of hidden states propagated through the transformer layerswhich are distinct from model's pre-existing knowledge ( § 3.2). building on this finding, additional interpretability-based analysis reveals that these residual streams reflect a direct activation for the generation of the target token o * , bypassing the model's natural reasoning process ( § 3.3). this behavior exposes a fundamental shortcoming of current editing strategies and underscores the need for approaches that promote semantic-level integration for coherent and reliable knowledge editing. to address this challenge, we propose steam (semantic-level knowledge editing framework), a novel approach that enhances knowledge integration and is easily compatible with existing locate-and-edit approaches. steam augments the conventional locate-and-edit approach through two main components: (1) latent positioning, which identifies semantic anchors for edited knowledge, and (2) latent-level alignment, which guides the edited knowledge representation towards these anchors in the latent space. our experimental results demonstrate that steam significantly improves the model's reasoning ability with edited knowledge and enhances overall semantic consistency, supporting the advantages of semantic-level integration for reliable knowledge editing. we summarize our contributions as follows: ## locate-and-edit approach locate-and-edit approach consist of two stages: (1) locating, which identifies the parameters where a fact (s, r, o) is stored, and (2) editing, which modifies those parameters to encode the updated fact (s, r, o * ). ## latent space analysis of edited knowledge although an edited language model can generate the updated fact, this alone does not confirm its integration into the model's knowledge structure. to investigate this, we conduct two complementary analyses focusing on the residual stream, the sequence of hidden states propagated through the transformer layers: for this analysis, we sample 1,000 instances from the counterfact @xcite dataset, specifically focusing on cases where the factual association (s, r) → o is preserved in the model, and both the original target o and the edited target o * are single-token entities. ## method to address the semantic isolation of edited knowledge observed in section 3, we introduce steam, a framework designed to enhance the semantic coherence of updated knowledge with the model's internal knowledge representations. to achieve this, steam augments the standard locate-and-edit process, specifically enhancing the optimization of the value vector v * through two main components: (1) latent positioning, which identifies appropriate semantic anchors for edited knowledge in the model's latent space, and (2) latent-level alignment, which incorporates an additional objective to steer the edited knowledge representation towards these anchors during the optimization process. ## conclusion in this paper, we addressed a limitation of current knowledge editing methods: their failure to coherently integrate updated knowledge into the model's existing structure despite successful output changes. our analysis revealed that existing methods often induce isolated latent representations, hindering consistent and reliable inference with edited knowledge. to address this issue, we introduced steam, a framework designed to promote deeper semantic integration. our experimental results demonstrate that enhancing semantic alignment improves the model's reasoning capabilities with edited knowledge and increases semantic coherence. these findings emphasize the importance of semantic-level integration for developing more robust and coherent knowledge editing techniques.