How to Make LLM s Forget: On Reversing In-Context Knowledge Edits (2025)

Abstract: In-context knowledge editing (IKE) enables efficient modification of large language model (LLM) outputs without parameter changes and at zero-cost. However, it can be misused to manipulate responses opaquely, e.g., insert misinformation or offensive content. Such malicious interventions could be incorporated into high-level wrapped APIs where the final input prompt is not shown to end-users. To address this issue, we investigate the detection and reversal of IKE-edits. First, we demonstrate that IKE-edits can be detected with high accuracy (F1 > 80%) using only the top-10 output probabilities of the next token, even in a black-box setting, e.g. proprietary LLMs with limited output information. Further, we introduce the novel task of reversing IKE-edits using specially tuned reversal tokens. We explore using both continuous and discrete reversal tokens, achieving over 80% accuracy in recovering original, unedited outputs across multiple LLMs. Our continuous reversal tokens prove particularly effective, with minimal impact on unedited prompts. Through analysis of output distributions, attention patterns, and token rankings, we provide insights into IKE’s effects on LLMs and how reversal tokens mitigate them. This work represents a significant step towards enhancing LLM resilience against potential misuse of in-context editing, improving their transparency and trustworthiness.

## introduction in-context knowledge editing (ike) is an effective and efficient knowledge editing method (ke), aimed at updating factual knowledge in llms @xcite . ike essentially uses incontext learning with few demonstrations to cause the llm to output predictions that are aligned with the knowledge provided in the prompt. different from traditional kes, which are based on fine-tuning @xcite , locating and editing the parameters responsible for the outputs @xcite , or meta-learning @xcite @xcite , ike does not require any training or changing the llms' parameters, but only a few demonstrations in the prompt that teach the model to update its knowledge. the advantage of using the context @xcite allows ike to achieve high editing performance @xcite at low cost. despite kes being useful for updating facts in llms, kes can be intentionally misused to insert inaccurate or unsafe information into llms @xcite . for example, badedit @xcite and megen @xcite use kes to efficiently modify a small subset of parameters to inject backdoors into llms. the development of zero-cost ike presents new opportunities for misuse, such as inserting misinformation and toxic content into llms. in this work, we first examine detecting edits conducted with @xcite . ike-edits can be used to poison prompts, where the user's prompt is maliciously modified by an attacker to cause the llm's outputs to change (e.g., to spread misinformation, bias llms, and to manipulate llms-trusting users). detecting ike-edits is therefore essential for identifying such attacks. detecting ike-edits is a more challenging task than detecting edits conducted with parameter-modifying methods @xcite , because ike-edits do not change the internal llm parameters. we show that even in a very constrained setting, where only the llm's output probabilities are available, ike-edits can be detected with high accuracy. going beyond detecting edits, we introduce the novel task of reversing ike-edits. reversing edits allows retrieving the llm's original unedited outputs, evading potential malicious attacks, and ensuring high transparency towards end users. our approach for reversing edits relies on learning reversal tokens, continuous or discrete, that cause llms to ignore the provided context, and focus more on parametric knowledge. we make our code available. 2 our contributions are the following: ## related work knowledge editing. the efficient updating of pre-trained models has garnered increased attention due to the computationally expensive nature of full model retraining. in particular, several model editing methods have been proposed that can efficiently update discrete model knowledge without retraining the entire model @xcite @xcite . however, these efficient update methods can potentially be exploited for malicious manipulation of model behaviors @xcite . notably, knowledge editing approaches make such malicious use even more accessible and cost-effective. for example, previous work shows that knowledge editing can be used to embed backdoors into llms @xcite and insert counterfactual and toxic information effectively @xcite . furthermore, model editing has been observed to potentially introduce unintended bias into llms @xcite or cause unsafe model behavior @xcite . in addition to investigating potential risks, @xcite have explored methods for detecting edited knowledge by parametermodifying kes (e.g., memit @xcite and malmen @xcite ). @xcite further investigates detecting types of edits (e.g., misinformation, bias, etc.). to the best of our knowledge, no previous work has explored detecting and reversing in-context edits. knowledge unlearning. knowledge unlearning is to selectively remove or ignore parts of the models' potentially harmful parametric knowledge, e.g., private information and copy-righted materials, while ideally not affecting unrelated knowledge @xcite @xcite . current knowledge unlearning methods can be generally categorized into two groups: parameter-based methods, such as fine-tuning @xcite , gradient ascent @xcite , inserting new components to the model @xcite , merging parameters @xcite and locate-and-modify specific neurons @xcite ; and in-context learning methods that use to-be-forgotten and normal samples as examples @xcite . our work can be viewed as an extension of the knowledgeunlearning landscape. while we similarly aim to make the model selectively disregard parts of the model's knowledge, we focus not on the model's parametric knowledge, as explored by @xcite , but on knowledge introduced through in-context learning, i.e., malicious examples inserted into user input. knowledge conflicts. models rely on their parametric knowledge from pre-training or the provided context from inputs to make predictions. knowledge conflicts arise when these sources of knowledge contain contradicting facts @xcite . ike creates conflicts between an llm's parametric knowledge and contradictory contextual facts in the ike prompts. our work resolves these conflicts by prioritizing parametric knowledge over in-context information, effectively aligning the model's outputs with its inherent knowledge base. prompt tuning. tuned prompts can either be discrete (hard), i.e., concrete tokens from the model's vocabulary, or continuous (soft), vectors in the embeddings space that do not correspond to any concrete tokens @xcite . discrete prompts can be found by mining @xcite , paraphrasing existing prompts @xcite or gradient-based search @xcite . in continuous prompts optimization, new embedding vectors are added to the vocabulary and are optimized to decrease the loss @xcite . additionally, more parameters can be added in each layer @xcite . initially, we follow @xcite and add new tokens to the vocabulary and tune the parameters of these tokens only. we later consider discrete prompt tuning. however, instead of considering the top-k tokens at each step to decrease the loss @xcite , we add a cosine loss term to keep the learned embedding vectors close to the model's original vocabulary, and consider the top-k candidates from the original vocabulary at the last step. this significantly decreases the computational cost for discrete prompt tuning. ## background and problem statement in this section, we provide a brief background on incontext knowledge editing, and define the detection and reversal problems we address in this work. ## detecting in-context knowledge edits informing end-users about post-training edits within llm-generated content is a straightforward approach to improving transparency in llm-based api applications. to offer an add-on solution, we frame this challenge as a detection task. specifically, given any llm-based api, our goal is to detect edited llm generations. this section focuses on detecting edits conducted with ike, a zero-cost knowledge editing method, and introduces an effective approach for doing so. dataset and models. following zheng et al. ( 2023 ), we edit facts from counter-fact @xcite using ike-editing prompts from @xcite . counter-fact includes counterfactual statements as the target fact, i.e., false facts (e.g., "the eiffel tower is in rome"). to create our detection dataset, we use 2,000 edits as instances of the edited class. these prompts contain p edit (s, r, o ′ ) for editing and p(s, r) for querying. as instances of the unedited class, we use the corresponding 2,000 prompts that contain only p(s, r), i.e., without ike-editing prompts. the dataset is then split into two balanced and equally sized parts: a training set and a test set, each containing 2000 instances. we experiment with gpt2-xl @xcite , gpt-j (wang and komatsuzaki, 2021) and llama3.1 8b @xcite . detection method. we train a simple logistic regression classifier with l1 regularization, using the top-10 output probabilities for the next token, given a prompt, as features to classify the output object as either edited or unedited. our approach relies on output probabilities, as previous work has demonstrated that parameter-modifying kes increase the llm's certainty about its outputs @xcite . we hypothesize that this effect also applies to ike-edits as well. to train the classification model, we assume access to a training set composed of edited prompts p edit (s, r, o ′ ) ⊕ p(s, r) and unedited prompts p(s, r). results. the results in tab. 1 show that the detection classifier performs well on all three llms (@xmath0. using only the top-10 output probabilities of the next token as features enables using this classifier in restrictive settings, where one does not have access to the model's internals. to better understand the effect of ike, we analyze the shift in the output distributions after editing, i.e., we examine how often the top-1 outputs of the model before editing can be found in the top-10 outputs of the model after editing. our comparison ## reversing in-context knowledge edits following our detection results, we now address reversing in-context knowledge edits. reversing edits allows for recovering the original outputs. ## experimental setup we evaluate how both continuous and discrete prompt tuning affect the model's outputs. we use the same models and datasets as the ones previously used for detecting ike-edits (cf. sec. 4). we train for 3 epochs with adam as optimizer using the default parameters. we keep the model's parameters frozen and tune only the embeddings of the reversal tokens. in discrete prompt tuning, we set @xmath1. we experiment with different numbers of reversal tokens. for each setting, we experiment with 5 seeds and report the maximum and mean accuracy with standard deviation. evaluation. we calculate the matching accuracy, i.e., the agreement of the original output and the output of the edited prompt with reversal tokens. for a perfect reversal method, i.e., optimal reversal tokens, the matching accuracy would be 1.0. matching accuracy is calculated as ## analysis we analyze the found reversal tokens under the discrete setting. our goal is to investigate which natural tokens help reverse edits. we show examples with 1 reversal token in tab. 4, and more examples in fig. 5 in the appendix. the found tokens contain some non-english tokens. in gpt2-xl and gpt-j, we notice that the prompts that contain the beginning of sentence token (<@xmath2. @xmath3. it is therefore intuitive for @xmath4. we experimented with @xmath5.1, but this did not lead to improved performance. this might be due to differences in the training procedure in llama3.1 compared to the gpt-models. ## conclusion in this work, we studied detecting and reversing ike-edits in llms. our work demonstrated that it is possible to detect ike-edits with high accuracy, even in scenarios where the model's parameters are not changed and where only limited information about the model's outputs is available. we further showed that ike-edits can be neutralized by tuning special reversal tokens. by achieving over 80% accuracy in recovering original outputs, continuous reversal tokens offer a promising method for mitigating the effects of malicious edits, and making llms intrinsically more secure. the effectiveness of continuous reversal tokens across different llms suggests a degree of generalizability that could be valuable for widespread implementation. we believe our work contributes significantly to the ongoing efforts to make llms more secure, transparent, and trustworthy. our future research will focus on improving the accuracy and robustness of both continuous and discrete reversal tokens. 12663 following zheng et al. (2023), we experimented with counterfact only, since ike-edits require demonstrations that share the same relation as the edited facts, and such relations/demonstrations are not easily identifiable in other datasets. as a first venture for studying reversing edits, we assumed white-box access to llms to add and tune special reversal tokens. however, such access might be difficult to attain for users in practice.