Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning (2025)

Abstract: As large language models (LLMs) are applied across diverse domains, the ability to selectively unlearn specific information is becoming increasingly essential. For instance, LLMs are expected to selectively provide confidential information to authorized internal users, such as employees or trusted partners, while withholding it from external users, including the general public and unauthorized entities.Therefore, we propose a novel method termed ìn-context knowledge unlearning”, which enables the model to selectively forget information in test-time based on the query context.Our method fine-tunes pre-trained LLMs to enable prompt unlearning of target knowledge within the context, while preserving unrelated information. Experiments on TOFU, AGE and RWKU datasets using Llama2-7B/13B and Mistral-7B models demonstrate that our method achieves up to 95% forget accuracy while retaining 80% of unrelated knowledge, significantly outperforming baselines in both in-domain and out-of-domain scenarios.Further investigation of the model’s internal behavior revealed that while fine-tuned LLMs generate correct predictions in the middle layers and preserve them up to the final layer. However, the decision to forget is made only at the last layer, i.e. LLMs pretend to forget”.Our findings offer valuable insight into the improvement of the robustness of the unlearning mechanisms in LLMs, laying a foundation for future research in the field.

## introduction large language models (llms), such as @xcite , have significantly transformed various sectors by providing advanced capabilities in information processing and text generation. (3) our method enables llms to selectively unlearn knowledge in a timely manner by inputting the knowledge we want llms to forget in a prompt (e.g., «unl»paris«/unl»). in contrast to in-context unlearning (icul) @xcite , our method causes no hallucination by outputting "forget" in response to a question. however, the widespread deployment of such models introduces complex challenges related to privacy and the ethical use of information. in particular, the indiscriminate provision of sensitive or domain-specific information by llms raises significant concerns, necessitating mechanisms for selective information handling based on the user context @xcite . to improve the privacy and ethical use of llms, previous works have explored several approaches, including differential privacy @xcite , federated learning @xcite , and knowledge distillation @xcite . despite their contributions, these methods often compromise between privacy and model performance. the concept of "test-time adaptation" @xcite or "in-context learning" @xcite offers a dynamic approach to model adaptation, yet it fails to adequately address selective forgetting of sensitive information. for example, an llm deployed in a corporate environment to streamline project management should retain substantial industry-specific knowledge while selectively "forgetting" proprietary company data or sensitive information when accessed by unauthorized external consultants. this scenario underscores the critical need for a mechanism that enables llms to selectively forget or withhold sensitive information based on the query context without compromising their overall utility and performance. this paper introduces "in-context knowledge unlearning", a novel approach that enables llms to selectively forget information at test-time based on the query context. the overview of our method is provided in figure 1 . we developed unlearning tokens that, when applied during inference, enable the model to selectively ignore information pertaining to specified domains. through comprehensive experimentation, we validated the efficacy of our approach in facilitating domain-specific unlearning without compromising the general performance of the model. specifically, we conducted experiments on the tofu, age, and rwku datasets @xcite @xcite using llama2-7b/13b and mistral-7b models, showing that our method achieves up to 95% forget accuracy while retaining 80% unrelated knowledge, significantly outperforming baselines in both in-domain and out-ofdomain scenarios. moreover, further investigations into the model's internal behavior revealed that while fine-tuned llms generate correct predictions in the middle layers and sustain them through to the final layer, the decision to forget occurs only at the last layer, i.e., "llms pretend to forget". this finding enriches our understanding of selective information handling in llms and lays a foundation for future research to improve the robustness of models across sensitive and regulated domains. ## related work in-context unlearning. our method leverages in-context learning (icl) for knowledge unlearning. icl enables llms to adapt flexibly to new tasks by incorporating data within the input sequence context, rather than through weight updates as in fine-tuning @xcite @xcite . exploring the full capabilities of icl remains an active area of research, with recent studies empirically investigating its potential by examining in-context example design @xcite @xcite . @xcite explored methods for performing in-context unlearning. this study focuses on text classification tasks where the labels of specific instances are flipped to facilitate in-context unlearning. however, this approach has limitations as it primarily assesses unlearning in terms of text classification ability rather than actual knowledge. furthermore, the method trains the model to generate incorrect outputs, which does not constitute true forgetting. in contrast, our study introduces unique characteristics that address these issues. we specifically investigated knowledge unlearning within an incontext learning framework. moreover, by defining unlearning as the ability to "forget," we ensure that our approach avoids merely generating errors or irrelevant information, thereby achieving an effective and appropriate form of unlearning. ## conclusion in this study, we introduced and explored the concept of "in-context knowledge unlearning" within the framework of large language models (llms) by employin fine-tuning. our findings demonstrate that this approach enables llms to dynamically "forget" or selectively disregard information in test-time, as well as uncovers a nuanced behavior of llms: where they "pretend to forget" rather than actually eliminating the information from their knowledge base. the ability of llms to effectively "unlearn" in both in-domain and outof-domain scenarios, without compromising their overall performance, represents a significant advancement toward more ethically responsible and privacy-conscious ai technologies. this capability is crucial for applications that require careful handling of sensitive or confidential information, such as those in the healthcare, legal, and educational sectors.