Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective (2024)

Abstract: This paper investigates Who’s Harry Potter (WHP), a pioneering yet insufficiently understood method for LLM unlearning. We explore it in two steps. First, we introduce a new task of LLM targeted unlearning, where given an unlearning target (e.g., a person) and some unlearning documents, we aim to unlearn only the information about the target, rather than everything in the unlearning documents. We further argue that a successful unlearning should satisfy criteria such as not outputting gibberish, not fabricating facts about the unlearning target, and not releasing factual information under jailbreak attacks. Second, we construct a causal intervention framework for targeted unlearning, where the knowledge of the unlearning target is modeled as a confounder between LLM input and output, and the unlearning process as a deconfounding process. This framework justifies and extends WHP, deriving a simple unlearning algorithm that includes WHP as a special case. Experiments on existing and new datasets show that our approach, without explicitly optimizing for the aforementioned criteria, achieves competitive performance in all of them.

## introduction machine unlearning in large language models (llms) has attracted wide research attention amidst the rising privacy and security concerns of llms, such as potential leakage of copyright content, personal information, and misuse in developing bioweapons and cyberattacks @xcite @xcite @xcite @xcite . one pioneering work in llm unlearning is who's harry potter (whp) @xcite , which introduces a novel unlearning approach based on name changes. specifically, as shown in figure 1 , to "forget the link" between an entity (e.g., harry potter) and its associated knowledge (e.g., hogwarts), they obtain a teacher prediction by substituting the name of harry potter in the input with a generic name like jon and then fine-tune the llm to approach the teacher prediction on the original input. in addition to its simplicity and efficacy, whp enjoys a unique advantage compared with other existing unlearning algorithms -the ability to perform targeted unlearning. rather than forgetting all information mentioned in the forget documents, whp can unlearn only a subset of concepts by only replacing their names, and retaining the other names. as shown in figure 2 , the targeted unlearning can forget the information about the unlearning target, wilhelm wattenbach, while retaining other information, such as the fact that rantzau is in holstein, even though the latter information also appears in the document. compared with the original unlearning setting, targeted unlearning is more flexible and practical in many real-world applications, such as the privacy preservation scenario, where only personal information needs to be removed. despite the great potential in whp, this pioneering unlearning algorithm, as well as the targeted unlearning setting, remains under-explored. on the one hand, there have been few attempts to create benchmarks for the targeted unlearning, including creating datasets and defining metrics. therefore, it is unclear what constitutes a satisfactory targeted q: where was wilhelm wattenbach born? a: i don't have his personal information. ## experiments we evaluate our framework on different unlearning targets. first, we describe the construction of the new dataset for targeted unlearning in §4.1. then, we discuss experiments on forgetting persons and authors plus books in §4.2 and §4.3 respectively. ## conclusion in this paper, we examine the pioneering who's harry potter for llm unlearning. we introduce a new task called targeted unlearning and design comprehensive evaluation metrics. we then propose a causal intervention framework for targeted unlearning, which justifies and improves the algorithm in whp. experiments on new and existing datasets show the effectiveness of our framework.