Which Retain Set Matters for LLM Unlearning? A Case Study on Entity Unlearning (2025)

Abstract: Large language models (LLMs) risk retaining unauthorized or sensitive information from their training data, which raises privacy concerns. LLM unlearning seeks to mitigate these risks by selectively removing specified data while maintaining overall model performance. However, most existing work focuses on methods to achieve effective forgetting and does not provide a detailed analysis of the retain set, the portion of training data that is not targeted for removal. In this paper, we investigate the effects of unlearning on various subsets of the retain set through a case study on entity unlearning. We introduce the \textit{Syntactically Similar Neighbor Set} , a group of queries that share similar syntactic structures with the data targeted for removal, and show that this subset suffers the greatest performance drop during unlearning. Moreover, when used for regularization, this set not only preserves performance on syntactically similar queries but also delivers comparable or improved results across other data subsets. Our results highlight that syntactic similarity is a critical factor, potentially more so than domain or entity relationships, in achieving effective and practical LLM unlearning.

## introduction large language models (llms), trained on vast text corpora, exhibit remarkable capabilities @xcite . however, their deployment raises concerns about retaining unauthorized content, posing risks in copyright @xcite , privacy @xcite . these issues are critical under regulations like gdpr @xcite , which mandates post-training data removal and the right to erasure. to address these challenges, language model unlearning @xcite has emerged as a promis-after unlearning figure 1 : impact of unlearning across different neighbor sets. syntactically similar neighbors are most affected (in red). in contrast, entity and domain neighbors retain correct knowledge (in blue). in this paper, we take a closer look at which areas of the retain set are significantly affected by unlearning through a case study on entity unlearning. entity unlearning @xcite ., neighbor set type entity example example question example answer domain neighbor set mark twain (a writer, of the same profession) on which river did mark twain work as a steamboat pilot? mississippi river entity neighbor set emma watson (the lead actress in rowling's works) what was emma watson's first lead role? harry potter syntactically similar neighbor set when was x born? (similar syntactic structure as in the forget set) when was nelson mandela born? july 18, 1918 (b) examples of types of neighbor sets. 2024) aims to remove knowledge about particular entities, typically expressed through qa pairs. since it is not practical to test the whole retain set, previous work has used smaller groups called neighbor sets @xcite . these neighbor sets have similar properties to the data being removed, but they do not include the target data. they are particularly important as they are expected to experience significant performance degradation during the unlearning process. building on previous work, we conduct an in-depth analysis of these neighbor sets and address two key research questions: ## dataset construction we consider two scenarios for entity unlearning: the fictitious author scenario (tofu) and a realworld scenario involving actual individuals. this section details the construction of the forget set and the various neighbor sets for each scenario. ## how does performance degradation vary across different neighbor sets? this section investigates how performance degradation after unlearning varies across different neighbor sets. first, we examine which neighbor sets experience the most significant performance degradation. (section 5.1) if similar syntactic structures sets are the most vulnerable to forgetting, we further examine whether domain differences within these structures lead to varying effects. (section 5.2) we then examine the robustness of these forgetting patterns when questions are paraphrased. (section 5.3) finally, we analyze gradient updates during unlearning to understand the underlying mechanisms driving the observed patterns. (section 5.4) ## what is the optimal neighbor set for effective regularization? to preserve model utility during unlearning, regularization losses on a subset of the retain set are commonly employed during the unlearning process @xcite . based on the findings of the previous section, we aim to identify the optimal configuration of the retain set used for regularization, to optimize model utility while ensuring successful forgetting, specifically from a data perspective. domain neighbors entity neighbors syntactically similar train retain set domain neighbors entity neighbors syntactically similar test retain set -55.1 -50.2 -53.7 -57.3 -50.1 -53.7 -61.9 -61.0 -46.8 gd regularization domain neighbors entity neighbors syntactically similar train retain set -62.1 -59.6 -53.8 -59.4 -62.0 -53.8 -63.9 -67.4 -58.3 kl regularization 70 65 60 55 50 45 relative utility drop (%) our analysis considers two primary regularization approaches: gradient descent (gd) and kullback-leibler divergence (kl). a comprehensive explanation of these methods is provided in appendix b. ## related work llm unlearning @xcite @xcite has gained significant attention as a method to enhance privacy. various approaches @xcite have been proposed to ensure that models effectively erase specific information while maintaining overall performance. a key challenge in unlearning is assessing whether knowledge unrelated to the forget set is inadvertently affected. to evaluate this, researchers commonly examine general knowledge @xcite as well as a designated subset of the retain set that shares a similar distribution with the forget set but excludes the targeted information. these subsets, often referred to as neighbor sets @xcite , help determine the extent of unintended degradation in model performance. in hazardous knowledge unlearning, prior work has leveraged domain-relevant general knowledge as a benchmark. for instance, @xcite employs general biology knowledge to assess the impact of bioweapon-related unlearning and general computer security knowledge to evaluate the removal of information related to attacking critical infrastructure. for entity unlearning @xcite , previous studies have used entities from similar professions or those closely linked to the target entity as neighbor sets. while these approaches provide an initial framework, they lack a systematic investigation of which aspects of the retain set suffer the most from unlearning. our study addresses this gap by systematically investigating the impact of unlearning on different types of neighbor sets more clearly and identifying which knowledge components experience the highest degree of forgetting. ## conclusion in this paper, we examine unlearning's impact on retain sets and highlight the syntactically similar neighbor set as key to forgetting patterns. our results show syntactic similarity, not domain or entity ties, drives retained knowledge degradation. experiments confirm that syntactically similar neighbors face the highest utility drop, challenging prior assumptions. we also find that using such data for regularization improves performance retention. these findings refine unlearning strategies and emphasize the role of syntactic structure in minimizing unintended knowledge loss.