REVS : Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Space (2025)

Abstract: Language models (LMs) risk inadvertently memorizing and divulging sensitive or personally identifiable information (PII) seen in training data, causing privacy concerns. Current approaches to address this issue involve costly dataset scrubbing, or model filtering through unlearning and model editing, which can be bypassed through extraction attacks. We propose REVS, a novel non-gradient-based method for unlearning sensitive information from LMs. REVS identifies and modifies a small subset of neurons relevant for constituent tokens which form sensitive information. To adequately evaluate our method on truly sensitive information, we curate three datasets: an email and URL datasets naturally memorized by the models, and a synthetic social security number dataset that we tune the models to memorize. Compared to other methods, REVS demonstrates superior performance in unlearning sensitive information and robustness to extraction attacks, while retaining underlying model integrity.

## introduction language models (lms) exhibit a concerning tendency to memorize information from their training data @xcite @xcite . while factual recall is a desirable property when dealing with general knowledge, memorization and regurgitation of sensitive private information, such as personal and contact details, is a security concern regulated by laws like the general data protection regulation (gdpr; european @xcite . to ensure such information does not get inadvertently leaked, it is paramount to develop techniques that detect and erase sensitive information from lms or their training data. current approaches for handling sensitive information in lms fall into two groups. exact unlearning approaches tackle the problem from the data perspective, either erasing information from datasets @xcite or applying differential privacy @xcite @xcite to training data. such approaches are costly and time-consuming, as each iteration of scrubbing requires retraining the entire model. moreover, they reduce but do not entirely prevent the risk of information leakage @xcite . machine unlearning approaches @xcite discourage models from generating sensitive information through in-context unlearning (icl; @xcite @xcite or gradient ascent @xcite @xcite . these approaches do not ensure erasure of sensitive information from model parameters, rendering them vulnerable to extraction attacks @xcite . model editing, a variant of localization-based unlearning, localizes and edits a subset of parameters to erase sensitive information @xcite @xcite . in contrast to icl and optimization-based methods, which prevent models from generating sensitive content but do not fundamentally erase the underlying knowledge, editing methods hold greater potential to resist extraction attacks @xcite . we introduce rank editing in the vocabulary space (revs), a novel non-gradient-based modelediting approach that enables robust unlearning of sensitive information from lms while maintaining model performance and offering strong robustness against extraction attacks. adopting the view that transformer mlp layers construct predictions by promoting specific tokens in the output vocabulary space @xcite , revs locates the layers and particular subsets of neurons that promote the tokens corresponding to the targeted sensitive information. by modifying these neurons with respect to the relevant tokens, revs surgically removes the model's encoded tendency to generate that sensitive data while preserving its broader knowledge and remaining robust to extraction attacks. see figure 1 for an illustration. we aim to prevent models from generating specific token sequences rather than erasing broader conceptual knowledge evaluated in datasets such as wmdp @xcite and tofu @xcite . to this end, we curate three benchmark datasets containing sensitive information: emails and urls, containing real addresses inadvertently memorized by the model during pretraining, and ssn, a synthetic dataset where we instilled social security numbers into lms through finetuning. unlike prior work @xcite , which evaluated editing methods on non-sensitive data, our benchmarks contain actual private information, enabling rigorous assessment of unlearning efficacy and robustness against extraction attacks. we conduct experiments on llama-3-8b @xcite and gpt-j-6b @xcite , which we find to naturally memorize sensitive information from their training data (the pile; @xcite . our experiments demonstrate that revs outperforms six strong baselines in unlearning sensitive information while maintaining model performance and robustness against extraction attacks. our main contributions are: ## problem setup and preliminaries consider the case where a lm is prompted with text from its training dataset that contains sensitive information, e.g., "contact david lewis at ". if the model has memorized the completion of this prompt, it will generate the relevant email address, such as "lewis.david@email.com", illustrated in figure 1 . this scenario highlights a prevalent issue with lms: their ability to memorize can lead to unintended divulgence of sensitive data, in particular pii. to address this issue, dataset scrubbing approaches tackle it from a data perspective, preventing sensitive information from being stored in the model during training. machine unlearning approaches take on a post-hoc approach, modifying either all, or just a subset of parameters that contain sensitive information. the goal of these approaches is to efficiently and effectively prevent the model from generating each sequence containing sensitive information in a way robust to extraction attacks, while preserving general capabilities of the model @xcite . background on transformer lms. we briefly describe the main components in transformer lms; more details are found elsewhere @xcite . omitting some technical details, a transformer lm is made of a sequence of blocks. each block contains self-attention and multi-layer perceptrons (mlps) that write and read to the residual stream @xcite . the mlp is defined as mlp(⃗ x) = ff 2 σ(ff 1 ⃗ x), and is added to the residual stream. the final layer hidden state is projected to the vocabulary space via the unembedding matrix u , followed by a softmax to obtain next token probabilities. due to the residual stream tying hidden spaces across layers, applying the unembedding matrix to residual states at intermediate layers results in meaningful token logits (nostalgebraist, 2020), while applying it to mlp weights reveals stored knowledge @xcite . in fact, the second mlp layer, ff 2 , acts as a memory store @xcite , and model editing methods often target this layer to update a model's knowledge @xcite . we adopt this view, and focus on information stored in ff 2 columns, henceforth neurons, for unlearning. ## methodology we prompt a transformer lm with a prompt p and observe its behavior when generating the next token. let ⃗ @xmath0, while ⃗ @xmath1, for a given input ⃗ x. leveraging the insight that applying the unembedding matrix u to hidden states of intermediate layers produces meaningful logits, we define r t as the rank of a target token t among other logits in the vocabulary projection of hidden state h: r t := rank(t, u ⃗ h) the higher the rank, the more likely the token is to be generated, where rank r @xmath2. to unlearn sensitive information, revs aims to increase r t value to a desired r d , thus decreasing its rank by identifying and modifying f f 2 columns that contribute most to generating the target token t. this unlearning process operates in two main phases: localization (section 3.1), where we identify the layers and neurons (f f 2 columns) that contain information relevant for generating t, and editing (section 3.2), where we overwrite information relevant for t while not affecting knowledge used for generating other tokens. we provide a high-level overview of revs in algorithm 1. algorithm 1 revs: rank editing in the vocabulary space 1: procedure unlearntoken(m, p, t, r d , nmax) 2: input: model m , prompt p , target token t, 3: target desired rank r d , max neurons nmax 4: ## results as effectiveness. revs achieves the best efficacy and generality on ssn and urls and strong efficacy on emails. as shown in figure 4e , perfect efficacy is possible but compromises specificity. notably, methods surpassing revs in efficacy on emails achieve a lower unlearning score due to poor specificity. ## conclusion we introduce revs, a novel unlearning method which demonstrates superior performance to stateof-the-art baselines in effectively unlearning sensitive information from lms while preserving general capabilities and remaining robust to extraction attacks. we curate three datasets containing sensitive information, two naturally memorized by the models. through detailed analyses we show revs' robustness to varying candidate token set sizes, number of unlearned targets and the trade-off between efficacy and specificity.