Towards Robust Evaluation of Unlearning in LLM s via Data Transformations (2024)

Abstract: Large Language Models (LLMs) have shown to be a great success in a wide range of applications ranging from regular NLP-based use cases to AI agents. LLMs have been trained on a vast corpus of texts from various sources; despite the best efforts during the data pre-processing stage while training the LLMs, they may pick some undesirable information such as personally identifiable information (PII). Consequently, in recent times research in the area of Machine Unlearning (MUL) has become active, the main idea is to force LLMs to forget (unlearn) certain information (e.g., PII) without suffering from performance loss on regular tasks. In this work, we examine the robustness of the existing MUL techniques for their ability to enable leakage-proof forgetting in LLMs. In particular, we examine the effect of data transformation on forgetting, i.e., is an unlearned LLM able to recall forgotten information if there is a change in the format of the input? Our findings on the TOFU dataset highlight the necessity of using diverse data formats to quantify unlearning in LLMs more reliably.

## introduction large language models (llms) have shown remarkable performance on a variety of tasks @xcite @xcite and a broad range of applications going beyond regular nlp tasks @xcite . however, llms have been trained using vast sources of texts, which may include personal information of an individual as well. it has encouraged researchers to develop methods for forcing llms to forget undesirable information without degrading the performance on regular tasks, giving rise to the area of machine unlearning (mul) @xcite @xcite @xcite . moreover, recently, user privacy in terms of unintended use of personal data has gained some interest, such as the general data protection regulation (gdpr) and the california consumer privacy act, which empower users with the "right to be forgotten" (rtbf), i.e., an organization must remove/delete all the information if a user wants to revoke access to their information, with a minimal delay. researchers in the mul community have proposed various methods @xcite @xcite ) and text-based benchmarks @xcite . for example, to evaluate forgetting in llms @xcite have created the tofu benchmark built using a dataset having facts about various fictitious entities. the tofu dataset uses a particular format (e.g., q&a (questions and answers)); however, the same information can be expressed in multiple ways in natural language. in this work, we investigate if unlearning algorithms are sensitive to data formats, i.e., we experiment with a setting where the learning/unlearning happens in one default format and study how the unlearning performance varies when the same information is presented in a different format. in a nutshell, we make the following contributions: llms, despite their significant advancements @xcite @xcite , are susceptible to inadvertently disclosing sensitive information or personal details as billions of trainable parameters are utilized during training. recent studies have adopted different approaches using machine unlearning @xcite to alleviate this issue and achieve trustworthiness @xcite and fairness @xcite by removing sensitive information @xcite . the primary objective of machine unlearning is to modify the weights of a pre-trained model, allowing it to unlearn the knowledge acquired from a specific subset of data intended to be erased while maintaining performance on the retained set. recently, the notion of exact unlearning has garnered significant attention. this method involves re-training the model from scratch after removing specific training data points, which are considered the gold standard for unlearning. nevertheless, this method entails substantial computation cost and demands access to the whole training set @xcite . to overcome these challenges, recent research efforts have shifted focus towards developing scalable and effective approximate unlearning @xcite @xcite @xcite methods. one of the concurrent works by @xcite , emphasizes on usage of data transformation techniques to evaluate unlearning effectiveness in llms. in this work, we provide a medium to achieve this by creating an extended version of the tofu benchmark. ## problem definition and methodology problem setup: a broader applicability of llms considers using an open-weight model m θ with parameters θ as a base to enhance them with new proprietary information d p . a general machine learning/unlearning pipeline follows training/finetuning the base model over new information d p by constructing a training set d @xmath0, y i )} n @xmath1, where f i denotes the transformation of the information into a format, such as q&a. the model m θ is trained/finetuned over the created d train to obtain a finetuned-llm m θ where θ represents the updated model parameters. since the new proprietary information is user-specific, user(s) may ask to remove/erase their data, leading to a forget set split from the d @xmath2. the goal of an unlearning algorithm is to update the fine-tuned llm m θ to obtain an unlearned version mθ (here θ represents model parameters after unlearning) that shows behavior similar to m θ over the held-out forget-set d f orget . benchmarking of the unlearning algorithms usually relies on a single format (f i ). however, the same information d p can be represented in m different format f 1 , f 2 , . . . f @xmath3. when unlearning, it becomes imperative to ensure the information in the forget set is removed from model parameters θ and does not depend on the transformation style f i , i.e., the model performance on d f orget should be similar for all the formats in which the dataset can be represented. fig. 1 explains the entire process with an example. ## discussion in this work, we extend the existing tofu benchmark for a more robust unlearning evaluation by creating additional resources and framing a better evaluation scheme. we keep the primary focus of our study to highlight the sensitivity towards dataset transformation (aka same information being present in different formats) in the unlearning methods, pointing towards a need for better and more reliable unlearning evaluation. we create 5 new variants of the tofu dataset using formats widely used in nlp, including q&a, mcqa, cloze, analogy, comprehension, and odd-one-out. in general, these formats are inspired by recent llm benchmarking papers, q&a is the default (already existing in the tofu dataset) and is used by @xcite for evaluating llms. mcqa @xcite ) has become a new information evaluation format used by benchmarks/datasets like bigbench (bench authors, 2023), mmlu @xcite , mmlu-pro @xcite , arc @xcite , etc. cloze @xcite test is another format used by brown et al. ( 2020 ) and the following approaches: llama @xcite and @xcite . analogy was majorly inspired by in-context learning examples @xcite , where some examples are given in the context/prompt to evaluate if the model can retrieve/understand the relationship from the examples and some of the recent works @xcite . comprehension (inspired by squad @xcite ) is again useful in assessing the quality of the model in general q&a if the relevant information is provided in the context (should have no effect after updates by the unlearning algorithm). finally, odd-one-out takes inspiration from the mia attack @xcite in the unlearning literature and frames the query using natural language to assess if the model can differentiate between the forget and the retain set samples. we believe these created formats, though limited in number, provide an initial step towards robust evaluation of unlearning methods. in the future, it would be interesting to consider more number of formats for a better evaluation. the current state of the unlearning benchmarks is limited, and the way of maintaining knowledge depends on only one dataset format. for future approaches, we recommend a few settings that could be tried aiming at different unlearning objectives, utilizing various dataset formats. in this work, we only considered previous approaches where learning and unlearning happen only in one format (q&a in our case). however, the knowledge represented by these formats is the same, and one could learn in one format and try unlearning in another format. in another setting, one could assume the model is being trained on multiple formats (for example, q&a and mcqa), where one of the formats remains unavailable for unlearning (mcqa). in this case, a better unlearning algorithm would be able to sufficiently unlearn the requested knowledge from the single available formats. moreover, a wide combination of learning and unlearning formats can be chosen to quantify the robustness of future unlearning approaches. ## conclusion in this work, we study the role of dataset transformation in unlearning. we enhance an existing dataset with multiple new formats, validating the effectiveness of unlearning algorithms. we further experiment with open-weight models over the created evaluation settings , highlighting the impact of data transformation. with quantitative and qualitative analysis, our empirical findings point towards reaching a better validation criterion for unlearning algorithms. we find that evaluation over a single format may lead to unreliable improvements, and unlearning benchmarks should consider evaluation over multiple formats. we hope the curated dataset transformation in 5 different formats will be a useful resource for future benchmarking of unlearning algorithms.