Paper: Unlearning via Sparse Representations (ICLR-2024)

Reviewer comment (Experiments):
I think that the experimental evaluation is a bit lacking and could be expanded to larger datasets such as ImageNet, as the proposed method is almost compute-free.

Author response:
We evaluate our approach on a larger scale by training the baseline and DKVB models on the ImageNet-1k and attempting to unlearn class number 1.

* Experimental Setup - We train the DKVB and baseline model (i.e. Frozen backbone + trainable linear layer) on ImageNet-1k up to 68% classification accuracy. ImageNet-1k consists of 1.28 million training examples and 50000 test examples divided into 100 classes. Next, we choose class 1 as the class to be unlearnt, and perform the same set of experiments as performed for the three datasets in Section 5.2 and Section 5.3 in the paper.

* Results - We present a comparative analysis between the baseline and the proposed approach on ImageNet-1k in the table given below. Further, we have also included scatter plots for the retain set accuracy vs forget set accuracy in Appendix I of the paper. We observe that our approach performs equally well on ImageNet as compared to the three datasets already included in the paper, with similar hyperparameters This demonstrates that the proposed approach is equally effective in large-scale settings as well. The given table presents a comparison between the proposed methods and the baselines on ImageNet-1k. We compare the change in performance on the retain and forget test data relative to the originally trained models.

|  | $D_{retain}^{test}$ | $D^{forget}_{test}$ |
|----------|----------|----------|
| DKVB via Activations (sec 5.2) | 0.15 % | $\textbf{-100}$ % |
| DKVB via Examples (sec 5.2) | -0.03 % | $\textbf{-100}$ % |
| Linear Layer + SCRUB | $\textbf{7.31\}$ % | $\textbf{-100}$ % |

Reviewer comment (Reproducibility):
Can you expand more on Sec.D of the Appendix, as I think it is relevant from a practical point of view?

Author response:
First, we would like to clarify that our approach is not a-priori suited for selective unlearning, i.e. the setting where we want the model to forget specific examples or a small subset of examples instead of removing the information about an entire class. The KV bottleneck induces clusters of representation, where the members of a particular cluster correspond to the representations belonging to the same class. When we try to unlearn the representations corresponding to one particular example belonging to a particular class, the KV bottleneck routes the selection to other (key-)representations within the same cluster. Since these representations also contain information about the same class as the examples we intend to unlearn, the model would still predict the class to be unlearnt.

Due to the same reason our approach is also not designed for working against traditional Membership Inference attacks. According to the basic attacks setup as explained in their paper  the objective is to obtain a model that has unlearnt a small subset of specific examples (i.e. selective unlearning) such that the loss of the model on the unlearnt subset of examples should be indistinguishable from examples that the model never saw during training.

Nevertheless, we attempt to modify the above setup to that of "Class Membership Inference Attacks (CMIA)". In CMIA, the aim is to defend against an attacker whose aim is to determine whether a model that has undergone unlearning ever saw a particular class as a part of its training data. Thus, we want the model to unlearn a particular class such that the losses/performance of the model on the unlearnt class is indistinguishable from a held-out class that the model never saw during its training. We describe the experimental setup and results below:

* Experimental Setup: We perform the experiment for CIFAR10. We divide the dataset into training data ($\mathcal{D_{Train}}$), validation data ($\mathcal{D_{Val}}$) and test data ($\mathcal{D_{Test}}$). Training Data consists of 4000 examples per class; validation and test data consist of 1000 examples per class. We first trained a model on the first 9 classes of CIFAR10. Here, class number 10 is the held-out class. Next, we unlearn class 1 from the model using the Unlearning via Activations approach introduced in the paper. We unlearn the model until the loss on the validation sets of the forget class and the held-out class are similar. In our experiments, we find that we reach this point at approximately $N_a = 240000$. The loss $l(x, y)$ in our case would be the cross-entropy loss.

* Next, we label the losses corresponding to the validation and test set of the forget class as 1 and those corresponding to the validation and test set of the held out class as 0. We train a binary classifier as done in the "Basic Attack" setting in [6] on the validation losses of both sets and test it on the test losses. We follow a similar setting for the baseline model, where we obtain the model suitable for MIA defense by following a procedure similar to SCRUB+R as introduced in [6]. For a successful defense, we would want the accuracy of the classifier to be close to 50%, indicating that it is unable to distinguish between the unlearnt class and the held-out class. Same as Kurmanj [1], we use sklearn.logistic_regression as our attacker (the binary classifier). We call the approach described above Partial UvA (Partial Unlearning via Activations).  We run experiments for 3 random seeds, and the mean of the attacker performance is reported. 
* Observations and Results: We report the results of the experiment described above in the table given below. We observe that although the baseline performs better, the proposed approach performs competitively, even though we have not intended to develop the method for this scenario. 

| Approach | Accracy of the attacker |
|-----------|-----------|
| Partial UvA | 53.5 |
| Linear Layer + SCRUB + R | 51.50 |

Reviewer comment (Evaluation):
Even though the whole motivation of this approach is to implement a new approach with better computational efficiency no experiments on computational cost are done. This approach is not measured with current approaches of class unlearning in terms of computational efficiency.

Author response:
**Regarding Experiments around the computational cost of the approaches discussed**

We measure the runtimes of the experiments discussed in Section 5.3 and Table 2 as a proxy to the computational costs required by the approaches discussed. The results are presented in the table below. We note that both the proposed approaches are several times faster than the baseline

| | CIFAR-10 | CIFAR-100 | LACUNA-100 |
|----------|----------|----------|----------|
| Runtimes (in seconds) | | | |
| DKVB via Activations (sec 5.2) | $\textbf{5.02}$ | $\textbf{1.57}$ | $\textbf{2.74}$ |
| DKVB via Examples (sec 5.2) | 13.98 | 6.65 | 13.03 |
| Linear Layer + SCRUB | 288.80 | 921.56 | 553.31 |

We find that Unlearning via Activations is consistently the most compute efficient approach while both Unlearning via Activations and Unlearning via Examples being at least ~20x faster than the baseline

Reviewer comment (Experiments):
Is this approach model agnostic? I am not sure if this can applied to other models if the bottleneck is not present.

Author response:
**Regarding the approach being model agnostic**

The reviewer correctly notes that the proposed approaches are conditioned on the presence of a Discrete Key Value Bottleneck. However, it is agnostic to the choice of the frozen backbone and the decoder architectures. This can be supported by evidence from our new experiments with a ResNet50 backbone (see Appendix Section H) which show that the proposed approach also works with backbones other than a CLIP pretrained ViT-B/32.

Reviewer comment (Experiments):
A more diverse range of benchmarks is necessary to test the unlearning capabilities of DKVB. The evaluation benchmarks are quite similar to each other (CIFAR10/100, LACUNA100), so it is unclear how successful the unlearning method is. It might be interesting to evaluate the method on more diverse benchmarks such as those with outlier classes that are quite similar to each other, or tasks involving distinguishing between superclasses and unlearning a specific class within a superclass.

Author response:
**Evaluating the approach on more benchmarks**

We tested the proposed approach on ImageNet-1k dataset and the results are as follows:

* Experimental Setup - We train the DKVB and baseline model (i.e. Frozen backbone + trainable linear layer) on ImageNet-1k up to 68% classification accuracy. ImageNet-1k consists of 1.28 million training examples and 50000 test examples divided into 1000 classes. Next, we choose class 1 as the class to be unlearnt, and perform the same set of experiments as performed for the three datasets in Section 5.2 and Section 5.3 in the paper.
* Results - We present a comparative analysis between the baseline and the proposed approach on ImageNet-1k in the table given below. Further, we have also included scatter plots for the retain set accuracy vs forget set accuracy in Appendix I of the paper. We observe that our approach performs equally well on ImageNet as compared to the three datasets already included in the paper, with similar hyperparameters This demonstrates that the proposed approach is equally effective in large-scale settings as well. The table given below provides a comparison between the proposed methods and the baselines on ImageNet-1k. We compare the change in performance on the retain and forget test data relative to the originally trained models.

|  | $D_{retain}^{test}$ | $D^{forget}_{test}$ |
|----------|----------|----------|
| DKVB via Activations (sec 5.2) | 0.15 % | $\textbf{-100}$ % |
| DKVB via Examples (sec 5.2) | -0.03 % | $\textbf{-100}$ % |
| Linear Layer + SCRUB | $\textbf{7.31\}$ % | $\textbf{-100}$ % |

Reviewer comment (Evaluation):
The authors choose to unlearn a class until 0% accuracy. Is this correct, or should unlearning stop at random chance accuracy (10%)?

Author response:
**Complete Unlearning vs Unlearning until random accuracy**

As discussed in the literature, unlearning completely vs random accuracy depends on the task at hand. For instance, in cases where we care about removing harmful biases from a model (e.g. remove a users data completely), complete unlearning is more desirable. In other scenarios such as Membership Inference Attacks (MIAs), the focus is on user privacy, it is desired that the model unlearns only to an extent such that the performance of the model on the unlearnt class is equivalent to that on a class that it has never seen. For the proposed approaches of Unlearning via Examples and Unlearning via Activations, this “extent of unlearning” can be controlled using the parameters $N_e$ and $N_a$ respectively as shown in Figures 2 and 3. Further, our new experiments of MIAs as discussed in Appendix D. Here, we show that the proposed approach is also useful in cases where complete unlearning is not desired.

Reviewer comment (Experiments):
Section 5.4 is unclear regarding several aspects: What do you mean by “implications of additional compute”? Are you retraining the DKVB models for longer? What do you mean by “retraining”? In Figure 4, you state “retraining…using the proposed methods” (it says “proposed models” but possibly meant “proposed methods”). What does it mean to retrain using SCRUB? Or is the SCRUB’ed model then retrained to learn the forgotten class through normal supervised training? It also seems strange that the performance would ever drop during retraining with SCRUB.

Author response:
**Clarifications regarding Section 5.4**

* Implications of using additional compute - The proposed approaches do not require additional compute for unlearning a class apart from the compute required for inference. The baseline, on the other hand, requires additional compute in the form of training the model and optimizing it w.r.t. a given objective. Thus, we attempt to investigate what would happen if the proposed approach also somehow made use of additional compute. In order to make use of some additional compute, once the DKVB models have undergone unlearning of the forget class, we train them on just the retain set in order to cover up any damage in performance on the retain set that might have occurred due to the unlearning.
* Use of the term “retraining” - We use the term “retraining” to refer to the above-explained process of training the unlearnt DKVB models on only the retain set in order to improve the performance on the retain set.

Reviewer comment (Experiments):
**Unlearning Scenarios**: The focus on 'complete unlearning' may limit the paper's scope, given the broader spectrum of unlearning scenarios presented in [2-7], where the goal is for the unlearned model to closely resemble a model that has been retrained without the forgotten data. Although the authors claim in Appendix D that their method is applicable to traditional machine unlearning problems, detailed results and discussions within these contexts would strengthen the paper's contributions.

Author response:
First, we would like to clarify that our approach is not a-priori suited for selective unlearning, i.e. the setting where we want the model to forget specific examples or a small subset of examples instead of removing the information about an entire class. The KV bottleneck induces clusters of representation, where the members of a particular cluster correspond to the representations belonging to the same class. When we try to unlearn the representations corresponding to one particular example belonging to a particular class, the KV bottleneck routes the selection to other (key-)representations within the same cluster. Since these representations also contain information about the same class as the examples we intend to unlearn, the model would still predict the class to be unlearnt.

Due to the same reason our approach is also not designed for working against traditional Membership Inference attacks. According to the basic attacks setup as explained in Kurmanji et al., 2023, the objective is to obtain a model that has unlearnt a small subset of specific examples (i.e. selective unlearning) such that the loss of the model on the unlearnt subset of examples should be indistinguishable from examples that the model never saw during training.

Nevertheless, we attempt to modify the above setup to that of "Class Membership Inference Attacks (CMIA)". In CMIA, the aim is to defend against an attacker whose aim is to determine whether a model that has undergone unlearning ever saw a particular class as a part of its training data. Thus, we want the model to unlearn a particular class such that the losses/performance of the model on the unlearnt class is indistinguishable from a held-out class that the model never saw during its training. We describe the experimental setup and results below:
 
* Experimental Setup: We perform the experiment for CIFAR10. We divide the dataset into training data ($\mathcal{D_{Train}}$), validation data ($\mathcal{D_{Val}}$) and test data ($\mathcal{D_{Test}}$). Training Data consists of 4000 examples per class; validation and test data consist of 1000 examples per class. We first trained a model on the first 9 classes of CIFAR10. Here, class number 10 is the held-out class. Next, we unlearn class 1 from the model using the Unlearning via Activations approach introduced in the paper. We unlearn the model until the loss on the validation sets of the forget class and the held-out class are similar. In our experiments, we find that we reach this point at approximately $N_a = 240000$. The loss $l(x, y)$ in our case would be the cross-entropy loss.  

* Next, we label the losses corresponding to the validation and test set of the forget class as 1 and those corresponding to the validation and test set of the held out class as 0. We train a binary classifier as done in the "Basic Attack" setting in [6] on the validation losses of both sets and test it on the test losses. We follow a similar setting for the baseline model, where we obtain the model suitable for MIA defense by following a procedure similar to SCRUB+R as introduced in [6]. For a successful defense, we would want the accuracy of the classifier to be close to 50%, indicating that it is unable to distinguish between the unlearnt class and the held-out class. Same as Kurmanj [1], we use sklearn.logistic_regression as our attacker (the binary classifier). We call the approach described above Partial UvA (Partial Unlearning via Activations).  We run experiments for 3 random seeds, and the mean of the attacker performance is reported. 
* Observations and Results: We report the results of the experiment described above in the table given below. We observe that although the baseline performs better, the proposed approach performs competitively, even though we have not intended to develop the method for this scenario. 

| Approach | Accracy of the attacker |
|-----------|-----------|
| Partial UvA | 53.5 |
| Linear Layer + SCRUB + R | 51.50 |

Reviewer comment (Experiments):
**Model Diversity**: The exclusive use of the ViT model in the experiments may raise questions about the method's generalizability. As DKVB [1] also considers ResNet architectures, a discussion on the application or limitation of the proposed method to other architectures would provide deeper insights into its adaptability and utility across various models.

Author response:
As requested by the reviewer, we performed the same set of experiments presented in the paper in Sections 5.2 and 5.3 for CIFAR-10, with a backbone from the ResNet family. 
* We use an ImageNet supervised pretained ResNet50  model as the backbone for our approach as well as the baseline model. 
* The results are as presented in the table below. We have also attached the scatter plots for the retain set accuracy vs forget set accuracy in Appendix H of the paper. We observe that the results with ResNet backbone are fairly similar to the ones with ViT backbone, discounting the difference in absolute performances. Importantly, even in this scaled-up setting we are able to perform unlearning with minimal compute without damaging the performance on the retain set.
* We compare the change in performance on the retain and forget test data
relative to the originally trained models. For the baseline, we report two cases: Case B where the
model unlearns the forget set completely but the retain set performance is not preserved very well,
and Case A where the retain set performance is not preserved very well, but the model does not
unlearn the forget set completely


|  | $D^{retain}_{test}$ | $D^{forget}_{test}$ |
|----------|----------|----------|
| DKVB via Activations (sec 5.2) | 0.04% | $\textbf{-100}$% |
| DKVB via Examples (sec 5.2) | -0.07% | $\textbf{-100}$% |
| Linear Layer + SCRUB (A) | $\textbf{-0.44\}$% | -96.04% |
| Linear Layer + SCRUB (B) | -11.75% | $\textbf{-100}$% |

Reviewer comment (Novelty):
**Limited Literature Review**: Given the recent surge in research surrounding machine unlearning, the paper's literature review could be perceived as insufficiently comprehensive. I included some papers published recently in the following references.

Author response:
We thank the reviewer for pointing us to these important papers. We have discussed these ([1], [3], [4], [5]) works as well as drawn a comparison of the proposed approach to [1] and [5] in the "Additional Related Works" section in the Appendix (section G).

 > [1] Jia, Jinghan, et al. "Model sparsification can simplify machine unlearning." arXiv preprint arXiv:2304.04934 (2023). \
> [2] Chundawat, Vikram S., et al. "Zero-shot machine unlearning." IEEE Transactions on Information Forensics and Security (2023). \
> [3] Warnecke, Alexander, et al. "Machine unlearning of features and labels." arXiv preprint arXiv:2108.11577 (2021). \
> [4] Chen, Min, et al. "Boundary Unlearning: Rapid Forgetting of Deep Networks via Shifting the Decision Boundary." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\
> [5] Mehta, Ronak, et al. "Deep unlearning via randomized conditionally independent hessians." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022 \
> [6] Kurmanji, Meghdad, Peter Triantafillou, and Eleni Triantafillor> "Towards Unbounded Machine Unlearning." arXiv preprint arXiv:2302.09880 (2023).