Paper: Deep Unlearning: Fast and Efficient Training-free Approach to Controlled Forgetting (ICLR-2024)

Reviewer comment (Reproducibility):
The authors only describe the representations collection of linear and convolutional layers, but how to deal with the transformer layer, BatchNorm, etc.?

Author response:
We have added a line in section 4.3 under “Discussion” to clarify this.  We apply our algorithm to all the linear layers in the transformer architecture. For an attention block, the linear layer weights updated are the Key Weights, Query Weights, Value Weights, and Output weights. We do not perform any unlearning of the normalization layers as the fraction of total parameters for these layers is insignificant and empirically is not seen to affect the performance of our algorithm.

Reviewer comment (Experiments):
The authors perform SVD operation based on the representations matrix of $X_r$ and $X_f$. Do the sample size and sampling strategy of $X_r$ and $X_f$ affect the results of the algorithm?

Author response:
Samples from each class are necessary to ensure a good estimation of retain space and forget space. We chose $X\_r$ and $X\_f$ to be around 1000 to ensure we have at least 1 sample from each class (ImageNet). Empirically we observed that having 1000 samples was enough and we did not increase it to ensure the compute cost of Representation collection and SVD over $X\_r$/$X\_f$ is kept low.

Reviewer comment (Theory):
Why did the authors consider designing importance-base space scaling? Won't this cause the deformation of the forget space and retain space?

Author response:
In our algorithm  (line 14, Algorithm 1) we find $P\_{dis}$, which is the class discriminatory activation space. We want to find a class discriminatory space where forget samples have the ‘highest’ activations whereas retain samples have the 'lowest'. By removing this space (from the entire space) we perform our unlearning. To assign this connotation of ''highest/lowest' activation with the original $P\_f$/$P\_r$ space we need to assign importance - thus scaling was used.  We agree this causes deformation of the space - which is a design choice.

Reviewer comment (Experiments):
The choice of best $\alpha$ in Eq.(1) seems to be a trick as different datasets give different $\alpha$ sets. Is there a more reasonable way to choose, like learnable $\alpha$?

Author response:
Our algorithm is gradient free algorithm and obtains the parameter update in a one-shot fashion. As we did not use gradient information it was not straightforward to come up with a learning scheme for $\alpha$. However, in the future training algorithms can be designed that could learn $\alpha$ to be used for unlearning if needed.

Reviewer comment (Experiments):
The authors mention that the experimental results come from 10 different target unlearning classes, and CIFAR-100 is evaluated for every 10th. Does it mean that the [10, 20, …, 100]-th class was used as an unlearning class to conduct the experiment?

Author response:
For CIFAR100 we use [1, 11,21, … 91 ]. We update the paper stating  “CIFAR10 dataset is accessed for unlearning on each class and CIFAR100 is evaluated for every 10th starting from the first class.”

Reviewer comment (Evaluation):
Does the "Original" represent the well-train model? Are the results reported on all retain and forget classes?

Author response:
Yes, original refers to a well trained model which is trained on $D\_{train}$. The accuracy numbers in the previous version were the overall accuracy of the network on the samples of retain and forget class. We have updated these numbers to report retain class accuracy and forget class accuracy separately.

Reviewer comment (Reproducibility):
How is the classification head designed? For example, for CIFAR-100, the classification head of the well-train model outputs a 100-dimensional vector. What about the unlearning model?

Author response:
To evaluate the unlearnt model we assume that the unlearning algorithm gives us the unlearnt model without changing the classifier output neurons similar to the baseline SoTA methods Tarun et al and Kurmanji et al.

Reviewer comment (Evaluation):
In Figure 4, the proposed method does not have an advantage in efficiency compared to the Tarun et al. (2023)? Could the authors provide further analysis?

Author response:
We thank the reviewer for pointing this out. We realized that the runtimes depend on the implementation of the algorithm and the background hardware it is running on. We analytically find the compute costs of different algorithms and plot them in Figure 5 and show the details in Appendix A.7. Our algorithm has at least 6.5x lower compute than other methods.

Reviewer comment (Experiments):
While I think the multi-class forgetting experiment is interesting, what the authors provide is not sufficient. First, simply conducting experiments on CIFAR10 is not convincing. Second, the effects of baselines were not compared. Third, the result analysis is insufficient. For example, why is the original model of resnet18 better in Figure 5a, but not as good as the unlearning model in Figure 5b?

Author response:
We present the results for superclass unlearning on CIFAR100 in Figure 6 and have moved the results of CIFAR10 to Appendix A.8. Further, we have added a comparison with baselines in Table 6 in Appendix A8. We see our model performs better on MIA scores and retain accuracy over SoTA Tarun et al. NegGrad+ is seen to outperform our method on ResNet18. We saw Kurmanji et al to have extremely low retain accuracies and hence have not reported them in the table. There was a mistake in the plot of Figure 5b in the original submission (10b in updated). We did not plot the original model’s forget accuracy in the figure.

Reviewer comment (Evaluation):
This paper only evaluates the effectiveness of the method based on the classification accuracy of forgetting and retaining classes. Have the authors considered verification metrics with more theoretical guarantees? If the forgetting class is known, the output probability of the forgetting class can be forced to 0 without changing the model parameters to ensure the effect of forgetting and retaining classes, but this approach is obviously contrary to the motivation of unlearning.

Author response:
We have added membership inference attack evaluations in our experiments. The results show our method performs well across different datasets and architectures obtaining an average improvement of 7.8% in MIA accuracy over other baselines. 
Our MIA attack has access to the true confidence scores of the model on the correct class. Hence an algorithm that post hoc manipulates the output would fail to obtain good MIA scores.

Reviewer comment (Novelty):
Motivation: the way that the problem of unlearning is motivated in this paper (data deletion; user privacy) seems at odds with the problem of unlearning classes (as it would correspond to unlearning individual data points that don’t necessarily belong to the same ‘class’, depending on the definition of class). What are application scenarios for class unlearning? While the authors have motivated the problem of unlearning well, motivation of missing for class unlearning in particular.

Author response:
**Motivation for Class Unlearning -** Class unlearning is important where entire user data belongs to a single class. Consider a scenario where we train a classification model for face recognition[1]. In this case, an entire class belongs to a single person and we would be required to remove this class on request. Further, in the cases where a collaborative multi-task model is shared by a few companies where a task is classification over a group of classes and one of the companies decides to withdraw its data, the multi-class removal algorithm becomes relevant.

Reviewer comment (Novelty):
The authors claim that Stable Ascent is one of the contributions of the paper but this baseline has already been proposed in previous work that the authors did not cite ([A] – see References below, where it is referred to as NegGrad+ and the authors of that work also find that it is a strong baseline, surpassing previous SOTA in several scenarios).

Author response:
**NegGrad+ baseline-** We thank the reviewer for pointing this out. We have added the citation and updated the contributions. Further, we have renamed Naive Ascent as NegGrad and Stable Ascent as NegGrad+ to be consistent with the literature.

Reviewer comment (Experiments):
Ablations are missing. For example, how large is the contribution of the proposed scaling? It would be good to investigate a version of the proposed method without this. Further, how important is it to use activations from all layers for SVD versus just the top layer(s)?

Author response:
**ablations are missing-** 
    -  **Proposed scaling:** Without the proposed scaling the $\Lambda\_r$ and $\Lambda\_f$ in line 12 of Algorithm 1 become Identity. Now as $U$ matrix obtained through SVD is an orthonormal matrix, $P\_r$ and $P\_f$ in line 13 of Algorithm 1 become Identity when $\Lambda\_r$ and $\Lambda\_f$ are Identity. When $P\_r$ and $P\_f$ is Identity $P\_{dis} = P_f(I-P\_r)$ computed in line 15 becomes a zero matrix. Projection of weights on $I-P\_{dis}$ in line 15 would hence cause no change in the weight as $I-P\_{dis}$ is Identity essentially leading to no unlearning. This necessitates the scaling in line 12 of the algorithm. We have clarified this under the “Importance-base Space Scaling” in Section 4.2.
    -  **Impact of layer:** We have added a section on “Layer-wise analysis” where our results show that later layers play a more significant role in unlearning. Further, in Figure 8 of Appendix A.6 we see that removing our algorithm from the first 4 layers has a very low impact on forget accuracy.

Reviewer comment (Novelty):
It would be good to motivate the scaling a bit more. Is such a scaling used in the continual learning literature / related methods? If not, what is different about this application that necessitates it?

Author response:
**Scaling Motivation-** In continual learning, scaling has been used to modulate the gradient descent steps during training so that old tasks are not forgotten and new tasks are learned with high accuracy.  In this scenario, we apply one-shot scaling of the weights on a pre-trained model. Effectively, we apply scaling to restrict the layer-wise activation of the input samples (during inference) to remove activations belonging to class discriminatory space $P\_{dis}$. Further, we have added a few lines to explain the need for importance scaling under “Importance-base Space Scaling” in Section 4.2.

Reviewer comment (Evaluation):
The evaluation metrics seem to be at odds with the goal of class unlearning that the authors state in the Introduction, namely that “the unlearning algorithm should produce parameters that are equivalent to those of a model trained without the target class”. Despite this definition, the authors don’t look at proximity in weight space or related metrics and instead rely primarily on accuracy.

Author response:
**Class Unlearning goal in Introduction-** We meant “The unlearning algorithm should produce model (parameters) that are functionally equivalent to those of a model trained without the target class.” by the statement and have updated this in the paper. We show such equivalence by our experiment using ACC_r, ACC_f, MIA, confusion matrix, and gradient based saliency maps.

Reviewer comment (Theory):
In section 3, the description of the problem of class unlearning isn’t precise enough. It’s defined as producing a set of unlearned parameters such that two conditions (test retain examples are correctly classified and test forget examples incorrectly classified) are satisfied for ‘many samples’. But how many samples? Usually, the accuracy on each of those two sets is desired to be just as high/low as it would be for retrain-from-scratch. Is there a reason that this is not the definition used? Also, how come this definition refers only to test examples? Usually it is also desired to have similar conditions hold for the training set (retain and forget partitions).

Author response:
**Problem description in Section 3-**  Thanks for this suggestion. We have modified the writing to address reviewers concern. 
We clarify the definition in the following places in the updated paper.
Class Unlearning  ( Section 1 Introduction) “For a class unlearning setup, the primary goal of the unlearning algorithm is to eliminate information associated with a target class from a pretrained model. This target class is referred to as the forget class, while the other classes are called the retain classes.” Additionally, we have formally presented “class unlearning” in Section 3: “The parameters $\theta^*\_f$ must be functionally indistinguishable from a network with parameters $\theta^*$, which is retrained from scratch on the samples of $\mathcal{D}\_{train\_r}$ in the output space. In other words, these parameters must satisfy $f(x\_i, \theta^*) \simeq f(x\_i, \theta^*\_f)$ for $(x\_i, y\_i) \in D\_{test}$ or $D\_{train}$.”

Reviewer comment (Writing):
Clarity: the algorithm is presented in terms of linear and convolutional layers. But in their experiments, the authors also use Transformers. It’s not directly obvious how the proposed method is used for attention layers.

Author response:
**Transfomer Layers -** We have added a line in section 4.3 under “Discussion” to clarify this.  We apply our algorithm to all the linear layers in the transformer architecture. For an attention block, the linear layer weights are the Key Weights, Query Weights, Value Weights, and Output weights. We do not perform any unlearning of the normalization layers as the fraction of total parameters for these layers is insignificant and empirically is not seen to affect the performance of our algorithm.

Reviewer comment (Writing):
Clarity: “for both linear and convolutional layers, where l is a layer and i is retain sample” – i was not mentioned in that context.

Author response:
**typo-** We updated the line to  “ for both linear and convolutional layers, where $l$ is layer.”

Reviewer comment (Writing):
‘produce parameters that are equivalent to those of a model trained without the forget set’ – not clear what the word ‘equivalent’ means here.

Author response:
**Class Unlearning goal in Introduction-** We meant “The unlearning algorithm should produce model (parameters) that are functionally equivalent to those of a model trained without the target class.” by the statement and have updated this in the paper. We show such equivalence by our experiment using ACC_r, ACC_f, MIA, confusion matrix, and gradient based saliency maps.

Reviewer comment (Writing):
In several places, some articles are missing, and there are more typos than these mentioned here. Please proofread the paper and check for grammatical errors.

Author response:
**Response to Minor Issues and typos**
- We thank the reviewer for pointing this out. We went over the document and corrected grammatical errors.

Reviewer comment (Theory):
What is it about this method that makes it specific to class unlearning? Could one use this method to unlearn a random subset of the training dataset? If not then why not?

Author response:
1. Our method relies on SVD which analyzes the underlying statistics of the activation spaces. We believe our method would work on any unlearning problem where there are differences in the data distributions of the forget samples and the retain samples. Class Unlearning is one such case where there is a significant difference in the data distributions, which makes it easier for SVD to figure out distinct spaces. When we assign a random subset of training samples as forget set, SVD can not distinguish well between the activations of forget samples and retain samples. This is because Retain samples are also drawn similarly (Independently Identically Distributed) from the training set. This is why the SVD based method would need further modifications to be fully effective for random subset unlearning problems.

Reviewer comment (Evaluation):
In figure 4, I was surprised that the method of Tarun et al. is more efficient than the proposed method, since Tarun et al uses SGD (in several phases too) while the proposed method only requires forward passes of a few samples (and no backward passes). Why is that?

Author response:
2. We thank the reviewer for pointing this out. We realized that the runtimes depend on the implementation of the algorithm and the background hardware it is running on. We analytically find the compute costs of different algorithms and plot them in Figure 5 and show the details in Appendix A.7. We see 7.3x lesser compute for our approach than the baseline.

Reviewer comment (Evaluation):
In figure 5b, why are the confidence intervals so large? Can we draw any meaningful conclusions from this figure?

Author response:
3. The mean forget accuracy in Figure 5b in the original version (it has been moved to Figure 10 in the updated version) is less than 0.37 on average. In all the experiments we ran for 5 class unlearning the forget accuracy was 0 in 6 of the 10 experiments (the other experiments have forget accuracy of 0.32, 0.7, 0.74, 1.88). In the experiments for 2 class unlearning 9 of the 10 experiments have 0 forget accuracy while one experiment had a forget accuracy of 2.5. These experiments show very low forget accuracies on the model and hence we believe are significant.

Reviewer comment (Evaluation):
Why is the reference point / oracle of Retraining not included in all tables? Is this because it is too computationally expensive to compute this for some models / datasets? Without that reference point, it is not possible to know what the target accuracy/error is for the forget set (because the goal is usually defined as matching / being as close as possible to the accuracy/error of Retrain on the forget set). Could you explain how the results are interpreted given the absence of that reference point?

Author response:
4. Yes, it was computationally expensive to compute these numbers for ImageNet dataset. To address reviewer concern we have updated Section 6 with lines “Due to the training complexity of the experiments, we were not able to obtain retrained models for ImageNet. We observe that the results on CIFAR10 and CIFAR100 datasets consistently show $ACC_f$ to be 0 and the $MIA$ performance to be $100\%$. We, therefore, interpret the model with high $ACC_r$, $MIA$, and low $ACC_f$ as a better unlearnt model for these experiments.”

Reviewer comment (Writing):
The paper is lacking a clear and precise definition of unlearning. Its is important to show the definition of unlearning that you want to achieve through your algorithm.

Author response:
Thanks for this suggestion. We have modified the writing to address this. 
We clarify the definition in the following places in the updated paper.
Class Unlearning  ( Section 1 Introduction) “For a class unlearning setup, the primary goal of the unlearning algorithm is to eliminate information associated with a target class from a pretrained model. This target class is referred to as the forget class, while the other classes are called the retain classes.” Additionally, we have formally presented “class unlearning” in Section 3: “The parameters $\theta^*\_f$ must be functionally indistinguishable from a network with parameters $\theta^*$, which is retrained from scratch on the samples of $\mathcal{D}\_{train\_r}$ in the output space. In other words, these parameters must satisfy $f(x\_i, \theta^*) \simeq f(x\_i, \theta^*\_f)$ for $(x\_i, y\_i) \in D\_{test}$ or $D\_{train}$.”

Reviewer comment (Theory):
The proposed algorithm is an empirical algorithm without any theoretical guarantees. It is important for unlearning papers to provide unlearning guarantees against an adversary.

Author response:
We pose our work as empirical and evaluate our work on large-scale experiments with various sized dataset (including ImageNet) and SoTA deep neural network models (e.g. ViTs etc’). The SoTA baselines we compare against [Tarun et al] and [kurmanji et al] are also empirically evaluated and the scale of their experiments is limited. Further, We have added experiments for Membership Inference Attacks to demonstrate the efficacy of forgetting.

Reviewer comment (Novelty):
The approach is very similar to this method (http://proceedings.mlr.press/v130/izzo21a/izzo21a.pdf) applied on each layer, which is not cited.

Author response:
We thank the reviewer for providing the reference. We have added reference to this work in our paper. We would like to point out the fundamental differences between this work and our work. 
    -  It uses synthetic data and gradient-based optimization/finetuning of the model on these data for unlearning. Whereas we do not use any synthetic data and we don’t need any gradient-based finetuning of the given pre-trained model.
    - The scale of experiments is very different. In particular, this work proposes data deletion with a particular focus on linear and logistic regression whereas we propose an unlearning method for deep learning models that give SoTA performance. 
    - At its core this work relies on being able to calculate the logits of the retrained model without having access to the weights. Essentially calculating  $f(x\_i,\theta^*)$ without knowing $\theta^*$, where $x\_i$ is a forgot sample and $\theta^*$ is retrained model weights. This is a challenging problem to solve in the case of deep learning models and datasets where data might not be very separable which limits the algorithm's application in deep networks. Our algorithm does not rely on the knowledge of logits of the retrained model and hence easily scales for challenging datasets and networks.

Reviewer comment (Experiments):
A simple baseline is just applying all the unlearning algorithm mentioned in the paper to the last layer vs the entire model. This comparison is missing.

Author response:
We add the results for applying our method to the top layer in Appendix A6. Figure 8 shows applying our approach to the last layer only (indicated by layer=10) has a forget accuracy 65% which indicates that removing information from other layers is also necessary.

Reviewer comment (Evaluation):
All the unlearning verification are only show wrt accuracy of the model or the confusion matrix, however, the information is usually contained in the weights of the model, hence other metrics like membership attack or re-train time after forgetting should be considered.

Author response:
We have added membership inference attack evaluations in our experiments. The results show our method performs well across different datasets and architectures obtaining an average improvement of 7.8% in MIA accuracy over other baselines.

Reviewer comment (Theory):
The authors should also consider applying this method as a linear perturbation of the network, as in those settings you will be able to get theoretical guarantees in regards to the proposed method, and also get better results.

Author response:
Can the reviewer clarify this point e.g. what is linear perturbation of the network in the context of unlearning? Some reference would be helpful.

Reviewer comment (Evaluation):
Since the method is applied on each layer, the authors should provide a plot of how different weights of the model move, for instance plot the relative weight change after unlearning to see which layers are affected the most after unlearning.

Author response:
We have added this plot in Figure 4. We see that later layers are more affected by unlearning through our approach. This is expected as the later layers are expected to learn complex class discriminatory information while the initial layers learn edges and simple textures [1].

Reviewer comment (Theory):
There is a more fundamental question about the class unlearning setup. The whole point of unlearning is to replicate a completely retrained model in parameter space or in output space. Now for the preliminary section 3 it is mentioned that for an unlearned model the output label of a datapoint belonging to an unlearned class, is not a true label in this case the unlearned class i.e. y_i != f(x_i,\theta_f). Why is this the case? Is it not that the unlearned model output should be exact/almost the same as the retrained model? If the retrained model gives output as the true label on very few samples why can’t the unlearned model give the same output? In other terms, if the retrained model gives accuracy let's say 3% on the test forget set and the unlearning model also gives similar accuracy on the forget set, the unlearning method is valid.

Author response:
We have rewritten the definition in Section 3(Class Unlearning) as "The parameters $\theta^*\_f$ must be functionally indistinguishable from a network with parameters $\theta^*$, which is retrained from scratch on the samples of $\mathcal{D}\_{train\_r}$ in the output space. In other words, these parameters must satisfy $f(x\_i, \theta^*) \simeq f(x\_i, \theta^*\_f)$ for $(x\_i, y\_i) \in D\_{test}$ or $D\_{train}$." We thank the reviewer for pointing this out.