Paper: QuickDrop: Efficient Federated Unlearning by Integrated Dataset Distillation (ICLR-2024)

Reviewer comment (Novelty):
The technical contribution is incremental. The proposed method uses a gradient ascent + fine-tuning framework, which has been introduced in [1, 2]. The main contribute of this paper is the use of data distillation (a direct application of existing algorithm), which is similar to the idea of representative data selection in [3]. Thus, this paper seems to be a combination of existing methods.

Author response:
We appreciate the reviewer's examination of our proposed method and references to related work. While some aspects of our approach are built upon previous work, we argue that our work extends beyond a mere combination of existing methods. Although SGA and DD techniques are well-known, it is not immediately evident if their combination can perform well for FU. We conducted thorough empirical assessments on three diverse datasets to understand and demonstrate the effectiveness and adaptability of our approach across different unlearning scenarios, e.g., unlearning on class and client levels. Furthermore, our work emphasizes minimizing the overhead associated with DD generation, addressing a critical aspect often overlooked in previous studies.

**Essential difference between DD and representative data selection**:

Besides, we have carefully studied the related work indicated by the reviewer ([3]) about representative data selection and find there is a key difference between our approach and the representative data selection. 
DD in QuickDrop can absorb all the crucial information from the whole original dataset by condensation, while the approach in [3] can only choose part of the dataset and lose extensive information in those unselected data. The unlearning performance with representative data selection therefore has a worse accuracy degradation compared with QuickDrop. For example, Figure 6 in [3] reveals that if 25% data are selected from MNIST as representative dataset, the accuracy on the representative dataset is merely 60%, compared with 95% when using the full original dataset. However, in QuickDrop with scale=100 (i.e., the size of the distilled dataset is only 1% compared with the original dataset), the achieved test accuracy with the distilled dataset is 91%.

Reviewer comment (Experiments):
The proposed method is applicable to sample-wise unlearning, but the experiments are conducted on class-wise and client-wise unlearning. I was expected to see the evaluation of sample-wise unlearning, since it has broader applicability.

Author response:
We agree that sample-wise unlearning presents a more general scenario compared to class- and client-wise unlearning. While it would be valuable to tackle sample-wise unlearning, our framework in its current form requires additional adaptations to achieve this, and we should have been more clear about this limitation. We currently generate class-wise distilled samples that can be reused for downstream class-wise or client-wise unlearning. To achieve sample-wise unlearning, we would need to understand how to generate targeted distilled data for specific forgetting samples, and we left this out of this submission due to the lack of space and to keep the discussion more focused. We look forward to extending our work with sample-wise unlearning.

Reviewer comment (Experiments):
Lack of baseline methods. I suggest that the authors consider including some representative ones or at least one method from more different approaches. Additionally, PU-MP is a CNN-specific method. It may not be appropriate to limit the target model to CNN and compare it with the proposed method, which is more general. I list some of the approaches for your consideration:  
i. Efficient retraining: [4, 5]  
ii. Influence function: [6]  
iii. Gradient ascent + regularization: [7, 8] ([8] is compared in the paper)  
iv. Roll back gradient + knowledge distillation: [1]  
v. Scaling: [9]

Author response:
Because of the time limitation, we can only add one more baseline for comparison during this rebuttal period, which is “IJCAI’22, ARCANE：An Efficient Architecture for Exact Machine Unlearning”. We will report these experiment results soon.

Reviewer comment (Evaluation):
The paper does not discuss communication overhead of exchanging distilled datasets, which is the main concern I have regarding whether the proposed QuickDrop could be useful.

Author response:
There may be a misunderstanding here. 

QuickDrop does not exchange distilled datasets: each distilled dataset is generated locally and remains local. Therefore, this should not be a concern at all. Unlearning, however, does proceed in a federated manner involving the exchange of updated and global models between clients and the server. 

We have provided in Section 3.2 a detailed explanation of the unlearning algorithm that might help clarify the point. If specific points still need to be clarified, we would be happy to provide additional details or discuss them further.

Reviewer comment (Experiments):
Some of the important details are not clear, for example, how non-IID data affects QuickDrop and the impact of distillation hyperparameters.

Author response:
**The effect of Non-IIDness on QuickDrop**:

From an algorithm perspective, the data distribution of the dataset used does not impact how distilled data is generated on each client. From a performance perspective, in Appendix C (Table 7 and 8), we have conducted experiments with IID and non-IID data distributions, showing that QuickDrop can still achieve similar accuracy compared to baselines while maintaining its significant efficiency benefit.

In general, non-IIDness affects us the same as in other unlearning algorithms/baselines. Understanding the impact of data distribution on unlearning algorithms and their proper assessment with well-defined metrics is still an open question for the machine unlearning community.

**Impact of other hyperparameters for distillation**:

**Scale**: An important hyperparameter that influences our distillation is the scale, which defines the size of the distilled dataset on each client. We currently use a scale of 100, i.e., we condense client local datasets by a hundred times. If we reduce the scale (e.g., to 10), it is obvious that we will obtain a larger distilled dataset, which can absorb more information to represent the characteristics of the original dataset and improve the effectiveness of unlearning.

Given the importance of this parameter, we conduct additional experiments to test the influence of scale on unlearning accuracy and computation cost of QuickDrop (on CIFAR-10, alpha=0.1, 10 clients). We list the results in the tables below, where the first table shows the accuracy on the R-set after recovery and the second table shows the compute time to conduct unlearning:

| Scale | Acc. on R-set  (QuickDrop) | Acc.on R-set  (Retraining) |
|:-----:|:------------------------------:|:-------------------------------:|
|  100  |             70.48%             |              74.95%             |
|   10  |             73.69%             |              74.95%             |

| Scale | Compute time  (QuickDrop) | Compute time (Retraining) |
|:-----:|:-------------------------:|:-------------------------:|
|  100  |           15.61s          |          7239.58s         |
|   10  |          154.79s          |          7239.58s         |

In conclusion, the scale is a trade-off between the unlearning accuracy and efficiency, and can be adjusted according to the different demands of clients or the FU environment. We will add these experimental results to the paper for a final version.

We also identify other hyperparameters that influence the performance of QuickDrop, like the current global round K and local update steps T. In our work, we fix K=200 and T=50. Since QuickDrop combines DD and FL, the distilled dataset is a by-production of the normal FL training process. The local update step T and global round K required to obtain a trained FL model is much larger than the requirements of DD. Therefore, we are guaranteed that the distilled dataset with K=200 and T=50 are of high quality.