Paper: Rethinking Adversarial Robustness in the Context of the Right to be Forgotten (ICLR-2024)

Reviewer comment (Experiments):
One of my main concerns is the robustness of the results. The experimental setup section describes the experiments use 4 datasets, 6 model architectures, 3 adversarial attacks, and 2 unlearning techniques. This setup enables more than 140 different dataset-model-attack-unlearning combinations, however, the results in each section tend to only show a very select subset of these combinations. For example, Figure 6 only shows 2 attacks on the CIFAR-10 dataset, and 1 attack for the SVHN dataset; why aren't all the results shown for all the dataset-model-attack combinations, or aggregated in some way to give readers a general sense of AdvUA's effectiveness across different settings?

Author response:
**Q4.1. Discussing the robustness of experimental results across 140 different dataset-model-attack-unlearning combinations in experiments (since there are 4 datasets, 6 model architectures, 3 adversarial attacks, and 2 unlearning techniques).**

**A4.1:** Thanks for the valuable comment. We want to first clarify that in the initial submission, we adopted 4 unlearning techniques instead of 2 unlearning techniques. Due to space limitation, we deferred the experimental results on another two unlearning methods to the Appendix (see Figure 12 in the Appendix with detailed discussion in Section E.2.3.). 

Additionally, we appreciate the suggestion to exhaustively test every possible scenario across all of the 140 different dataset-model-attack-unlearning combinations in experiments. Considering the limited rebuttal time, we conducted the following new attack transferability experiments to show the good attack transferability of AdvUA across various settings (e.g., different unlearning methods). Specifically, in Table 1, we illustrate the attack transferability of AdvUA across different unlearning methods. From this table, we can see that the malicious unlearning samples generated using the first-order based unlearning method are also effective in increasing the models’ adversarial vulnerabilities when applied with other unlearning methods. The reason is that different unlearning methods are designed to achieve similar unlearning performance to remove the impact of the given same set of unlearning samples, and the unlearning samples we found are crucial against the robust training. Note that in Table 1, we adopted the ResNet-18 model, the adversarial attack method of FGSM, and the CIFAR-10 dataset. Additionally, in Table 2 and Table 3, we conducted experiments to verify the attack transferability of AdvUA across different models under our black-box settings. Specifically, in Table 2, AdvUA demonstrates good attack transferability across different models when transferring unlearning samples. Here, we adopted the first-order based unlearning method and the adversarial attack method of FGSM. Moving to Table 3, AdvUA also exhibits effective attack transferability across different models when transferring adversarial samples. More specifically, we first obtained different unlearned models by transferring the same set of unlearning samples to different models using the first-order based unlearning method, and then transferred adversarial samples across these obtained unlearned models for PGD adversarial attacks. 

Table 1: Transferability of AdvUA across different unlearning methods. The first-order based unlearning method serves as the surrogate unlearning method.
| # of unlearning samples | Second-order based | Unrolling SGD |
| :--- | :--- | :--- |
| 20 | 52.0% ± 4.4% | 58.0% ± 6.3% |
| 40 | 60.0% ± 6.7% | 60.0% ± 6.0% |
| 60 | 78.0% ± 8.1% | 60.0% ± 6.7% |
| 80 | 86.0% ± 3.1% | 66.0% ± 6.7% |
| 100 | 88.0% ± 4.4% | 70.0% ± 4.5% |

Table 2:  Attack transferability (transferring unlearning samples) across different models. ResNet-50 serves as the surrogate model.
| # of unlearning samples | VGG-19 | DenseNet-121 | ResNet-152 |
| :--- | :--- | :--- | :--- |
| 10 | 88.0% ± 2.9% | 91.0% ± 4.3% | 85.0% ± 3.4% |
| 20 | 93.0% ± 2.6% | 93.0% ± 3.3% | 88.0% ± 4.2% |
| 30 | 94.0% ± 2.7% | 96.0% ± 2.2% | 93.0% ± 3.3% |

Table 3: Attack transferability (transferring adversarial samples) across different models. ResNet-50 serves as the surrogate model.
| # of unlearning samples | VGG-19 | DenseNet-121 | ResNet-152 |
| :--- | :--- | :--- | :--- |
| 10 | 88.0% ± 3.3% | 93.0% ± 1.5% | 94.0% ± 2.2% |
| 20 | 91.0% ± 2.8% | 95.0% ± 2.2% | 97.0% ± 1.5% |
| 30 | 93.0% ± 2.1% | 97.0% ± 1.5% | 97.0% ± 1.5% |

In the final version, we will also incorporate this helpful suggestion. By conducting the above attack transferability across various settings (e.g., varying unlearning methods), we believe that they can provide a good sense of the attack generalization performance of our proposed method across diverse scenarios.

Reviewer comment (Experiments):
I think there could be a more in-depth discussion between the $k$NN based approach and AdvUA using local density only. A comparison of $k$NN to this ablated version of AdvUA may be insightful and beneficial to readers. It's also not clear to me how $k$ is chosen in the experiments, and how impactful the choice of $k$ is on the results.

Author response:
**Q4.3. Giving a more in-depth discussion between the kNN based approach and AdvUA using local density only. A comparison of kNN to this ablated version of AdvUA may be insightful and beneficial to readers. It's also not clear to me how $k$ is chosen in the experiments, and how impactful the choice of $k$ is on the results.** 

**A4.3:** Thanks for the valuable comments. The kNN method, relying on absolute distances, removes the k-nearest points around the victim sample, which limits its ability to accurately assess local data structures. In contrast, our local density measure $l_{wd}$ for AdvUA, as defined in Equation (1),  leverages relative distances and takes into account broader data areas. This allows for a more precise characterization of local structures and is more effective in identifying crucial points that are key to maintaining model performance near victim samples.

Additionally, following your suggestions, in Table 4 below, we conducted experiments to compare the attack success rates of the kNN method and our AdvUA approach using local density only, where $k$ equals the number of unlearning samples for fair comparison. This comparison is conducted across various numbers of unlearning samples with FGSM on the CIFAR-10 dataset. These experimental results verify that the AdvUA method with local density only, outperforms the kNN approach under different values of $k$. In addition, we can find that increasing the value of $k$ in the kNN method does not significantly enhance attack performance. In contrast, our approach substantially improves attack performance as the number of unlearning samples increases. This is attributed to the effectiveness of our method in effectively reducing local density by removing more crucial training samples.

Table 4: Attack success rate comparison between AdvUA with local density and kNN.
| # of unlearning samples    | 10   | 20   | 30   | 40   | 50   |
| :--- | :--- | :--- | :--- | :--- | :--- |
| kNN baseline | 32.0% ±  8.0% | 30.0% ± 7.5% | 34.0% ± 9.5% | 32.0% ± 9.5% | 30.0% ± 10.4% |
| AdvUA with local density | 52.0% ±  5.3% | 54.0% ± 6.7% | 62.0% ± 6.3% | 70.0% ± 5.4% | 72.0% ± 4.4%|

Lastly, as explained above, in our experiments, $k$ is set as the number of unlearning samples for fair comparison with AdvUA. Note that in our experiments, we also set the attack budget for the launched adversarial unlearning attacks by restricting the maximal number of unlearning samples to be deleted.

Reviewer comment (Experiments):
It's not clear to me how robust AdvUA is to different unlearning techniques. I think a plot comparing results using the two different unlearning techniques would provide more evidence to the generalizability of the proposed approach. Have the authors also experimented with the gold standard unlearning technique of retraining from scratch on a small dataset/model?

Author response:
**Q4.4. How robust AdvUA is to different unlearning techniques? Giving a plot to compare results using the 2 different unlearning techniques to show the generalizability of the proposed method.** 

**A4.4:** Thanks for highlighting this. Due to space limitations in our main submission, we deferred the experiments about the comparison of AdvUA's performance across various unlearning methods in the Appendix. Detailed experimental results are available in Figure 12, along with a comprehensive discussion in Section E.2.3. For your convenience, we have also provided the following table (Table 5) to show the effectiveness of AdvUA with second-order based, unrolling SGD, and SISA unlearning methods to attack the defended model against FGSM attacks on CIFAR-10.

Table 5: Attack success rate of AdvUA with different unlearning methods.
| # of unlearning samples | 10 | 20 | 30 | 40 | 50 |
| :--- | :--- | :--- | :--- | :--- | :--- |
| Second-order based | 50.0% ± 6.8% | 72.0% ± 4.4% | 82.0% ± 5.5% | 84.0% ± 5.8% | 84.0% ± 5.8% |
| Unrolling SGD | 64.0% ± 5.0% | 66.0% ± 7.3% | 70.0% ± 6.1% | 72.0% ± 6.1% | 72.0% ± 6.1% |
| SISA | 50.0% ± 5.4% | 50.0% ± 5.4% | 52.0% ± 6.1% | 54.0% ± 6.7% | 64.0% ± 5.8% |

Reviewer comment (Writing):
There are some minor grammatical errors throughout the paper, consider using a service like Grammarly to fix these issues.

Author response:
**Q4.6. Correcting minor grammatical errors.**

**A4.6:** Thanks for the helpful suggestion. Following your suggestions, in the revised version, we have thoroughly checked the paper and carefully corrected grammatical errors.

Reviewer comment (Presentation):
Consider listing what datasets and models are being shown in Figure 6.

Author response:
**Q4.7. Listing what datasets and models are being shown in Figure 6.**

**A4.7:** Thanks for the comment. The details of Figure 6 are provided at the top of Page 8 in the main manuscript. Specifically, Figure 6a and Figure 6b utilize the ResNet-18 model on the CIFAR-10 dataset, and Figure 6c employs ConvNet on the SVHN dataset.

Reviewer comment (Presentation):
Figures 1, 7, and 8 are not colorblind friendly.

Author response:
**Q4.8. Changing Figures 1, 7, and 8 to be colorblind friendly.**

**A4.8:** Thanks for the valuable suggestions. Following your suggestions, in the revised version, we have revised Figures 1, 7, and 8 to make them colorblind friendly.

Reviewer comment (Evaluation):
What is the non-robust accuracy of the different models in Figure 1? It may be beneficial to readers to see that comparison.

Author response:
**Q4.9. What is the non-robust accuracy of the different models in Figure 1?**

**A4.9:** Thanks for bringing attention to the non-robust accuracy in Figure 1. As you suggested, we have conducted experiments to evaluate the non-robust accuracy of the different models in Figure 1. Table 7 below reports the non-robust accuracy on the SVHN dataset, which is the test accuracy of the naturally trained model under adversarial attacks (e.g., FGSM, PGD, and CW attacks). Specifically, we randomly removed 10% of the training samples and re-evaluated the non-robust accuracy of the unlearned model. As observed in this table, the random deletion strategy adversely affects the non-robust accuracy against various adversarial attacks. For example, the non-robust accuracy of the FGSM attack is 22.62% before unlearning and drops to 9.56% after unlearning. In other words, even random unlearning can also exacerbate the adversarial vulnerabilities of the undefended models.

Table 7: Comparison of non-robust accuracy on SVHN.
| Adversarial attack | Before unlearning | After unlearning |
| :--- | :--- | :--- |
| FGSM | 22.62% ± 0.00% | 9.56% ± 0.27% | 
| PGD | 3.75% ± 0.01% | 1.37% ± 0.12% | 
| CW | 1.13% ± 0.01% | 0.32% ± 0.05% |

Reviewer comment (Presentation):
Is Figure 2 showing the density of training examples before or after unlearning?

Author response:
**Q4.10. Is Figure 2 showing the density of training examples before or after unlearning?**

**A4.10:** To clarify, Figure 2 shows the density of training examples after unlearning, as detailed in Section 3.2 FORMULATION OF ADVUA.

Reviewer comment (Experiments):
Where are the kNN and random baseline methods in Figure 7?

Author response:
**Q4.12. Where are the kNN and random baseline methods in Figure 7?**

**A4.12:** Thanks for the questions. As you suggested, following the same experimental setup in Figure 7, we have conducted new experiments on the kNN and Random baseline methods, and presented the obtained experimental results in Table 9 and Table 10 below. Specifically, Table 9 shows the obtained experimental results of AdvUA and the suggested baselines on the certified defended models using spectral normal regularization against FGSM attacks. Table 10 shows the obtained experimental results of AdvUA and the suggested baselines on the certified defended models using interval bound propagation against PGD attacks. It is evident that AdvUA outperforms the kNN baseline and Random baseline across varying unlearning samples, demonstrating the efficacy of our proposed method against certified defenses.

Table 9: Attack success rate of AdvUA on certified defended models using spectral norm regularization. 
| # of unlearning samples | 20 | 40 | 60 | 80 | 100 | 120 | 140 | 160 | 180 | 200 |
| :--- | :--- | :--- | :--- | --- | :--- | :--- | :--- | :--- | :--- | :--- |
| AdvUA | 42.0% ± 4.7% | 48.0% ± 5.3% | 46.0% ± 7.9% | 52.0% ± 5.3% | 52.0% ± 6.1% | 60.0% ± 7.9% | 68.0% ± 4.4% | 72.0% ± 6.8% | 70.0% ± 6.8% | 76.0% ± 5.8% | 
| kNN | 38.0% ± 4.7% | 22.0% ± 7.0% | 42.0% ± 8.1% | 30.0% ± 6.8% | 40.0% ± 8.4% | 36.0% ± 6.5% | 40.0% ± 8.4% | 40.0% ± 8.3% | 44.0% ± 8.3% | 52.0% ± 7.4% | 
| Random | 26.0% ± 7.9% | 22.0% ± 9.6% | 28.0% ± 4.4% | 26.0% ± 3.1% | 34.0% ± 7.9% | 34.0% ± 5.2% | 42.0% ± 7.6% | 42.0% ± 11.3% | 44.0% ± 7.2% | 46.0% ± 5.2% | 

Table 10: Attack success rate of AdvUA on certified defended models using interval bound propagation. 
| # of unlearning samples | 20 | 40 | 60 | 80 | 100 | 120 | 140 | 160 | 180 | 200 |
| :--- | :--- | :--- | :--- | --- | :--- | :--- | :--- | :--- | :--- | :--- |
| AdvUA | 38.0% ± 7.6% | 44.0% ± 9.3% | 48.0% ± 6.8% | 50.0% ± 6.1% | 50.0% ± 6.8% | 60.0% ± 3.0% | 64.0% ± 5.0% | 76.0% ± 5.8% | 74.0% ± 4.3% | 80.0% ± 5.2% |
| kNN | 28.0% ± 6.1% | 38.0% ± 7.6% | 30.0% ± 7.5% | 46.0% ± 6.0% | 32.0% ± 9.0% | 40.0% ± 6.0% | 32.0% ± 6.8% | 52.0% ± 6.1% | 48.0% ± 5.3% | 60.0% ± 6.7% | 
| Random | 28.0% ± 7.4% | 34.0% ± 6.7% | 42.0% ± 3.6% | 40.0% ± 4.2% | 44.0% ± 7.2% | 34.0% ± 5.2% | 40.0% ± 6.7% | 42.0% ± 3.6% | 42.0% ± 6.3% | 52.0% ± 4.4% |

Reviewer comment (Experiments):
Have the authors thought about how AdvUA can be used to train more robust models?

Author response:
**Q4.13. Have the authors thought about how AdvUA can be used to train more robust models?**

**A4.13:** Thanks for raising the discussion on using AdvUA to train more robust models. Indeed, AdvUA can also be employed to enhance model robustness by targeting the local density regions during the training process. It's important to note that in our paper, AdvUA is originally designed to unlearn certain training samples to strategically place target victim samples in low-density areas. By considering this process in reverse, AdvUA can be adapted to target the low-density areas during the training process to enhance model robustness. Specifically, this can be achieved by combining AdvUA with current data augmentation techniques to produce synthetic augmented data that targets low-density regions at different training stages. Notably, this strategy is complementary to existing adversarial robust training methods. It can be readily combined with these methods to develop enhanced adversarial robust training strategies, which is also observed from our experiments.

Reviewer comment (Theory):
The intent of Theorem 2 is not clear. Could you please explain what theoretical justification this theorem gives about the proposed method?

Author response:
**A3.1:** Thanks for the constructive comment. Theorem 2 illustrates the relation between the number of training samples and the associated $l_{\infty}^{\epsilon}$-adversarial robust error. This theorem shows that when the training set size diminishes from $n_1$ to $n_2$ under the conditions mentioned in this theorem, there is a high probability (at least $1-2 \exp \left(-\frac{d}{8(\sigma^2+1)}\right)$) that the robust error will increase. Note that our proposed AdvUA is designed to unlearn the strategically selected malicious samples to reduce the models’ adversarial robustness. In the revised version, we have provided more explanations for the intent of this theorem in the paragraph following Theorem 2 in Page 6.

Reviewer comment (Writing):
Unlearning U should be defined as a mapping with specifying inputs and output.

Author response:
**A3.2:** We appreciate the helpful suggestion. Following your suggestions, in the revised version (please refer to Definition 1 on Page 3), we have defined Unlearning $U$ by specifying inputs and output.

Reviewer comment (Writing):
H should be defined before it appears for the first time.

Author response:
**A3.3:** Thanks for pointing this out. Following your suggestions, in the revised version (please refer to the first paragraph in Section 2), we have first defined $H$ before it appears for the first time.

Reviewer comment (Evaluation):
Although the experimental section shows promising gain on ASR after applying AdvUA, an important metric is obviously ignored: **the benign accuracy** of the model after unlearning. Ideally, a powerful attack will not only improve the ASR but also has marginal impact on the benign function of the threat model. Therefore, I would suggest authors report both ACC and ASR instead of ASR alone.

Author response:
**A2.1:** Thanks for the constructive suggestion. Following your suggestion, in Table 1 below, we report the attack success rate and the benign accuracy of the models after unlearning. We followed the experimental setup in Figure 6a that uses ResNet-18 on CIFAR-10 and against FGSM attacks. It's worth noting that the initial model achieved a benign accuracy of 83.53%. From this table, we can see that AdvUA not only achieves high attack success rates but also demonstrates a minor impact on the benign accuracy of the threat model. 

Table 1: Benign accuracy and attack success rate of AdvUA.
| # of unlearning samples | 10 | 20 | 30 | 40 | 50 |
| :--- | :--- | :--- | :--- | :--- |
| Attack success rate | 62.0% ± 6.3% | 64.0% ± 5.8% | 74.0% ± 6.0% | 76.0% ± 6.5% | 78.0% ± 5.5% |
| Benign accuracy | 81.52% ± 0.27% | 80.86% ± 0.29% | 80.22% ± 0.33% | 79.33% ± 0.42% | 78.68% ± 0.37% |

Reviewer comment (Writing):
Some minor error exists: For example, "PDG" in Page 7. I would suggest authors re-read the whole paper for further corrections.

Author response:
**A2.2:** Thanks for the helpful suggestions. Following your suggestions, in the revised version, we have corrected the mentioned minor error, and thoroughly re-read the whole paper for further corrections.

Reviewer comment (Theory):
The computational cost of AdvUA could be unaffordable when the dataset is huge. Algorithm 1 shows that AdvUA needs to at least iterate through the data $|S_e|\times B\times |K_v|$ times. When $|S_e|$ is huge, large $|K_v|$ is needed to ensure the accuracy of assessment. It may sharply increase the computational expenses of AdvUA.

Author response:
**A2.4:** Thanks for the valuable comment. Note that due to the discrete deletion nature of unlearning samples and the corresponding time complexity of combinatorial optimization, solving the proposed loss in Equation (3) in the main manuscript is computationally infeasible. To address this, we instead adopt the empirical search based method (as outlined in Algorithm 1) to effectively optimize the loss. First, we would argue that from the aspects of practical attack constraints and attack stealthiness, the computational cost of AdvUA is relatively low and affordable. In practice, using a large number of malicious unlearning samples (i.e., $B$) could alert the model owner to potential malicious attack risks, thereby increasing the likelihood of detecting the launched attacks. Therefore, in Algorithm 1, we set the maximum number of malicious unlearning samples as $B$. Additionally, the attacker in practice usually has a limited search set $S_{e}$, and consequently, the number of the top nearest neighbors (i.e., $K_{v}$) is also limited. In our experiments, we also typically opted for small values  for $|S_{e}|$, $K_{v}$ ($K_{v} < |S_{e}|$), and the malicious unlearning samples. For example, as depicted in Figure 6, we set $|S_{e}|$ as 500, and with a maximum of 40 unlearning samples, we achieved a 76% attack success rate. Note that in all of the experiments in our paper, we set $K_{v}$ as 100. We also want to emphasize that the attack performance being constrained by a limited budget is also of great practical significance in existing literature.

Additionally, we want to clarify that in our experimental attack implementation of AdvUA in the initial submission, when attacking the victim samples, we did not unlearn each unlearning sample individually. Instead, in our experiments, we first split all the available unlearning samples from the overall search set (i.e., $S_{e}$) into a set of local search subsets. Then, during our empirical search based optimization procedure, at each step, we selected an unlearning search subset, whose removal will largely increase the loss value associated with the distance evaluation metric and cosine similarity. In the revised version, we have revised Algorithm 1 in the Appendix to include this detail. We also conducted new experiments to compare the effectiveness of searching through the entire pool (global search) versus a local pool (local search), in terms of both running time and attack success rate. In Table 2 and Table 3 below, we presented the obtained experimental results. Specifically, we set the candidate set sizes (i.e., $|S_{e}|$) to 100, 250, and 500, with the objective of unlearning a total of $B=50$ samples in each pool. In the global search, our method involves iterating over each sample in the candidate set to identify the effective malicious unlearning samples. On the other hand, in local search, we split the set $S_{e}$ into multiple subsets based on the above procedure. From the below table, we can see that our adopted strategy has less running time while achieving comparable attack performance. 

Table 2: Running time (min) of AdvUA with global search and local search.
| $\|S_e\|$ | 100 | 250 | 500 | 
| :--- | :--- | :--- | :--- |
| Global search | 4.05 ± 0.10 | 11.91 ± 0.31 | 25.30 ± 0.53 |
| Local search | 0.29 ± 0.00 | 0.87 ± 0.0 | 1.66 ± 0.04 |

Table 3: Attack success rate of AdvUA with global search and local search.
| $\|S_e\|$ | 100 | 250 | 500 | 
| :--- | :--- | :--- | :--- |
| Global search | 56.0% ± 7.2% | 60.0% ± 9.4% | 64.0% ± 6.5% |
| Local search | 58.0% ± 8.7% | 56.0% ± 6.5% | 62.0% ± 6.3% |

Reviewer comment (Presentation):
Why don't the starting points of AdvUA, kNN, and Random overlap in Figure 6?

Author response:
**A2.5:** Thanks for the comment. To clarify, the starting point for the number of unlearning samples in Figure 6 is 10 for Figure 6a & Figure 6b (CIFAR-10), and 20 for Figure 6c (SVHN), rather than zero, explaining why AdvUA, kNN, and Random don't overlap. In the revised version, we have corrected this, shifting all the starting points to zero. Note that the samples we choose to attack are all hard samples, which cannot be successfully attacked on the original robust model before unlearning. Thus, in this figure, the attack success rate when no samples are unlearned is just zero.

Reviewer comment (Experiments):
Almost all experiments are performed on CNNs. ViT are another prevailing architecture in computer vision. Can AdvUA help existing attacks better evade robust ViTs [1,2] during unlearning?

Author response:
**A2.6:** Thank you very much for bringing our attention to the application of our proposed AdvUA to invade robust ViTs [1, 2]. Following your suggestions, we have conducted experiments to evaluate the attack performance of AdvUA on robust ViTs [1]. We adopted the vit_base_patch16_224_in21k model in [1] and trained an adversarial robust model on 20% of the CIFAR-10 dataset using the adversarial training technique. We then conducted our AdvUA attack on this robust model against PGD attacks, and our experimental results show that AdvUA can effectively attack robust ViTs. For example, when unlearning 50 unlearning samples, AdvUA can achieve an attack success rate of 56.0% ± 4.0% while the kNN baseline achieves 30.0% ± 6.8%; when unlearning 100 unlearning samples, AdvUA can achieve an attack success rate of 68.0% ± 6.1% while the kNN baseline achieves 42.0% ± 6.3%. Note that in this experiment, we chose the hard samples to attack, which led to a zero attack success rate on the original robust model before unlearning. Therefore, these experiments further underscore the significance of AdvUA, and help us to well understand adversarial unlearning attacks in different settings.

Reviewer comment (Experiments):
The authors only consider one kind of black-box setting: The attacker uses a surrogate model to unlearn various numbers of training samples with the first-order based method and then generate the adversarial examples. However, another black-box setting is that the attackers use a surrogate model to select samples for unlearning and victims use the selected samples to unlearn their own models. Is AdvUA effective in this situation?

Author response:
**A2.7:** Thanks for highlighting another black-box setting. Following your suggestions, in Table 4 below, we evaluate the attack performance of AdvUA against FGSM in another black-box setting. Specifically, we first crafted malicious unlearning samples by utilizing the ResNet-50 model, and then transferred these crafted unlearning samples to the black-box victim models (i.e., VGG-19, DenseNet-121, and ResNet-152). After that, we unlearned these crafted samples from the black-box victim models (i.e., VGG-19, DenseNet-121, and ResNet-152), resulting in the corresponding unlearned models, which were then used to generate adversarial perturbations. As shown in the table below (i.e., Table 4), AdvUA can also achieve high attack performance in this suggested black-box setting.

Table 4: Attack success rate of AdvUA in another black-box setting. ResNet-50 serves as the surrogate model.
| # of unlearning samples | VGG-19 | DenseNet-121 | ResNet-152 |
| :--- | :--- | :--- | :--- |
| 10 | 88.0% ± 2.9% | 91.0% ± 4.3% | 85.0% ± 3.4% |
| 20 | 93.0% ± 2.6% | 93.0% ± 3.3% | 88.0% ± 4.2% |
| 30 | 94.0% ± 2.7% | 96.0% ± 2.2% | 93.0% ± 3.3% |

Reviewer comment (Experiments):
The authors should conduct further evaluation on real-world systems to better understand the practical impact of AdvUA.

Author response:
**Q1.2. Evaluation on real-world systems.** 

**A1.2:** Thanks for the valuable comment. Following your suggestions, we have conducted new experiments by considering real-world settings. However, due to the constraints (e.g., ethical and legal considerations) of attacking real-world running systems, we followed [2] and used a real-world medical dataset to simulate a healthcare diagnosis system, a domain that commonly integrates deep learning models to enhance diagnostics. Building upon this simulation, we extended the method in [2] by employing AdvUA on medical deep learning based systems. Specifically, we adopted the dermoscopy image dataset, a large collection of multi-source dermatoscopic images, for building an automated diagnosis system. Next, we trained a robust ResNet-18 model using the adversarial training technique on this dataset. Then, we conducted our AdvUA method on the defended model against PGD attacks, and the obtained experimental results also verify the effectiveness of AdvUA. For example, when unlearning 100 training samples, AdvUA can achieve an attack success rate of 86.0% ± 3.1%, while the kNN baseline achieves 22.0% ± 7.6%. Therefore, these conducted experiments further highlight the practical importance and impact of AdvUA on real-world system applications.

Reviewer comment (Evaluation):
Results show that AdvUA can perform reasonably well against certified defense methods, even when unlearning only a small number of training samples. How does the effectiveness of AdvUA against certified defense methods vary depending on the specific certified defense method used?

Author response:
**Q1.3. How does the effectiveness of AdvUA against certified defense methods vary depending on the specific certified defense method used?**

**A1.3:** Thanks for the insightful question. In our experiments, we have indeed observed variations in attack effectiveness against certified defense methods. Specifically, in our conducted experiments, we found that if the adopted certified defense methods (e.g., interval bound propagation-based certified defenses adopted in our paper) modify the training data to provide certified guarantees, they are highly susceptible to malicious unlearning samples. On the other hand, for the certified defense methods that rely on upper bound-based regularizers (e.g., spectral norm based methods adopted in our experiments) to provide their guarantees, we observed that under the same number of malicious unlearning samples, these methods typically show less vulnerability to security threats when subjected to malicious unlearning samples.

Reviewer comment (Theory):
It is about the AdvUA attack’s computational time complexity in black box scenarios. How does the computation time complexity of AdvUA against certified defense methods vary based on the number of unlearning samples?

Author response:
**Q1.4. In the black-box setting, e.g., how does the computation time complexity of AdvUA against certified defense methods vary based on the number of unlearning samples?**

**A1.4:** Thanks for the valuable question. Note that in the main manuscript, for the black-box setting discussed in Page 6, we assume that the adversary only knows the output of the target model through predictions. In this black-box setting, we can first follow existing model stealing attacks to construct a surrogate model, which leverages shared decision boundaries among various models. Then, based on this constructed surrogate model, we can perform our proposed adversarial unlearning attacks (i.e., AdvUA) to attack the target victim samples. After that, we can transfer the crafted malicious unlearning samples (that are generated based on the constructed surrogate model) to attack the victim black-box model, which means that the computation cost of AdvUA in terms of the number of unlearning samples in this black-box setting is similar to that in the white-box setting (as discussed earlier in Q1.2). Notably, in this black-box setting, the additional computational cost is incurred in the construction of the surrogate model. This step is essential for the black-box attack to approximate the behavior of the target victim model, thereby enabling the effective transfer of malicious unlearning samples.