Paper: Adversarial Machine Unlearning: A Stackelberg Game Approach (ICLR-2024)

Reviewer comment (Theory):
Details of the proposed method is not clear. Regarding Eq.(5), what is the precise formulation for $f$? Do the stationarity conditions account for $\nabla V(\tilde{D}_{\theta_u}^{tr}; \theta_a^{'})$? It would enhance comprehension if the complete equation for $f$ were explicitly presented.

Author response:
1.  The precise formulation of $f$ depends on the specific model used in the MIA. The stationarity condition is indeed a part of $f$. For illustrative purposes, we provide an example of $f$ for linear SVM in the appendix (A.6).

Reviewer comment (Writing):
The manuscript could benefit from more meticulous notation for clarity. In Eq.(4), should $D_{\theta_u}$ actually be $D_{\theta_u}^{val}$? Alternatively, is it meant to be $\tilde{D}_{\theta_u}^{tr}$?

Author response:
2.1  We thank the reviewer for pointing this out. We have updated Eq.(4) in the revised paper.

Reviewer comment (Writing):
The manuscript could benefit from more meticulous notation for clarity. In Section 5, when referencing $n$ as indicative of size, is this referring to the size of the attack model parameters?

Author response:
2.2 In section 5, $n$ is the number of variables  plus the number of constraints in the attacker’s optimization problem.

Reviewer comment (Writing):
The manuscript could benefit from more meticulous notation for clarity. For example, in Section 3, $i$ in $(x_i, y_i)$ denotes the indices, it might be more clear to use $(x_j^f, x_j^f)$ rather than $(x_f^j, x_f^j)$.

Author response:
2.3 We will double check the notations and improve the clarity.

Reviewer comment (Experiments):
Missing details in the experiments and unconvincing results. All experiments only consider the setting where the forget set is randomly selected, accounting for 10% of the training data, and only employs ResNet-18. A broader set of conditions (e.g., 20% forget set, class-specific forgets, poisoned sample forgets, and varying network architectures) would have enriched the analysis.

Author response:
3.1 We selected 10% to ensure consistent comparison with prior studies. However, we acknowledge that exploring the impact of the forget set size on unlearning is an important direction for future research. To show that the SG framework is applicable to more than just image datasets and ResNet architectures, we present supplementary results on text data and transformer architectures in section A.1 of the appendix.

Reviewer comment (Experiments):
Missing details in the experiments and unconvincing results. How does the unscrubbed model perform? Also, results in tables for methods like FF and IU fail to unlearn. How do these methods perform with increased noise level?

Author response:
3.3  The performance of the unscrubbed models (e.g.,  test accuracy and the MIA accuracy) are displayed in Figure 2 at epoch = 0. The noise level of FF and IU is selected on the validation data. The criterion is to select the noise level that maximizes: $\text{prediction accuracy}−\text{MIA accuracy}$. We exhaustively search five values for the noise level: 1e-9, 1e-8, 1e-7, 1e-6, 1e-5. IU is not sensitive to the noise level. However, the test accuracy of FF significantly drops below 0.4 when the noise level exceeds 1e-7, which makes the unlearned model useless for prediction tasks.

Reviewer comment (Evaluation):
Missing details in the experiments and unconvincing results. Considering the MIA is already integrated into the unlearning algorithm, KS and MIA metrics are quite related to the performance of MIA. Incorporating common metrics like Weight Distance [1], Activation Distance [2–3], Relearn Time [4], and Epistemic Uncertainty [5] might offer a more holistic performance view.

Author response:
3.4 We would like to clarify that the KS and MIA metrics reported in Table 1 are evaluated on an auditing dataset that is separate from the one used in unlearning. Although the two auditing datasets share the same forget set, they have different test instances. In the revised paper, we have made this distinction clearer. Additionally, we have included two new additions: 1) running time comparisons and 2) the Wasserstein distance between the cross-entropy losses of the forget and test instances. The Wasserstein distance reported in Table 1 is computed on an auditing dataset that is disjoint from the one used in unlearning.

Reviewer comment (Experiments):
Missing details in the experiments and unconvincing results. It would be better if the loss/accuracy curve (from the retrained model and the proposed method) could be provided to show the convergence.

Author response:
3.5 We added the training loss for the retrain baseline to Figure 7 in the appendix.

Reviewer comment (Experiments):
Missing details in the experiments and unconvincing results. The literature review appears to overlook some recent developments in the field (e.g., [5–8]). It would strengthen the paper's comparative analysis if at least one contemporary method were included for benchmarking.

Author response:
3.6  We will go through the literature and discuss the connections to the current paper in the next revision.

Reviewer comment (Theory):
It does not provide any theoretical analysis of the game, such as convergence, complexity, or optimality guarantees or discussions.

Author response:
We thank the reviewer’s comments for improving the paper!

1. Please refer to comment 5 in “To all reviewers” at the top. We will add a subsection to discuss the convergence in details. 

2. We have a time complexity analysis of Algorithm 1 in section 5, just above the pseudocode of the algorithm. We will revise the writing to emphasize the analysis.

Reviewer comment (Experiments):
It does not consider multiple attackers or multiple attack methods, which may pose stronger or more stealthy threats to the unlearning process.

Author response:
We thank the reviewer’s comments for improving the paper!

3. We acknowledge the significance of investigating the setting of multiple attack methods as an important direction for future research. It is worth noting that the single-attack setting has not been explored in the context of machine unlearning, and this paper takes a first step towards addressing the unlearning problem from a game theoretic perspective.

Reviewer comment (Experiments):
It does not conduct experiments on more datasets or more complex models, to demonstrate the generalization and scalability of the framework and algorithm.

Author response:
We thank the reviewer’s comments for improving the paper!

4. We have added additional experimental results on the SVHN dataset, and results on text data and transformer architectures. 
  - SVHN dataset: see A.2 in the appendix.
  - Text data and transformer architectures: see A.1 in the appendix. 
  - Running time comparison has been added to Table 1.

Reviewer comment (Theory):
The authors consider the setting where the forget set is randomly sampled. First, It is not clear to me why this assumption will hold in real-world scenario when the machine unlearning is motivated by regulations in certain geographical regions. Second, if the forget set is randomly sampled, from a statistical point of view, the problem of machine unlearning becomes the problem of understanding how training dataset size can affect the model performance (the difference between the original performance and the retraining performance).

Author response:
Uniform sampling is used as an approximation of the distribution of real-world forget requests. For instance, [2] demonstrates that requests to delete URLs originate from multiple countries, although some countries have more requests than others. We acknowledge that studying a more realistic distribution of the forget set is an important direction for future research. Another reason we use the uniform sampling is for having a consistent comparison with prior studies. 
- [2] Five years of the right to be forgotten, Bertram, et al. 

3. We acknowledge the importance of understanding the impact of the training set size on unlearning. However, we believe that the issue of unlearning extends beyond the size of the training data. For example, SG and FT are trained using the same retain set, but the attacker's perspective reveals significant differences in their MIA accuracy and KS (as shown in Table 1). Furthermore, we have included Figure 2 in the revised paper, which demonstrates that adversarial modeling has a significant impact on the learning dynamics. Please refer to 1c & 1d in "To all reviewers" at the top for further details.

Reviewer comment (Evaluation):
In the empirical results, the authors only demonstrated the effectiveness of the framework. The missing piece is the effectiveness of the proposed gradient-based method. It seems that the paper is lack of demonstration on which point the gradient-based method is converging to. For example, how the utilities of both players evolve during the training process? And which solution concept the algorithm is converging to if it is converging?

Author response:
Please refer to comment 1b and 5 in “To all reviewers” at the top (also Figure 2 in the revised paper). We will add a subsection to discuss the convergence in details.