Paper: A Comprehensive Study of Privacy Risks in Curriculum Learning (ICLR-2024)

Reviewer comment (Novelty):
The motivation is not well-established. The authors mentioned "it’s crucial to investigate how training techniques affect privacy" -- I generally agree with this point, but this is not sufficient to convince me that studying the privacy risk in curriculum learning is important. 1) From a practical perspective, curriculum learning is a relatively niche subject, and is very different from self-supervised learning or unsupervised learning which have attracted extensive interest in the ML community. I'm not aware of any of its real-world applications and am not sure whether privacy is indeed a valid issue within. 2) From a research perspective, the author briefly mentioned "data ordering could have negative impacts on privacy" -- this is a valid hypothesis, and could be a good starting point to perform some thought experiments to obtain a more fine-grained argument (e.g., what kind of data ordering, what is the connection to curriculum learning, what data points will be affected the most). Directly jumping to large-scale evaluations based on an unmature hypothesis seems too hasty to me.

Author response:
**Motivation**: First, We argue that this work is well-motivated, as evidenced by the application of CL in emerging language models [1]. Researchers have proposed a CL approach termed 'length-based CL,' which uses the length of input text as a measure of difficulty. 

Secondly, the study referenced in [2] investigated data ordering for data poisoning in machine learning, confirming that this concept is more than just a hypoyhesis. While “data ordering” might seem vague, its definition in the context of CL is well-defined— it refers to the sequence in which data are presented to the model. This definition aligns with that in [2], lending further support to our motivation for investigating the potential negative impacts of such specific data ordering. Regarding your comment about 'which data points will be affected the most’, identifying these is one of the goals of our study. We began with the overarching question of how CL influences model privacy in general and then proceeded to a sample-wise investigation to determine which types of data points are most susceptible.

Reviewer comment (Experiments):
In algorithm 1, it seems that for later iterations in a given epoch (i.e., large $i$), $X_i'$ will still contain easy samples. This suggest that easy samples will still appear in later iterations of training (since $B_i$ is sampled from $X_i'$), meaning that they will have high occurrence compared to hard samples. 1) I'm not familiar with curriculum learning, but I feel a more natural way is to segment the ordered samples (based on difficulty scores) into *disjoint* batches, and then progressively train on them in a single epoch; 2) If you indeed used algorithm 1, then the number of occurrence will be another factor (other than difficulty) that will impact the vulnerability of each sample, and this is not taken into account when designing experiments or drawing conclusions.

Author response:
**Question 1**: Both definitionsare recognized in curriculum learning literature[3]. We specifically use 2) and have previously tested out 1) too, our preliminary results showed that they lead to same conclusion. 

I find the concept of sample occurrence very interesting and would love to add more discussion and experiment for it. However, we argue that occurrence is closely linked to difficulty; generally, a higher occurrence implies easier samples (based on your listed definition 2) of CL). Based on our current results, data ordering exerts a more significant influence on performance than the frequency of data occurrence. For example, in normal training (as shown in Figure 1), hard samples appear in all epochs, while in Curriculum Learning (bootstrapping and transfer learning), there hard samples appear less especially in the early epochs. Nonetheless, the MIA accuracy for hard samples in CL (e.g., samples with difficulty level of nine) still surpasses that in normal training.

To further explore the impact of occurrence, especially on easier samples which present a less intuitive case, we plan to train a CL model with disjoint batches. This approach will help us isolate and assess the effect of occurrence. It's important to note that our preliminary results indicate that using disjoint batches leads to the same key conclusions as presented in this paper. This implies that the occurrence is not the primary factor causing vulnerability in CL; rather, it is the data ordering. Samilar conclusion can be drawn from Figure 1 for the hard samples (see last paragraph).

Reviewer comment (Evaluation):
In Fig 1a and 2a, the harder samples actually have lower attack accuracy and confidence scores (for all curricula and normal training). This is very counter-intuitive and not explained. Particularly, I don't buy the results that harder samples will have low attack success rate for normal training. This makes me doubt that the difficult scores used in this paper is problematic. Also, I think it is misleading to claim that "harder samples are more vulnerable". Instead, what you can actually infer from the results is that "CL will introduce stronger effects to harder samples compared to normal training" (i.e., the *gap* is large).

Author response:
**Question2**: We're interested in understanding why you find the concept of 'harder samples having a lower attack success rate in normal training' counter-intuitive. Could you please elaborate on that?  To explain, let's first define what makes a sample 'difficult.' A common measure of difficulty is the loss score of a sample, where a higher loss score indicates greater difficulty. In the context of transfer learning, this score is derived from a pre-trained model. A high loss score could mean that the pre-trained model struggles to 'remember' this sample, possibly because the sample is rare, its features are too distinctive, or for other reasons. If a target model has difficulty remembering a sample, it's logical to assume that such a sample would be less vulnerable to attacks, as the model reveals less information about it.

We also want to clarify that when we say 'harder samples are more vulnerable,' we specifically refer to their vulnerability in the context of CL compared to a non-CL setting. This might be more an issue of how we present our ideas. Phrasing it as 'CL introduces stronger effects on harder samples compared to normal training' seems to convey the same idea in my opinion. However, we're open to rephrasing this for greater clarity for the audience.

Reviewer comment (Experiments):
This paper limits the datasets to image and tabular datasets. It will be interesting to study other data types like text and graphs, to make this study more comprehensive.

Author response:
**Weakness 1**: We can possibly add text classification task upon request with data set from torchtext.datasets to this study to make it more comprehensive. More specifically, we plan to use the example shown in [1] for this study.

Reviewer comment (Evaluation):
While the paper evaluates various defense methods, it does not have a deeper investigation into why certain defenses underperform and how they could be potentially improved.

Author response:
**Weakness 2**: In general, many of these defenses operate under different assumptions and vary in effectiveness depending on the setting. 

DP-SGD trains models with noise, which fundamentally reduces the model’s ability to memorize individual samples, thus leading to superior defensive performance. 

MemGuard, on the other hand, alters a model’s output to defend against MIA. This is the reason that it doesn’t affect the target model yet can still be effective. However,  it also does not offer protection against label-only attacks due to its lack of changes to the target model.

MixupMMD and AdvReg both modify the posterior to enhance privacy. MixupMMD is more effective because it brings the member and non-member posterior distributions even closer than AdvReg by introducing the new penalty  termed Maximum Mean Discrepancy.

From all the defenses mentioned above, we've learned that a key factor in making a defense effective is to minimize the difference between member and non-member posterior distributions. Therefore, a potential direction for future research in improving defenses could involve exploring other methods to further align these posterior distributions. We plan do add this discussion to the paper.

Reviewer comment (Experiments):
Could the proposed Diff-Cali method be combined with existing defense mechanisms to create a more robust defense against privacy attacks in Curriculum Learning?

Author response:
**Question 1**: Could you please further elaborate on this question? If we understand your question correctly, are you asking whether modifying the difficulty score can make Curriculum Learning (CL) less vulnerable to attacks, and whether the difficulty score can be utilized as part of a defense strategy? Based on our study across various CL settings (e.g., baseline), we found that even with randomized difficulty scores, CL remains more vulnerable compared to normal training. Therefore, the key to a more robust defense lies in minimizing the gap between member and non-member posterior distributions.

Reviewer comment (Evaluation):
How would the introduction of federated learning or decentralized training scenarios affect the privacy risks and defense mechanisms evaluated in this study?

Author response:
**Question 2**:
We consider federated learning and decentralized training a parallel line of study to ours. However, when combined with our study, the key conclusions remain the same for all decentralized models, and we believe the central model will be less intrusive compared to our findings. This is because, fundamentally, federated learning and decentralized training are designed to offer enhanced privacy.

Reviewer comment (Experiments):
The paper misses an important SOTA NN-based MIA attack LiRA (from the paper “Membership Inference Attacks From First Principles” in IEEE S&P 2022), making the evaluation of NN-based less convincing.

Author response:
**W1-LiRA**: In general, not all attack methods (e.g., adaptive attack and LiRA) and defense techniques have been examined in our paper. Although LiRA is considered state-of-the-art, it requires lots of shadow models (for example, 256 shadow models for CIFAR100, as demonstrated in [1]), whereas all other attacks in our paper require only one. To compare fairly with LiRA, we would need to divide the current datasets into much smaller subsets, which would likely degrade the performance of all target and shadow models. Moreover, training 256 shadow models for both CL and no-CL settings would be very costly. Due to these reasons, LiRA was not investigated in this study. However, we believe our key findings (e.g., the increased vulnerability of difficult samples when trained with CL) are broadly applicable, given the fundamental principles of curriculum design.  

Furthermore, we can possibly run a downscaled version of LiRA upon request (i.e., with fewer shadow models) in a case study to demonstrate that our key conclusions remain valid. More specifically, we intend to re-divide the dataset into six subsets, $D_i$  where $i = 0,1,2,3,4,5$. We will use $D_0$ to train the target model, $D_1$, $D_2$, $D_3$, and $D_4$ to train shadow models for LiRA and NN-based methods, and then use $D_5$ as the target and shadow test dataset.

Reviewer comment (Experiments):
The author does not justify the threat model clearly. For example, the Difficulty Calibrated MIA implicitly assumes the target model training is using CL. However, such type of information is not usually available to the attack. In this case, the author might also want to evaluate the proposed attack under a non-CL setting and compare its performance with other MIA attacks.

Author response:
**W3**: We tested Diff-Cali in both non-CL settings and various CL settings (refer to Figures 3 and 5). In these figures, we used bootstrapping as the default curriculum referenced in Diff-Cali and then applied this attack to models trained with all non-CL and CL methods (the "normal" lines in Figures 3 and 5 represent the non-CL setting). As observed, difficult samples become more vulnerable in all cases, regardless of whether CL is used in the target model. However, when the matching CL is applied, the improvement in accuracy on difficult samples is more pronounced. For instance, comparing Figure 1a with Figure 5a, the improvement in accuracy at difficulty level 9 is approximately 2.5 for anti-CL, while for bootstrapping, it is nearly 7.5.

Reviewer comment (Experiments):
In the defense part, it is natural to extend the idea of allocating privacy budget accordingly based on the difficulty level of the training examples. Have you tried this in the defense evaluation?

Author response:
**Question-Defense**: We have not yet explored this, but we believe the fundamental conclusions would remain unchanged, given DP-SGD's proven success in defending against MIA. Allocating the privacy budget based on the difficulty level could potentially yield a better balance between privacy and utility. However, this approach would impact the moments accountant technique used by DP-SGD for total budget accounting and require an examination of privacy composition. Additionally, incorporating the difficulty level might alter the privacy guarantee, possibly requiring the addition of noise to this difficulty level as well.