Paper: Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance (ICLR-2024)

Reviewer comment (Theory):
Computing the basis using private dataset compromises privacy, as the basis is directly depending on the private data. Although the authors argue that this can be considered as tuning hyperparameters, such an argument is unconvincing. In particular, this work does not conserve privacy for basic hyperparameter-tuning such as learning rate. Compared to related works or baselines, the proposed method definitely loses more privacy.

Author response:
We agree that, _in theory,_ the computation of GSD does cause privacy leakage.  However, our empirical evaluation using membership inference attacks reveals that an overestimated adversary has a success rate similar to _random guessing_ in practice. Based on realistic assumptions, the additional information an adversary can obtain from GSD is limited to "CIFAR-100 is the chosen public dataset for CIFAR-10." This information is difficult to construct a meaningful attack. Therefore, we further assume a powerful yet unrealistic adversary: we assume that the adversary knows: 1) the model used by GSD and all the related computations such as gradients, hidden layer activation, 2)  90% of the private data examples used by GSD, and all the rest private examples in the private dataset. The adversary tries to use membership inference attacks to tell if a particular image $x_i$ is used by GSD (i.e., see if $x_i$ is in the rest 10%). We apply one of the most well-studied and well-cited white-box membership inference attacks and the experiments show that: even under this overly estimated and powerful adversary, the attack success rate is as good as random guesses (see Appendix D.1 for more details).

Additionally, we nevertheless discuss DP methods for GSD computation in Appendix C. We hope that this work will serve as a starting point and inspire future research to effectively solve this scientific question with differential privacy.

Reviewer comment (Experiments):
The authors claim to focus on smaller public datasets due to computational resource limitations. For instance, instead of pre-training a network in CIFAR100, a batch of CIFAR100 data is selected for the projected DP-SGD. However, this argument seems weak. In my experience, projected DP-SGD is resource-intensive due to the basis calculation and gradient matrix storage. I suspect that conducting PEFT on the entire CIFAR100 dataset followed by vanilla PEFT DP-SGD could be faster and yield better results. This raises questions about the motivation and necessity of devising such a multi-stage training pipeline. The authors could conduct some additional experiments and provide details on running time, memory usage and configurations to justify the merits of their framework.

Author response:
The reviewer has raised concerns about the claims we made in the paper regarding computational resource limitations. We address these concerns with the following points:

* Indeed we focus on the scenario where practitioners only have a small amount of public datasets. Due to 1) this is consistent with previous literature setting [1][2] 2) this setting is applicable in certain key settings. For example, consider a clinical environment. Due to privacy and safety concerns, it may be inappropriate to use large, uncurated Internet datasets as public data (e.g., such a dataset may be infected by a data poisoning attack). Instead, the practitioner may have a small set of trusted public datasets. In our experiments, we set the number of public data examples to 2000 or 300, depending on the size of the private dataset. 
* Under this smaller public datasets setting, projected DP-SGD is not resource-intensive. It is indeed less time-efficient than “public pre-training, DP fine-tuning”, but all the experiments in this paper can be done on a single NVIDIA A100 GPU with 40GB VRAM. More importantly, as indicated in previous studies on public-data-assisted private ML, projected DP-SGD does improve accuracy. This is why this work is based on them and efficiently addresses the problem of selecting the best public dataset.
* When discussing computation resource limitations, we refer to large-scale pre-training on datasets like ImageNet, JFT-300, or even foundation models. These computational resource requirements are not on the same scale as the projected DP-SGD. Most practitioners can neither afford the computation resources for such pre-training, nor afford the time, effort and money to collect and create such a dataset. That’s why most practitioners will turn to off-the-shelf pre-trained models, and this advocates the necessity of devising such a “multi-stage training pipeline”. In addition, irrelevant pre-training datasets do not improve accuracy [3][4], highlighting the need for GSD to identify the best public datasets .
* Additionally, if the reviewer agrees with the setting together with the reasoning we give, “PEFT on the entire CIFAR100 dataset followed by vanilla PEFT DP-SGD” would be inappropriate because we cannot assume that we have enough public examples available. 
* To make this more clear, the “multi-stage training pipeline” is designed as an alternative to “PEFT, vanilla PEFT DP-SGD”, based on the reasoning we give aforementioned. Projected DP-SGD is one of the pre-conditioning methods. We also evaluated other pre-conditioning methods other than projected DP-SGD (see Appendix D.3) and GSD also works. That said, GSD works for all kinds of public-data-assist private ML methods, based on our experiments.

[1] Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=7aogOj_VYO0.

[2] Yingxue Zhou, Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=7dpmlkBuJFC.

[3] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In arXiv [cs.CV]. http://arxiv.org/abs/2103.00020

[4] Da Yu, Sivakanth Gopi, Janardhan Kulkarni, Zinan Lin, Saurabh Naik, Tomasz Lukasz Religa, Jian Yin, and Huishuai Zhang. Selective pre-training for private fine-tuning, 2023.

Reviewer comment (Experiments):
The best utility gains reported in the experiments are primarily achieved when the private dataset is used as the public dataset. This result is trivial and does not demonstrate the necessity of employing GSD. Additionally, only a limited number of candidates are reported in each experiment, making it unclear whether the GSD order is aligned well with the actual utility gain.

Author response:
We disagree that the finding is "trivial": indeed, if it were trivial, then it would always be true that using public data drawn from the same distribution as the private data would be ideal. However, in one of our more surprising results, we find this is _not_ true -- that public data from a separate distribution may be better than the original one! For example, CIFAR-100 is consistently a better dataset for CIFAR-10 than itself, and this answers the reviewer’s question i). Furthermore, public data from the same distribution as the private dataset may not always be available: consequently, GSD is useful in determining which alternative dataset one should use. See another example: at first we also assume that MNIST-M (colorful MNIST), is a good public dataset for SVHN, as they are all images of digits. BUT GSD shows that MNIST-M is even worse than CIFAR-100!

Reviewer comment (Evaluation):
Most of the utility gains are marginal. The reported results are poor given that the networks are pre-trained. Specifically, the best results of CIFAR10 and FMNIST are worse than or only comparable to some basic baselines, e.g., Tramer & Boneh, DIFFERENTIALLY PRIVATE LEARNING NEEDS BETTER FEATURES (OR MUCH MORE DATA), 2021. Although the original paper of this baseline does not report the results of $\epsilon=2$, the authors can run its code to verify this.

Author response:
First, the results on CIFAR10 and FMNIST are achieved without pre-training---we DP train a Resnet20 from _scratch_. Second, we respectfully disagree with the argument that the utility gains are insignificant. For instance, when examining the results of ChestX-ray14 dataset, our approach demonstrates a significant ~3% accuracy improvement over the DP-SGD baseline. This improvement is particularly crucial in clinical environments. Furthermore, this paper's primary contribution lies in identifying and addressing an important scientific question: determining which public datasets perform well for specific private tasks. Our solution to this problem is both straightforward and highly effective.

Reviewer comment (Experiments):
GSD-based public data set selection may leak sensitive information.

Author response:
We acknowledge that GSD is indeed not differentially private, and thus may leak sensitive information.  We give the following reasons to explain that 1) GSD is unlikely to breach privacy in practice, and 2) this work still makes meaningful contributions

* The selection of a public dataset is a hyperparameter: in essentially all work on private ML, hyperparameter tuning is performed non-privately, since the privacy impact is considered to be minimal [1][2].
* Under this suggestion, in the context of public data sets selection, we assume what an adversary can know from GSD is “CIFAR-100 is the chosen public dataset for CIFAR-10”. Since this provides little information to the adversary, we also assume an unrealistic adversary that knows: 1) the model used by GSD and all related computations, such as gradients and hidden layer activation, and 2) 90% of the private data examples used by GSD, along with all the remaining private examples in the dataset. The adversary attempts to use membership inference attacks to determine if a particular image $x_i$ is used by GSD (if $x_i$ is in the remaining 10%). When we apply a well-studied and well-cited white-box membership inference attack, the experiments show that even under this overly powerful adversary, the attack success rate is as good as random guesses (see Appendix D.1 for more details).
* We believe the main contribution of this paper is: identifying this scientific question (exploring which public datasets perform well for a particular private task), and giving a simple yet effective solution. 
* Nevertheless, we discuss DP methods for GSD computation in Appendix C. We hope that this work will serve as a starting point and inspire future research to effectively solve this scientific question with differential privacy.

[1] Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential privacy. In Proceedings of the 10th International Conference on Learning Representations, ICLR ’22, 2022.

[2] Shubhankar Mohapatra, Sajin Sasy, Xi He, Gautam Kamath, and Om Thakkar. The role of adaptive optimizers for honest private hyperparameter selection. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, volume 36 of AAAI ’22, pp. 7806–7813, 2022.

Reviewer comment (Theory):
What is running time (time complexity) of Algorithm 1 Gradient Subspace Distance (GSD) ?

Author response:
One thing worth noticing about GSD is: it doesn’t update or iterate over the model, meaning the gradient subspace distance is given only by one pass through the model. _For a simple theoretical analysis_, we assume that the model used for GSD is a linear model with one hidden layer of size $n$, the number of examples are $m$, the input dimension is $d$, the output is of size $c$, and the lower dimension is $k$. Then the time complexity of GSD is $O(2mdn + 2mnc + 2m(dn+nc)\log(k) + 2(m+dn+nc)k^2)$ as we only need to get top-$k$ basis. Empirically, in our experiments, GSD computation takes only a _negligible_ amount of time. For example, giving a gradient subspace distance number in Table 1, like 0.15, will only take ~12s with one NVIDIA A100 (40GB) GPU (here #parameters is $O(10^4)$). For fine-tuning LoRA on ViT (#params becomes $O(10^5)$), the time cost is ~41s.

Reviewer comment (Theory):
It is unclear why Gradient Subspace Distance is a good measure of the similarity of data sets in nature.

Author response:
The reviewer raises an interesting question: how will GSD reflect the similarity between data sets in nature? _GSD is designed to rank the utility of potential public data sets in the context of private machine learning._ Thus, it is natural to think that data sets that are semantically close, or say similar in nature, will have smaller GSD, meaning higher utility. However, our experiments show that this is not always true. We surprisingly find that CIFAR-100 is even ‘closer’ to SVHN w.r.t. gradient subspace distance, compared with MNIST-M (colorful MNIST). But in nature, MNIST-M should be closer to SVHN than CIFAR-100, as MNIST-M and SVHN are all images with digits. On the other hand, Task2Vec, the method we compare GSD with, is good at describing data set similarities in nature, but fails to give correct predictions for the utility w.r.t. private machine learning. 

That said, it is natural to think that smaller GSD means two data sets are more similar in nature. But this isn’t always true, and vice versa: two naturally similar data sets may not have smaller GSD. For example, CIFAR-10 should be most naturally similar to itself, but our experiments show that CIFAR-100 gives the smallest GSD. Again, the idea of GSD is to rank the utility of a list of potential public data sets in the context of private machine learning, and GSD is effective in doing so.

Reviewer comment (Theory):
In Lemma 4.1 what is the relationship between singular values and gradient subspace distance, which one is larger?

Author response:
We appreciate the reviewer for proposing this interesting question. We are unsure if we understand the question, but we will attempt to provide an answer. _Empirically_, top singular values are usually larger. We have an empirical evaluation in Appendix D.2. Gradient subspace distance is bounded by $\sqrt{k}$, and is usually much smaller than top singular values. _Theoretically,_ this question can be extended to “does singular value itself have a mathematical relation with the rank of the matrix?” To the best of our knowledge, we cannot find a good mathematical relation between these two quantities. 

Please let us know if this answers your question. We are happy to discuss more.

Reviewer comment (Theory):
Overall, I think the presented ideas are intuitive and I believe useful but on a theoretical level the contribution is not big, the value is on the experimental side. All in all this is a nice contribution and I appreciate also the "second phase fine-tuning" part of the paper and the careful experiments. I think this paper would fit well to this venue.

Author response:
We appreciate the reviewer's insightful and encouraging feedback on the utility of public datasets in private machine learning. We are encouraged by the reviewer's interest in our idea and the acknowledgment of the paper's contribution, as well as its fit with ICLR. The summary given by the reviewer is thorough and concise, and we are lucky to have a reviewer who understands our work well.

Below, we respond to your specific concerns.

Reviewer comment (Experiments):
I have mostly questions related to the computational efficiency (see above). In the experiments of this paper, how big was the computational burden of choosing and using the projectors? I mean if you compare, e.g., to DP-SGD?

Author response:
The reviewer raised concerns about the computational efficiency of our method, which is an important aspect for all projection-based preconditioning methods. The short answer is, computational efficiency is not a problem for our methods. We give the following reasons:
* We assume that the practitioner has a relatively small number of public data examples, which is consistent with projection-based pre-conditioning methods literatures[1][2]. This setting is especially applicable in certain key settings, such as a clinical environment where privacy and safety concerns make it inappropriate to use large, uncurated Internet datasets as public data due to the risk of data poisoning attacks. Instead, the practitioner may have a small set of trusted public datasets. In our experiments, we set the number of public data examples to 2000 or 300, depending on the size of the private dataset. 
* Another advantage of GSD is, GSD doesn’t update or iterate over the model, meaning the gradient subspace distance is given only by one-pass through the model. In Section 4.2, we empirically validate that the order of GSD is preserved over training, thus only two computation (one for $V_{pub}$ and one for $V_{priv}$) is needed to give the gradient subspace distance. With this advantage, combined with small number of public examples assumption, GSD computation only takes negligible time. For example, giving a gradient subspace distance number in Table 1, like 0.15, will only take ~12s with one NVIDIA A100 (40GB) GPU (here #parameters is $O(10^4)$). For fine-tuning LoRA (#params becomes $O(10^5)$), the time cost is ~41s.
* For those projection-based methods, like GEP, yes, these methods will cost an extra computational burden. As pointed out by the reviewer, computing the basis can be quite expensive, and for GEP, $V_{pub}$ needs to be updated per iteration. However, extra computational burden makes GSD valuable as we can identify the best one from a list of public datasets in advance and thus, saving the time to run GEP over all the public datasets.
* Practical scenario: as mentioned earlier, GSD is applicable in certain key settings, like clinical environments. To improve accuracy using private ML methods with public data assistance, practitioners can follow these steps: 1) Use GSD to rank the utility of available public datasets, and select the best one; 2) Use the same public examples in step 1, run algorithms like GEP to enhance accuracy.

[1] Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=7aogOj_VYO0.

[2] Yingxue Zhou, Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=7dpmlkBuJFC.

Reviewer comment (Presentation):
I think the form of the "projection metric" with those cosine principal angles as given in Definition 2 is quite intuitive, but I think the form where it is written with the Frobenius norm (used e.g. in the proof of Lemma 4.1) makes it even clearer, perhaps you could consider moving that to the main text? Just to quickly mention it.

Author response:
We thank the reviewer for pointing out Forbenius norm is more intuitive from an audience’s perspective. We also thank the reviewer for pointing out the minor mistakes made in this paper. We will correct these in the final camera-ready paper (if possible).

Reviewer comment (Writing):
Dot missing after Eq. 4.

Author response:
We thank the reviewer for pointing out Forbenius norm is more intuitive from an audience’s perspective. We also thank the reviewer for pointing out the minor mistakes made in this paper. We will correct these in the final camera-ready paper (if possible).

Reviewer comment (Presentation):
There are some dots left to the tables all over, e.g. on page 8 and in the appendix.

Author response:
We thank the reviewer for pointing out Forbenius norm is more intuitive from an audience’s perspective. We also thank the reviewer for pointing out the minor mistakes made in this paper. We will correct these in the final camera-ready paper (if possible).

Reviewer comment (Writing):
Page 20, paragraph "Experiment Setting", third line: bracket missing.

Author response:
We thank the reviewer for pointing out Forbenius norm is more intuitive from an audience’s perspective. We also thank the reviewer for pointing out the minor mistakes made in this paper. We will correct these in the final camera-ready paper (if possible).

Reviewer comment (Presentation):
Page 21, before D.3 title: dot in the middle of the page.

Author response:
We thank the reviewer for pointing out Forbenius norm is more intuitive from an audience’s perspective. We also thank the reviewer for pointing out the minor mistakes made in this paper. We will correct these in the final camera-ready paper (if possible).